The FOSDEM infrastructure has spawned another side project. The video box hardware has performed pretty well for this years event and a few of the boxes have been sent out to other conferences around the world to do streaming. These other conferences don’t have the exact same room setup as what has been planned for FOSDEM so this exposes some nice edge cases and missing features.
In the current FOSDEM setup two of the boxes are used. One for capturing the camera in the room and one for capturing the HDMI output of the slides laptop and the wireless microphones. These two boxes then send their streams over the network to the rack of “voctops” that run voctomix and combine these streams in the livestream/recording you’ll see online.
But it turns out the Radxa x4 in the box has almost enough CPU power to do all these tasks since other conferences setups have been made without voctops and a second box by using USB webcams instead of HDMI cameras and running voctomix on the box itself.
So back in march I tried making a small prototype application that does the picture-in-picture layout used in conferences with the two pictures being a V4L2 device and composites them on the GPU.
From my experience with Megapixels I know you have to make a bit of a threading mess to get a V4L2 pipeline performant on slow hardware and I didn’t want to go through that mess again in C so I decided this is the perfect oppertunity to learn Golang since supposedly that’s good at concurrency.
I don’t think any line of code from this prototype has survived all the refactoring and improvements that have been made since actually makes it a useable piece of software.
Why not OBS
Since this is the first question everyone has when you don’t use OBS for anything video related… This is a headless remote controlled video mixer that’s build around memory bandwidth constraints of an intel atom CPU. It allows to have an arbitrary amount of inputs and outputs where the inputs are mostly V4L2 devices or ffmpeg processes and the output is either the window or another ffmpeg process to do streaming.
The current code makes a few assumptions about the internal architecture of the hardware which allows it to optimize a bit more. Like assuming the GPU is onboard and sharing system memory.
How it works
The initial revision of Fazantix (that’s what it’s called) was basically an OpenGL demo application and a V4L2 capture example glued together. For every capture device a goroutine is spawned that fetches V4L2 frames from the kernel and puts them in a channel. Then on the main thread of the program the OpenGL render loop runs that fetches the frames from every channel and uploads them to a GPU texture. This then gets composited together in a fragment shader and shown on the screen.
This design has many performance flaws but still was enough to have the composited view of 2 camera streams and a rendered output of 720p60. To get this to actually perform at the level required for FOSDEM a few things needed to be overhauled and made way more complicated. The initial revision is only 427 lines of Go but lacks many important features like not having all the sources and layout hardcoded in the application and multiple outputs. I’m still pretty happy with how much functionality I’ve been able to build in the application while also learning OpenGL and Golang. The most featureful version of this weekend project ended up at 609 LoC which includes some hacks to render YUV encoded textures in OpenGL.
The rework
With the initial idea proven working it was time to do a massive restructuring of the project. This was done during the FOSDEM infrastructure hackathon which helped massively with this since there were people in attendance that actually know Golang and watching the codebase transform in something that’s more idiomatic Go code has been an immensely helpful learning experience. This rework contains:
- Deciding on and implementing a config file format for the application. This ended up being YAML for describing the sources, sinks and scenes.
- Replacing the channels pushing frames around with a custom FrameForwarder class that implements a ringbuffer of byte slices to have less copies and allocations happening to the video data.
- De-duplicating a lot of code that I’ve had just copy-pasted around for the initial version. Like having a proper function to set up an OpenGL texture instead of having the OpenGL calls everywhere.
- Splitting of the various video source types to seperate classes and implementing more of them. The initial version can stream V4L2 video and load static images which are now seperate classes but there’s also an ffmpeg source that supports launching a process and receiving frames over a pipe. This has been great to load in test videos as sources instead of having multiple cameras.
- Decoupling the framerate of the composited output from incoming frames. The initial implementation would get stuck if none of the sources would send a new frame.
- Implementing FBOs to render multiple scenes per frame in order to render a seperate layout for the live stream as what’s shown on the projector in the room.
- Adding an FFmpeg sink that accepts the rendered output from an FBO and send it over a pipe to an FFmpeg process to become a network stream.
- A HTTP api that provides a REST interface for switching scenes.
At the end of this long weekend of hacking Fazantix was in a almost-real-world-ready half usable state but at this point was outperforming the other streaming solutions that were tried. Mainly because Fazantix due to keeping copies to a minimum uses less memory bandwidth which the hardware platform was desperately lacking. The post-hackathon version of the software was able to run the whole pipeline at a comfortable 1080p60 in my tests which was more than enough for the FOSDEM event that runs at 1080p30.
In the weeks since a lot of work has happened to the codebase to make things even more efficient like refactoring the entire codebase for a third time (but it looks really neat now!) and fixing some efficiency issues and race conditions in the FrameForwarder. This also made it a lot easier to implement PBOs (Pixel Buffer Objects) in the pipeline to make the main render loop not block anymore on reading the pixels from the framebuffer.
This current version has now been running stable at 1440p60 for me with 2 capture cards inputting an 1440p60 stream, still all running on that Intel N100 CPU/GPU.