Building a software video mixer

The FOSDEM infrastructure has spawned another side project. The video box hardware has performed pretty well for this years event and a few of the boxes have been sent out to other conferences around the world to do streaming. These other conferences don’t have the exact same room setup as what has been planned for FOSDEM so this exposes some nice edge cases and missing features.

FOSDEM pipeline

In the current FOSDEM setup two of the boxes are used. One for capturing the camera in the room and one for capturing the HDMI output of the slides laptop and the wireless microphones. These two boxes then send their streams over the network to the rack of “voctops” that run voctomix and combine these streams in the livestream/recording you’ll see online.

But it turns out the Radxa x4 in the box has almost enough CPU power to do all these tasks since other conferences setups have been made without voctops and a second box by using USB webcams instead of HDMI cameras and running voctomix on the box itself.

So back in march I tried making a small prototype application that does the picture-in-picture layout used in conferences with the two pictures being a V4L2 device and composites them on the GPU.

first working build of Fazantix

From my experience with Megapixels I know you have to make a bit of a threading mess to get a V4L2 pipeline performant on slow hardware and I didn’t want to go through that mess again in C so I decided this is the perfect oppertunity to learn Golang since supposedly that’s good at concurrency.

I don’t think any line of code from this prototype has survived all the refactoring and improvements that have been made since actually makes it a useable piece of software.

Why not OBS

Since this is the first question everyone has when you don’t use OBS for anything video related… This is a headless remote controlled video mixer that’s build around memory bandwidth constraints of an intel atom CPU. It allows to have an arbitrary amount of inputs and outputs where the inputs are mostly V4L2 devices or ffmpeg processes and the output is either the window or another ffmpeg process to do streaming.

The current code makes a few assumptions about the internal architecture of the hardware which allows it to optimize a bit more. Like assuming the GPU is onboard and sharing system memory.

How it works

The initial revision of Fazantix (that’s what it’s called) was basically an OpenGL demo application and a V4L2 capture example glued together. For every capture device a goroutine is spawned that fetches V4L2 frames from the kernel and puts them in a channel. Then on the main thread of the program the OpenGL render loop runs that fetches the frames from every channel and uploads them to a GPU texture. This then gets composited together in a fragment shader and shown on the screen.

Fazantix V4L2 usage with channels

This design has many performance flaws but still was enough to have the composited view of 2 camera streams and a rendered output of 720p60. To get this to actually perform at the level required for FOSDEM a few things needed to be overhauled and made way more complicated. The initial revision is only 427 lines of Go but lacks many important features like not having all the sources and layout hardcoded in the application and multiple outputs. I’m still pretty happy with how much functionality I’ve been able to build in the application while also learning OpenGL and Golang. The most featureful version of this weekend project ended up at 609 LoC which includes some hacks to render YUV encoded textures in OpenGL.

The rework

With the initial idea proven working it was time to do a massive restructuring of the project. This was done during the FOSDEM infrastructure hackathon which helped massively with this since there were people in attendance that actually know Golang and watching the codebase transform in something that’s more idiomatic Go code has been an immensely helpful learning experience. This rework contains:

Fazantix V4L2 usage with the frameforwarder

At the end of this long weekend of hacking Fazantix was in a almost-real-world-ready half usable state but at this point was outperforming the other streaming solutions that were tried. Mainly because Fazantix due to keeping copies to a minimum uses less memory bandwidth which the hardware platform was desperately lacking. The post-hackathon version of the software was able to run the whole pipeline at a comfortable 1080p60 in my tests which was more than enough for the FOSDEM event that runs at 1080p30.

Fazantix streaming

In the weeks since a lot of work has happened to the codebase to make things even more efficient like refactoring the entire codebase for a third time (but it looks really neat now!) and fixing some efficiency issues and race conditions in the FrameForwarder. This also made it a lot easier to implement PBOs (Pixel Buffer Objects) in the pipeline to make the main render loop not block anymore on reading the pixels from the framebuffer.

This current version has now been running stable at 1440p60 for me with 2 capture cards inputting an 1440p60 stream, still all running on that Intel N100 CPU/GPU.