A Networking Protocol Built For Low Latency Interactive Game Streaming

BUD is our proprietary networking protocol built with one goal in mind — low latency, interactive game streaming from anywhere. It is based off of UDP and encrypted with DTLS 1.2 (via OpenSSL). It adds your basic reliability semantics like TCP, but with a highly tuned and custom congestion control algorithm. Early on, we made the decision that every part of our software had to be built purposefully for game streaming for there to be any chance that we could produce high frame rates and low enough latency to make gaming remotely over the internet viable. We started with our streaming software — no shortcuts were taken in developing the capture, encode, decode, and render pipeline. From the beginning, we built Parsec to take advantage of the lowest-level technologies available on the hardware to improve performance by milliseconds.

When we released Parsec a year ago, the software relied on TCP. This was okay to start, but nowhere near where it needed to be. Even in small quantities, TCP tends to freak out with loss (see video below) and adds a ton of latency. UDP doesn’t mind loss — you’ll continue to get fluid streaming movement — but the reliability breaks down and the screen begins to look like a tie-dye shirt (which, as you can imagine, isn’t great for gaming). In the case of lots of loss, the best move with a video stream is to tweak the encoder’s bitrate (H264 is a lossy codec). We tried several solutions, like WebRTC, plain UDP, and others, but nothing could achieve the reliability of TCP with the latency of UDP for interactive game streaming. So, we did the only thing obsessed engineers do — we built our own networking protocol to shave milliseconds while maintaining high definition video without sacrificing frames.

When building Parsec, we had to make tradeoffs for what was most important. From our streaming and networking perspective, we prioritize latency, frame rates, and then video quality, in that order. Our networking protocol works hand-in-hand with our encoding and decoding modules in a way that using an off-the-shelf networking protocol could never do. When there are issues in the networking between you and your friend’s gaming PC, BUD is sharing information with our streaming software and vice-versa, making split-second decisions and changing variables to deliver the lowest lag possible.

The Downsides Of Building Your Own Networking Protocol

Gamers who use Parsec definitely notice the work that went into building our networking protocol.

When you finish coding an intro to networks project and realize how hard delivering a file reliably is over UDP, let alone streaming input/video over [the network]…Shoutouts to all the Parsec devs, your job is hard as ****. — @Labryz in our Discord Server

But there are some serious downsides to building your own protocol beyond the difficulty of developing it. Things that other developers who use off-the-shelf networking take for granted, we cannot. In fact, we had to develop our own version of cURL to handle some of the work we did. The impact of developing all of this in-house is the potential for lots of networking issues. We’ve been battling those for the last 6 months and are still working on them. At one point, 85% of our users experienced an error when using Parsec during a day. Fortunately, with recent releases, this has come down significantly. We still run into issues with symmetric NATs, firewalls, and overall degraded networking performance, but we’ve made tons of progress.

The Next Frontier Of Our Game Streaming Networking Protocol

Up until this week, we always assumed a user had a great network without interruption on both sides of the connection. This assumption was bad, we knew that. It led to lots of errors of the 12007 and 12010 variety (if you’ve been a long-time Parsec user, you’ve definitely come across plenty of those). Prior to this week, if there was even a momentary blip in connectivity, Parsec bailed hard and threw a nebulous/confusing error. But yesterday, we released new functionality, similar to FaceTime and other P2P products, that just says, “your network is bad, please hang tight.” Hopefully, this happens so fast that you don’t notice it, but if it’s really bad, we’ll stop the stream and give you time to catch up when the internet is restored. If it’s not resolved in 60 seconds, we end the session. If multiple friends are connected to the gaming PC at the same time, each connection is treated individually, so an interruption in one person’s networking (unless it is the host), doesn’t impact everyone else.

Now that we have reduced our run-time errors, the next frontier is algorithmically adjusting our networking to account for lots of different network conditions. We’ve made the aggressive assumption that anyone who makes a Parsec connection has less than a 50 millisecond ping to their gaming PC and up/down bandwidth of 30 Mb/s. The up bandwidth only matters for the host of the gaming PC. Parsec is optimized for that type of connection. Going forward, however, our networking will adjust for your specific networking conditions. We know that most people may have 30 Mb/s down (at least in the US), but almost everyone has just single digits up bandwidth unless they’re on fibre connections.

Let’s say you have a 12 Mb/s connection. The goal is to make the most of that 12 Mb/s. Since Parsec has no buffers of any kind on video, we basically have to detect a congestion event before it starts. That means reading the tea leaves of all of our networking metrics, crunching numbers really fast, and making a split micro-second decision. Right now, it tends to be too conservative for people with higher latencies. But the end goal is to basically saturate that 12 Mb/s connection right at the edge without introducing loss or latency. At the end of the day though, the primary concern is making the performance as good as possible while you are playing your game. That means low level networking and video processing and no extra bloat in our codebase.

A visualization of what’s happening in our networking

If you really are interested in digging into the weeds, here’s a presentation that highlights how our co-founder went about the process of building our BUD protocol for gaming:

Building a UDP Protocol For Cloud Gaming from Benjy Boxer

Be sure to follow us on Twitter for future updates, and join our Discord for support and finding friends to game with.