Customize your notifications!
👋 Hey, you haven’t set your favorites yet. Hit ‘Star’ to deal with it anytime.
This material was created with the support of our Patrons. You can support us!
Become a Patron
I am Graziano “Gillstolemyride” Fabiano, active fighting games community member and I have the pleasure of interviewing Jeff Vavasour, Founder of Code Mystics, a company specializing in high fidelity multi-platform emulation and netplay solutions in gaming.
In short, Prometheus is our cross-platform engine. This allows us to develop our games abstractly and then deploy on a variety of platforms. Though there are a number of generic options for cross-platform engines out there like Unity, Unreal, or GameMaker, there wasn't anything suited to our specific needs, particularly when we developed the technology 12 years ago.
FOCAL is our emulation technology. Our three main emulation architects, myself included, have been doing emulation since the end of the 80s. Back then emulation was quite a challenge since there wasn't such a vast performance difference between the machine doing the emulation and the machine it was emulating, and emulation has a lot more overhead than porting. We developed a lot of tricks to get decent performance. FOCAL is built upon the expertise in emulation development we've accumulated over the last 30+ years. We kept an emphasis on code optimization and efficiency, which allows us to do emulation that may not otherwise be feasible. In fact, that's part of the reason we needed a unique engine like Prometheus. If we're trying to get our emulation running as fast as possible, the last thing we want to do is to sacrifice those hard-won gains to the platform engine.
Gemini is a suite of online multiplayer solutions that grew out of the desire to take emulation online. On the netcode side, it's a framework for synchronizing inputs on multiple instances of the game via peer-to-peer networking, for minimizing latency under varying network conditions, and also, of course, managing game state in rollback. It also provides some developer features that allow us to debug desync issues and the like. Beyond that, we have our own server solution to complement it that can support matchmaking, leaderboards, etc. as you would find on most major platforms; but, like the rest of our technology, it is not tied to any specific platform.
When we started Code Mystics we'd already been doing this for about 20 years, so the goal was to build a base set of tools that we'd own that would allow us to continue to ply our trade. They were built and streamlined with the benefit of hindsight based on what was and wasn't successful in our earlier work prior to Code Mystics. In our early meetings with clients, we were trying to sell them on emulation, but on platforms that they might've not realized were viable at the time, like Nintendo DS and web browsers. So we built some prototypes to demo in meetings at conventions.
Prometheus was a "means to an end" to work through how to get that emulation working on these modest platforms without too much overhead. With both FOCAL and Prometheus we took a slightly different approach where we could have the code rewrite itself offline to be more efficient, rather than spending extra CPU power to do the abstraction at runtime. Again, the goal was to achieve things on modest hardware that weren't otherwise achievable.
We also created a small casual game, Dreamwalker, to give us some context to flesh out Prometheus's features beyond emulation, such as UI support.
Even now, we spend a portion of our time always focusing on R&D to expand what we're able to offer, anticipating demand. Years before Microsoft asked us to put online multiplayer in Killer Instinct Classic, we'd been experimenting with adding online multiplayer to our web-browser-based Atari games. In particular, we were working on being able to play Atari Combat browser-to-browser over the Internet. We were tweaking the netcode to reduce and balance latency, so when KI came along, we already had some of the logistics worked out for peer-to-peer. We just needed to add the support for Xbox Live matchmaking, etc. to get our games connected.
As the demand for online multiplayer rose, we recognized a need to flesh out our offerings there, so set aside time to build it up, speculatively. Sometimes it took a few years, but that R&D effort always paid off. Sooner or later, publishers wanted to make use of something we'd been developing.
Backbone Entertainment was the middle of that story. I started with Andrew Ayre and Digital Eclipse, back in 1994, serving as founder/leader of their emulation development team and CTO overall. We eventually grew a studio around my team in Vancouver, BC, while Andrew grew a studio around him in Emeryville, CA, where they focused primarily on Game Boy development.
As time went on and we continued to grow, Digital Eclipse eventually merged with ImaginEngine, an edutainment software developer, and became Backbone Entertainment. Each studio also expanded its offerings outside its core specialty. For example, on top of being the in-house emulation team, the Vancouver studio became the home of development for the first three Spyro the Dragon games on Game Boy Advance, Age of Empires for Nintendo DS, etc.
We later merged again with The Collective and Pipeworks to form Foundation 9 and acquired more studios like Shiny and Sumo. It became quite a different thing than I started out with. I missed the small, intimate projects we used to do. So, in 2006 when there were plans to reorganize F9 after another acquisition, I decided that was a good time to move on. I did some consulting for a couple of years, including a time with cartoon-maker Nerd Corps. With the 2008 economic collapse, Foundation 9 shuttered the Vancouver studio. With my old teammates available, it seemed like the perfect time to get the team back together and get back to doing what we loved. That's when I formed Code Mystics.
The bar keeps getting raised. We recognize the importance of staying on top of that, which is why we dedicate our efforts to R&D. When our partners ask us to add features in response to these shifting expectations from players, we want to be sure we have the tools ready to hit the ground running. Nothing has brought the expectations for online gaming to the forefront quite like the last year of COVID. With many tournaments going virtual, online gaming is more important than ever. Many of these features are key to making online tournaments the best experience they can be. Like I said, our R&D invariably pays off with clients eventually asking us for the very features we've been developing, so we're hopeful we'll be able to keep pushing forward with these qualities of life improvements for fans.
In a strange way, you can think of rollback as a purely cosmetic feature. The "mainline" game logic plays out the same regardless of whether it's input delay or rollback. Only, with rollback, the render stage includes a "what if" step. "What if we pretend we know what the user's inputs will be for the next X frames? What would the screen have looked like after that?" You have the code run that "what if" simulation and render the result. On the next logic frame, that "what if" scenario is thrown away and a new one is built based on what we now know for certain, a frame later. The notion of "rollback" comes in, in that, after you render the frame, "throwing away" the prediction essentially involves creating a save state before you begin making the prediction, then rolling back to that save after the prediction is rendered. It's a little like using a rewind feature every single frame.
The trick is you need to rewind absolutely everything important to game logic back to exactly what it was before the prediction. Because of network conditions, there's no guarantee you and your opponent will be looking ahead at the exact same number of frames. So if you need to predict 5 frames and they need to predict 6, you need to be sure that going ahead 5 frames and back 5 frames gets you back to the exact same spot as going ahead six frames and back 6. In theory that sounds straightforward but there's a lot of nitty-gritty game state to keep track of any mistakes or omissions that can be made.
In emulation, it's a bit easier because you built a simulation of an entire machine, so you already have the container that all the game state lives in. Make a copy of that, and you're good to go. If it's not emulation -- and especially if the game wasn't built with full save states in mind -- going through the code afterward and finding all that state and wrangling it is a huge treasure hunt. If you miss something, then going ahead 5 frames and back 5 frames may not get you back where you started, or get you to the same place as going ahead 6 and back 6. If the two peers in a match end up in different states after a rollback frame, you get a dreaded desync. Once a desync happens, they often get worse with time through the match, until eventually, something important doesn't agree (like your health bar or even if one player is alive and the other is dead!). Even in a 2D game like King of Fighters 2002 Unlimited Match, which is not an emulation, a lot of work went into desync detection. We spent three intense weeks in Community Beta with tools checking hundreds of thousands of matches to report on and analyze any desyncs detected.
3D games are necessarily complex. They may be built atop third-party engines (e.g. Unreal) where you may not have access to all the pieces that govern a state. But you still need to keep track of everything important and make sure you put it right back where you found it after you're done making your rollback prediction for render, just like in the 2D games.
The further problem with modern games is that logic may be entwined with the rendering. In a 2D game, the logic tells the render what to draw. But in a 3D game, the animation system may also tell the logic it's time to do something, e.g. when an animation makes contact. These feedback loops can make it harder to keep everything "wrangled."
Another big issue is CPU performance. The 2D games we're talking about can be 20 years old, and easier to run on a modern system without taxing it. This is important if you're making a "prediction" about what's going to happen 6 frames from now. You're going to need to simulate all 6 of those frames and the frame you're really playing, all in one "real" frame's worth of time. In other words, to render at 60 FPS, the game has to effectively be capable of running its logic at 420 FPS. For a 20-year-old game, that's feasible. For a modern game (with physics simulations and the like), you may not have the CPU power to do that.
Not a lot of people know this, but we actually added rollback to Killer Instinct Classic, first, about two years after it was released. It was an earlier rendition of our netcode, so not as refined, but that was actually the first time we released a game with rollback. Our first game with SNK was actually King of Fighters 98 Ultimate Match Final Edition in 2014, but KOF '97 was the first one we were able to add rollback to.
Certainly, the excitement with the new rollback update of KOF '97 was palpable. The fan reaction was overwhelming. And no doubt that positive and vocal enthusiasm had to have helped make it possible for us to retrofit rollback into our earlier Garou, Last Blade 2, Samurai Shodown V Special, and King of Fighters 2002 Unlimited Match releases with SNK.
Having tweaked our rollback code since KI, I'd been wanting an opportunity to try it out with the SNK games for some time, so it was quite gratifying to see it received so positively. We want people to have the best experience they can. At the time there was a growing movement amongst fans to advocate for rollback, but there were others who were skeptical and content with input delay. It's long been a philosophy with our company that games should never "take away" features, so it was important to us to keep the input delay support in there too. Our UI allows you to select input delay, rollback, or a blend of the two. I think that helped it gain acceptance. Rollback fans got what they wanted, and skeptics could still play with input delay if they chose. And, if they were curious, they could dip their toes in the water and see what the rollback fuss was about, just by moving the netcode slider.
We've only used our in-house rollback code, so I can't comment on what it would be like to use someone else's solution. Now, as to fitting our rollback code to someone else's game, that's definitely something we're familiar with. To oversimplify it, adding rollback to someone else's game (not an emulation) really just becomes a systematic sweep of the code to find all the states that need wrangling. Some people's code is just a little easier to follow and wrangle than others. Of course, it is a big challenge to debug and make sure you've got all the "corner cases", which is why things like last year's Community Beta can be so valuable. (That's another way fan support really helps make things like this happen!)
Of course, the challenges in a more modern game multiply, as I mentioned above.
I can only speculate, but it certainly takes time and work to add these features. Games can be in production for years. This shift in the importance of rollback due to lockdown has only been about a year. If a game was 40% into development, it might be a challenge to retool and rebudget, mid-stride. It can be a challenge enough to make your game bug-free in time to ship. Making it watertight so that it can be perfectly stable when thrashing load/save states over and over, or for that matter even implementing load/save states that work absolutely anywhere in the game, can no doubt prove a daunting challenge for any development team that already has a full plate.
And then there's the performance angle: if you are trying to create a state-of-the-art cutting-edge game for a modern platform, you're pushing the platform to the limits. Imagine someone comes to the team, 40% in, and says "OK, now we need all this cutting edge stuff to run 5x faster." Something has to give. As long as there's been a video game development there's always been a tug-of-war between adding features, hitting your ship date, meeting system requirements, keeping costs under control, etc. Modern engines standardize development and so can reduce some of those pressures, but the demands of rollback bring it back to the forefront.
Thanks to you and the fans for your continued interest and support!
There's even more to come on the topic! Next time we’ll take a look at the SNK community points of view.
You can find more about Code Mystics on Twitter and by visiting their Website
Read my fighting games takes on: Twitter and come learn how to press buttons on YouTube and Twitch. Hell! If you ask nicely I will even coach you on Metafy
This material was created with the support of our Patrons. You can support us!
Become a Patron
T8
MK1
SF6
Scouterz
GBVSR
Project L
SSBU
T7
DBFZ
BH
GGST
SC6
SG2E
KI
MVS
KoFXV
MBTL
SF5
MK11