Beyond emulation: the massive effort to reverse engineer N64 source code

0

Aurich Lawson

Earlier this week, with little warning, the internet was given a Windows executable containing a fully playable PC port of Super Mario 64. Far from being just a usual emulated ROM, this standalone program enables features such as autoscale to any screen resolution, and players are already experimenting the addition of simple reshapers at the level of the graphics card, including ray tracing, also.

The PC port, which was released with little build-up and almost no promotion, was not built from the ground up in a modern game engine in the way of another now missing Super Mario 64 portage projects. And its output has nothing to do with a recent leak of internal Nintendo files dating from the Gamecube era.

Instead, the port appears to be the direct result of a multi-year effort to decompile it Super Mario 64 ROM to parsable C code. This kind of reverse engineering from raw binary to easy-to-read code is not a simple process, but it is an effort that a growing community of amateur decompilers is committed to unlocking the secrets of some of their favorite games.

Decomposition 101

The two-year effort to decompile Super Mario 64 was not started with a Windows executable in mind. Instead, he was primarily driven by speedrunners who wanted to “better understand the game code in order to help find these [speedrunning] exploits, ”according to Kenix, who currently helps lead the Zelda Reverse Engineering Team (ZRET) that resulted from this effort.

A hidden piece of a debugging version of<em>Ocarina of Time</em> which unlocked the game’s reverse engineering secrets. “src =” https://cdn.arstechnica.net/wp-content/uploads/2020/05/zeldadebug-300×225.jpg “width =” 300 “height = “225” srcset = “https://cdn.arstechnica.net/wp-content/uploads/2020/05/zeldadebug.jpg 2x”/></a><figcaption class=
Enlarge / A hidden piece of a debugging version of Ocarina of time which helped unlock some of the game’s reverse-engineering secrets.

The first step in reverse-engineering an N64 ROM is as basic as determining which specific version of the Silicon Graphics IDO Compiler was used to create the ROM in the first place. This requires a lot of trial and error testing, aided in some cases by careful analysis of disclosed debug versions and source code snippets left buried in the ROM files themselves. And even running such an outdated compiler for testing means emulating the SGI workstation system calls that the original N64 developers used, Kenix told Ars.

The next step is just to figure out how the ROM is organized. “How do you know where the functions are? How do I know where the polygon or texture data is? »Said Kénix. “You have to analyze the internal components of the ROM and then generate a way to split it into these different files.”

Fortunately, N64 games organize their files into 16-byte chunks, making it easier to see the empty padding at the end of a file. And a game, like Ocarina of time, use an easy to analyze direct memory access table which sets most of the file limits found in the original ROM (nowadays a tool like N64Split can automate this process). Debug versions of a game can also help reverse engineers document its structure, thanks to the presence of uncompressed game files and C macros like __FILE__ and __LINE__ that reveal internal file names used by Nintendo.

Craft functions

With the known compiler and basic ROM structure, simple decompilation techniques can generate a sprawling list of raw assembly language instructions that are passed to N64 hardware. But converting these instructions into parsable and easily editable C code by humans is far from a straightforward process (and automated tools that convert this assembly code to C often introduce logical errors or obscure the code too severely).

So, a true reverse engineering of an N64 ROM is to go through these assembly code files function by function, converting them by hand into usable C code. And unlike emulation, where “close enough” is sometimes sufficient, precision is important here. “Our goal is to match byte by byte the original assembly code of all functions in the game. [after running through the compiler]”Kenix said.

An example programming environment that allows coders to easily compare the compiled results of their code to the actual game ROM in near real time.
Enlarge / An example programming environment that allows coders to easily compare the compiled results of their code to the actual game ROM in near real time.

Even converting a small function of some assembly instructions in this way can be a complicated process. But individual N64 functions can execute thousands of instructions, and a single N64 set can have thousands of such functions (over 15,700 in the case of Ocarina of time, for an example).

The difficulty may also vary depending on the game. Super Mario 64, Nintendo has compiled its source code without any fancy compilation options, which means that the decompiled assembly language is easier to convert back to C code. For a game like Ocarina of time, however, Nintendo used optimization flags to generate faster code, making the resulting ROM much more difficult to disentangle from its source.

“When there are optimization flags, you have a harder time matching a loop to a ‘for’ vs ‘while’ [statement] etc. “Kenix said.” You should try all matching code patterns until you find the one that matches. “

More than ports

https://www.youtube.com/watch?v=rJUT1el0qFg

Mario looks great in high definition thanks to a PC port, but that’s not the main point of the decompilation efforts.

While ZRET management understands that PC ports are going to be the natural result of their efforts, Kenix said the reverse engineers “take this outside the scope of what we’re doing. We’re just decompiling the game. someone else will inevitably pick it up and write the PC port. “

But even with decompiled C code in hand, creating a PC port is “not as easy as [saying] “Compile it for Windows,” ZRET member Rozlette noted. “There is a lot of code that deals with communicating with N64 hardware. The N64 rendering pipeline is very different from modern OpenGL, for example. “

The process is “close but not quite” as complex as just writing an N64 emulator in the first place, Kenix said. “This is still quite difficult, especially when you consider changes that are considered implicit with a PC target, such as being able to change the resolution or the frame rate,” added Roman, member of ZRET.

Ports aside, having the source code opens up a potential new world of mods and hacks that would be difficult or impossible by simply building on binary ROM. Before Zelda decompilation efforts even started in earnest, for example, more basic reverse engineering efforts were essential to uncover the amazing method to obtain Star Fox 64‘s Arwing in Ocarina of time.

After a few months of work, Zelda’s reverse engineering team only deployed about 15% of the Ocarina of timein C code. Over time, however, they hope to bring the game’s source code to a point of “portability”, where overall changes to the game can be easily C-coded rather than assembler-coded. This is already the case for games like Super Mario 64—Since the source code for the game was released last September, modders have created new tools which allow easy editing of the world, background images, level distortion areas, etc.

For Zelda, the “modifiable” source code could also lead to a new, more complete version of the Ocarina of time Randomizer, which moves game objects and objectives. A randomizer built from source code could exist as a standalone ROM rather than a patch that needs to be applied to the game ROM with a new seed each time, for example. example.

“The knowledge gained through binary hacking of the game over the years has made [the Ocarina of Time randomizer] possible without decompilation, ”said ZRET member Fig. ” We can do a lot with just some editing and general knowledge of the game. C just makes it easier and will allow things that would generally be considered too difficult. ”

For some retro-engineers, however, unraveling the mysteries of the N64 code is its own reward. “I do this because of my childhood love for the game,” Rozlette told Ars. “It sounds like a big puzzle to me where every function is a piece. It’s very rewarding for me when I work on an unknown code function and then realize that I recognize what it does in the game, like, “Hey! This is the function that generates rubies when you cut grass! ‘”

Zelda’s reverse engineering team is always on the lookout for more volunteers. You can register via Team Discord channel.


Source link

Share.

About Author

Comments are closed.