He's a total legend, yet apparently he's never met Bill Gates in person from what he said in an interview in the Dave's Garage YouTube channel a few years ago. You'd think that someone who's been that prominent for so long in the company would have been invited to a company dinner where he was present or something.
IMHO, if something isn’t part of the contract, it should be randomized. Eg if iteration order of maps isn’t guaranteed in your language, then your language should go out of its way to randomize it. Otherwise, you end up with brittle code: code that works fine until it doesn’t.
There are various compiler options like -ftrivial-auto-var-init to initialize uninitialized variables to specific (or random) values in some situations, but overall, randomizing (or zeroing) the full content of the stack in each function call would be a horrendous performance regression and isn't done for this reason.
Regarding contracts, there's an additional lesson here, quoting from the source:
> This is an interesting lesson in compatibility: even changes to the stack layout of the internal implementations can have compatibility implications if an application is bugged and unintentionally relies on a specific behavior.
I suppose this is why Linux kernel maintainers insist on never breaking user space.
I agree, this can also detect brittle tests (e.g, test methods/classes that only pass if executed in a particular order). But applying it for all data could be expensive computation-wise
Not really the ethos of C(++), though of course this particular bug would be easily caught by running a debug build (even 20 years ago). However, this being a game "true" debug builds were probably too slow to be usable. That was at least my experience doing gamedev in that timeframe. Then again code holding up for 20 years in that line of biz is more than sufficient anyway :)
> Not ignore the compilation warnings – this code most likely threw a warning in the original code that was either ignored or disabled!
What compiler error would you expect here? Maybe not checking the return value from scanf to make sure it matches the number of parameters? Otherwise this seems like a data file error that the compiler would have no clue about.
The compiler has no way of knowing that the memory would be undefined, not unless it somehow can verify the data file. The most I think it can do is flag the program for not checking the return value of scanf, but even that is unlikely to be true since the program probably was checking for end of file which is also in the return value. It was failing to check the number of matched parameters. This is the kind of error that is easy to miss given the semantics of scanf.
i think the shift will be from craftmens to trademens in regards to general software engineers, but these are type of writes up stem of a artisan style all to its own.
We have been seeing this shift for a while, where "software engineers" graduate from 3 month bootcamps. Except now most likely they will not be earning 500k making crud apps.
When I worked at Microsoft and I had downtime I would sometimes read the code for app compatibility shims out of pure curiosity.
Win9x video games that made bad assumptions about the stack were a theme I saw. One of the differences between win9x and NT based windows is that kernel32 (later kernelbase) is a now user mode wrapper atop ntdll, whereas in the olden days kernel32 would trap directly into the kernel. This means that kernel32 uses more user mode stack space in NT. A badly behaving app that stored data to the left of the stack pointer and called into kernel32 might see its data structures clobbered in NT and not in 9x. So there were compatibility hacks that temporarily moved the stack pointer for certain apps.
> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. The real issue here is the game relying on undefined behavior (uninitialized local variables), and to be honest, I’m shocked that the game didn’t hit this bug on so many OS versions, although as I pointed out earlier, it was extremely close
This sentence is the real takeaway point of the article. Undefined behavior is extremely insidious and can lull you into the belief that you were right, when you already made a mistake 1000 steps ago but it only got triggered now.
I emphasized this point in my article from years ago (but after the game was released):
> When a C or C++ program triggers undefined behavior, anything is allowed to happen in the program execution. And by anything, I really mean anything: The program can crash with an error message, it can silently corrupt data, it can morph into a colorful video game, or it can even give the right result.
> If you’re lucky, the program triggering UB will show an appropriate error message and/or crash, making you immediately aware that something went wrong. If you’re unlucky, the program will quietly mangle data, and by the time you notice the problem (via effects such as crashes or incorrect output) the root cause has been buried in the past execution history. And if you’re very unlucky, the program will do exactly what you hoped it should do, until you change some unrelated code / compiler versions / compiler vendors / operating systems / hardware platforms – and then a new bug becomes visible, and you have no clue why seemingly correct code now fails to work properly.
As I wrote in my article, this point really got hammered into me when a coworker showed me a patch that he made - which added a couple of innocuous, totally correct print statements to an existing C++ program - and that triggered a crash. But without his print statements, there was no crash. It turned out that there was a preexisting out-of-bounds array write, and the layout of the stack/heap somehow masked that problem before, and his unlucky prints unmasked the problem.
Okay so then, how can we do better as developers today?
0) Read, understand, and memorize what actions in C or C++ are undefined behavior. Avoid them in your code at all costs. Also obey the preconditions of any API you use, whether in the standard library, operating system, etc.
1) Compile your application in Debug mode and compare its behavior to Release mode. If they differ by anything other than speed, then you have a serious problem on your hands.
2) Compile and run with sanitizers like -fsanitize=undefined,address to catch undefined behavior at runtime.
3) Use managed languages like Java, C#, Python, etc. where you basically don't have to worry about UB in normal day-to-day code. Or use very well-designed low-level languages like Rust that are safe by default and minimize your exposure to UB when you really need to do advanced things. Whereas C and C++ have been a bonanza of UB like we have never seen before in any other language.
Other than C#, there is no reason to use those other languages for game dev. Unless the game is fairly simple, or you want to risk a fairly long project by employing a language that hasn't been proven in tge space yet (Rust). No shade at any of those languages, I don't even like C#, just being pragmatic.
I would add: code defensively. Initialize your variables (either to a sensible value, or an outrageously wrong value) before passing pointers to them, even when you "know" that the value will be overwritten. Check for errors. Always consider what happens when things go wrong, not just when things go right. Any time you find yourself thinking, "condition X is guaranteed to hold, so I don't need to check for it" consider checking it anyway just in case you're wrong about that, or it changes later.
I learned this lesson many moons ago, on a Fortran code I wrote for a university assignment. It was a basic genetic algorithm, and for some reason it was converging much more slowly than expected. So I was sprinkling some WRITEs to debug, and suddenly the code converged a hundred times faster.
Surprised to see the return value of sscanf being ignored, that seems like a pretty rookie mistake, and this bug would never have made it out of the original programmer's system if they had bothered to check it.
Once this category of error is raised to your attention, you start to notice it more and more.
A little piece of technology made sense in the original context, but then it got moved to a different context without realizing that move broke the contract. Specifically in this case a flying boat became an airplane.
---
I recently worked a bug that feels very similar:
A linux cups printer would not print to the selected tray, instead it always requested manual feed.
Ok. Try a bunch of command line options, same issue.
Ok. Make the selection directly in the PPD (postscript printer definition) file. Same issue.
Ok! Decompile the PXL file. Wrong tray is set in pxl file... why?
Check Debug2 log level for cups - Wrong MediaPosition is being sent to ghostscript (which compiles the printer options into the print job) by a cups filter... why?
Cups filter is translating the MediaPosition from the PPD file... because the philosophy of cups is to do what the user intended. The intention inferred from MediaPosition in the PPD file (postscript printer definition) is that the MediaPosition corresponds to the PWG (Printer Working Group) MediaPosition, NOT the vendor MediaPosition (or local equivalent - in this case MediaSource).
AHA!! My PPD file had been copied from a previous generation of server, from a time when that cups filter did NOT translate the MediaPosition, so the VENDOR MediaSource numbers were used. Historically, this makes sense. The vendor tray number is set in the vendor ppd file because cups didn't know how to translate that.
Fast forward to a new execution context, and cups filters have gotten better at translating user intention, now it's translating a number that doesn't need to be translated, and silently selecting the wrong tray.
TLDR; There is no such thing as a printer command, only printer suggestions.
tl;dr of the explanation: the Skimmer vehicle is missing a wheel scale definition, so its wheel scale gets read from uninitialized memory. On previous versions of Windows, this happened to be the wheel scale of the previously-loaded vehicle, so things happened to work fine. Starting on Windows 11 24H2, LeaveCriticalSection (which gets called between loading vehicles) uses more stack space than before, so it now overwrites that memory with a gigantic value, resulting in the Skimmer spawning so high up that it may as well not exist at all.
My takeaway, speaking as someone who leans towards functional programming and immutability, is "this is yet another example of a mutability problem that could never happen in a functional context"
(so, for example, this bug would have never been created by Rust unless it was deeply misused)
This is more of a problem of the C/C++ standard that it allows uninitialized variables but doesn't give them defined values, considering it "undefined behavior" to read from an uninitialized variable. Java, for example, doesn't have this particular problem because it does specify default values for variables.
FWIW I think a linter or other similar code quality checker would have caught this as well. From a practical perspective (e.g., how do you prevent this from happening again in your game studio's multi-million line code base) that would have been the right thing to do here.
The code would have failed because you can't use an uninitialized variable, so you would have had to set it to a default. You don't just get random garbage from the stack.
It would have forced you to either specify a default or fail pretty loudly as soon as you launched the game, both much better than leaving a bug there just for it to resurface 20 years later.
Most popular languages would prevent this. In this case it’s as simple as having more sensible reader API than sscanf in standard library and forcing variables to be initialized.
I'd actually say that Rust is a third option between "everything is immutable" and "mutable soup". Rust is more of "one mutator at a time". Because, Rust really embraces being able to mutate stuff (so not functional in that sense), it just makes sure that it's in a controlled way.
Could you elaborate? I cannot see how a functional programming language would have protected you from reading a non existing value while not providing a default
I've never heard of a functional language that would allow you to initialize a value to whatever value the system memory already had in that memory location. In languages that allow nil, it would at least be nil; in languages that don't, you would have gotten an error about an uninitialized and undefaulted value. In any typed language, you would have also gotten an error.
It's true that C may be unique-ish in this regard though- this bug also couldn't happen in Ruby, which is not a functional language, but Ruby certainly still makes undefined behaviors much more possible than in other languages like Elixir.
To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world.
On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
JSON didn't save them: This is the same studio that handrolled a JSON parser with accidentally quadratic time complexity, making most players wait 3 to 10 minutes to load GTA Online, for 7 years, until a player got tired and found the root cause.
The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler.
Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
While it doesn't excuse the bad habits, we do have to keep in mind C++98 (or whatever more ancient was used back then) didn't have the simple initializers we now take for granted. You couldn't just do 'Type myStruct = {};' to null-initialize it, you had to manually NULL all nested fields. God forbid you change the order of the variables in the struct if you're nesting them and forget to update it everywhere. It was just considerably more practical to do 'Type myStruct;' then set the fields when needed.
I haven't been using C++ for a number of years but I think you could set the default values of fields even back then. Something like
struct test {
int my_int = 0;
int* my_ptr = std::nullptr;
};
Or is this something more recent ?
You cannot initialize them with a different value unless you also write a constructor, but it not the issue here (since you are supposed to read them from the file system)
This is the thing that drives artists and craftsmen to despair and drink: That a flawed, buggy, poor quality work can be "successful" while something beautiful and technically perfect can fail.
San Andreas might be rough under the hood, but on the surface it was nothing short of a masterpiece of game design. The engine was so complex and the cities felt alive, and the game could handle a lot of general nonsense. Still one of my favorite go-to games.
The job of the artist is to take the years of expertise and distill it down into something "enjoyable." The hardest mental hurdle to get over is that people just don't care about the technicals being perfect. Hell, the final product doesn't even need to be beautiful; it just needs to be arresting.
One artist can take months painting a picture of a landscape where everything is perfect. And the next artist can throw 4 colors of paint at a wall. The fact that lots of people enjoy the work of the second artist doesn't invalidate the work of the first. The two artists are focusing on different things; and it's possible for both of them to be successful at reaching their goals.
Let's be clear that it was a success very much in spite of UB, not because of it. And there was still a cost--likely at least hundreds of person-hours spent fixing other similar bugs due to UB (if not more).
I worked in gamedev around the time this game was made and this would have been very much an ordinary, everyday kind of bug. The only really exceptional thing about it is that it was discovered after such a long time.
> it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler.
Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.
You’re not entirely wrong, but a library doesn’t have to be “heavyweight” in order to be bulletproof. And you can load the library during startup and then unload it after; it doesn’t have to stick around for the whole run time of the game. Modern OSes will reclaim the pages after you stop using them, if there is memory pressure. Of course the PS2 didn’t do that I am sure.
Meanwhile, in a certain modern OS, unloading a library is too broken to the point that people are discouraged to do so... Try to unload GLib [0] from your process :p
Unloading C libraries is fundamentally fraught with peril. It's incredibly difficult to ensure that no dangling pointers to the library remain when it's unloaded. It's really fun to debug, too. The code responsible for the crash literally is not present in the process at the time of the crash!
Wow I had no idea YAML was that old. I always thought it was created some time around when CI/CD became popular. Now I'm really curious how it ended up as a superset of JSON.
Easier for internal development. Non- or less technical team members can tweak values without having to rebuild these binary files. Possibly also easier for lightweight modding externally as well.
This isn't that uncommon - look at something like Diablo 2 which has a huge amount of game data defined from text files (I think these are encoded to binary when shipped but it was clearly useful to give the game a mode where it'd load them all from text on startup).
Definitely, and architectures back then were far less standardized. The Xbox 360 was a big-endian PowerPC CPU, the PS2 had a custom RISC-based CPU. On the desktop, this was still the era of PowerPC-based Macs. Far easier (and I would argue safer) to use a standard, portable sscanf-like function with some ascii text, than figure out how to bake your binaries into every memory and CPU layout combination you might care about.
Video games are made by a lot of non-programmers who will be much more comfortable adjusting values in a text file than they are hex editing something.
Besides, the complaint about not having a heavyweight parser here is weird. This is supposed to be "trusted data", you shouldn't have to treat the file as a threat, so a single line sscanf that's just dumping parsed csv attributes into memory is pretty great IMO.
Definitely initialize variables when it comes to C though.
The flaw isn't the language. The issue is a 0.5x programmer not knowing to avoid sscanf() and failing to default and validate the results. This could be handled competently with strtok() parsing the lines without needing a more complicated file format.
Worked fine on the target machines and the "0.5x programmer" got to see their family for winter holiday. Or are you saying they should have defensively programmed around a bug manifesting 21 years later and skip seeing their family during crunch time?
To be honest, I just don't like how you disparaged the programmer out-of-context. Talk is cheap.
Well-written 3rd party serialization libraries weren't exactly easy to come by 20 years ago, at least from what I can recall. Your best bet was using an XML library, but XML parsing was quite resource heavy. Many that seemed well designed turned out to be a security nightmare (Java serialization).
I disagree. JSON is 25 years old, and SAX parsers are 22. A SAX parser is the opposite of “resource heavy”, since it is event driven. The parser does not maintain complex state, although you might have to manage some state in order to correctly extract your objects from the XML. Granted, it wouldn’t have integrated nicely with C to generate the parser code from your struct definition back then, but the basics were there.
But it is even more important for today’s game studios to see and understand the mistakes that yesterday’s studios made. That’s the only way to avoid making them all over again.
I’ll be the first to defend the greybeards I’ve befriended and learned from in AAA, but having seen codebases of that age and earlier, the “meta” around game development was different back then. I think the internet really changed things for the better.
Your average hire for the time might have been self-taught with the occasional C89 tutorial book and two years of Digipen. Today’s graduates going into games have fallen asleep to YouTube lectures of Scott Meyers and memorized all the literature on a fixed timestep.
They wrote a JSON “parser” using sscanf. sscanf is not bulletproof! Just use an open source library instead of writing something yourself. You will still be a real programmer, but you will finish your game sooner and you won't have embarrassing stories written about you.
Yes, but now it's in the realm of ~3 minutes, and not ~8 minutes even on a top-spec PC, right? I really liked the game, but waiting 8 minutes to load just to get griefed by hackers within seconds of walking outside... I don't understand how that game makes any money.
IMO it varies widely. This past weekend it was taking me multiple attempts to get logged in to a public lobby— after waiting ~5-10 minutes!
Nothing has changed appreciably. If they would let you login to a private invite-only lobby that would likely speed things up greatly— but it’ll never happen.
I don’t think its ever been an option to login directly to an invite-only lobby. But then I have taken multiple multi-year breaks! I was pleasantly surprised you can actually play most of the game in a private lobby now… that is a huge change and I am not at all certain when it occurred.
Nitpicking: What you're describing is called a "variable ratio reinforcement schedule", and is considered to be the most effective form of operant conditioning.
However, it's not even remotely "like crack". Crack is really really really really fun, period, no "just enough of the time" about it. The reason people get hooked on crack is because it's guaranteed to be fun.
If I had to choose a substance that most closely mirrored variable ratio reinforcement conditioning, it'd probably be ketamine.
Putting the (very valid) reasons for not having human-readable game saves aside, are you sure it's worse than using a 3rd party library that's built to accept semi-valid input values, possibly evaluates user input in some way and has difficult to debug bugs that occur only under certain inputs? I agree that writing a stable and safe parser for a binary data file isn't easy, but there's less things that can go wrong when you can hardcode it to reject any remotely suspicious input. Third party XML/JSON libraries OTOH try to interpret as much as possible, even when the values are bogus. Also no need to deal with different text encoding bugs, line endings...
You misunderstood. Game developers should use a _good_ third–party library, not a _bad_ one. At a minimum they should be able to read the source code so that they know it is good. Thus open source libraries should be at the top of the list.
If you don't know what “good” looks like, take a look at [Serde](https://serde.rs/). It’s for Rust, but its features and overall design are something you should attempt to approach no matter what language you’re writing in.
I disagree. Serde is not merely good, it is excellent.
The only C code that I have recently interacted with uses a home–grown JSON “library” that is actually pretty good. In particular it produces good error messages. If it were extracted out into its own project then I would be able to recommend it as a library.
But how is that C project using a custom made JSON library doing better than Rockstar games doing the same? Because that library has good error messages?
Apart from that, many of us thought that Java serialization was good if just used correctly, that IE's XML parsing capabilities were good if just used correctly, and so on. We were all very wrong. And a 3rd party library would be just some code taken from the web, or some proprietary solution where you'd once again have to trust the vendor.
It’s good because they have spent hundreds or thousands of hours polishing and improving it. It’s paid off too, because stable releases never have broken data files any more. Any mistakes that do get made are usually found and fixed before the code is ever committed. Even the experimental branch rarely sees broken data files. It’s more likely to see these error messages when loading save files, because the code to read old save files and convert them for newer versions of the game is the hardest to write and test.
> And a 3rd party library would be just some code taken from the web, or some proprietary solution where you'd once again have to trust the vendor.
Open source exists for a reason, and had already existed for ~15 years by the time this game was begun. 20 years later there are even fewer excuses to be stuck using some crappy code that you bought from a vendor and cannot fix.
But also keep in mind in 2004 the legality of many open source projects was not really tested very well in court. Pretty sure that was right around the time one of the bigger linux distros was throwing its weight around and suing people. So you want to ship on PS2 and XBOX and PC and GameCube. Can you use that lib from inside windows? Not really. Can you build/vs buy? Buy means you need the code and probably will have to port it to PS2/GameCube yourself. Can you use that opensource lib? Probably, but legal is still dragging its feet, and you get to port it to PS2. Meanwhile your devs need a library 3 weeks ago and have hacked something together from an older codebase that your company owns and it works and means you can hit your gold master date.
Would you do that now? No. You would grab one of the multitudes of decent libs out there and make sure you are following the terms correctly. Back then? Yeah I can totally see it happening. Open source was semi legally grey/opaque to many corporations. They were scared to death of losing control of their secret sauce code and getting sued.
Before game companies earned all their profit through selling cosmetics and premium currency nobody cared if you cheated at your single player game and nobody SHOULD care if you want to give yourself extra money.
It's only now that single player progress is profitable to sell that video games have taken save game encryption to be default.
The trouble is that if some weirdness happens because of the edit, you've got to handle it even if you say it would be reasonable to assume that it's outside of being supported. Maybe you spend a bit more time defensive coding around what inputs it reads from the file, maybe a certain proportion of users doing the save edit see bugs in an apparently unrelated part of the game and seek support (and their bug report might not be complete with all the details), developers spend time to chase down what went wrong, maybe they bad-mouth it on forums which affects sales - there's going to be some cost to handling all of that.
One of the anecdotes from Titan Quest developed by Iron Lore is that their copy protection had multiple checks, crackers removed the early checks to get the game running but later 'tripwires' as you progress through the game remained and the game appeared to crash. So the game earned a reputation for being buggy for something no normal user would hit running the game as intended.
>The trouble is that if some weirdness happens because of the edit, you've got to handle it even if you say it would be reasonable to assume that it's outside of being supported.
What? No. What even are you suggesting? Hell, games with OFFICIAL MODDING SUPPORT still require you submit bug reports with no mods running.
Editing game files has always been "you are on your own", even editing standard Unreal config files is something you wont get support for, and they are trivial human readable files with well known standards.
>One of the anecdotes from Titan Quest
Any actual support for this anecdote? Lots of games have anti-piracy features that sneakily cause problems, and even could fire accidentally. None of those games get a reputation for being buggy. Games like Earthbound would make the game super hard and even delete your save game at the very end. Batman games would nerf your gliding ability. Game Dev Tycoon would kill your business due to piracy.
None of these affected the broad reputation of the game. Most of them are pretty good marketing in fact.
Use a debugger folks. A 10x dev cited this story to me about the ills of not using one.
This is the kind of thing I'd expect from Raymond Chen - which is extremely high praise!
I'm glad they tracked it down even further to figure out exactly why.
Raymond is a wizard. Read his blogs for many years and love his style and knowledge.
He's a total legend, yet apparently he's never met Bill Gates in person from what he said in an interview in the Dave's Garage YouTube channel a few years ago. You'd think that someone who's been that prominent for so long in the company would have been invited to a company dinner where he was present or something.
Raymond knows everything. From microcode bugs on Alpha AXP to template meta programming to UI.
IMHO, if something isn’t part of the contract, it should be randomized. Eg if iteration order of maps isn’t guaranteed in your language, then your language should go out of its way to randomize it. Otherwise, you end up with brittle code: code that works fine until it doesn’t.
There are various compiler options like -ftrivial-auto-var-init to initialize uninitialized variables to specific (or random) values in some situations, but overall, randomizing (or zeroing) the full content of the stack in each function call would be a horrendous performance regression and isn't done for this reason.
Regarding contracts, there's an additional lesson here, quoting from the source:
> This is an interesting lesson in compatibility: even changes to the stack layout of the internal implementations can have compatibility implications if an application is bugged and unintentionally relies on a specific behavior.
I suppose this is why Linux kernel maintainers insist on never breaking user space.
I agree, this can also detect brittle tests (e.g, test methods/classes that only pass if executed in a particular order). But applying it for all data could be expensive computation-wise
Then you are wasting runtime clock cycles randomizing lists.
Not really the ethos of C(++), though of course this particular bug would be easily caught by running a debug build (even 20 years ago). However, this being a game "true" debug builds were probably too slow to be usable. That was at least my experience doing gamedev in that timeframe. Then again code holding up for 20 years in that line of biz is more than sufficient anyway :)
iirc, Go intentionally randomizes map ordering for just this reason.
Yep, and then you get crash reports you can’t reproduce.
> Not ignore the compilation warnings – this code most likely threw a warning in the original code that was either ignored or disabled!
What compiler error would you expect here? Maybe not checking the return value from scanf to make sure it matches the number of parameters? Otherwise this seems like a data file error that the compiler would have no clue about.
Undefined behavior to access the uninitialized memory. A sanitizer would have flagged that.
The compiler has no way of knowing that the memory would be undefined, not unless it somehow can verify the data file. The most I think it can do is flag the program for not checking the return value of scanf, but even that is unlikely to be true since the program probably was checking for end of file which is also in the return value. It was failing to check the number of matched parameters. This is the kind of error that is easy to miss given the semantics of scanf.
Uninitialized variables are a really common case.
Yeah, the debugging here is great, but the actual cause is super mild.
I always enjoy reading deeply technical writeups like these. I only wonder how much more rare they may or may not get in the AI era.
i think the shift will be from craftmens to trademens in regards to general software engineers, but these are type of writes up stem of a artisan style all to its own.
We have been seeing this shift for a while, where "software engineers" graduate from 3 month bootcamps. Except now most likely they will not be earning 500k making crud apps.
For anyone with access issues
https://web.archive.org/web/20250423144746/https://cookieplm...
I'm more curious in what changed with the critical section locking/unlocking implementation in this version of Windows!
It looks like the utilized stack, or a stack protection area, increased.
When I worked at Microsoft and I had downtime I would sometimes read the code for app compatibility shims out of pure curiosity.
Win9x video games that made bad assumptions about the stack were a theme I saw. One of the differences between win9x and NT based windows is that kernel32 (later kernelbase) is a now user mode wrapper atop ntdll, whereas in the olden days kernel32 would trap directly into the kernel. This means that kernel32 uses more user mode stack space in NT. A badly behaving app that stored data to the left of the stack pointer and called into kernel32 might see its data structures clobbered in NT and not in 9x. So there were compatibility hacks that temporarily moved the stack pointer for certain apps.
> all these findings prove that the bug is NOT an issue with Windows 11 24H2, as things like the way the stack is used by internal WinAPI functions are not contractual and they may change at any time, with no prior notice. The real issue here is the game relying on undefined behavior (uninitialized local variables), and to be honest, I’m shocked that the game didn’t hit this bug on so many OS versions, although as I pointed out earlier, it was extremely close
This sentence is the real takeaway point of the article. Undefined behavior is extremely insidious and can lull you into the belief that you were right, when you already made a mistake 1000 steps ago but it only got triggered now.
I emphasized this point in my article from years ago (but after the game was released):
> When a C or C++ program triggers undefined behavior, anything is allowed to happen in the program execution. And by anything, I really mean anything: The program can crash with an error message, it can silently corrupt data, it can morph into a colorful video game, or it can even give the right result.
> If you’re lucky, the program triggering UB will show an appropriate error message and/or crash, making you immediately aware that something went wrong. If you’re unlucky, the program will quietly mangle data, and by the time you notice the problem (via effects such as crashes or incorrect output) the root cause has been buried in the past execution history. And if you’re very unlucky, the program will do exactly what you hoped it should do, until you change some unrelated code / compiler versions / compiler vendors / operating systems / hardware platforms – and then a new bug becomes visible, and you have no clue why seemingly correct code now fails to work properly.
-- https://www.nayuki.io/page/undefined-behavior-in-c-and-cplus...
As I wrote in my article, this point really got hammered into me when a coworker showed me a patch that he made - which added a couple of innocuous, totally correct print statements to an existing C++ program - and that triggered a crash. But without his print statements, there was no crash. It turned out that there was a preexisting out-of-bounds array write, and the layout of the stack/heap somehow masked that problem before, and his unlucky prints unmasked the problem.
Okay so then, how can we do better as developers today?
0) Read, understand, and memorize what actions in C or C++ are undefined behavior. Avoid them in your code at all costs. Also obey the preconditions of any API you use, whether in the standard library, operating system, etc.
1) Compile your application in Debug mode and compare its behavior to Release mode. If they differ by anything other than speed, then you have a serious problem on your hands.
2) Compile and run with sanitizers like -fsanitize=undefined,address to catch undefined behavior at runtime.
3) Use managed languages like Java, C#, Python, etc. where you basically don't have to worry about UB in normal day-to-day code. Or use very well-designed low-level languages like Rust that are safe by default and minimize your exposure to UB when you really need to do advanced things. Whereas C and C++ have been a bonanza of UB like we have never seen before in any other language.
Other than C#, there is no reason to use those other languages for game dev. Unless the game is fairly simple, or you want to risk a fairly long project by employing a language that hasn't been proven in tge space yet (Rust). No shade at any of those languages, I don't even like C#, just being pragmatic.
I would add: code defensively. Initialize your variables (either to a sensible value, or an outrageously wrong value) before passing pointers to them, even when you "know" that the value will be overwritten. Check for errors. Always consider what happens when things go wrong, not just when things go right. Any time you find yourself thinking, "condition X is guaranteed to hold, so I don't need to check for it" consider checking it anyway just in case you're wrong about that, or it changes later.
I learned this lesson many moons ago, on a Fortran code I wrote for a university assignment. It was a basic genetic algorithm, and for some reason it was converging much more slowly than expected. So I was sprinkling some WRITEs to debug, and suddenly the code converged a hundred times faster.
Much love to Silent, who’s been improving my favorite game for over... a decade now?
20 years and still nobody has realized what the back of this sign is about. IYKYK.
https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...
fixed link (the above did not work for me) https://static.wikia.nocookie.net/gta-myths/images/f/f1/Egg_...
And if I don't know?
Apparently the planes are buggy in all sorts of ways, although I don't remember them being this bad... https://youtu.be/hrJ0eVY5ACw
Surprised to see the return value of sscanf being ignored, that seems like a pretty rookie mistake, and this bug would never have made it out of the original programmer's system if they had bothered to check it.
Once this category of error is raised to your attention, you start to notice it more and more.
A little piece of technology made sense in the original context, but then it got moved to a different context without realizing that move broke the contract. Specifically in this case a flying boat became an airplane.
---
I recently worked a bug that feels very similar:
A linux cups printer would not print to the selected tray, instead it always requested manual feed.
Ok. Try a bunch of command line options, same issue.
Ok. Make the selection directly in the PPD (postscript printer definition) file. Same issue.
Ok! Decompile the PXL file. Wrong tray is set in pxl file... why?
Check Debug2 log level for cups - Wrong MediaPosition is being sent to ghostscript (which compiles the printer options into the print job) by a cups filter... why?
Cups filter is translating the MediaPosition from the PPD file... because the philosophy of cups is to do what the user intended. The intention inferred from MediaPosition in the PPD file (postscript printer definition) is that the MediaPosition corresponds to the PWG (Printer Working Group) MediaPosition, NOT the vendor MediaPosition (or local equivalent - in this case MediaSource).
AHA!! My PPD file had been copied from a previous generation of server, from a time when that cups filter did NOT translate the MediaPosition, so the VENDOR MediaSource numbers were used. Historically, this makes sense. The vendor tray number is set in the vendor ppd file because cups didn't know how to translate that.
Fast forward to a new execution context, and cups filters have gotten better at translating user intention, now it's translating a number that doesn't need to be translated, and silently selecting the wrong tray.
TLDR; There is no such thing as a printer command, only printer suggestions.
I like the one where you shoot at the moon and it gets closer
That isn't a bug. https://x.com/ObbeVermeij/status/1757572432863384046
[dead]
tl;dr of the explanation: the Skimmer vehicle is missing a wheel scale definition, so its wheel scale gets read from uninitialized memory. On previous versions of Windows, this happened to be the wheel scale of the previously-loaded vehicle, so things happened to work fine. Starting on Windows 11 24H2, LeaveCriticalSection (which gets called between loading vehicles) uses more stack space than before, so it now overwrites that memory with a gigantic value, resulting in the Skimmer spawning so high up that it may as well not exist at all.
I wonder if they fixed the vehicle definition file as well, or just the parser. The latter would be an incomplete fix.
Given that those parameters are for wheels on a plane that doesn’t have wheels, I would say fixing the parser is the better fix
It is, but who enjoys fixing parser issues...
My takeaway, speaking as someone who leans towards functional programming and immutability, is "this is yet another example of a mutability problem that could never happen in a functional context"
(so, for example, this bug would have never been created by Rust unless it was deeply misused)
This is more of a problem of the C/C++ standard that it allows uninitialized variables but doesn't give them defined values, considering it "undefined behavior" to read from an uninitialized variable. Java, for example, doesn't have this particular problem because it does specify default values for variables.
FWIW I think a linter or other similar code quality checker would have caught this as well. From a practical perspective (e.g., how do you prevent this from happening again in your game studio's multi-million line code base) that would have been the right thing to do here.
Rust protects you from external file data you read being incorrect?
That's one hell of a language!
The code would have failed because you can't use an uninitialized variable, so you would have had to set it to a default. You don't just get random garbage from the stack.
It would have forced you to either specify a default or fail pretty loudly as soon as you launched the game, both much better than leaving a bug there just for it to resurface 20 years later.
Of course not, but this here was a memory access error and rust would have prevented this.
Most popular languages would prevent this. In this case it’s as simple as having more sensible reader API than sscanf in standard library and forcing variables to be initialized.
I'd actually say that Rust is a third option between "everything is immutable" and "mutable soup". Rust is more of "one mutator at a time". Because, Rust really embraces being able to mutate stuff (so not functional in that sense), it just makes sure that it's in a controlled way.
Could you elaborate? I cannot see how a functional programming language would have protected you from reading a non existing value while not providing a default
I've never heard of a functional language that would allow you to initialize a value to whatever value the system memory already had in that memory location. In languages that allow nil, it would at least be nil; in languages that don't, you would have gotten an error about an uninitialized and undefaulted value. In any typed language, you would have also gotten an error.
It's true that C may be unique-ish in this regard though- this bug also couldn't happen in Ruby, which is not a functional language, but Ruby certainly still makes undefined behaviors much more possible than in other languages like Elixir.
[dead]
Just like $dayjob.
pretty wild how bugs can stick around that long - id never think something from 20 years ago would pop up just cause windows changed
For some reason this domain is blocked by work dns filtering?
Let your IT department know that their denylist is broken.
To u/db48x whose post got flagged and doesn't reappear despite me vouching for it as I think they have a point (at least for modern games): GTA San Andreas was released in 2004. Back then, YAML was in its infancy (2001) and JSON was only standardized informally in 2006, and XML wasn't something widely used outside of the Java world.
On top of that, the hardware requirements (256MB of system RAM, and the PlayStation 2 only had 32MB) made it enough of a challenge to get the game running at all. Throwing in a heavyweight parsing library for either of these three languages was out of the question.
JSON didn't save them: This is the same studio that handrolled a JSON parser with accidentally quadratic time complexity, making most players wait 3 to 10 minutes to load GTA Online, for 7 years, until a player got tired and found the root cause.
https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
The comment reappeared, and while you're right about using proper libraries to handle data, it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that can be fed to the compiler.
Most of the time, the programmers who do this do not follow the simple rule that Stroustrup said which is to define or initialize a variable where you declare it (i.e. declare it before using it), and which would solve a lot of bugs in C++.
While it doesn't excuse the bad habits, we do have to keep in mind C++98 (or whatever more ancient was used back then) didn't have the simple initializers we now take for granted. You couldn't just do 'Type myStruct = {};' to null-initialize it, you had to manually NULL all nested fields. God forbid you change the order of the variables in the struct if you're nesting them and forget to update it everywhere. It was just considerably more practical to do 'Type myStruct;' then set the fields when needed.
I haven't been using C++ for a number of years but I think you could set the default values of fields even back then. Something like
Or is this something more recent ?You cannot initialize them with a different value unless you also write a constructor, but it not the issue here (since you are supposed to read them from the file system)
You could always `bzero` or `memset` the entire struct to 0.
But only if it contains strictly POD members, otherwise it's UB.
And it did not matter at all. The game shipped and was a success.
This is the thing that drives artists and craftsmen to despair and drink: That a flawed, buggy, poor quality work can be "successful" while something beautiful and technically perfect can fail.
San Andreas might be rough under the hood, but on the surface it was nothing short of a masterpiece of game design. The engine was so complex and the cities felt alive, and the game could handle a lot of general nonsense. Still one of my favorite go-to games.
The job of the artist is to take the years of expertise and distill it down into something "enjoyable." The hardest mental hurdle to get over is that people just don't care about the technicals being perfect. Hell, the final product doesn't even need to be beautiful; it just needs to be arresting.
Heck, sometimes the thing that's most interesting about a work is people arguing over whether or not it's art.
One artist can take months painting a picture of a landscape where everything is perfect. And the next artist can throw 4 colors of paint at a wall. The fact that lots of people enjoy the work of the second artist doesn't invalidate the work of the first. The two artists are focusing on different things; and it's possible for both of them to be successful at reaching their goals.
Let's be clear that it was a success very much in spite of UB, not because of it. And there was still a cost--likely at least hundreds of person-hours spent fixing other similar bugs due to UB (if not more).
I worked in gamedev around the time this game was made and this would have been very much an ordinary, everyday kind of bug. The only really exceptional thing about it is that it was discovered after such a long time.
> it doesn't excuse the "undefined behavior (uninitialized local variables)" that I still see all the time despite all the warning and error flags that you can feed to the compiler.
Yeah but we're talking about a 2004 game that was pretty rushed after 2002's Vice City (and I wouldn't be surprised if the bug in the ingestion code didn't exist there as well, just wasn't triggered due to the lack of planes except that darn RC Chopper and RC plane from that bombing run mission). Back then, the tooling to spot UB and code smell didn't even exist or, if at all, it was very rudimentary, or the warnings that did come up were just ignored because everything seemed to work.
You’re not entirely wrong, but a library doesn’t have to be “heavyweight” in order to be bulletproof. And you can load the library during startup and then unload it after; it doesn’t have to stick around for the whole run time of the game. Modern OSes will reclaim the pages after you stop using them, if there is memory pressure. Of course the PS2 didn’t do that I am sure.
Meanwhile, in a certain modern OS, unloading a library is too broken to the point that people are discouraged to do so... Try to unload GLib [0] from your process :p
[0] https://docs.gtk.org/glib/
Unloading C libraries is fundamentally fraught with peril. It's incredibly difficult to ensure that no dangling pointers to the library remain when it's unloaded. It's really fun to debug, too. The code responsible for the crash literally is not present in the process at the time of the crash!
Wow I had no idea YAML was that old. I always thought it was created some time around when CI/CD became popular. Now I'm really curious how it ended up as a superset of JSON.
Why weren't binary files used like I would expect in the 1990's DOS game? fread into a struct and all that
Easier for internal development. Non- or less technical team members can tweak values without having to rebuild these binary files. Possibly also easier for lightweight modding externally as well.
This isn't that uncommon - look at something like Diablo 2 which has a huge amount of game data defined from text files (I think these are encoded to binary when shipped but it was clearly useful to give the game a mode where it'd load them all from text on startup).
By the 2000s, portability was a concern for most titles. Certainly anything targeted at a rapidly changing console market back then.
Definitely, and architectures back then were far less standardized. The Xbox 360 was a big-endian PowerPC CPU, the PS2 had a custom RISC-based CPU. On the desktop, this was still the era of PowerPC-based Macs. Far easier (and I would argue safer) to use a standard, portable sscanf-like function with some ascii text, than figure out how to bake your binaries into every memory and CPU layout combination you might care about.
Video games are made by a lot of non-programmers who will be much more comfortable adjusting values in a text file than they are hex editing something.
Besides, the complaint about not having a heavyweight parser here is weird. This is supposed to be "trusted data", you shouldn't have to treat the file as a threat, so a single line sscanf that's just dumping parsed csv attributes into memory is pretty great IMO.
Definitely initialize variables when it comes to C though.
Vouching seems to be time-lagged and require more than one.
XML was everywhere.
The flaw isn't the language. The issue is a 0.5x programmer not knowing to avoid sscanf() and failing to default and validate the results. This could be handled competently with strtok() parsing the lines without needing a more complicated file format.
Worked fine on the target machines and the "0.5x programmer" got to see their family for winter holiday. Or are you saying they should have defensively programmed around a bug manifesting 21 years later and skip seeing their family during crunch time?
To be honest, I just don't like how you disparaged the programmer out-of-context. Talk is cheap.
Using a well–written third–party library would not increase the development time; it would in fact reduce it. No risk of missing Christmas there.
Well-written 3rd party serialization libraries weren't exactly easy to come by 20 years ago, at least from what I can recall. Your best bet was using an XML library, but XML parsing was quite resource heavy. Many that seemed well designed turned out to be a security nightmare (Java serialization).
I disagree. JSON is 25 years old, and SAX parsers are 22. A SAX parser is the opposite of “resource heavy”, since it is event driven. The parser does not maintain complex state, although you might have to manage some state in order to correctly extract your objects from the XML. Granted, it wouldn’t have integrated nicely with C to generate the parser code from your struct definition back then, but the basics were there.
But it is even more important for today’s game studios to see and understand the mistakes that yesterday’s studios made. That’s the only way to avoid making them all over again.
I’ll be the first to defend the greybeards I’ve befriended and learned from in AAA, but having seen codebases of that age and earlier, the “meta” around game development was different back then. I think the internet really changed things for the better.
Your average hire for the time might have been self-taught with the occasional C89 tutorial book and two years of Digipen. Today’s graduates going into games have fallen asleep to YouTube lectures of Scott Meyers and memorized all the literature on a fixed timestep.
Otoh, the Internet has meant that nothing is ever finished, there's always an update to download.
[dead]
[flagged]
funnily enough, there was an infamous bug in GTA5 for a long time that was related to using JSON : https://nee.lv/2021/02/28/How-I-cut-GTA-Online-loading-times...
I remember that one too :)
They wrote a JSON “parser” using sscanf. sscanf is not bulletproof! Just use an open source library instead of writing something yourself. You will still be a real programmer, but you will finish your game sooner and you won't have embarrassing stories written about you.
Loading times are still absurd, fwiw.
Yes, but now it's in the realm of ~3 minutes, and not ~8 minutes even on a top-spec PC, right? I really liked the game, but waiting 8 minutes to load just to get griefed by hackers within seconds of walking outside... I don't understand how that game makes any money.
IMO it varies widely. This past weekend it was taking me multiple attempts to get logged in to a public lobby— after waiting ~5-10 minutes!
Nothing has changed appreciably. If they would let you login to a private invite-only lobby that would likely speed things up greatly— but it’ll never happen.
That's probably just the nature of P2P networking code.
> If they would let you login to a private invite-only lobby that would likely speed things up greatly— but it’ll never happen.
Did they remove this option in the last couple years?
I don’t think its ever been an option to login directly to an invite-only lobby. But then I have taken multiple multi-year breaks! I was pleasantly surprised you can actually play most of the game in a private lobby now… that is a huge change and I am not at all certain when it occurred.
It was like crack. People put up with a lot of problems and bugs just because it was really fun just enough of the time to get them hooked.
Nitpicking: What you're describing is called a "variable ratio reinforcement schedule", and is considered to be the most effective form of operant conditioning.
However, it's not even remotely "like crack". Crack is really really really really fun, period, no "just enough of the time" about it. The reason people get hooked on crack is because it's guaranteed to be fun.
If I had to choose a substance that most closely mirrored variable ratio reinforcement conditioning, it'd probably be ketamine.
Definitely similar to a drug addiction, speaking from firsthand experience with both. GTA has been harder to give up than cocaine was.
Putting the (very valid) reasons for not having human-readable game saves aside, are you sure it's worse than using a 3rd party library that's built to accept semi-valid input values, possibly evaluates user input in some way and has difficult to debug bugs that occur only under certain inputs? I agree that writing a stable and safe parser for a binary data file isn't easy, but there's less things that can go wrong when you can hardcode it to reject any remotely suspicious input. Third party XML/JSON libraries OTOH try to interpret as much as possible, even when the values are bogus. Also no need to deal with different text encoding bugs, line endings...
You misunderstood. Game developers should use a _good_ third–party library, not a _bad_ one. At a minimum they should be able to read the source code so that they know it is good. Thus open source libraries should be at the top of the list.
If you don't know what “good” looks like, take a look at [Serde](https://serde.rs/). It’s for Rust, but its features and overall design are something you should attempt to approach no matter what language you’re writing in.
There are no good third party libraries
I disagree. Serde is not merely good, it is excellent.
The only C code that I have recently interacted with uses a home–grown JSON “library” that is actually pretty good. In particular it produces good error messages. If it were extracted out into its own project then I would be able to recommend it as a library.
But how is that C project using a custom made JSON library doing better than Rockstar games doing the same? Because that library has good error messages?
Apart from that, many of us thought that Java serialization was good if just used correctly, that IE's XML parsing capabilities were good if just used correctly, and so on. We were all very wrong. And a 3rd party library would be just some code taken from the web, or some proprietary solution where you'd once again have to trust the vendor.
It’s good because they have spent hundreds or thousands of hours polishing and improving it. It’s paid off too, because stable releases never have broken data files any more. Any mistakes that do get made are usually found and fixed before the code is ever committed. Even the experimental branch rarely sees broken data files. It’s more likely to see these error messages when loading save files, because the code to read old save files and convert them for newer versions of the game is the hardest to write and test.
> And a 3rd party library would be just some code taken from the web, or some proprietary solution where you'd once again have to trust the vendor.
Open source exists for a reason, and had already existed for ~15 years by the time this game was begun. 20 years later there are even fewer excuses to be stuck using some crappy code that you bought from a vendor and cannot fix.
You both are correct.
But also keep in mind in 2004 the legality of many open source projects was not really tested very well in court. Pretty sure that was right around the time one of the bigger linux distros was throwing its weight around and suing people. So you want to ship on PS2 and XBOX and PC and GameCube. Can you use that lib from inside windows? Not really. Can you build/vs buy? Buy means you need the code and probably will have to port it to PS2/GameCube yourself. Can you use that opensource lib? Probably, but legal is still dragging its feet, and you get to port it to PS2. Meanwhile your devs need a library 3 weeks ago and have hacked something together from an older codebase that your company owns and it works and means you can hit your gold master date.
Would you do that now? No. You would grab one of the multitudes of decent libs out there and make sure you are following the terms correctly. Back then? Yeah I can totally see it happening. Open source was semi legally grey/opaque to many corporations. They were scared to death of losing control of their secret sauce code and getting sued.
> Putting the (very valid) reasons for not having human-readable game saves aside,
I don't follow. What would the reasons be?
A human-readable game save file is presumably human-editable.
Most binary save game files are human editable, too; unless they go through a separate encoding stage.
Editting simcity saves was my introduction to hex editing...
Require a hash in the file to match the rest of the file if you want to avoid effortless changes to the file.
(There is no way to prevent changes by a knowledgeable person with time or tools, so that's not a goal)
Before game companies earned all their profit through selling cosmetics and premium currency nobody cared if you cheated at your single player game and nobody SHOULD care if you want to give yourself extra money.
It's only now that single player progress is profitable to sell that video games have taken save game encryption to be default.
It's so stupid.
The trouble is that if some weirdness happens because of the edit, you've got to handle it even if you say it would be reasonable to assume that it's outside of being supported. Maybe you spend a bit more time defensive coding around what inputs it reads from the file, maybe a certain proportion of users doing the save edit see bugs in an apparently unrelated part of the game and seek support (and their bug report might not be complete with all the details), developers spend time to chase down what went wrong, maybe they bad-mouth it on forums which affects sales - there's going to be some cost to handling all of that.
One of the anecdotes from Titan Quest developed by Iron Lore is that their copy protection had multiple checks, crackers removed the early checks to get the game running but later 'tripwires' as you progress through the game remained and the game appeared to crash. So the game earned a reputation for being buggy for something no normal user would hit running the game as intended.
>The trouble is that if some weirdness happens because of the edit, you've got to handle it even if you say it would be reasonable to assume that it's outside of being supported.
What? No. What even are you suggesting? Hell, games with OFFICIAL MODDING SUPPORT still require you submit bug reports with no mods running.
Editing game files has always been "you are on your own", even editing standard Unreal config files is something you wont get support for, and they are trivial human readable files with well known standards.
>One of the anecdotes from Titan Quest
Any actual support for this anecdote? Lots of games have anti-piracy features that sneakily cause problems, and even could fire accidentally. None of those games get a reputation for being buggy. Games like Earthbound would make the game super hard and even delete your save game at the very end. Batman games would nerf your gliding ability. Game Dev Tycoon would kill your business due to piracy.
None of these affected the broad reputation of the game. Most of them are pretty good marketing in fact.
Mostly to prevent people and programs from editing them, obfuscating implementation details, reducing file sizes (say had they used XML vs. binary)...
Higher barrier to cheating.
It's a single player game. Cheat codes are built into it by design.
HESOYAM
Telling a 3d engine programmer not to have opinions on data formats? Good luck with that.
[dead]