A few years ago I designed a way to detect bit-flips in Firefox crash reports and last year we deployed an actual memory tester that runs on user machines after the browser crashes. Today I was looking at the data that comes out of these tests and now I'm 100% positive that the heuristic is sound and a lot of the crashes we see are from users with bad memory or similarly flaky hardware. Here's a few numbers to give you an idea of how large the problem is. 🧵 1/5
You can’t effect the number of bit flips your users hardware has, but you can affect how often buggy code corrupts their memory or otherwise crashes your program.
Let’s say any app will crash about once a year on my machine due to a bit flip. If the app is crap and crashes hundreds of times for other reasons, the bit flip is irrelevant. If the app is robust enough that the bit flip accounts for 10 % of the crashes, that basically means the app is pretty much never crashing due to poor code.
Anecdotal evidence, but I had both a 13th gen and 14th gen Intel CPU with the bug that caused them to over time, destroy themselves internally.
The most-user-visible way this initially came up, before the CPUs had degraded too far, was Firefox starting to crash, to the point that I initially used Firefox hitting some websites as my test case when I started the (painful) task of trying to diagnose the problem. I suspect that it’s because Firefox touches a lot of memory, and is (normally) fairly stable — a lot of people might not be too surprised if some random game crashes.
As a long time Firefox user, I believe Firefox sees orders of magnitude more RAM issues than other apps because it is using orders of magnitude more RAM than other apps.
I suspect the stuff Firefox stores in ram is more sensitive, too. A lot of games load tens of gigabytes of textures, but a bitflip in that stuff will lead to a pixel somewhere being the wrong colour instead of a crash.
You’re assuming that app quality is constant. But if I made an app that crashes on launch, I can confidently say 0% of those crashes would be from bitflips.
Firefox isn’t special in some way that could cause bitflips, but it’s 1) where this data was collected (and why this post isnt talking about some other product) and 2) speaks to the quality of FF, because crashes are rare enough for bit flips to be a significant crash factor.
The takeaway is that for the FF team, and anyone using ram (everyone), bitflips are more common than expected
It would be interesting to see how this works in Chrome. I would guess that it could be the same - people tend to leave their browsers open with hundreds of tabs and will never reboot their laptops. If you play a random game for 2 hours, bit flips shouldn’t be a problem. But if you keep your browser open for weeks or months with hundreds of tabs, that may cause problems.
… I can’t imagine having a browser with hundreds of open tabs. That would tend me of the old days of Netscape Navigator and all the popups and browser add on cancer.
Ahh the nostalgic days of the early Dotcom era. I sometimes miss you geocities
Wouldn’t that mean ten percent of all crashes in all apps would be caused by bit flips? What makes Firefox special?
You can’t effect the number of bit flips your users hardware has, but you can affect how often buggy code corrupts their memory or otherwise crashes your program.
Let’s say any app will crash about once a year on my machine due to a bit flip. If the app is crap and crashes hundreds of times for other reasons, the bit flip is irrelevant. If the app is robust enough that the bit flip accounts for 10 % of the crashes, that basically means the app is pretty much never crashing due to poor code.
That’s the way people should be looking at it. It basically means hard crashes are extremely rare in the firefox ecosystem.
To be fair, I can’t remember the last time a browser crashed on me in general.
I’ve had Safari of all things crash on me a couple of times. Still, not enough to actually be disruptive.
Anecdotal evidence, but I had both a 13th gen and 14th gen Intel CPU with the bug that caused them to over time, destroy themselves internally.
The most-user-visible way this initially came up, before the CPUs had degraded too far, was Firefox starting to crash, to the point that I initially used Firefox hitting some websites as my test case when I started the (painful) task of trying to diagnose the problem. I suspect that it’s because Firefox touches a lot of memory, and is (normally) fairly stable — a lot of people might not be too surprised if some random game crashes.
As a long time Firefox user, I believe Firefox sees orders of magnitude more RAM issues than other apps because it is using orders of magnitude more RAM than other apps.
I suspect the stuff Firefox stores in ram is more sensitive, too. A lot of games load tens of gigabytes of textures, but a bitflip in that stuff will lead to a pixel somewhere being the wrong colour instead of a crash.
You Firefox also hoards RAM? I thought it’s just mine.
You’re assuming that app quality is constant. But if I made an app that crashes on launch, I can confidently say 0% of those crashes would be from bitflips.
Firefox isn’t special in some way that could cause bitflips, but it’s 1) where this data was collected (and why this post isnt talking about some other product) and 2) speaks to the quality of FF, because crashes are rare enough for bit flips to be a significant crash factor.
The takeaway is that for the FF team, and anyone using ram (everyone), bitflips are more common than expected
It would be interesting to see how this works in Chrome. I would guess that it could be the same - people tend to leave their browsers open with hundreds of tabs and will never reboot their laptops. If you play a random game for 2 hours, bit flips shouldn’t be a problem. But if you keep your browser open for weeks or months with hundreds of tabs, that may cause problems.
… I can’t imagine having a browser with hundreds of open tabs. That would tend me of the old days of Netscape Navigator and all the popups and browser add on cancer.
Ahh the nostalgic days of the early Dotcom era. I sometimes miss you geocities