Reminds me of: People said you can’t write a financial trading system in a garbage collected language. Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.
For many processes where the lifecycle is limited, freeing memory can be waste of time; most notably freeing memory just before exiting is complete waste.
Well, for ballistic missiles a large portion of the flight will be the mid-course phase (which is typically unpowered) and the terminal phase may use control surfaces for maneuvering. For cruise missiles that's a good rule of thumb.
Yeah, a ballistic missile's bus still does a lot of horizontal maneuvering during the mid-course phase to actually get on target; the main boost phase is mainly just for getting it pointed in the general direction and proving verticality
> People said you can’t write a financial trading system in a garbage collected language
People have been writing HFT systems in GC languages (Java, OCaml) for a while, and as you mentioned, you need to know when to pause/unpause/prevent collection in the critical section.
This [1] is an excerpt from a talk with Dave Lauer [1], who has done it in Java - it took his system ~40 μs from receiving data to producing market order, and he explains in great detail what amount of work the system had to do. And this was >10 years ago.
> Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
The surprising thing here is that it is considered worth it with those limitations.
My hope/guess is that because of the GC hype there wasn't any good alternatives.
With the alternatives gaining maturity now I'd hope that that decision wouldn't be made today (when starting from a clean slate).
> it is considered worth it with those limitations.
These people make decisions based on Excel spreadsheets that are not properly documented. I wouldn't trust their judgement, despite the fact they managed not to crash the global financial system too frequently.
> The surprising thing here is that it is considered worth it with those limitations.
It shouldn't be that surprising. Languages like Java and C# are perfectly sensible languages for writing the vast majority of a lot of trading systems. So it's eminently reasonable to use them, or equivalent, and deal with the bits that are GC-phobic as special cases.
I've worked with/on trading systems written is mainstream languages including C++, Java, C# and Python, as well as less mainstream such as APL and Smalltalk. I can safely say that there are way more critical concerns than whether or not it uses a GC.
How? GC does its work in large batches once in a while, whereas for example Arc (the only serious alternative to what GC brings, which is the ability to have arbitrary graphs of objects) does its work much more frequently, with additional effort expended on atomic operations in a multithreaded environment, and with additional memory writes to boot. Somehow I doubt that GC "just increases your energy bill".
[Citation needed] Rust encourages reference counting, a particularly inefficient form of GC which actually turns reads into writes(!), and it cannot move objects to compact memory except when it can prove that every object has only a single reference.
I usually hear those sorts of rationalizations in companies that have already become feature factories. We just have to go fast. We can't make people slow down and learn discipline, or if we have, we can't go back and fix the past because we have to go go go.
In retrospect I think I should have put more effort into breaking the feature factory culture than the discipline culture. It's hard to get clean if your drug of choice is ready at hand.
Is that really an old story? There's usually no memory allocation after initial startup in safety- and mission-critical systems. Rolling over past the max value of an integer data type is a thing though, and they do reboot for that.
This is why we need a Software Engineering license.
When the point of the product is to kill someone, you can't just stochastically measure shit like this. If the device is ever out of control before it ceases operations, you're looking at Geneva Convention level offenses.
Even software designed to save lives can't get away with this sort of thing.
Presumably undefined unless specifically designed to fail safe but a logical thought would be it keeps burning at present trajectory until it falls and blows up at end of trajectory if device uses an impact fuse. I would be very interested in a more precise answer if you can find one.
Isn't the reason for Software to flourish in last 20 years is that we have had no regulation? Adding regulation would then lead to contraction of software industry as we know it. May be the regulation should be not on employees, but the products in categories where they are subjected to life/death situations.
If you think getting a license is the way to fix quality issues, I have a bridge to sell you.
Well no it doesn’t but it does however assure the public that all certified teachers have a duty to care for your children in a safe environment. In which teachers are also duty bound to report anything illegal or potentially harmful as they are liable in court if something does end up happening
Doctors have licences and that doesn't necessarily make them any good at doctoring either, but that doesn't mean you should prefer one without a license.
>you're looking at Geneva Convention level offenses.
That's a joke. Russia is intentionally bombing Ukrainian civilians, you think they're going to have a single War Crimes charge put against them? How many civilians did the US unintentionally kill across Afghanistan and Iraq, hundreds of thousands? See any charges there either?
Just because charges don't get brought doesn't mean that you didn't commit a crime.
I don't know how I could possibly get one but I'd love to read an independent comparison of this sort of behaviour between US operations and Russian ones. My belief is that the US aims not to kill civilians but is often careless and nets a lot of collateral damage, whereas Russia doesn't care at all and will happily bomb schools if it thinks there's a target in there. But of course I mostly read Western media and writing so my view is potentially very biased.
You've just asserted that it's stochastic with no basis. It could easily be (and in fact is much more likely to be) a periodic operation that leaks memory.
This sort of analysis is completely normal in software. E.g. in safety critical software you often analyse maximum possible stack depth to check that your stack is big enough (one of the reasons why recursive code is sometimes disallowed). This is exactly the same class of analysis.
Reminds me of a sales call to a military contractor. They were interested in an Open Source OS because of the licensing costs. The embedded OS they were using had a license that said you could run as many instances as you liked on a "single device".
Their problem was that when they operated their device it was indeed a single device, but later became many more. If they paid for the extra licenses they didn't need it would be ripping off the government; if they failed to pay for the extra license at some point they would be in breach of their contract with the government as all these unlicensed devices were released.
It rapidly became obvious that their application was a MIRV and that they weren't really going to change but had asked us to talk with them as part of their process of negotiating a deal with their current supplier.
It's hard to imagine really worrying about your license agreements if there actually were a global thermonuclear war.
That reminds me of a short sci-fi story I read about an alien invasion, in which humans were losing horribly until we learned about their legal system and sued them into submission
I immediately though of the GetTickCount win32 function, which, as you said, wraps around to zero every fifty days or so. A lot of programs would crash if you left them open for this long.
I worked at Microsoft on the NetShow Encoder, later known as the Windows Media Encoder, in the Windows 95/Windows NT4 era. I remember carefully writing millisecond code that would deal with differences, instead of absolutes, so that we could run longer than 49 days. Honestly I don't recall whether the limitation was GetTickCount, the recording API, the file format, or all three.
In our debug build, I added a registry key that would start the concept of time at 49-ish days, so that we could test that everything was working without waiting 50 days. However, before we shipped, the intrepid test team did actually run the encoder for 50 days to prove that it worked. (On Windows NT, of course. Windows 95 with good hardware and drivers was perhaps more stable than its reputation, but still...) Though TBH we definitely wouldn't have delayed a release by 49 days if it had failed at the last minute.
FTA: Editor's note: An earlier version of this article mentioned the Boeing 787 CCS uses a Wind River VxWorks real-time OS product at its heart. While this is true, Wind River has been in touch to remind us "the CCS is made up of 80 to 100 applications," as well as VxWorks, and said the bug described in this article is not the fault of its operating system.
"The functions of VxWorks have nothing to do with the data issue you are highlighting in the 787," a spokesperson added. We are happy to clarify our coverage.
VxWorks is the only software in the entire system that I would not question. Sadly, somebody has already added this thing about the airplane to VxWorks' wikipedia page.
I didn't mean to imply that the 787 ran critical infrastructure on Windows 95, just to give an example of a system which was affected by a timer overflow of about that duration.
Your analogy made me laugh, but also made me think about how common fatal mental illness is among humans, and contrast that rate with the fatal plane crashes attributable to computing errors.
It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
You need more scare quotes around “reliable” than brain. The airplane requires constant almost daily maintenance by dozens of highly trained professionals and yet none of its chips will survive decades save for a few lucky passive components.
Whereas going decades without seeing a qualified doctor has been the default state since the dawn of homo sapiens with nothing more than a good night’s sleep.
> humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them
I would not compare human brains to tech. Maybe you can call them simpler and thus more reliable. The human brain is processing far more data than any processor humanity has made. The only comparison is humans handle various types of data whereas computers are designed to handle just a small batch.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Airplanes are not reliable. That's why they require frequent and regular maintenance.
Ignoring the poor analogy linking brains/humans to human built systems, if every human maintained their mental and physical health as much as airplanes get, then they'd all be much better off.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Load the airplane's "brain" with algorithms for how to survive, cooperate, adapt and replicate under every conceivable condition and you'll end up with a comparable system more complicated and worse-performing than ourselves. Not really a fair comparison.
Self-driving cars (and robotics) would be better examples, since they attempt to emulate a broader swath of our sensory capabilities.
I did an ultra running race that took 37 hours last month. I got maybe 10minutes sleep. I had hallucinated before during (shorter) long races but I had no idea that auditory hallucinations [1] were a thing. it was trippy "hearing" things for a solid 12 hours - mainly off-trail "voices" and the sound of "runners" approaching behind me. No matter how many times I stepped to the side to allow a non-existant runner go by, I kept doing it. It was that realistic
How do you rationalize running for 37 hours while having hallucinations, I am legitimately curious. At what point do you say "I might be pushing myself too much"?
I’m not a runner but pushing yourself to your limits is the point of many sports. Humans like seeing how close they can get to death without succeeding.
Your body goes into an autopilot trance. It's happened to me on long bike marathons where as long as your water, electrolytes and fuels are still coming in your brain is in this primal mode where your body knows what to do, and your brain shuts off. You will hear conversations in memories vividly, or the 'singing' of the frame of the bicycle and its connection with the pavement through your tires becomes readily apparent.
Later Hominids are meant for this. Without being able to pass long time chasing our prey to the moment of its exhaustion, we'd not have been able to secure the cholesterol needed to further evolve our brains.
"exhaustion chase" never lasted that long, I don't remember the times exactly but animals that don't sweat are exhausted pretty quickly, a couple hours at most.
Doesn't make any sense to me either. Not a biologist, but AFAIK animals known for being sprinters are able to outperform you in bursts, with refractory periods of extreme exhaustion and vulnerability. Especially if already wounded. You turn the chase into a game of hide-and-seek.
Small fish/shrimp can appear to "teleport" since they dart away so quickly, but if you harass them enough, you exhaust them into a state of shock. It's no longer a chase, it's a harvest.
Even in manhunting, you get nowhere chasing the subject since you move at the same speed and the element of uncertainty tilts in their favor. So you try to anticipate where they're going and ambush them there. You turn the chase into a trap.
We're apex predators for a reason. Exhaustion chases don't make sense.
Well, I don't think it was a rational thing to do, but that's not a reason not to do it. The race [1] took place on circular hiking route 118 miles with 19,000ft of climbing. I'd hiked sections of the route (which typically takes 7-10 days for hikers) but was really drawn to the idea of doing the route in a "single day" (actually starts at 6am Friday, cut-off is 10pm Saturday - 40 hours). It was also my fourth time attempting to finish the race. After each of the first three times I failed, my immediate reaction was "It is too hard. I'm not going to sign up again." That is usually followed within 24-48 hours with an intense desire to sign up and try again. The hallucinations were not something I sought, and as I said, the auditory was not something I even know existed. The overall feeling was trippy rather than discomfort. Unlike the many blisters on my feet. I'm definitely older and chunkier than most doing the race I'm proud as hell that I persevered and finished especially after failing repeatedly. During the years of training and attempts I learned a lot about myself - good and bad.
I've never experienced auditory hallucinations before. I did however do a 42 hour self navigated run last year with 0 sleep. After about 30 hours I was visually hallucinating pretty much consistently. Amongst the things I saw was Darth Vader, a giant lego man, our cat sleeping on the trail, spectators that weren't there, a banker sitting at a desk staring at me, a church building that wasn't there, signs that didn't exist, and more.
I think it has to do with the way the human visual system effectively does edge detection and then groups those into objects and attaches meaning. When you are extremely sleep deprived, the edges gets grouped differently and you start seeing things that aren't there.
I could judge what was real or not based on likelihood of me seeing said things in the wilderness, or if I got close enough I would be able to tell what I was really looking at. Another runner later informed me that I had told him he wasn't real when he passed me.
I once saw death (exactly as most of us imagine it, dark robes and a scythe) while driving on a highway late at night, I understood the message my brain was sending me: I just took the next exit and slept for an hour (it was too cold to sleep longer, but it made a big difference on my concentration anyway).
I can't believe physical exhaustion is not at least as important a part in this. Personally, I skip a night's sleep every now and then to readjust my sleeping patterns, and I've never experienced anything strange after something like 32-36 hours without sleep (but note: no significant physical exertion involved).
It doesn't even feel that bad once you get past the slump in the 20-24 h region.
It might be. I ran 30h race earlier this year and had no hallucinations. I've not done _exhaustive_ research on this, I'm just recalling some undergrad neuropsychology. It could also be the environment. I suspect it's a lot less likely that you'll be seeing stuff that isn't there when you are in a familiar environment as opposed to running alone in the bush.
Chapter 1 of "The Inmates are Running the Asylum," by Alan Cooper (1999) opens with the riddle: "What do you get when you cross a computer with an airplane?"
This was my first thought. Surely there are other systems on the plane that require maintenance or checks that are even more disruptive. Rebooting is a switch, button, or command. Inspecting engines, pipes, or compartments requires somebody to get out of a chair.
This seems like some kind of overflow problem, but 2^32 milliseconds is 49.7 days. Is it possible that the 51 day number came from a combination of rounding and an off-by-one error in the reporting?
I'm just thinking about a Liam Neeson movie in a 787 with on flight tank refill and the reboot after 2^32 milliseconds because the plane could crash and it is the president pilot in the middle of something bad like a WW3..
Can you imagine being a pilot halfway through a flight and while you are at 40,000 feet your airlines tech support asks you "Have you tried turning it off and on again?" :-D
One time I was flying inside China, and as the plane reached cruising altitude it felt eerily quiet, as if the engines were off and we were just gliding. I honestly wondered "Is this it? Am I going to die? Surrounded by people I won't even be able to communicate with?".
Of course since I'm still here, it wasn't it. After a while I felt the vibration of the engines again and was relieved. Maybe the pilots had to reboot the plane.
I have lab equipment that needs a power cycle after 48 hours of power on time for it to continue to produce high quality data. This is because the internal calibration code only runs upon boot. Maybe these planes have something similar.
So, to me, the power cycle thing is not super surprising. That fact that the FAA felt the need to order Boeing to do it seems a bit suspect though. That's probably the real story.
The only reason this seems to be a problem is that apparently operators (or the manufacturer) didn't know about it, since I don't think there's an actual usecase for the aircraft to be powered on continuously for that long.
FTA: It is usual for commercial airliners to spend weeks or more continuously powered on as crews change at airports, or ground power is plugged in overnight while cleaners and maintainers do their thing.
Ryanair a private ULCC of Europe and one of the largest ULCC of the world having a TAT of 25 minutes does not have a seatback pocket.
Wondering why?
Back in 2004, removal of seat back pockets greatly reduced the time for which the Ryanair aircraft stayed on the ground. That’s is because it takes more time to clean an airplane with seatback pockets.</i>
So, that’s 25 minutes from touching down to being airborne again. Chances are the plane is standing still for less than 15 minutes. Long-haul flights take longer (up to two hours, according to https://simpleflying.com/aircraft-turnaround-process/)
Also: it wouldn’t surprise me if you could not refuel an airliner while it is rebooting, all lights may be off for a while, etc.
If you take off on day 50 23 hours type of thing is more of what I was getting at. It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot". As long as it doesn't end during your flight, why not push it for someone else? Otherwise, you just make it part of every flight's routine to reboot.
I think this is one of the areas that airlines are super good at. Every single plane needs strict regular maintenance and tons of checklists in order to take off, so something like a 50-day requirement for a reboot would be in that list of items, and if the people aren't following those lists of items you're going to have things like turbine failures, so they get done.
Maybe if there's a really badly run airline it might be a problem, but there are much more severe problems than software in a case like that.
Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
>Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
Yes, like the mundane software overriding pilot's intent from software written specifically to not retrain pilots on the system? Yup, sounds mundane to me.
When I said I don't trust anything from Boeing, it had little to do with this 50 day reboot, but much more in line with the Boeing is allowed to self certify and has been shown to not do that in a good way. They have ruined that trust because of greed. You might think that's a broad brush? At this point, every little bit is just another instance of death from a thousand cuts.
Airplane always want to fly, give them enough speed, they will take off on their own.
Thats why you have bumpy landings sometimes because pilots didn’t slow down/flare properly to slowdown the aircraft instead forced it to ground. With enough wind some general aviation aircraft are capable of takeoff/land vertically like a chopper.
Usually these aircraft are power cycled often. There is no way they are flying these planes 51 days straight with no power cycle. Maintenance checks would necessitate a restart.
I’m pretty sure most of the computers can be reset during flight. You may lose flight protections, but you can fly an airplane on basic instruments. You can even handle unreliable speed by flying at a specific angle of attack and a proper thrust setting. (According to the Air France 447 report)
On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
>On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
I would honestly hope that would be true about any airplane from any manufacture. Fly-by-wire makes some of that more difficult when the wires go through the computer, but even then I'd have assumed some sort of 100% manual capability. Like when Y2K nutters were predicting airplanes falling out of the sky because of 2-digit years rather than the logical the planes would behave according to laws of physics for an intact air frame.
I learned programming in the late '70s on a Wang 2200T. The manual had a procedure it would recommend called "Master Initialization." Yes: turning the power off and on.
I would not be so brazen as to call the engineers behind this "average" or their work so. Unless you can make the argument that a plane would fly for 51 days straight, being required to reboot the on-board systems is not a major deal-breaker, especially as the larger Boeing's take 60 minutes to refuel.
The engineers behind Boeing planes have built an incredibly sophisticated machine which omitting a very nessescary but not so inconvient task of rebooting every n days, performs its primary directive well.
one bug implies two bugs only in the sense that any given software has at least one bug and any given software has at least one more bug than you know exists ;)
Reminds me of: People said you can’t write a financial trading system in a garbage collected language. Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
Or the old story of the junior engineer who finds a memory leak in a missile’s guidance system and the senior engineer says the memory leak is fine as long as you don’t run out of memory before the missile completes its mission.
For many processes where the lifecycle is limited, freeing memory can be waste of time; most notably freeing memory just before exiting is complete waste.
Depending on your OS. And if you want to profile or move code around, it's better to free it asap.
Or to put it in other terms, you want to consume memory at a rate lower than propellant.
Well, for ballistic missiles a large portion of the flight will be the mid-course phase (which is typically unpowered) and the terminal phase may use control surfaces for maneuvering. For cruise missiles that's a good rule of thumb.
Not typically, always. “Ballistic” in “ballistic missile” means “moving under the force of gravity only”
Might you not use some fuel for countermeasures during the midcourse phase?
Yeah, a ballistic missile's bus still does a lot of horizontal maneuvering during the mid-course phase to actually get on target; the main boost phase is mainly just for getting it pointed in the general direction and proving verticality
I imagine RAM weight is not a relevant parameter on the rocket equation.
There's a reason why lots of HFT firms have massive amounts of RAM. It's easier to get the code out first, and handle leaks with a reboot later.
And RAM is quite cheap these days compared to the money the hardware is supposed to make.
> People said you can’t write a financial trading system in a garbage collected language
People have been writing HFT systems in GC languages (Java, OCaml) for a while, and as you mentioned, you need to know when to pause/unpause/prevent collection in the critical section.
This [1] is an excerpt from a talk with Dave Lauer [1], who has done it in Java - it took his system ~40 μs from receiving data to producing market order, and he explains in great detail what amount of work the system had to do. And this was >10 years ago.
[1] https://youtu.be/1ah7XokvcwA?t=785
sub 10 in Java was very doable 10 years ago, fwiw. even better than collecting once a day is doing everything off-heap — C-style java :D
Just keep your objects live, attach reusable scratch space to them, and memory pool them instead of throwing them away.
(from my experience anyway in writing critical-path allocation-free C# with Unity).
ya, also keep everything in cache to be below a few mics
> you need to know when to pause/unpause/prevent collection in the critical section.
That's his point. It's like saying humans can walk over a lake, as long as it's dry season and the lake is now 10 inches deep.
You can use garbage collected languages as long as you don't collect. (!?!?!)
I'm even more hardline. IMO everything time or life-critical should have static memory allocation, period.
> Turns out you can as long as you don’t garbage collect during the trading day by carefully managing allocations and manually running GC each day before trading starts.
The surprising thing here is that it is considered worth it with those limitations.
My hope/guess is that because of the GC hype there wasn't any good alternatives.
With the alternatives gaining maturity now I'd hope that that decision wouldn't be made today (when starting from a clean slate).
> it is considered worth it with those limitations.
These people make decisions based on Excel spreadsheets that are not properly documented. I wouldn't trust their judgement, despite the fact they managed not to crash the global financial system too frequently.
> The surprising thing here is that it is considered worth it with those limitations.
It shouldn't be that surprising. Languages like Java and C# are perfectly sensible languages for writing the vast majority of a lot of trading systems. So it's eminently reasonable to use them, or equivalent, and deal with the bits that are GC-phobic as special cases.
I've worked with/on trading systems written is mainstream languages including C++, Java, C# and Python, as well as less mainstream such as APL and Smalltalk. I can safely say that there are way more critical concerns than whether or not it uses a GC.
We have Rust now, and some safe memory usage patterns for C++. GC just increases your energy bill and helps heat up the planet.
>We have Rust now, and some safe memory usage patterns for C++.
>GC just increases your energy bill and helps heat up the planet.
Were C++ compilation times that use 100% of my PC taken into consideration?
And not just for final binary, but also the compilations that happen everyday for development purpose too
> GC just increases your energy bill
How? GC does its work in large batches once in a while, whereas for example Arc (the only serious alternative to what GC brings, which is the ability to have arbitrary graphs of objects) does its work much more frequently, with additional effort expended on atomic operations in a multithreaded environment, and with additional memory writes to boot. Somehow I doubt that GC "just increases your energy bill".
[Citation needed] Rust encourages reference counting, a particularly inefficient form of GC which actually turns reads into writes(!), and it cannot move objects to compact memory except when it can prove that every object has only a single reference.
I usually hear those sorts of rationalizations in companies that have already become feature factories. We just have to go fast. We can't make people slow down and learn discipline, or if we have, we can't go back and fix the past because we have to go go go.
In retrospect I think I should have put more effort into breaking the feature factory culture than the discipline culture. It's hard to get clean if your drug of choice is ready at hand.
Is that really an old story? There's usually no memory allocation after initial startup in safety- and mission-critical systems. Rolling over past the max value of an integer data type is a thing though, and they do reboot for that.
The parent probably meant it more as an urban legend rather than some sort of documented event.
No, it's a real story. See sibling replies.
"junior"? :P
https://devblogs.microsoft.com/oldnewthing/20180228-00/?p=98...
_________
From: k...@rational.com (Kent Mitchell)
Subject: Re: Does memory leak?
Date: 1995/03/31
Norman H. Cohen (nco...@watson.ibm.com) wrote:
: The only programs I know of with deliberate memory leaks are those whose
: executions are short enough, and whose target machines have enough
: virtual memory space, that running out of memory is not a concern.
: (This class of programs includes many student programming exercises and
: some simple applets and utilities; it includes few if any embedded or
: safety-critical programs.)
This sparked an interesting memory for me. I was once working with a
customer who was producing on-board software for a missile. In my analysis
of the code, I pointed out that they had a number of problems with storage
leaks. Imagine my surprise when the customers chief software engineer said
"Of course it leaks". He went on to point out that they had calculated the
amount of memory the application would leak in the total possible flight time
for the missile and then doubled that number. They added this much
additional memory to the hardware to "support" the leaks. Since the missile
will explode when it hits its target or at the end of its flight, the
ultimate in garbage collection is performed without programmer intervention.
--
Kent Mitchell | One possible reason that things aren't
Technical Consultant | going according to plan is .....
Rational Software Corporation | that there never was a plan!
This is why we need a Software Engineering license.
When the point of the product is to kill someone, you can't just stochastically measure shit like this. If the device is ever out of control before it ceases operations, you're looking at Geneva Convention level offenses.
Even software designed to save lives can't get away with this sort of thing.
I do wonder what happens if the missile runs out of memory
It just drops?
Presumably undefined unless specifically designed to fail safe but a logical thought would be it keeps burning at present trajectory until it falls and blows up at end of trajectory if device uses an impact fuse. I would be very interested in a more precise answer if you can find one.
https://www.scienceabc.com/innovation/why-do-some-missiles-e...
There's a star trek episode about such a thing, https://memory-alpha.fandom.com/wiki/Warhead_(episode)
Apparently it will "hijack" more memory and attempt to reconstruct it's orders.
But, real world, I would say that data corruption would occur and outside of chemical/physics, it will just drop and lockup.
May have a failsafe to detonate, but that is a fun question indeed.
Isn't the reason for Software to flourish in last 20 years is that we have had no regulation? Adding regulation would then lead to contraction of software industry as we know it. May be the regulation should be not on employees, but the products in categories where they are subjected to life/death situations.
If you think getting a license is the way to fix quality issues, I have a bridge to sell you.
Teachers have licenses. It does not make them any good at teaching though.
Well no it doesn’t but it does however assure the public that all certified teachers have a duty to care for your children in a safe environment. In which teachers are also duty bound to report anything illegal or potentially harmful as they are liable in court if something does end up happening
Doctors have licences and that doesn't necessarily make them any good at doctoring either, but that doesn't mean you should prefer one without a license.
Bad analogies are bad analogies. A doctor can be a matter of life and death.
>you're looking at Geneva Convention level offenses.
That's a joke. Russia is intentionally bombing Ukrainian civilians, you think they're going to have a single War Crimes charge put against them? How many civilians did the US unintentionally kill across Afghanistan and Iraq, hundreds of thousands? See any charges there either?
Just because charges don't get brought doesn't mean that you didn't commit a crime.
I don't know how I could possibly get one but I'd love to read an independent comparison of this sort of behaviour between US operations and Russian ones. My belief is that the US aims not to kill civilians but is often careless and nets a lot of collateral damage, whereas Russia doesn't care at all and will happily bomb schools if it thinks there's a target in there. But of course I mostly read Western media and writing so my view is potentially very biased.
You've just asserted that it's stochastic with no basis. It could easily be (and in fact is much more likely to be) a periodic operation that leaks memory.
This sort of analysis is completely normal in software. E.g. in safety critical software you often analyse maximum possible stack depth to check that your stack is big enough (one of the reasons why recursive code is sometimes disallowed). This is exactly the same class of analysis.
Reminds me of a sales call to a military contractor. They were interested in an Open Source OS because of the licensing costs. The embedded OS they were using had a license that said you could run as many instances as you liked on a "single device".
Their problem was that when they operated their device it was indeed a single device, but later became many more. If they paid for the extra licenses they didn't need it would be ripping off the government; if they failed to pay for the extra license at some point they would be in breach of their contract with the government as all these unlicensed devices were released.
It rapidly became obvious that their application was a MIRV and that they weren't really going to change but had asked us to talk with them as part of their process of negotiating a deal with their current supplier.
It's hard to imagine really worrying about your license agreements if there actually were a global thermonuclear war.
That is an amazing story.
That reminds me of a short sci-fi story I read about an alien invasion, in which humans were losing horribly until we learned about their legal system and sued them into submission
49.7 days is equal to 2**32 milliseconds, after which Windows 95 crashes (https://web.archive.org/web/20010331174127/http://support.mi...). That's close-ish to 51 days. Maybe they start the counter with a value other than zero.
I immediately though of the GetTickCount win32 function, which, as you said, wraps around to zero every fifty days or so. A lot of programs would crash if you left them open for this long.
I worked at Microsoft on the NetShow Encoder, later known as the Windows Media Encoder, in the Windows 95/Windows NT4 era. I remember carefully writing millisecond code that would deal with differences, instead of absolutes, so that we could run longer than 49 days. Honestly I don't recall whether the limitation was GetTickCount, the recording API, the file format, or all three.
In our debug build, I added a registry key that would start the concept of time at 49-ish days, so that we could test that everything was working without waiting 50 days. However, before we shipped, the intrepid test team did actually run the encoder for 50 days to prove that it worked. (On Windows NT, of course. Windows 95 with good hardware and drivers was perhaps more stable than its reputation, but still...) Though TBH we definitely wouldn't have delayed a release by 49 days if it had failed at the last minute.
I would hope there's a margin of error baked in and stuff doesn't break on day 52 exactly.
The 787 runs on VxWorks though.
https://blogs.windriver.com/wind_river_blog/2011/09/boeing-7...
Running in a VM running on Windows.
FTA: Editor's note: An earlier version of this article mentioned the Boeing 787 CCS uses a Wind River VxWorks real-time OS product at its heart. While this is true, Wind River has been in touch to remind us "the CCS is made up of 80 to 100 applications," as well as VxWorks, and said the bug described in this article is not the fault of its operating system.
"The functions of VxWorks have nothing to do with the data issue you are highlighting in the 787," a spokesperson added. We are happy to clarify our coverage.
VxWorks is the only software in the entire system that I would not question. Sadly, somebody has already added this thing about the airplane to VxWorks' wikipedia page.
I didn't mean to imply that the 787 ran critical infrastructure on Windows 95, just to give an example of a system which was affected by a timer overflow of about that duration.
Though Somehow I would not be surprised if it were true
Or maybe there’s an error in that calculation of 51 days. 49.7 times 1.024 is 50.8928, so there could be a 2¹⁰ = 10³ error in there.
I would hope the advisory leaves some room for error in the reboot interval, though.
Humans have to be rebooted at most every 2 days to avoid misleading auditory data among other things.
Your analogy made me laugh, but also made me think about how common fatal mental illness is among humans, and contrast that rate with the fatal plane crashes attributable to computing errors.
It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
That development was built up over time and very much in blood though. Channels like Mentour are quite illuminating on that subject.
And in fairness, reliability is a major point of automation.
You need more scare quotes around “reliable” than brain. The airplane requires constant almost daily maintenance by dozens of highly trained professionals and yet none of its chips will survive decades save for a few lucky passive components.
Whereas going decades without seeing a qualified doctor has been the default state since the dawn of homo sapiens with nothing more than a good night’s sleep.
> humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them
I would not compare human brains to tech. Maybe you can call them simpler and thus more reliable. The human brain is processing far more data than any processor humanity has made. The only comparison is humans handle various types of data whereas computers are designed to handle just a small batch.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Airplanes are not reliable. That's why they require frequent and regular maintenance.
Ignoring the poor analogy linking brains/humans to human built systems, if every human maintained their mental and physical health as much as airplanes get, then they'd all be much better off.
You can also use simple tools to create incredibly accurate telescope mirrors.
> It's a pretty amazing accomplishment for humanity to have developed systems as complex as modern jumbo jets with "brains" that are essentially more reliable than the ones that created them.
Load the airplane's "brain" with algorithms for how to survive, cooperate, adapt and replicate under every conceivable condition and you'll end up with a comparable system more complicated and worse-performing than ourselves. Not really a fair comparison.
Self-driving cars (and robotics) would be better examples, since they attempt to emulate a broader swath of our sensory capabilities.
I did an ultra running race that took 37 hours last month. I got maybe 10minutes sleep. I had hallucinated before during (shorter) long races but I had no idea that auditory hallucinations [1] were a thing. it was trippy "hearing" things for a solid 12 hours - mainly off-trail "voices" and the sound of "runners" approaching behind me. No matter how many times I stepped to the side to allow a non-existant runner go by, I kept doing it. It was that realistic
[1] - https://en.wikipedia.org/wiki/Auditory_hallucination
How do you rationalize running for 37 hours while having hallucinations, I am legitimately curious. At what point do you say "I might be pushing myself too much"?
I’m not a runner but pushing yourself to your limits is the point of many sports. Humans like seeing how close they can get to death without succeeding.
Your body goes into an autopilot trance. It's happened to me on long bike marathons where as long as your water, electrolytes and fuels are still coming in your brain is in this primal mode where your body knows what to do, and your brain shuts off. You will hear conversations in memories vividly, or the 'singing' of the frame of the bicycle and its connection with the pavement through your tires becomes readily apparent.
Later Hominids are meant for this. Without being able to pass long time chasing our prey to the moment of its exhaustion, we'd not have been able to secure the cholesterol needed to further evolve our brains.
"exhaustion chase" never lasted that long, I don't remember the times exactly but animals that don't sweat are exhausted pretty quickly, a couple hours at most.
Doesn't make any sense to me either. Not a biologist, but AFAIK animals known for being sprinters are able to outperform you in bursts, with refractory periods of extreme exhaustion and vulnerability. Especially if already wounded. You turn the chase into a game of hide-and-seek.
Small fish/shrimp can appear to "teleport" since they dart away so quickly, but if you harass them enough, you exhaust them into a state of shock. It's no longer a chase, it's a harvest.
Even in manhunting, you get nowhere chasing the subject since you move at the same speed and the element of uncertainty tilts in their favor. So you try to anticipate where they're going and ambush them there. You turn the chase into a trap.
We're apex predators for a reason. Exhaustion chases don't make sense.
It took just 4 miles (approximately 40 minutes) to wear off some cheetahs: https://www.bbc.com/news/world-africa-24953910
The hotter the weather, the easier this gets.
and the effort to track these animals sometimes could take a long time and a lot of foot travel
Well, I don't think it was a rational thing to do, but that's not a reason not to do it. The race [1] took place on circular hiking route 118 miles with 19,000ft of climbing. I'd hiked sections of the route (which typically takes 7-10 days for hikers) but was really drawn to the idea of doing the route in a "single day" (actually starts at 6am Friday, cut-off is 10pm Saturday - 40 hours). It was also my fourth time attempting to finish the race. After each of the first three times I failed, my immediate reaction was "It is too hard. I'm not going to sign up again." That is usually followed within 24-48 hours with an intense desire to sign up and try again. The hallucinations were not something I sought, and as I said, the auditory was not something I even know existed. The overall feeling was trippy rather than discomfort. Unlike the many blisters on my feet. I'm definitely older and chunkier than most doing the race I'm proud as hell that I persevered and finished especially after failing repeatedly. During the years of training and attempts I learned a lot about myself - good and bad.
[1] - https://www.kerrywayultra.com/
Assuming it is safe (I don't know whether it is), it sounds like a fun experience.
I've never experienced auditory hallucinations before. I did however do a 42 hour self navigated run last year with 0 sleep. After about 30 hours I was visually hallucinating pretty much consistently. Amongst the things I saw was Darth Vader, a giant lego man, our cat sleeping on the trail, spectators that weren't there, a banker sitting at a desk staring at me, a church building that wasn't there, signs that didn't exist, and more.
I think it has to do with the way the human visual system effectively does edge detection and then groups those into objects and attaches meaning. When you are extremely sleep deprived, the edges gets grouped differently and you start seeing things that aren't there.
I could judge what was real or not based on likelihood of me seeing said things in the wilderness, or if I got close enough I would be able to tell what I was really looking at. Another runner later informed me that I had told him he wasn't real when he passed me.
Why would you inflict that on yourself?
I once saw death (exactly as most of us imagine it, dark robes and a scythe) while driving on a highway late at night, I understood the message my brain was sending me: I just took the next exit and slept for an hour (it was too cold to sleep longer, but it made a big difference on my concentration anyway).
For the free T-shirt, of course. The finisher's medal, if there is one, is an added bonus. (Not sarcasm)
Ha ha, the t-shirt wasn't even that nice :p
I can't believe physical exhaustion is not at least as important a part in this. Personally, I skip a night's sleep every now and then to readjust my sleeping patterns, and I've never experienced anything strange after something like 32-36 hours without sleep (but note: no significant physical exertion involved).
It doesn't even feel that bad once you get past the slump in the 20-24 h region.
It might be. I ran 30h race earlier this year and had no hallucinations. I've not done _exhaustive_ research on this, I'm just recalling some undergrad neuropsychology. It could also be the environment. I suspect it's a lot less likely that you'll be seeing stuff that isn't there when you are in a familiar environment as opposed to running alone in the bush.
How do you stay on trail when you’re hallucinating signposts?
GPS watch. Although I did accidentally run back where I came from for a bit O_o
Oof. You ultramarathoners need a watch that signals when you’re going backwards or in circles, like in a car racing game!
Some people would pay for that effect.
51 days is nothing. The Airbus A350 had to be rebooted every 149 hours: https://www.theregister.com/2019/07/25/a350_power_cycle_soft...
Chapter 1 of "The Inmates are Running the Asylum," by Alan Cooper (1999) opens with the riddle: "What do you get when you cross a computer with an airplane?"
The answer, of course, is "a computer."
I would expect rebooting to be a part of a routine maintenance and a health check between each flight.
Now I'm curious: how long does it take to reboot Boeing 787.
This was my first thought. Surely there are other systems on the plane that require maintenance or checks that are even more disruptive. Rebooting is a switch, button, or command. Inspecting engines, pipes, or compartments requires somebody to get out of a chair.
This seems like some kind of overflow problem, but 2^32 milliseconds is 49.7 days. Is it possible that the 51 day number came from a combination of rounding and an off-by-one error in the reporting?
Airworthiness directive indicates 51.
https://ad.easa.europa.eu/blob/2020-06-14.pdf/AD_US-2020-06-...
I'm just thinking about a Liam Neeson movie in a 787 with on flight tank refill and the reboot after 2^32 milliseconds because the plane could crash and it is the president pilot in the middle of something bad like a WW3..
Edit: patent pending
Can you imagine being a pilot halfway through a flight and while you are at 40,000 feet your airlines tech support asks you "Have you tried turning it off and on again?" :-D
One time I was flying inside China, and as the plane reached cruising altitude it felt eerily quiet, as if the engines were off and we were just gliding. I honestly wondered "Is this it? Am I going to die? Surrounded by people I won't even be able to communicate with?".
Of course since I'm still here, it wasn't it. After a while I felt the vibration of the engines again and was relieved. Maybe the pilots had to reboot the plane.
It's very close to the limit of a 32 bit integer in miliseconds. Comes out to 47 days according to google.
Perhaps you should switch to a search engine that's better at math.
My Windows desktop and iPhone need to be cold booted now and then. My Ubuntu box hasn't needed that.
I have lab equipment that needs a power cycle after 48 hours of power on time for it to continue to produce high quality data. This is because the internal calibration code only runs upon boot. Maybe these planes have something similar.
So, to me, the power cycle thing is not super surprising. That fact that the FAA felt the need to order Boeing to do it seems a bit suspect though. That's probably the real story.
The only reason this seems to be a problem is that apparently operators (or the manufacturer) didn't know about it, since I don't think there's an actual usecase for the aircraft to be powered on continuously for that long.
FTA: It is usual for commercial airliners to spend weeks or more continuously powered on as crews change at airports, or ground power is plugged in overnight while cleaners and maintainers do their thing.
I don’t know how long rebooting a 787 takes, but turn Around Time of an airliner may be too short to do a reboot. http://aviationavi.com/turn-around-time-tat/:
Ryanair a private ULCC of Europe and one of the largest ULCC of the world having a TAT of 25 minutes does not have a seatback pocket.
Wondering why?
Back in 2004, removal of seat back pockets greatly reduced the time for which the Ryanair aircraft stayed on the ground. That’s is because it takes more time to clean an airplane with seatback pockets.</i>
So, that’s 25 minutes from touching down to being airborne again. Chances are the plane is standing still for less than 15 minutes. Long-haul flights take longer (up to two hours, according to https://simpleflying.com/aircraft-turnaround-process/)
Also: it wouldn’t surprise me if you could not refuel an airliner while it is rebooting, all lights may be off for a while, etc.
Assuming the plane spends 50% of its time flying, it kinda lines up with an A check interval.
https://en.wikipedia.org/wiki/Aircraft_maintenance_checks
The old turn it off and on again...
mid-flight though?
How many flights last for 51 days?
If you take off on day 50 23 hours type of thing is more of what I was getting at. It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot". As long as it doesn't end during your flight, why not push it for someone else? Otherwise, you just make it part of every flight's routine to reboot.
> It's very easy to just hit skip when presented dialogs like "3 more days to reset password" or "3 more days until scheduled reboot"
This simply does not and cannot happen to an airline. If it does people will be fired and processes revised.
oh boy, you sound like someone that is big into the brochures.
with the current state of Boeing, nothing would surprise me, and there's very little i would accept at face value from them.
Wrong, planes are powered down often.
I think this is one of the areas that airlines are super good at. Every single plane needs strict regular maintenance and tons of checklists in order to take off, so something like a 50-day requirement for a reboot would be in that list of items, and if the people aren't following those lists of items you're going to have things like turbine failures, so they get done.
Maybe if there's a really badly run airline it might be a problem, but there are much more severe problems than software in a case like that.
Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
>Boeing gets away with this software issue because it's not really a big issue. The scary point the media can tell you because you don't have context to understand why it's mundane.
Yes, like the mundane software overriding pilot's intent from software written specifically to not retrain pilots on the system? Yup, sounds mundane to me.
When I said I don't trust anything from Boeing, it had little to do with this 50 day reboot, but much more in line with the Boeing is allowed to self certify and has been shown to not do that in a good way. They have ruined that trust because of greed. You might think that's a broad brush? At this point, every little bit is just another instance of death from a thousand cuts.
I would hope aircraft have some sort of "glider mode" that allows them time to reboot critical systems in an emergency.
They do, it's called "being an airplane."
Airplane always want to fly, give them enough speed, they will take off on their own. Thats why you have bumpy landings sometimes because pilots didn’t slow down/flare properly to slowdown the aircraft instead forced it to ground. With enough wind some general aviation aircraft are capable of takeoff/land vertically like a chopper.
Usually these aircraft are power cycled often. There is no way they are flying these planes 51 days straight with no power cycle. Maintenance checks would necessitate a restart.
I’m pretty sure most of the computers can be reset during flight. You may lose flight protections, but you can fly an airplane on basic instruments. You can even handle unreliable speed by flying at a specific angle of attack and a proper thrust setting. (According to the Air France 447 report)
On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
>On some (all?) Airbus planes, the basic flight controls don’t require the flight computers to function.
I would honestly hope that would be true about any airplane from any manufacture. Fly-by-wire makes some of that more difficult when the wires go through the computer, but even then I'd have assumed some sort of 100% manual capability. Like when Y2K nutters were predicting airplanes falling out of the sky because of 2-digit years rather than the logical the planes would behave according to laws of physics for an intact air frame.
It looks like the mechanical backup for modern Airbus planes is stabilizer trim and hydraulic rudder. https://skybrary.aero/articles/flight-control-laws
No idea about 787, but usually there’s a bunch of fuses in the cockpit (eg https://thumbs.dreamstime.com/b/fuse-panel-airliner-18627188...), and they are made easily accessible for a reason.
I learned programming in the late '70s on a Wang 2200T. The manual had a procedure it would recommend called "Master Initialization." Yes: turning the power off and on.
commercial air travel is all about critical maintenance. Add it to the checklist and move on. Feels like the article pandering to that y2k nostalgia.
Absolutely. It's like the regular social media mini uproar when someone discovers duct tape on the plane they are flying. It's speed tape.
Reminds me of the fact that a lot of Boeing planes still operate on floppy disks for data https://www.popularmechanics.com/science/a33612762/boeing-74...
Well a lot of planes are decades old, the fact some use floppy disks doesn't surprise me at all
Nitpick: the floppy disks are used for software updates, not flight operations.
I wonder what's being used to update B-52s.
And now we know the 787 runs on Windows.
Hopefully the forced Windows 11 upgrade isn't scheduled for when you're over the Atlantic.
It doesn't.
I believe it's called a joke.
It’s probably more complex than a slow memory leak, but is very reminiscent of one.
Maybe something to do with a clock. Space shuttle missions couldn't fly during the new year clock/date changeover...
Anything new here?
Some discussion from 6 years ago:
https://news.ycombinator.com/item?id=13094600
I think that's fine. I mean, what could possibly go wrong?
Here’s a properly technical explanation of the problem: https://news.ycombinator.com/item?id=33239443
Who are the programmers behind this average work and who are their superiors?
I would not be so brazen as to call the engineers behind this "average" or their work so. Unless you can make the argument that a plane would fly for 51 days straight, being required to reboot the on-board systems is not a major deal-breaker, especially as the larger Boeing's take 60 minutes to refuel.
The engineers behind Boeing planes have built an incredibly sophisticated machine which omitting a very nessescary but not so inconvient task of rebooting every n days, performs its primary directive well.
If this is a known issue, what are the chances of a related, unknown and potentially catastrophic issue?
Lower by one factor than before this issue was determined.
Considering it's likely an issue with an integer overflow for a millisecond clock: near zero.
One bug, does not imply two bugs.
one bug implies two bugs only in the sense that any given software has at least one bug and any given software has at least one more bug than you know exists ;)
This a known limitation, not an issue.
Chill out..
Every component in an aircraft require maintenance and inspections after so many operating hours. Why should software be any different?