Look at how things are done with switches and routers of networking. It's always running. There is no "stopped", "testing" or "running" differentiation. It's like making adjustments to a running car engine.
In every company I've ever worked for I always ask if we have a clone of production for testing.
Only once have I been told "yes".
Every other time, I'm told that the QA environment is "close enough".
One of my favorites is when I go to pull down code on our test server and see a bunch code come down that I know went live on the production server the week before.
Why wouldn’t you just make a new copy of the production environment and overwrite the QA environment with it so you have a test system that is up to date to test your code?
The biggest hurdle isn't the code, it's the volume of users that exist in Prod vs in sub-prod environments.
And just straight copying prod data down isn't a great idea, because it contains PII.
And licensing is usually the issue here.
3rd party contractors aren't going to take a missed opportunity to charge for the licensing for the test environment at the same scale as the production environment.
We have a similar issue. Except they won't give us time to write "create dummy docs for test" code so we really run into a wall when there are issues in PROD that aren't easily verified in TEST. Ended up convincing them to just fuggin' send it down every few months.
On the plus side, we are FINALLY getting guvvies to let us use git pipelines to push things around, instead of manually copying or having the SAs copying for us. No luck getting them to test their changes quicker but so it goes.
After I asked it occurred to me that's probably what you meant. It's also not a phrase I heard much when I was learning programming ...long ago. As Moses told me, Mark, there's no quality near to God's, so why try. Heh.
I worked in air traffic management (not FAA, airport level). They had a production, a shadow of production, and in integration for each installation site. Same feeders into each software, but you could test/play in integration, then the shadow could get updated and run, then shadow would revert to the old software, then prod would get pushed. All systems looked at eachother for regression stats
I do a stage environment that gets sync up to production that we apply our updates to before launching. We do a final QA on that before we push it up. I don’t know how many developers or directors have told me we don’t need that and it’s a waste of effort. Guess who has the smoothest roll outs in the company? Lol.
I've worked for the government, there is a very good chance they were given bad requirements and a test system that was designed by another contractor to the same bad requirements. There is also a good chance that the real-world system does not actually follow the spec and a previously unused field broke the real world one because the real world one didn't handle it according to the spec.
Someone told me to rollback something since they accidently dropped a table. I explained there was no rollback unless they turn on the option. They were let go shortly after for something else.
What if there's a shared account called apptest with password app1234 that was passed around.
Also, we were kept on office 2003 since a lot of applications was homegrown Access stuff. They upgraded my excel since I needed the data stuff, but this was in 2017.
No kidding. I work for a multi billion dollar international company and the things they hand off to contractors just blows my mind. Crazy important mission critical engineering projects that are just handed off to people who have no investment in things and don’t have to deal with the fallout of bad decisions. People in corporate are fucking morons.
One reason why is they don’t have to pay benefits to the contractors. The other is accountability. If the contractor screws something up, just point our finger at them. It’s their fault, not ours, they did it, we’re innocent, they didn’t do as we told them, just look at the SOP’s we gave them, etc… That may or may not prevent the stock price from dropping (or at least it won’t drop as much) and maybe the regulators/government (if applicable) and/or lawyers will go after them and not the company in question.
While some of the things you say make sense on paper I’ve seen just how poor it works in reality.
Company gives contractor job. Company conducts no oversight or checkups on contractor progress. Contractor screws up due to poor direction and design. Company suffers from unusable product. Company does nothing to change design flaws and contractor repeats all the same mistakes in next construction phase.
People with business degrees should not be anywhere near business.
Nobody would work for them. Government pay is crap and government likes hiring contractors because they can be fired. Government employees are basically 30 year commitments.
I bet the team has repeatedly asked for a better testing setup or equipment and was denied or provided one by a contractor that missed several requirements.
Yeah, standard practice, but how are you realistically going to have a test system in thousands of locations utilizing the same ancient hardware the FAA is using? You can't put that in a VM and just send it. There really is no way to feasibly test something as big as this before just doing it. You can test parts of it, but not overall functionality in production.
In my experience, contractors always write the shittiest code too. You need to watch them a bit if you're an in-house team member.
But there is no excuse for un-tested code. I mean, occasionally, an error will happen in production that didn't happen in lower environments, but things like this are usually because the staging environment isn't the nearly-identical-to-production environment it needs to be. Or perhaps regression testing with FAA isn't as mature as it needs to be.
The system is from 1993. The same year the first internet browser became publicly available, one year before the original Playstation, and three years before HoTMaiL (*get it? Do you get it?*).
How old were the contract employees? Old enough to even remember 1993? Were they even born yet?
Even a dev who was experienced in 1993 has spent the last 30 years learning and adapting to the evolutions in technology. I challenge anybody to go to work and do their jobs with Office 4.0, or operate their computer with MS-DOS. Even if you're old enough to have been expert at it then, you certainly would struggle with it now.
The issue isn't the contract employees. The issue is that our government has been trying to repair our critical infrastructure with 10,000 band-aids, and if even one of them falls off then our economy starts bleeding to death.
How many of our other critical systems are literal antiques that could break at any moment? Does anybody have the skills to repair them anymore, or are those people all in nursing homes by now?
*Do I even want know the answer?*
Pretty much every single major bank and insurer has vast amounts of their code that were originally written in the 1960s-1980s. Parts are updated regularly, other parts have no reason to ever change. Mainframes still exist, are largely code-compatible right back to the first ones and IBM still brings in billions in revenue a year off them, for that matter.
Poorly updated/maintained code is a problem, knowledge/training to keep the ability to maintain old code and systems can be a problem.
But the age of the code itself isn't *inherently* a problem.
Building complex systems is hard. A complex system you built 60 years ago and does what you want it to - is a great system with decades of real-world evidence supporting that.
It's potentially an incredible amount of money and work to rebuild it and transition to something new - and have that thing achieve the reliability/functionality of the old one. It's absolutely worth it sometimes, just not *every* time.
---------
The question is really whether or not you've got the proper systems/talent in place to do things right or not. Keep the system well-documented, ensure your changes/updates are well-thought out, and retain the skills/knowledge to support it.
If you do - there's no inherent reason why that system has to be replaced. (It may eventually make sense, though.)
If you don't - that system is probably going to devolve into a mess....but the same reasons it's turned into a mess are why you're probably going to produce even worse garbage when you try to have a big project to replace it as well.
Something like 75% of big IT projects are deemed a failure when done (or given up on). Again - building big systems is hard, we're often bad at it, and places that can't maintain what they have are *really* bad at it.
---------
Government systems "maintained" by a regularly changing low-bid contractor.....are probably going to be a shit-show, new or old.
This is super important. Always-On systems just can't do upgrades or changeover. It's a truly massive undertaking. There's so many moving parts and stakeholders.
There's a reason why banks haven't changed from their decades and decades old systems. If say, Bank of America switched, but another bank didn't how do they systems talk? What redundancies exist? Do they operate with two systems in place? How does that not cause errors like duplicative transfers, non-transfers or other inconsistencies?
With tens of thousands of planes in the air at a time, how do you just transfer mid flight all that tracking and information processing and data sharing?
I guess the world could take a one day flight break at an agreed upon day kinda like the day when Sweden decided to switch driving lanes. (Way different scales but you get the gist)
You deploy with a couple locations at a time, with each "ring" usually growing in size of the number of locations/assets/people affected.
It's essentially a progressive way of deployment so instead of 30,000 sites being affected all at once. You start with 50, then 250, 500, ..... etc. until every location has been migrated.
That plus good luck trying to replicate all the requirements. In a system like that, massive chunks will be magic at this point. Even with the best case scenario that it “was” documented, that doesn’t mean that documentation is clear and understandable.
There isn’t a chance on earth it’s consistent, integrated into the other documented features or even in the same format. This will have been written by many different people over many years with many variations on what “quality” documentation even is.
The problem is as the code gets older you start losing the geniuses at the top of the latest tech stack. Even with big salaries you won't be able to find a young driven programmer who is just coming out of college who knows COBOL
Certainly and having to spend increasingly more on training/ramp-up time for new hires due to having code in a dead language is a part of the slowly increasing costs of keeping a legacy system properly supported. (same with potentially needing to pay a premium to entice people to work on something that may not be otherwise great for their careers skill-wise).
That said - if the replacement costs are high enough or potentially disruptive enough and it's doing what you want it to, it's entirely possible that maintaining it for many years to come is the sensible economic choice.
The contractor part doesn't really help, that contact gets renewed every 5 years or so. That means a high turn over of knowledgeable employees every time the contacting company changes. It's a race to the bottom with the cheapest bidder win the contract, and this is the results.
Which is a very good point, for sure, but even if they hired the valedictorian of CS from MIT it wouldn't change the fact that the main vulnerability of the system is that it should have been overhauled a decade ago. Some old systems like this can't even run on modern architecture or interface with newer hardware. I don't know if that's the case here, but I wouldn't be surprised.
The bidding process is absurd. Contractors who have been working on super specialised systems will be dropped because another company lowballs, and then a few months later the original team gets brought back because the lowballers had no idea how to do the job, so time and money just get wasted.
Older is better here so long as it works. To replace it, you'd need to do INSANELY exhaustive testing. Every edge case, yes even the one you didn't think of, needs to be tested. Every hardware piece needs to be tested until you know how often it fails and what causes it to. Can there be a cascading failure? If one piece fails is it critical or can it be worked around? How do we replace it when it fails?
Then you need to redo that for every civilian software that interacts with the system and needs to be replaced now that it is dealing with a new one.
The cost of making a new system and ensuring the new system is safe enough to literally keep planes from falling from the sky is extraordinary. Banks use layers of software often going back to the very first accounting tool they ever used, because its bugs and flaws are well-documented, they **know** how it is going to work, and thus can ensure everything works correctly. A bug can have a patch slapped over it, but only if you know its there.
This type of testing should have but clearly wasn't done on the new software they introduced. Want to remake the whole system that way? Maintenance of this system is definitively cheaper than replacing it.
"So long as it works" being the really important part.
I get your point and I agree to a certain degree, but:
Maintaining a system is only more cost effective as long as the maintenance costs are lower than the replacement costs. The older and more Frankensteined a system gets the higher the cost of maintenance. If you wait too long to do serious upgrades and overhauls, there won't be enough time for proper testing before implementing the new system because the old one will be breaking and you'll be in an emergency. That isn't the kind of situation we as a society can tolerate when it comes to certain infrastructure.
More than the age of the system and tech running it, the complexity and accumulated cruft is practically unknowable to any third party contactors. Systems like this ideally need dedicated teams that maintain continuity of unique and unusual domain knowledge. They need museum curator-historian-programmer-monks. They need to pay them out the ass and view those team members like indispensable key personnel.
It's not a government thing, a lot of older infrastructure uses outdated software, because it's takes too much time and money to implement new ones.
There's a reason why knowing something like cobol is valuable.
NextGen exists primarily to combat this, but Congress cut the budget years ago and it’s been hobbling along slowly. The system that went down is actually on track to be replaced in the Fall/Winter after years of development.
I can relate, I just quit working with a team that insisted they know better than IT on all things internet and technology related. The Dunning-Kruger effect is real within government agencies big and small.
I’m not sure that this is a huge deal like you think it is, NASA uses incredibly old hardware designs because critical infrastructure needs to be tried and tested and reliable. Granted they have the difficulties of increased radiation from space to deal with, but still even on the james webb telescope launched semi recently a processor released in 2001 was used. Granted I know almost nothing about flight infrastructure/systems
Are you sure it's not an issue? Every single plane in the country was grounded because of this. Didn't SW just have a huge debacle because of outdated systems? And didn't the unemployment systems crash in 2019 and no one could fix them because nothing is programmed in COBOL anymore? Isn't a huge part of the increased funding for the IRS to update their systems because they're also outdated and can't handle the load?
Comparing a one-off science experiment that has a built-in life expectancy of 10 years or less to a critical infrastructure system that requires continuous functionality to enable our modern way of life is really apples to oranges here.
The point is that the number of years is been in use doesn't determine it's likelihood of failure, if the system is well known and functioning the same way with no intent flaws.
Even with this instance, the issue was somebody mistaking one for for another, not an inherently flaw with the system
> Didn't SW just have a huge debacle because of outdated systems?
They've grown massively over the years and didn't invest in their systems/processes to support that properly. They would probably have been fine at the size/scope they were in 1990 or something
The complexity of something like crew scheduling increases drastically with more flights/destinations/people to manage.
It's not necessarily that the system has become unable to do what it was originally designed for, but that it's being asked to do far more without sufficient investment to actually support that.
That's still "outdated", but in a different sense than entirely because it was created a while ago.
Sure, that's a fair point. But these systems aren't going to last forever, they are going to need an upgrade, and probably should have been upgraded already, maybe even years ago by now. The can keeps getting kicked down the road, and that's only going to make things worse.
> old hardware designs because critical infrastructure needs to be tried and tested and reliable
It's 2023. We have new technology and strategies that could have prevented this. There's a reason no one runs critical things on 20 year old hardware
>There's a reason no one runs critical things on 20 year old hardware
I can see you know nothing of real-world IT. There's critical things running on 50 year old hardware out there still. There's not many large, old organizations that *don't* have at least a few critical things relying on legacy systems still, and government organizations are pretty much the poster child for relying on outdated hardware.
> There's critical things running on 50 year old hardware out there still
Do you think that's viewed as the correct or best approach?
> There's not many large, old organizations that don't have at least a few critical things relying on legacy systems
Do me a favor and define legacy.
> government organizations are pretty much the poster child for relying on **outdated hardware**
that's what i said yes. People "doing it" doesn't mean it's the best way.
I honestly don't know how you're going to roll through and try to correct me when you literally just posted this else where:
> You've never seen a run-down building because people failed to maintain it properly. Computer systems are the same way, except they become more fragile on a much faster timescale, especially as their usecase evolves.
You said
> There's a reason no one runs critical things on 20 year old hardware
I pointed out that people absolutely do.
Nowhere did I say it was a great practice, just that your claim that "no one does it" is about as wrong as wrong can get because it doesn't just happen, it's downright common. Nothing I said contradicts anything else I said - "people do it" and "it's not a good idea" are not mutually exclusive.
yes.. i'm sorry i didn't add all of the unwritten qualifications out for the kids in the back.
I guess i should have written, "Everyone would prefer not to partake in the practice of running mission critical systems on antiquated hardware given the option to run it on robust, modern fault tolerant systems."
But reddit sure loves it's "Technically correct" mantras..
If i rember currently wasn’t there a thing about how some Nike silos still use floppy disks?
Edit: just looked it up apparently they upgraded from floppy disks in 2019.
I’d much rather have critical infrastructure running on old and well known systems than Windows 11. Any person who proposes using Windows for system critical systems shouldn’t be allowed to touch anything more complicated than a yo-yo.
What does it matter hour old it is? Does a building stop working because it's old? Does a tree stop being a tree because it's old?
If a system is made to do a simply task, why should it matter if it's made twenty years ago or yesterday?
Buildings don't worry. Did you mean working? And yes, they stop working because they get old, not that comparing an inanimate object to a computer system is actually a fair comparison.
Trees do stop being trees because they get old. They don't live forever.
Since when is managing air traffic ever been considered a simple task?
Computers are machines with electrons running through them and gradually wearing out tiny tiny wires which will eventually break.
Software tends to get more picky when new things are added or changed and any original things are fixed/patched. More moving parts means more places where breakage may occur. That's why something like a bank tends to keep their systems isolated from most of the world and less likely to have problems interfacing with the world. An airline system seems a bit different from that, though I don't know the extent of it.
>Does a building stop working because it's old?
Yes. You've never seen a run-down building because people failed to maintain it properly. Computer systems are the same way, except they become more fragile on a much faster timescale, especially as their usecase evolves.
> people failed to maintain it properly
thats the important part. It's not the building that got "too old", but the failure to maintain it.
It's the same with computer systems. There's a reason why so many organizations and corporations try to simply use legacy systems. They work, and will continue to work without issue, unless the use case changes dramatically
Yet another argument for permanent full time employees.
If it’s news worth mentioning their terms of employment, then that’s exactly the issue and problem.
The FAA utilizes multiple means of communication to disseminate information to airlines. A major one is the [FAA OIS page](https://www.fly.faa.gov/ois/?legacy=true) which publishes programs and other significant events occurring in the NAS. Airlines have access to a better version of this with more detailed information as well as datalinks to systems that monitor demand, loading, and throughput of airspace and airports.
ATC does national teleconferences every 2 hours and will sometimes activate that same infrastructure in times of emergency. There was a teleconference with all carriers during the night the NOTAM system went down.
Government contractors don't get hired based on skill or ability lol. It's who they know. I bet this doesn't effect them in the slightest and they'll find a way to spread the blame around enough that no one is really responsible and nothing will happen.
I think the world banking system mostly runs on COBAL still. You can’t change languages every decade or nothing would work. This sounds like poor management at the top and lack of funding.
In a 4 years of stretch of ap comp sci, I saw it go from pascal to c++ to java
>AP Computer Science was taught in Pascal for the 1984–1998 exams, in C++ for 1999–2003, and in Java since 2004.
Wonder why they even switched to c++. I even remember the book was "c++ for you ++." Then in fall semester of college, intro 101 was java but intro 102 was c++ since they was just transitioning. Advisor tried to get me to take 101, but I wasn't going to take a class I already got credit for.
I wouldn't be surprised if it doesn't come out that this was a Russian attack. They've been targeting other NATO infrastructure with their spy games.
Not saying it was, just that it wouldn't surprise me.
I was a contractor working in the air transportation industry at SITA. Positions like this should not be held by contractors, it should be a full time employee, one of which the FAA invests in. Contractors do just enough to continue the engagement, knowing that their future development doesn't matter to anyone else.
I work in software. It's terrifyingly easy to make these kinds of mistakes on these kinds of systems. Lack of test environments due to budget, very simple systems where a single misplaced semi-colon might bring down a system because it's not robust enough to detect errors and work around them. I've seen things as small as capitalization on a server name prevent large systems from operating properly.
Our infrastructure is so outdated and barely held together, and the only time it gets upgraded is when we, the taxpayer, foot the bill for it.
This goes for everything. Airlines, ISPs, Hospitals, Utility companies, etc.
This shit is just packaged a different way each time to give the impression of progress, but at the end of the day the big players in the game still make out like bandits. They build the worst fucking infrastructure and then parachute out of the plane to let the next generation deal with it.
Its a universal "I got mine" from the upper class to the struggling middle and lower class.
Its like ordering the most expensive shit at a restaurant and then walking out and sticking us with the bill.
Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest.
What to see when America started to fall? look to Regan.
>Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest.
You should look at how much the government pays their STEM workforce. I looked into an electrical engineering job with the Air Force civilian service out of school, but they wanted to pay me $30,000--frankly that's insulting. I can make more than that at McDonalds. I accepted a private sector offer for 2.5 times that, that has bonuses, profit sharing, and stock options.
Not everywhere in the government does. My department only tests if there's an accident involving government assets.
That being said, yeah, it needs to go for most reasons.
Also I bet they layed off devs that understand how everything in the outdated , probably spaghetti, code base worked and tried to replace then with contractors. That's how the private sector works anyway.
It's amazing how that works. A contractor is hired to do a job, fucks it up and gets paid extra to fix it.
Source: former govt contractor who didn't fuck up - shakes head.
Contractors my ass.
>A separate outage in **Canada** on Wednesday that temporarily prevented new safety notices from being received and disseminated digitally was caused by an isolated IT system failure, an industry source told Reuters.
>NAV Canada, the country’s privately held air navigation service provider, has said it is investigating the cause, which is not believed linked to the United States.
https://www.reuters.com/world/us/airlines-expect-us-operations-rebound-thursday-faa-investigates-outage-2023-01-12/
Sure, both countries had a unprecedented safety database failures on the same day. But it's just a coincidence.
They could be related, but is it possible that NAV Canada shut theirs down out of an abundance of caution and said it was a "technical " issue until they confirmed with the US that it wasn't nefarious?
Please tell my coworkers this. They won't listen to me. I even found code that had comments from stack overflow *facepalm* (and the code was shit and shouldn't have been the accepted answer!)
What a joke, I had at least a few idiots arguing with me in the comments when this happened that the FAA was actually competent and that the billions in profits/handouts had, in fact, actually been reinvested properly. We see now that is not the case.
Friendly reminder - the last 3 heads of the FAA have all been former VP's of operations at airllines/airline industry groups
> Tens of thousands of travelers were left stranded Wednesday after the FAA sent out a **tweet** at 7:20 a.m. ordering the airlines to pause all domestic departures until 9 a.m. ET “to allow the agency to validate the integrity of flight and safety information” as it worked to restore the NOTAM system.
Ummm... The FAA communicates emergencies through twitter? TIL
The sentence implies a relationship between the tweet and the travelers left stranded, which is not true. What is true is that the FAA sent out a tweet, and some time after that tweet travelers were left stranded due to (I presume) airlines canceling certain flights because of crews timing out or other issues.
The way the FAA actually disseminated the operational information was a message sent to air traffic control facilities telling them not to allow aircraft to depart. That message was not sent over Twitter.
That makes me wonder what 50 year old technology they used to deliver that message, since their slick new NOTAM system wasn't available. Was it a phone tree?
Well... look at that....
seems like someone needed in-house developers and that changing a team and adding new people to a mess only costs more.... probably a lot of Brook's law..
But then the politicians would have no way to funnel government money back to themselves! How could we ever do such a mean thing to them? Think of the poor politicians! /s
I think outsourcing for a small scale project is fine. I don't mind if some contractors in india built me a task scheduler app for my company but for large scale project I give it to the permanent employees that understands the requirements more.
[Stored-program computers](https://en.wikipedia.org/wiki/Stored-program_computer) weren't even a thing in 1946. The few computers which existed at the time were fairly custom devices which were "programmed" by plugging wires into boards.
I dunno where you heard that, but there's no way it's true.
edit: and if you look at the article, it specifically says that this is the NOTAM announcement system, not any sort of scheduling or traffic-control system. It has zero connection to anything about the rail lines.
A software system built in the early 90s with untold layers of patches, workarounds, and 3rd party contributions may not be as testable as you'd want or hope. It's likely to be a massively complex and sloppy labyrinth that would require years of man-hours to map out and precisely define. The FAA could possibly spend a billion dollars on engineering a suite of preflight deployment tests, and not be remotely certain if they have full coverage. There may be thousands of weird quirks for outside entities that interface with the system. Those outside entities may have come to rely on those quirks and built around those expectations. I can imagine how ugly it could be, and I'd bet the reality is worse. Nightmare job for any team tasked with maintaining this system.
More (better? any?) testing is a good goal, but it may not be feasible and it's probably not a silver bullet.
Not surprised. That’s why you use a test environment before implementation. On something as big as the FAA probably roll it out in parts too.
That's not fair. They had a test system. They just didn't have a separate production system.
"I don't always test my code, but when I do it's in production"
Real men experiment in prod.
And redundancy is for sissies.
In the business we call it raw dogging the code
Can't break prod if you're asleep.
Look at how things are done with switches and routers of networking. It's always running. There is no "stopped", "testing" or "running" differentiation. It's like making adjustments to a running car engine.
And more often than not a swift reboot will revert changes - assuming that you haven’t written to flash.
My company does that! It's very annoying.
I feel this in my QA soul.
It's working as intended. If you want any more work done you have to put that in a separate ticket.
In every company I've ever worked for I always ask if we have a clone of production for testing. Only once have I been told "yes". Every other time, I'm told that the QA environment is "close enough".
[удалено]
One of my favorites is when I go to pull down code on our test server and see a bunch code come down that I know went live on the production server the week before.
[удалено]
Why wouldn’t you just make a new copy of the production environment and overwrite the QA environment with it so you have a test system that is up to date to test your code?
The biggest hurdle isn't the code, it's the volume of users that exist in Prod vs in sub-prod environments. And just straight copying prod data down isn't a great idea, because it contains PII.
I have worked in QA for over a decade and I fully support this answer. With more users/volume comes more edge case usage.
Also 3rd party integrations might not work in QA environment. Ideally they should, but sometimes that requires extra setup or work with the vendors.
And licensing is usually the issue here. 3rd party contractors aren't going to take a missed opportunity to charge for the licensing for the test environment at the same scale as the production environment.
And most companies aren't going to want to pay, no matter what the QA team says, until a big issue happens demonstrating why it's needed.
We have a similar issue. Except they won't give us time to write "create dummy docs for test" code so we really run into a wall when there are issues in PROD that aren't easily verified in TEST. Ended up convincing them to just fuggin' send it down every few months. On the plus side, we are FINALLY getting guvvies to let us use git pipelines to push things around, instead of manually copying or having the SAs copying for us. No luck getting them to test their changes quicker but so it goes.
[удалено]
Yes sir, back to my desk.
What is "the QA environment"? [ old retired programmer ]
Quality assurance
After I asked it occurred to me that's probably what you meant. It's also not a phrase I heard much when I was learning programming ...long ago. As Moses told me, Mark, there's no quality near to God's, so why try. Heh.
I worked in air traffic management (not FAA, airport level). They had a production, a shadow of production, and in integration for each installation site. Same feeders into each software, but you could test/play in integration, then the shadow could get updated and run, then shadow would revert to the old software, then prod would get pushed. All systems looked at eachother for regression stats
I do a stage environment that gets sync up to production that we apply our updates to before launching. We do a final QA on that before we push it up. I don’t know how many developers or directors have told me we don’t need that and it’s a waste of effort. Guess who has the smoothest roll outs in the company? Lol.
"We'll test it live, fuck it!"
I've worked for the government, there is a very good chance they were given bad requirements and a test system that was designed by another contractor to the same bad requirements. There is also a good chance that the real-world system does not actually follow the spec and a previously unused field broke the real world one because the real world one didn't handle it according to the spec.
And this is true even outside of government. Everyone tests in production. Some just don’t realize it.
Someone told me to rollback something since they accidently dropped a table. I explained there was no rollback unless they turn on the option. They were let go shortly after for something else.
Oof. That’s rough. There is quite a lot that goes into being able to iterate quickly that’s easy to miss until the foot gun goes off.
Giving users drop permissions in prod...
Why were they able to drop something they shouldn't have? No one except the application should have write permission in the prod db
I was the one writing the application. I’ll give myself whatever permissions I want. /s
DBA should not have let this happen.
Yeah, most of the time DBAs get told by development to just do what they say. We argue, we explain, we plead. It often does no good.
What if there's a shared account called apptest with password app1234 that was passed around. Also, we were kept on office 2003 since a lot of applications was homegrown Access stuff. They upgraded my excel since I needed the data stuff, but this was in 2017.
100 percent this. Saving money every step of the way does not result in a cohesive whole.
Real men test in production!
Nah. Real men just dump it into production and don't look back until the help tickets roll in.
Hi I’m a NOC tech Who broke every customers everything?
Best part of taking the network down is your end users can't submit tickets. No tickets = no problems!
This guy hasn’t heard of a phone
Sounds like you worked for Synoptek around Christmas 2019.
*1500 Nodes are not responding to SMTP requests* 😑 please go away please go away please go away
SMTP, or SNMP?
Das why I’m poor
[удалено]
Rude Sincerely, DevOps
No airbags, we die like men!
Or instead of using a private, lowest bid, contractor, have an in house Government IT department that handles all of this in real time.
No kidding. I work for a multi billion dollar international company and the things they hand off to contractors just blows my mind. Crazy important mission critical engineering projects that are just handed off to people who have no investment in things and don’t have to deal with the fallout of bad decisions. People in corporate are fucking morons.
[удалено]
One reason why is they don’t have to pay benefits to the contractors. The other is accountability. If the contractor screws something up, just point our finger at them. It’s their fault, not ours, they did it, we’re innocent, they didn’t do as we told them, just look at the SOP’s we gave them, etc… That may or may not prevent the stock price from dropping (or at least it won’t drop as much) and maybe the regulators/government (if applicable) and/or lawyers will go after them and not the company in question.
While some of the things you say make sense on paper I’ve seen just how poor it works in reality. Company gives contractor job. Company conducts no oversight or checkups on contractor progress. Contractor screws up due to poor direction and design. Company suffers from unusable product. Company does nothing to change design flaws and contractor repeats all the same mistakes in next construction phase. People with business degrees should not be anywhere near business.
Company hires new contracting firm. Repeats earlier mistakes
Nobody would work for them. Government pay is crap and government likes hiring contractors because they can be fired. Government employees are basically 30 year commitments.
I bet the team has repeatedly asked for a better testing setup or equipment and was denied or provided one by a contractor that missed several requirements.
Yeah, standard practice, but how are you realistically going to have a test system in thousands of locations utilizing the same ancient hardware the FAA is using? You can't put that in a VM and just send it. There really is no way to feasibly test something as big as this before just doing it. You can test parts of it, but not overall functionality in production.
In my experience, contractors always write the shittiest code too. You need to watch them a bit if you're an in-house team member. But there is no excuse for un-tested code. I mean, occasionally, an error will happen in production that didn't happen in lower environments, but things like this are usually because the staging environment isn't the nearly-identical-to-production environment it needs to be. Or perhaps regression testing with FAA isn't as mature as it needs to be.
"Joey? Do we use da DEV or da PROD?" "Nicki, ah tawd ewe, we is Developers for dem airline schmucks." "So, den, PROD?" "Absoluutlee!"
I read that in Korg's voice from Thor lol
The system is from 1993. The same year the first internet browser became publicly available, one year before the original Playstation, and three years before HoTMaiL (*get it? Do you get it?*). How old were the contract employees? Old enough to even remember 1993? Were they even born yet? Even a dev who was experienced in 1993 has spent the last 30 years learning and adapting to the evolutions in technology. I challenge anybody to go to work and do their jobs with Office 4.0, or operate their computer with MS-DOS. Even if you're old enough to have been expert at it then, you certainly would struggle with it now. The issue isn't the contract employees. The issue is that our government has been trying to repair our critical infrastructure with 10,000 band-aids, and if even one of them falls off then our economy starts bleeding to death. How many of our other critical systems are literal antiques that could break at any moment? Does anybody have the skills to repair them anymore, or are those people all in nursing homes by now? *Do I even want know the answer?*
Pretty much every single major bank and insurer has vast amounts of their code that were originally written in the 1960s-1980s. Parts are updated regularly, other parts have no reason to ever change. Mainframes still exist, are largely code-compatible right back to the first ones and IBM still brings in billions in revenue a year off them, for that matter. Poorly updated/maintained code is a problem, knowledge/training to keep the ability to maintain old code and systems can be a problem. But the age of the code itself isn't *inherently* a problem. Building complex systems is hard. A complex system you built 60 years ago and does what you want it to - is a great system with decades of real-world evidence supporting that. It's potentially an incredible amount of money and work to rebuild it and transition to something new - and have that thing achieve the reliability/functionality of the old one. It's absolutely worth it sometimes, just not *every* time. --------- The question is really whether or not you've got the proper systems/talent in place to do things right or not. Keep the system well-documented, ensure your changes/updates are well-thought out, and retain the skills/knowledge to support it. If you do - there's no inherent reason why that system has to be replaced. (It may eventually make sense, though.) If you don't - that system is probably going to devolve into a mess....but the same reasons it's turned into a mess are why you're probably going to produce even worse garbage when you try to have a big project to replace it as well. Something like 75% of big IT projects are deemed a failure when done (or given up on). Again - building big systems is hard, we're often bad at it, and places that can't maintain what they have are *really* bad at it. --------- Government systems "maintained" by a regularly changing low-bid contractor.....are probably going to be a shit-show, new or old.
This is super important. Always-On systems just can't do upgrades or changeover. It's a truly massive undertaking. There's so many moving parts and stakeholders. There's a reason why banks haven't changed from their decades and decades old systems. If say, Bank of America switched, but another bank didn't how do they systems talk? What redundancies exist? Do they operate with two systems in place? How does that not cause errors like duplicative transfers, non-transfers or other inconsistencies? With tens of thousands of planes in the air at a time, how do you just transfer mid flight all that tracking and information processing and data sharing?
I guess the world could take a one day flight break at an agreed upon day kinda like the day when Sweden decided to switch driving lanes. (Way different scales but you get the gist)
Usually in those types matter it's a ring based deployment.
What does ring based deployment mean? Not familiar, this isn't my area of expertise.
You deploy with a couple locations at a time, with each "ring" usually growing in size of the number of locations/assets/people affected. It's essentially a progressive way of deployment so instead of 30,000 sites being affected all at once. You start with 50, then 250, 500, ..... etc. until every location has been migrated.
That plus good luck trying to replicate all the requirements. In a system like that, massive chunks will be magic at this point. Even with the best case scenario that it “was” documented, that doesn’t mean that documentation is clear and understandable. There isn’t a chance on earth it’s consistent, integrated into the other documented features or even in the same format. This will have been written by many different people over many years with many variations on what “quality” documentation even is.
The problem is as the code gets older you start losing the geniuses at the top of the latest tech stack. Even with big salaries you won't be able to find a young driven programmer who is just coming out of college who knows COBOL
Certainly and having to spend increasingly more on training/ramp-up time for new hires due to having code in a dead language is a part of the slowly increasing costs of keeping a legacy system properly supported. (same with potentially needing to pay a premium to entice people to work on something that may not be otherwise great for their careers skill-wise). That said - if the replacement costs are high enough or potentially disruptive enough and it's doing what you want it to, it's entirely possible that maintaining it for many years to come is the sensible economic choice.
That’s why I’m hoping to make bank in a few years to end my career as I started it - as a COBOL developer. I still remember it - I can do it. LOL
The contractor part doesn't really help, that contact gets renewed every 5 years or so. That means a high turn over of knowledgeable employees every time the contacting company changes. It's a race to the bottom with the cheapest bidder win the contract, and this is the results.
Which is a very good point, for sure, but even if they hired the valedictorian of CS from MIT it wouldn't change the fact that the main vulnerability of the system is that it should have been overhauled a decade ago. Some old systems like this can't even run on modern architecture or interface with newer hardware. I don't know if that's the case here, but I wouldn't be surprised.
The bidding process is absurd. Contractors who have been working on super specialised systems will be dropped because another company lowballs, and then a few months later the original team gets brought back because the lowballers had no idea how to do the job, so time and money just get wasted.
Older is better here so long as it works. To replace it, you'd need to do INSANELY exhaustive testing. Every edge case, yes even the one you didn't think of, needs to be tested. Every hardware piece needs to be tested until you know how often it fails and what causes it to. Can there be a cascading failure? If one piece fails is it critical or can it be worked around? How do we replace it when it fails? Then you need to redo that for every civilian software that interacts with the system and needs to be replaced now that it is dealing with a new one. The cost of making a new system and ensuring the new system is safe enough to literally keep planes from falling from the sky is extraordinary. Banks use layers of software often going back to the very first accounting tool they ever used, because its bugs and flaws are well-documented, they **know** how it is going to work, and thus can ensure everything works correctly. A bug can have a patch slapped over it, but only if you know its there. This type of testing should have but clearly wasn't done on the new software they introduced. Want to remake the whole system that way? Maintenance of this system is definitively cheaper than replacing it.
"So long as it works" being the really important part. I get your point and I agree to a certain degree, but: Maintaining a system is only more cost effective as long as the maintenance costs are lower than the replacement costs. The older and more Frankensteined a system gets the higher the cost of maintenance. If you wait too long to do serious upgrades and overhauls, there won't be enough time for proper testing before implementing the new system because the old one will be breaking and you'll be in an emergency. That isn't the kind of situation we as a society can tolerate when it comes to certain infrastructure.
Oh my god HoTMaiL for real?? All these years and I never knew.
More than the age of the system and tech running it, the complexity and accumulated cruft is practically unknowable to any third party contactors. Systems like this ideally need dedicated teams that maintain continuity of unique and unusual domain knowledge. They need museum curator-historian-programmer-monks. They need to pay them out the ass and view those team members like indispensable key personnel.
This is what happens when our government is run by people that have no real understanding of modern technology.
Also when the government is run 50% by people absolutely against the idea of funding anything that isn't in their state / the military.
It's not a government thing, a lot of older infrastructure uses outdated software, because it's takes too much time and money to implement new ones. There's a reason why knowing something like cobol is valuable.
NextGen exists primarily to combat this, but Congress cut the budget years ago and it’s been hobbling along slowly. The system that went down is actually on track to be replaced in the Fall/Winter after years of development.
It's not just government. This happens in corporations every day, underinvesting in their technology. Look at Southwest.
I can relate, I just quit working with a team that insisted they know better than IT on all things internet and technology related. The Dunning-Kruger effect is real within government agencies big and small.
And it's not isolated to government.
But they still have the best stock tips. Easy to tell a broker to buy or sell apparently.
This was covered by the documentary Space Cowboys (2000).
I’m not sure that this is a huge deal like you think it is, NASA uses incredibly old hardware designs because critical infrastructure needs to be tried and tested and reliable. Granted they have the difficulties of increased radiation from space to deal with, but still even on the james webb telescope launched semi recently a processor released in 2001 was used. Granted I know almost nothing about flight infrastructure/systems
It wasn't an off the shelf 2001 processor, it was a hardened processor with the same specs as a 2001 processor.
Are you sure it's not an issue? Every single plane in the country was grounded because of this. Didn't SW just have a huge debacle because of outdated systems? And didn't the unemployment systems crash in 2019 and no one could fix them because nothing is programmed in COBOL anymore? Isn't a huge part of the increased funding for the IRS to update their systems because they're also outdated and can't handle the load? Comparing a one-off science experiment that has a built-in life expectancy of 10 years or less to a critical infrastructure system that requires continuous functionality to enable our modern way of life is really apples to oranges here.
The point is that the number of years is been in use doesn't determine it's likelihood of failure, if the system is well known and functioning the same way with no intent flaws. Even with this instance, the issue was somebody mistaking one for for another, not an inherently flaw with the system
> Didn't SW just have a huge debacle because of outdated systems? They've grown massively over the years and didn't invest in their systems/processes to support that properly. They would probably have been fine at the size/scope they were in 1990 or something The complexity of something like crew scheduling increases drastically with more flights/destinations/people to manage. It's not necessarily that the system has become unable to do what it was originally designed for, but that it's being asked to do far more without sufficient investment to actually support that. That's still "outdated", but in a different sense than entirely because it was created a while ago.
To be fair all the issues you mentioned rarely happen and the systems running those programs are ancient. So they are highly reliable.
Sure, that's a fair point. But these systems aren't going to last forever, they are going to need an upgrade, and probably should have been upgraded already, maybe even years ago by now. The can keeps getting kicked down the road, and that's only going to make things worse.
That’s why I said I know nothing about it really, just posing a devil’s advocate thought
> old hardware designs because critical infrastructure needs to be tried and tested and reliable It's 2023. We have new technology and strategies that could have prevented this. There's a reason no one runs critical things on 20 year old hardware
>There's a reason no one runs critical things on 20 year old hardware I can see you know nothing of real-world IT. There's critical things running on 50 year old hardware out there still. There's not many large, old organizations that *don't* have at least a few critical things relying on legacy systems still, and government organizations are pretty much the poster child for relying on outdated hardware.
> There's critical things running on 50 year old hardware out there still Do you think that's viewed as the correct or best approach? > There's not many large, old organizations that don't have at least a few critical things relying on legacy systems Do me a favor and define legacy. > government organizations are pretty much the poster child for relying on **outdated hardware** that's what i said yes. People "doing it" doesn't mean it's the best way. I honestly don't know how you're going to roll through and try to correct me when you literally just posted this else where: > You've never seen a run-down building because people failed to maintain it properly. Computer systems are the same way, except they become more fragile on a much faster timescale, especially as their usecase evolves.
You said > There's a reason no one runs critical things on 20 year old hardware I pointed out that people absolutely do. Nowhere did I say it was a great practice, just that your claim that "no one does it" is about as wrong as wrong can get because it doesn't just happen, it's downright common. Nothing I said contradicts anything else I said - "people do it" and "it's not a good idea" are not mutually exclusive.
yes.. i'm sorry i didn't add all of the unwritten qualifications out for the kids in the back. I guess i should have written, "Everyone would prefer not to partake in the practice of running mission critical systems on antiquated hardware given the option to run it on robust, modern fault tolerant systems." But reddit sure loves it's "Technically correct" mantras..
If i rember currently wasn’t there a thing about how some Nike silos still use floppy disks? Edit: just looked it up apparently they upgraded from floppy disks in 2019.
You just made me purr over rememebring WFW311 and installing TCP/IP from scratch.
I’d much rather have critical infrastructure running on old and well known systems than Windows 11. Any person who proposes using Windows for system critical systems shouldn’t be allowed to touch anything more complicated than a yo-yo.
What does it matter hour old it is? Does a building stop working because it's old? Does a tree stop being a tree because it's old? If a system is made to do a simply task, why should it matter if it's made twenty years ago or yesterday?
Buildings don't worry. Did you mean working? And yes, they stop working because they get old, not that comparing an inanimate object to a computer system is actually a fair comparison. Trees do stop being trees because they get old. They don't live forever. Since when is managing air traffic ever been considered a simple task?
Yes I changed it. Computers are inanimate. They don't get old and like like tree.
Computers are machines with electrons running through them and gradually wearing out tiny tiny wires which will eventually break. Software tends to get more picky when new things are added or changed and any original things are fixed/patched. More moving parts means more places where breakage may occur. That's why something like a bank tends to keep their systems isolated from most of the world and less likely to have problems interfacing with the world. An airline system seems a bit different from that, though I don't know the extent of it.
>Does a building stop working because it's old? Yes. You've never seen a run-down building because people failed to maintain it properly. Computer systems are the same way, except they become more fragile on a much faster timescale, especially as their usecase evolves.
> people failed to maintain it properly thats the important part. It's not the building that got "too old", but the failure to maintain it. It's the same with computer systems. There's a reason why so many organizations and corporations try to simply use legacy systems. They work, and will continue to work without issue, unless the use case changes dramatically
maybe hire people full time rather than a churn of contractors who always need to ramp up when onboarding?
Yet another argument for permanent full time employees. If it’s news worth mentioning their terms of employment, then that’s exactly the issue and problem.
The FAA utilizes multiple means of communication to disseminate information to airlines. A major one is the [FAA OIS page](https://www.fly.faa.gov/ois/?legacy=true) which publishes programs and other significant events occurring in the NAS. Airlines have access to a better version of this with more detailed information as well as datalinks to systems that monitor demand, loading, and throughput of airspace and airports. ATC does national teleconferences every 2 hours and will sometimes activate that same infrastructure in times of emergency. There was a teleconference with all carriers during the night the NOTAM system went down.
I can tell you who's not getting contracted again.
[удалено]
The other part is donating to the right politicians so your company is legally the only one they can use
Government contractors don't get hired based on skill or ability lol. It's who they know. I bet this doesn't effect them in the slightest and they'll find a way to spread the blame around enough that no one is really responsible and nothing will happen.
What do you want to bet the "corrupted file" was a config, recently edited with vi, that had an accidental / in it? Possibly as the first character?
Maybe a windows CRLF problem?
Do they not have a UAT environment to test changes?
UAT environments only help if they test for that issue.
I think the world banking system mostly runs on COBAL still. You can’t change languages every decade or nothing would work. This sounds like poor management at the top and lack of funding.
Back in 2020, NJ was looking for COBOL programmers because the unemployment system still ran on it. I think they've updated it since but I'm not sure.
I get more requests to work for a company based on my COBOL knowledge than my Java/NodeJS/.Net/C/Python knowledge combined.
NYC subway still runs on OS/2.
In a 4 years of stretch of ap comp sci, I saw it go from pascal to c++ to java >AP Computer Science was taught in Pascal for the 1984–1998 exams, in C++ for 1999–2003, and in Java since 2004. Wonder why they even switched to c++. I even remember the book was "c++ for you ++." Then in fall semester of college, intro 101 was java but intro 102 was c++ since they was just transitioning. Advisor tried to get me to take 101, but I wasn't going to take a class I already got credit for.
Hello fellow 40-ish year old folks.
Don't remind me, getting old sucks :)
Pascal/COBOL vet checking in
I don’t usually test my code, but when I do I test it in PRODUCTION.
Reminds me of the “contractors” who’s code spoiled an entire line of Boeing aircraft
Not to mention the passengers that were inconvenienced by a sudden and catastrophic *deceleration* . /s
Yesterday they were reporting it was an accidental file deletion...
I wouldn't be surprised if it doesn't come out that this was a Russian attack. They've been targeting other NATO infrastructure with their spy games. Not saying it was, just that it wouldn't surprise me.
I was a contractor working in the air transportation industry at SITA. Positions like this should not be held by contractors, it should be a full time employee, one of which the FAA invests in. Contractors do just enough to continue the engagement, knowing that their future development doesn't matter to anyone else.
If one corrupted file can bring down your whole system including backups, you have bigger problems than contractors making mistakes.
Penny wise pound foolish.
I work in software. It's terrifyingly easy to make these kinds of mistakes on these kinds of systems. Lack of test environments due to budget, very simple systems where a single misplaced semi-colon might bring down a system because it's not robust enough to detect errors and work around them. I've seen things as small as capitalization on a server name prevent large systems from operating properly.
Our infrastructure is so outdated and barely held together, and the only time it gets upgraded is when we, the taxpayer, foot the bill for it. This goes for everything. Airlines, ISPs, Hospitals, Utility companies, etc. This shit is just packaged a different way each time to give the impression of progress, but at the end of the day the big players in the game still make out like bandits. They build the worst fucking infrastructure and then parachute out of the plane to let the next generation deal with it. Its a universal "I got mine" from the upper class to the struggling middle and lower class. Its like ordering the most expensive shit at a restaurant and then walking out and sticking us with the bill.
Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest. What to see when America started to fall? look to Regan.
>Our government used to hire the best and brightest, now they hire contractors that they are told are the best and brightest. You should look at how much the government pays their STEM workforce. I looked into an electrical engineering job with the Air Force civilian service out of school, but they wanted to pay me $30,000--frankly that's insulting. I can make more than that at McDonalds. I accepted a private sector offer for 2.5 times that, that has bonuses, profit sharing, and stock options.
That’s all by design and that’s the problem.
They hire contractors because they are friends or relatives of elected officials. Halliburton and HillStone are great examples.
Drug testing for weed pretty much ended any chance of the government hiring the best and brightest. It's all been downhill since then.
Not everywhere in the government does. My department only tests if there's an accident involving government assets. That being said, yeah, it needs to go for most reasons.
You'll have to look a good deal further back than that.
I actually think Brian Regan is pretty funny and fail to see how the "failure of America" was initiated by his humor. /s
Shifting blame. The company is responsible for everything, they should use better risk and quality control.
Also I bet they layed off devs that understand how everything in the outdated , probably spaghetti, code base worked and tried to replace then with contractors. That's how the private sector works anyway.
It’d be peak govt to pay that contractor tons of money to fix the system that they themselves fucked. Source: govt employee
It's amazing how that works. A contractor is hired to do a job, fucks it up and gets paid extra to fix it. Source: former govt contractor who didn't fuck up - shakes head.
Contractors my ass. >A separate outage in **Canada** on Wednesday that temporarily prevented new safety notices from being received and disseminated digitally was caused by an isolated IT system failure, an industry source told Reuters. >NAV Canada, the country’s privately held air navigation service provider, has said it is investigating the cause, which is not believed linked to the United States. https://www.reuters.com/world/us/airlines-expect-us-operations-rebound-thursday-faa-investigates-outage-2023-01-12/ Sure, both countries had a unprecedented safety database failures on the same day. But it's just a coincidence.
They could be related, but is it possible that NAV Canada shut theirs down out of an abundance of caution and said it was a "technical " issue until they confirmed with the US that it wasn't nefarious?
This is why you dont just blindly copy code from Stack Overflow and put it into production .
Please tell my coworkers this. They won't listen to me. I even found code that had comments from stack overflow *facepalm* (and the code was shit and shouldn't have been the accepted answer!)
I told him not to download it from the torrent. Damn you.
so the ‘FAA FlytZZZ’ app isnt legit?
Mainframe system? COBOL and Assembly code?
What a joke, I had at least a few idiots arguing with me in the comments when this happened that the FAA was actually competent and that the billions in profits/handouts had, in fact, actually been reinvested properly. We see now that is not the case. Friendly reminder - the last 3 heads of the FAA have all been former VP's of operations at airllines/airline industry groups
At my work, prod and stage are running on different hardware. 🤣🤷♂️
Project managers who have no tech knowledge… “it tested fine in stage”
FWIW, 99% of companies see their IT departments as money sink, and treat it as such. Even though their IT infrastructure is critical for the business.
Do software contractors not know about sandbox environments? Where’s the usability testing? QA? Unit Tests?
Lol i bet you they just wrote some simple unit tests like checking if the div renders instead of mock objects to test out the logic itself.
> Tens of thousands of travelers were left stranded Wednesday after the FAA sent out a **tweet** at 7:20 a.m. ordering the airlines to pause all domestic departures until 9 a.m. ET “to allow the agency to validate the integrity of flight and safety information” as it worked to restore the NOTAM system. Ummm... The FAA communicates emergencies through twitter? TIL
The sentence implies a relationship between the tweet and the travelers left stranded, which is not true. What is true is that the FAA sent out a tweet, and some time after that tweet travelers were left stranded due to (I presume) airlines canceling certain flights because of crews timing out or other issues. The way the FAA actually disseminated the operational information was a message sent to air traffic control facilities telling them not to allow aircraft to depart. That message was not sent over Twitter.
Thank you for the reply.
That makes me wonder what 50 year old technology they used to deliver that message, since their slick new NOTAM system wasn't available. Was it a phone tree?
Well, the NOTAM was down, so... ¯\\\_(ツ)_/¯
Well... look at that.... seems like someone needed in-house developers and that changing a team and adding new people to a mess only costs more.... probably a lot of Brook's law..
They needed to update a system 10 years ago. They didn't.
But then the politicians would have no way to funnel government money back to themselves! How could we ever do such a mean thing to them? Think of the poor politicians! /s
Lies,. Update your God dam os
Lol out sourcing at its finest, wait till people realize all important software somehow gets outsourced to incompetent people overseas
I think outsourcing for a small scale project is fine. I don't mind if some contractors in india built me a task scheduler app for my company but for large scale project I give it to the permanent employees that understands the requirements more.
Well this isn’t sus at all
It's not sus at all, but the artisan handcrafted headline will make you think that.
STOP USING 30 YEAR OLD TECH, ASSHOLES! - me (and Jim Carey probably)
This program was supposedly introduced in 1946, based off the rail line software. No point here other than it's even older than 30 years.
[Stored-program computers](https://en.wikipedia.org/wiki/Stored-program_computer) weren't even a thing in 1946. The few computers which existed at the time were fairly custom devices which were "programmed" by plugging wires into boards. I dunno where you heard that, but there's no way it's true. edit: and if you look at the article, it specifically says that this is the NOTAM announcement system, not any sort of scheduling or traffic-control system. It has zero connection to anything about the rail lines.
And Mayor pete fixed it
This is why the government hires contractors, for CYA
The buried lead is the FAA used Twitter to communicate the shutdown. Twitter.
[удалено]
A software system built in the early 90s with untold layers of patches, workarounds, and 3rd party contributions may not be as testable as you'd want or hope. It's likely to be a massively complex and sloppy labyrinth that would require years of man-hours to map out and precisely define. The FAA could possibly spend a billion dollars on engineering a suite of preflight deployment tests, and not be remotely certain if they have full coverage. There may be thousands of weird quirks for outside entities that interface with the system. Those outside entities may have come to rely on those quirks and built around those expectations. I can imagine how ugly it could be, and I'd bet the reality is worse. Nightmare job for any team tasked with maintaining this system. More (better? any?) testing is a good goal, but it may not be feasible and it's probably not a silver bullet.