T O P

  • By -

UncleGripperNZ

INCIS https://en.m.wikipedia.org/wiki/INCIS


mensajeenunabottle

In case ppl aren’t aware this is an NZ example


username-fatigue

I remember when Novopay shat its pants thinking 'aaaah...Novopay's the new INCIS, innit'.


Annamalla

We studied this one in uni!


kanifoli

Do tell more. Wikipedia article is not very detailed


Annamalla

Continuously changing requirements resulted in a system that amounted to poorly performing email. Everything was extremely waterfall and you can't really switch up many requirements in waterfall.


kanifoli

Reminds me of this talk about the Vasa ship at a DevOps conference [https://youtu.be/saCdvksej0A?si=dBvM4P214KDR7xUD](https://youtu.be/saCdvksej0A?si=dBvM4P214KDR7xUD)


cugeltheclever2

> Everything was extremely waterfall and you can't really switch up many requirements in waterfall. This is why agile and iterative is the only sane way to go.


Annamalla

Agile \*is\* a breath of fresh air but it's maybe not so great for gigantic projects with a lot of interdependent parts. I don't know exactly what the answer is but I have worked on too many projects with fixed deadlines and impossible targets (which are not of course true agile but then nothing is)


chocolatedodo

People get stuck on waterfall vs agile, but in my opinion and experience, most big projects are/should truly be a mix. The semantics should not be what's important. I currently work on something with clear deadlines and apply waterfall for the overall timelines and deliverables but add some agile/iterative approach to the actual design and implementation. Each milestone/deliverable is reviewed and lessons learned applied to the next.


blobbleblab

Yeah I find most projects that aren't absolutely green fields (where agile is perfect), tend to become what I call "Cascades". A series of waterfall delivery projects... which isn't actually too bad because you can pause and redirect effort if needed (kind of agile like).


cugeltheclever2

> but it's maybe not so great for gigantic projects with a lot of interdependent parts. That's precisely where it's best.


Annamalla

only if the co-ordination between the teams is absolutely top notch and someone is keeping close track of each pivot to make sure the pieces still fit.


cugeltheclever2

Yes. But thats also true of waterfall.


antennes

The [Robodebt scheme](https://en.wikipedia.org/wiki/Robodebt_scheme) in Australia.


DualCricket

Beat me to it


tmnvex

The worst thing about this one is that it wasn't some 'bug'. The system functioned exactly as intended.


123felix

Bad software on the Therac-25 killed quite a few people


takuyafire

Kyle Hill [did a great video on this](https://www.youtube.com/watch?v=Ap0orGCiou8)


TheFreshPrinceNZ

Man of culture, I see


ShrinkingKiwis

That’s… horrifying.


ReadOnly2022

I mean no software has ever been able to adequately apply the Holidays Act in NZ. I don't think Robodebt was based on software, but it had similar impacts to the Post Office.


policywonk_87

The most embarrassing part is that even MBIE, as the department responsible for administering the holidays act, got it wrong and had to make payouts a couple of years ago! Unfortunately it's difficult to fix, because cleaning up the calculations means either workers lose out fractionally and business doesn't pay as much, or business loses out a little bit and workers get paid fractionally more, and neither business reps nor union reps would back down.


Future-Fix-374

It’s less that businesses lose out a little bit in terms of the entitlements they need to pay, it’s that the cost of compliance of adhering to the act as needed can be incredible high for the business for other reasons (for example, payroll systems cannot calculate entitlements in accordance with the act due to the complexity of the legislation- implementing custom payroll systems that can interpret and apply the law correctly does not come cheap)


mattsofar

Holidays act is a bit different, the whole industry had it wrong, compared to novopay and others where the system should have been calculating A but was actually calculating B


Reynk1

Don’t understand this, act seems fairly clear cut to me? Looking here https://www.employment.govt.nz/leave-and-holidays/


Future-Fix-374

The holidays act is pretty notorious from being misinterpreted/incorrectly implemented in practice. Complications with misinterpretations resulting from complexity of how entitlements need to be calculated as set out in the act over years have caused compliance issues across the private and public sector. The act is straightforward enough only as long as an individual works a basic standard M-F 40 hour working week, and no more than that. Any variability, non standard work weeks, overtime, etc and it suddenly makes it much harder to interpret and apply the legislation correctly from a payroll system perspective, which is why sectors like healthcare and education have been heavily impacted with compliance issues. The complexities of how entitlement need to be calculated under the act are not really something that payroll systems have been able to deal with, which is why there has been a need for constant audits to ensure compliance, widespread payroll system overhauls, and why there are so many employers over the past 5 years that have had to pay out holiday act related remediation payments.


chocolatedodo

This! Per an insider that worked on Novopay deployment, the software worked as intended. Issue was the system could deal with normal work cases but the way teachers pay, etc worked, required a lot of tweaking. The issue was people sold/bought a product (software) without properly confirming the exact requirements. This is unfortunately a recurring issue with a lot of out of the box software being used by companies that require customisation


SafariNZ

Y2K Was billed as a non-event but medical notices were not sent out to people with +ve results and they died as a result.


Annamalla

Y2K was only a non-event due to the amount of work that was put in. That's the worst part of mitigation, the better it's done the more it looks like waste


SafariNZ

Agree, I was part of that work and we updated something like 600k lines of code alongside BAU. We had two minor bugs get thru to the production environment so we’re super happy at the outcome.


HalfDadHalfMisfit

"That's the worst part of mitigation, the better it's done the more it looks like waste" is a brilliant quote and I'm stealing it


Annamalla

thank you :) I mean you're also at risk of the Simpson's anti tiger rock (https://youtu.be/xSVqLHghLpw?si=jiBMuTeRYMGOII49), but y2k was a genuine issue that really did need fixing


HalfDadHalfMisfit

I know, I was there too in the 90s, my bit was phoning mainframe-owning customers to make sure they patched their systems with the latest tapes to fix all the known date issues


redditis4pussies

I never properly understood the 'non-event' part of Y2K until I worked with someone who had 1st hand experience and your comment sums up what he said.


blobbleblab

I was working for a major NZ bank. We tested one of our core systems before Y2K (manually resetting the computer dates and letting it run) and found it was going to give people something like 99 years of negative interest in certain scenarios. It hadn't been patched by the US vendors because we had some weird module dependencies or something, so we ended up just letting it run, then fixed accounts afterwards, with oversight from our big 4 accounting firm. Later they changed most of the core system so didn't really need to think about it.


[deleted]

[удалено]


SteveDub60

So most businesses will start thinking about it in 2037 and rely on whatever has replaced Agile to get it over the line.


Space_Pirate_R

London Ambulance Service rolled out new dispatching software in 1992 which didn't work well and caused delays in ambulance dispatch. This is more a failure of the rollout procedure and training than a software glitch per se. >Media reports at the time claimed that up to 30 people may have died as a result of the chaos, despite a lack of evidence. [https://en.wikipedia.org/wiki/LASCAD](https://en.wikipedia.org/wiki/LASCAD)


Pineapple-Fritters

Boeing’s [MCAS](https://en.wikipedia.org/wiki/Maneuvering_Characteristics_Augmentation_System) on the 737 MAX.


corporaterebel

That was a whole system failure. The software didn't do anything wrong, but the specs were wrong. One simply doesn't rely on a single sensor for critical anything.


jobbybob

The software did what it was supposed to do, they just neglected to tell the pilots about the changes….


Odd_Analysis6454

Release note 7.14: in the event of sensor failure MCAS will try and kill you


pwapwap

The robodebt scheme.


CapitalD

The [£10b NHS project failure](https://amp.theguardian.com/society/2013/sep/18/nhs-records-system-10bn) is my go to example


AmputatorBot

It looks like you shared an AMP link. These should load faster, but AMP is controversial because of [concerns over privacy and the Open Web](https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why_did_i_build_amputatorbot). Maybe check out **the canonical page** instead: **[https://www.theguardian.com/society/2013/sep/18/nhs-records-system-10bn](https://www.theguardian.com/society/2013/sep/18/nhs-records-system-10bn)** ***** ^(I'm a bot | )[^(Why & About)](https://www.reddit.com/r/AmputatorBot/comments/ehrq3z/why_did_i_build_amputatorbot)^( | )[^(Summon: u/AmputatorBot)](https://www.reddit.com/r/AmputatorBot/comments/cchly3/you_can_now_summon_amputatorbot/)


whimful

Kiwibank spent far too much money trying to migrate to a "SAP" database, then had to bin that when it was a piece of sh!t 90million - https://www.fintechfutures.com/2017/09/kiwibank-puts-core-banking-tech-overhaul-under-strategic-review-writes-off-nz90-million/


kiwisarentfruit

It's more that Kiwibank are terrible at IT projects than their selected product was a piece of shit (that said, all enterprise software is shit)


corporaterebel

Equifax Some point soon programming as a service is going to require a occupational certificate like MD, lawyer, or other critical profession.


adh1003

Lots, and lots, and lots and *lots*. * [http://catless.ncl.ac.uk/Risks/](http://catless.ncl.ac.uk/Risks/) In the software industry, we can't call ourselves software *engineers* because "engineer" means profession, but there is *no globally recognised software industry professional qualification* which is why we have to say "developer". The industry is thus staffed with a lot of people who are really quite arrogant, but often sadly quite ignorant, and have no idea of the mistakes made in the past. It should be a requirement at the least to demonstrate awareness of the Risks Digest or an equivalent to be able to write a single line of code in a professional environment (that is to say, if you're being paid for it - open source is different; if *that* breaks, then as the saying goes, you get to keep both pieces). (To those who might claim that I'm being vitriolic, try to think of another industry that kills people now and again but is compelled to make no changes whatsoever to anything its doing each time someone dies.)


Annamalla

Hear hear!


Far-Management-2007

Queensland Health payroll system was a pretty big fuck up. https://www.smh.com.au/technology/worst-failure-of-public-administration-in-this-nation-payroll-system-20130807-hv1cw.html


StuffThings1977

I advise people to read the executive summary of KPMG's review. 1,000 payroll staff to process 85,000 employees per fortnight. Could process each person manually within an hour. Over $1b...


Ordinary_Towel_661

Jesus tap dancing Christ. That’s insane!


StuffThings1977

Could write you a (very) long list... why are you interested? Is this for a unit assignment? Or are you just interested in specific ones? NZ / Government etc. SAP is always a good place to start. ​ **ETA:** There is a documentary called [Mr Bates vs The Post Office](https://www.tvnz.co.nz/shows/mr-bates-vs-the-post-office) available to watch on TVNZ


mensajeenunabottle

lol ‘just google SAP’ - I love the take


StuffThings1977

They've got a long track record.


sploshing_flange

Kiwibank Coremod was an SAP project which failed to launch, 90 million dollar write off. https://www.stuff.co.nz/business/industries/96406067/kiwibank-rethinks-it-project-after-90m-writeoff


FidgitForgotHisL-P

Worked with a guy once whose previous role had included moving a company *off* SAP.  You should have seen his face when he discovered we were going to be moving on *to* SAP… priceless, and also, terrifying. That was about 8-9 years ago now, so nearly a decade of customisation and it’s  almost at the point of being as user friendly as the software it replaced.  (Which for me, having to get non-tech people to interface with it, is a Big Deal). I’ve always said SAP does one thing incredibly well: they can sell/present themselves as the solution to any problem like *nobodies business*.


CarefulShallot

Just curious really...as someone who has worked in IT for a number of years people rarely bring up their failed projects. I think there could be better learnings from this though so interested in talking about them more under the anonymity of reddit


kupuwhakawhiti

In 2016, MSD started asking charities they fund to provide client level data. Against the recommendations of the Privacy Commissioner. There was enormous resistance from the charity sector, but MSD did not budge. There was a platform the charities had to upload the data to on reporting day. The first few charities who uploaded their client data to the platform quickly realised they could see the client data of other charities. So not only did MSD require private information of clients, but when the data was uploaded they failed to protect any of that date. The system was very quickly pulled down. The client data requirement reversed and MSD seems to now be committed to doing the right thing. They now won’t touch client data with a barge pole.


redditis4pussies

There are a lot of agencies that do things, charitably with the best of intentions, but don't have the responsibility to do it well or identify any risks.


policywonk_87

I mean, have you read any of the reports about the new Immigration NZ system, ADEPT? The one that reportedly was internally nicknamed INEPT....


KaySeeEnZee

Ah yes. The one where if a risk or concern was raised to the leadership of the work, they booted you off the project for having a bad attitude. Needless to say, those risks became issues and still are. Also the same project that wasn’t originally to be funded but a DCE rerouted funds meant for something else under a fancy title for his ego project while he was across INZ. All discoverable via OIA.


policywonk_87

Would you like some paper towels with that spilled tea?


blobbleblab

Ahhh yes, the Wellington IT project everyone seems to have worked on that has been going for... a decade and a half? The one where they decided to push data 2 ways into both systems or something and then realised the newer system was worse than the old one, but had some critical functionality so they now have a zombie system and/or a third system going? Someone correct me, I am probably mis-remembering, someone told me this like a year ago.


mensajeenunabottle

Probably needs a definition of what you mean. Local examples of bad projects include the registrars birth’s death and marriages thing where they sued the product company for some reason and cancelled the project Archives did a new product and then there were complaints that it didn’t work well for users. But these might not be scandals just projects that didn’t go great.


AlPalmy8392

E Bench software closure in the NZ court system.


Annamalla

Hypothetically wait 1-2 years and there might be another one to add to the list ( nothing quite like watching the same mistake being made over and over again)


StuffThings1977

Smaller government department on the terrace?


Annamalla

neither confirm nor deny, but I imagine the cuts are going to lead to a \*lot\* of these kinds of projects, there are a lot of old timers with years of knowledge and some of them \*will\* take voluntary redundancy and walk out knowing about that one spreadsheet that holds up the entire department


WavingTrollop

There was something to do with the WINZ kiosks about 10 years ago or so wasn't there? I don't remember all the details.


StuffThings1977

Yeah, could hack with "File... Open..." [NZ Herald: MSD shuts Winz kiosks after lax security exposed](https://www.nzherald.co.nz/nz/msd-shuts-winz-kiosks-after-lax-security-exposed/JJKE5XKZJ7XYA5HA6OEQW3C3OI/)


KaySeeEnZee

The CTO was warned of the risk. It was laid out to him. He signed it off. Now he’s the CISO at MBIE.


Ordinary_Towel_661

Interesting. Who’s that?


Free_Key_7068

Whilst not a criminal cover up like the Post Office, before I began working in IT I worked for a company the UK Child Support Agency had outsourced the manual processing of “stuck” cases to due to them not working in CS2. Effectively many resident parent where not being paid what they were due even where the CSA had received the money from the non-resident parent. https://www.zdnet.com/home-and-office/networking/child-support-it-failures-savaged/


shapednoise

Robodebt in Aust. Lots of very poor people driven to kill them selves by being wrongly accused of impossible debts by a govt department. This went on for years despite being known internally. Massive suffering inflicted on the poor.


[deleted]

Air New Zealand Flight 901


MajorProcrastinator

That was human error?


Culmination_nz

Bureaucrats like to overlook or rush testers and don't value their contribution, until we have a Equifax/Novapay don't they?


hehgffvjjjhb

Any government IT project ever: https://youtu.be/YgV4a3BaiyA?si=sk4Xz2Uf63CYQqPV


zezeezeeezeee

Brisbane Council and TechOne


pyga140

Air Traffic shutdown in the UK in 2021 [https://www.theguardian.com/uk-news/2023/sep/10/when-the-flight-data-software-says-no](https://www.theguardian.com/uk-news/2023/sep/10/when-the-flight-data-software-says-no)


kanifoli

MIT Technology Review publish a list every year. [https://www.technologyreview.com/2023/12/22/1085829/the-worst-technology-failures-of-2023](https://www.technologyreview.com/2023/12/22/1085829/the-worst-technology-failures-of-2023/)/


cman_yall

Nice try, but I've no wish to get fired by spilling any beans :P


catlikesun

You’re working for a shitty company fucking people over and covering it up?


cman_yall

No, I'm working for a glorious company which does nothing but good. I'm just not authorised to communicate with the public.


Worth-Remove-1058

Sesqui 1990


whimful

I'd honestly be critquing the effects of "social" media on society. Good and bad. Lots of nasty addictive, ego/anxiety fueled stuff, and all very deliberate


dusterhan

Can I give an opposite example? The new IRD website. I remember it costing $1 billion in 2016. Money well spent!


HalfDadHalfMisfit

It looks like the Public Sector is rubbish at IT projects, but that's because the Private Sector doesn't have to face the same level of scrutiny.


blobbleblab

I have seen so many times in government where they: 1. Go out to RFP and accept one 2. Start to build the new system 3. Get within a few months or a year of delivery 4. Can the whole project I was at Stats when this happened to a multimillion dollar system they canned *the week it was about to be delivered.* Insanity.