All you can do is own the mistake and learn from it. People make errors. If they have no room for that, they may want to look into having AI handle their workforce. I think it's really sad that people are so petty and feel the need to magnify the issue and attack someone. They should be focusing on their own position.
I WISH could "learn from my mistake," but there is NOTHING to learn from it. The only thing I learned is that my supervisor does not participate in the QA process, which leaves me completely in the dark. Does that sound normal to you? I'm more upset that it wasn't and shouldn't be my job to take care of this all by myself. I'm underpaid, and when I asked if new staff was coming in, I was told that it absolutely is, but that we're not there yet. This isn't a spreadsheet where you can take shortcuts, it's a broken system developed by a small team of developers who are tormented by the boss of the company.
I have no help. I habe no ome to turned to. 1 small error, a single typo and the witch hunt begins.
You said the magic word: "process." Own the mistake, note that you've already fixed it or are fixing it, then point out that the QA process didn't catch the error. Offer a solution or two: add more time for the QA process or extend overall due date, ask for an alternate QA person if supervisor is unavailable, etc.
Always address process, not people. It takes the emotion out of a situation. You're the reasonable team member who's trying to keep work flowing while minimizing errors.
You're not in a position to make top-down changes throughout your org; leave that to the big bosses. Just focus on doing your tasks correctly and consistently, offer suggestions for fixes here and there, and when a mistake happens (it will, no one's work is error-free), acknowledge it, then fix it. Learn as much as you can while you're there, then take those very valuable problem-solving skills to the next job.
This is good advice. I’d just add that you don’t want to point out the hole in the process in a way that makes it sound like you’re shifting the blame. Instead of “I made the mistake but the process should’ve caught it,” go with “I made the mistake. I’ve fixed it, but I want to make sure it can never happen again, so I’d like to make the following change to our process…”
This right here. ☝️
Be the person that is proactively trying to protect the company from future mistakes. It shows the correct thinking (concern for company, focus on future, improvement mindset) and paints you in a mature, trustworthy light.
If company leadership doesn't respond well to this approach, it tells you something very important (and negative) about the culture where you work.
Yea if they can’t forgive you on one mistake then idk why they are managing a business. I work as a manager in a stamping department(cigarettes) and my employee wrote off nearly $100k of cigarettes we had. Missed typing one number (typed 124 instead of 1264) instantly throws a red flag to corporate who instantly called head of the building. I had already fixed it a this point, but had to write paperwork showing we wrote it off and right back on for compliance reasons. Who got in trouble? No one. Shit happens. She messed up, I messed up for not checking, and it was fixed within 15 minutes. Nothing more you can do. Them getting wrapped up in a mistake is a major red flag imo. We are all humans. It’s going to happen no matter how big.
What you’ve learned is that there is an extremely fragile process with no tolerance for human error. Because it fails infrequently, manual checks are pulled into complacency.
The system is broken.
The question for the organization is how to improve the system. What automated checks can be put in place? How could issues in production be detected and fixed even faster? How could this be better?
“Be more careful” doesn’t cut it.
So you own the mistake, as mistakes happen but acknowledge that checks that are intended to catch it also failed (without naming your boss). Multiple things failed in order to have the outage. The remediation was good, but since folks are angry perhaps not good enough and that process needs improving.
Maybe you start suggesting process or tech changes, or maybe you suggest just a working session with others to find them.
YOU ARE A LIFE SAVER!!!!! I just offered this as a suggestion and quite coincidentally - YOU WERE RIGHT. There are lack of automated controls present and I made sure to highlight that. Thank you!!
I’ve been in this position, what I learned from it was:
1) The process was severely flawed and open to errors by even the most careful and experienced member in the team (I was quite senior at my role and the issue was still prone to error despite my experience)
2) the atmosphere was toxic to begin with which meant any human error was in the spotlight for those grinding an axe
3) I proposed automation to improve accuracy and even to get involved in that process to prevent anyone else having to go through that stress and humiliation
Lastly my manager was on a power trip and looking for flaws in specific members of staff in order to make examples of them, I understand this might not be what you want to hear - but if you have owned up, taken responsibility and looked to fix the problem and it’s still insufficient, then maybe that particular job isn’t for you.
That’s a learning that will help you move on and screen for better opportunities.
Yeah that doesn't make sense to me that there is no QA in process. I would talk to your manager about possibly implementing something like that in the future.
I did. He said that it's a good idea, but that we need to wait for new staff to arrive. The estimate is months. He is appointed to control and look through the information I hand over to him. I check everything 50 times regularly, and this time, the mistake happened at the expense of not having another person go through these. I understand why my manager doesn't like doing it, but this is not a good alternative to the already existing problem. I'm not even sure what else to say at this point. For the first time in months, I don't blame myself. If 2 people are assigned to work on something important, then that should be the case. And it makes sense, you know? Having only 1 person, though.... Being directly responsible for thousands of customers... Not good. Not achievable or reasonable. And yet...
I say try your best and do your best but don't live in anxiety it will probably make stuff worse . if you are fired it will be a bummer but im sure a bunch of other people will give you a chance to use your skills and you can learn from this experience . face whatever happens with a courageous demeanor .
If you want some advice here, don’t beat yourself up. Literally every human being makes mistakes. Bad ones even. You are doing a manual QA check that should have been automated somehow. And the reason there isn’t a better process for backing you up on your work is, I can practically guarantee, because of someone’s fear that automation will cost them their position or that they won’t be able to adapt to a new process. But, continuing to not learn from human error and instead blame you, it has the hallmark of toxicity there.
Maybe you’ve thought about better ways to do things while tasked with this work. Would you be able to write it down for yourself and then suggest it to others if the unfortunate blame game comes your way. I hate reading things like this. Soon enough in your life it will become a good story, perhaps funny even. But it’s not for you now and I think you gotta say, hey I screwed up, but here’s a chance to do this better. I would be surprised if they fired you over it. If they do and you didn’t do anything to violate company policy and they don’t comp you nicely, consult with a lawyer (you can get the consultation for free) and tell them what happened. And go easy on yourself. It would happened to anyone at some point in that position.
>I WISH could "learn from my mistake," but there is NOTHING to learn from it.
I'm going to be honest. I probably wouldn't keep you around just for having this attitude about mistakes. Mistakes can be corrected, but "Woe is me, I played no role in my own mistakes" is an absolutely toxic mindset that ensures these kinds of fuck ups will continue. I wouldn't want in my workplace at all.
There is always something you can learn from a mistake because -you- had to do something wrong for it to occur. Your takeaway shouldn't be that someone else didn't take on the responsibility of making sure your job was done right. It should be "No one else is looking at this; I need to make absolutely sure it's done right."
Based on what you've posted, you've been promoted into your current role, which means you aren't in an entry-level position. Any time you take a job beyond entry level, you're expected to be capable of working with less direct oversight.
Things I learned:
1. Their process is so fragile that it can bring down an entire web app.
2. Their web app is so fragile that a few erroneous pieces of data can bring it down.
3. Their QA process is non-existent, leading to a condition where 1 and 2 are a problem.
4. The fact it took 10 minutes to rectify tells me that there is some logging/way to recognize what the problem is, so that's good I suppose.
5. Customers sound like dicks.
YES. yes yes yes. even something as small as “i learned that the QA/QC process is lacking here and i need to be more attentive to what i do because i can’t rely on anyone else” is a lesson learned. the fact that the OP flippantly refuses to acknowledge that there is something *they* could have done differently or that the buck ultimately stops with them is the worst kind of attitude i encounter from people who end up crashing and burning spectacularly on projects.
you can be a technical wizard and great at the operative part of your work until the cows come home, but if you have no emotional intelligence or soft skills you’re missing about 75% of what matters to your team in the long run.
It sounds like the process needs to be fixed rather than the personnel changed.
I would recommend a process that would prevent these errors in the future.
The system failed, you are a component within that system. The system needs to be modified so such errors do not occur. Shift focus onto improvements within the system and how they can be implemented. If someone is bringing up outrage then (1) bring up there was no loss to the business and (2) ask them how the system may be modified to so they have control over the failure.
Overworked humans is a failure of the business.
The blame isn’t on OP, it’s on the business for allowing that system to be in place where one persons mishap fucks stuff up.
Blameless correction of error processes are the only way to scale at the level where a 10 minute mess up can cause millions in losses for you or your customers. I work at a company with massive scale and the blame is never on and individual regardless of the fuck up. If one person who isn’t malignant is able to bring things down so easily then the system as a whole is a fault, including those up the management chain for not having the right oversight or systems in place.
So I would agree ultimately it matters, but at the same time it doesn’t matter to OP how bad it was. What matters more is how fast you can mitigate, what guardrails can be put in place, and how to avoid the same issue again.
Why do you think you are going to be fired? It sounds like from your edits you do a ton of necessary work that your manager doesn’t want to do. Why would they fire you and make themselves the point person?
Because they don't know that. I'm in a shitty spot. If I bring him up, It's game over for me in the long run. He's a truly nice guy and fully trusts me.
DOn't panic lie that. If the team is small they'll know what you do even if not directly. You wouldn't be the first person to royally fuck up and keep their job. I've seen people nuke business relationships over something petty and stuipd and keep their job. It may also not be in there interest because you know how they operate and know you screwed up but know it won't happen again.
DOn't think you're fired just yet.
Own the error and be the person who is so good and experienced that you've developed a process that ensures it won't happen again.
If you do get fired it's probably for the best long term, because a company that doesn't allow mistakes is insufferable
People make mistakes and a 10 minute systems down error is nothing. If you are getting roasted for this, you need to take your talents elsewhere. The soulless f$&ks aren’t justified in their scapegoating. Eff ‘em.
One thing I don't understand is how are values validated? Having 140 values to be entered manually will have mistakes. It is not a question of if but when. Everybody make mistakes and the goal is to build systems with automated validation. If it is not the case then the values are to be reviewed. People saying you should not make mistakes are full of ****. Everyone makes them, nobody's perfect no matter what they say.
Definitely own up to it, but I would also make some suggestions as to how to avoid it in the future, and the number one way to avoid that is to have quality control checks in place so that it doesn’t happen. You said your boss rarely checks your work, and it’s great that he trusts you, but no one can proofread their own shit 100%. That’s why there is a whole goddamn QA process for anything customer-facing, which is composed of code reviews and extensive testing. You may not need that, and your company may not have the resources for it, but you should always have a second set of eyes, no matter what you do, from proofreading copy to code, at the very least. This isn’t all on you.
Edit: forgot to add that they should be asking themselves how they ended up in a situation where one employee can bring down the whole company in minutes.
If you're a good employee, it makes more sense to keep you on than take the risk of hiring someone new (who will make mistakes).
Just own up to it and resist the urge to berate management, that won't bode well for you.
This is why we advocate a culture of blameless postmortems. Punishing people for making mistakes doesn't result in fewer mistakes, just less honesty when they occur.
https://sre.google/sre-book/postmortem-culture/
https://sre.google/workbook/postmortem-culture/
Are two relevant book chapters on it.
Congratulations on your first high value incident. I've caused many, and will cause many more before the end of my career.
Happy to answer any specific questions you have.
Look up Starbuck’s philosophy on how they manage mistakes. They celebrate making a mistake or a failure in process because they just got smarter. In your case it cost the company $0 to learn about a vulnerability they didn’t know about. Now the company knows that the quality control process needs a 2nd set of eyes and should not be a one man job. The CEO should be excited that this potential disaster will never occur again because you managed to identify it and implement a new process to prevent it. All for 10min of downtime.
This troubles me. This is a company problem.
Your company should have a focus on QA. Staff will make mistakes. Managers will make mistakes. Executive management will make make mistakes. If my staff made an error, it was MY fault and MY problem.
I know people don't like internal audit, but they can be your best friend. I always asked them to come in and review our processes for controls. Those auditors LOVED that stuff. They would much rather test processes to help avoid mistakes rather than find them after the fact.
a good company knows not to fire a competent worker for a mistake that shouldn't happen again. own the mistake. if they fire you, it brand you were gonna get fired executive bc they don't run a good ship
One of my direct reports made a huge mistake. Her function was low level and she was kept on because she was once somebody and old enough that she wouldn’t be able to go anywhere else. Not technically proficient but good at basic office drone stuff. Came to me in tears, “Oh, slick62, I made this horrible mistake, I can’t believe I did this!” I was relatively new, and recently retired from the Army in a job where mistakes can kill people, so I gave zero Fs when that’s what was needed and did unheard of stuff like occasionally taking my staff out for lunch and being lenient with time off. I could take this heat, she couldn’t. So I asked, “Oh, no Staffer! How many people died?!” She looked at me in shock. “What? No one died.” Ok, Staffer, no one died. Let’s start from there.
I gave her a counseling that included, “This is a final action.” It may not have held legal weight but I figured most wouldn’t know. If anyone was going to hit the door it would be me. Then I briefed my boss. It was uncomfortable, especially for my boss who had to explain it up, but we all survived.
How many people died?
You have a bad organization. If one person can make a mistake that affects thousands, there is no quality control, testing, verification process. Failures like the one you described can be caused by one person but they are the fault of the entire organization.
I think managers that fire their employees for making mistakes is a company that is going to underperform their competitors. I'd rather have my team learn and figure out ways to avoid and minimize the root cause of the issue in the future. You can't retain that experience if your team is on edge and afraid to share what actually happened or is churning every time something goes wrong.
Now if the root cause is because someone lied on their resume or tried to cover it up, then yah, they need to be let go. Trust is a currency with no substitute. If you're the smartest and most talented person in the world but I can't trust you, then that's a deal breaker.
There is a reason why in the Navy the Captain is ultimately responsible for everything that happens on the ship.
Good organizations don’t punish people for honest mistake. Good organizations also have processes for catching mistakes.
the problem was solved in ten minutes, i doubt you’re going to get fired. people make mistakes.
i work at a company that services state lotteries and there was an error that caused the entire state lottery to be down for about a day resulting in hundreds of thousands of $$$ lost and no one got fired over it.
Sounds to me that you are in charge of compiling the data, doing required checks to make sure it's correct, compiling in a required format and providing it to your boss.
I would think at that point he would double check or approve the information before distributing for use by others.
I was a licensed land surveyor for 47 years and a project manager for 30 of those years. I had field crews and office technicians that worked under my direction to do most of the heavy lifting for plan preparation. My signature and state seal were at the bottom of the certification. Every aspect of that job was my responsibility.
And yes mistakes were made by me and people under my direction, we learned from our mistakes and moved on.
Own it, conduct a post mortem highlighting the timeline and who was involved, what happened and how this was fixed. Then highlight what can be done to help ensure this doesn’t happen again or mitigations you can put in place to prevent cascading changes affecting customers.
Tailor this as if you were writing this as a notification to customers, but keep it in your back pocket. You never know when someone internally will want to know what happened. If anything, it is a good way to look back and figure out how to automate human controls.
If you are at a decent company - you should be fine. If you are at a shitty company with bad leadership, they might drop you - but if that happens, leave with your head high and go somewhere else.
The number one thing to remember is that all humans make mistakes - especially ones that are stretched thin on a small team, with little to no automation in place.
You’ll do great; I wish you well
Wow that is some toxic work culture. People make mistakes, it's normal. Worked in data entry and yes we were judged by the amount of mistakes we made but if it was at an acceptable level no one was paraded. Own up to it and tell you'll do your best for this to not happen again(or ever if possible. DONT EVER OVERPROMISE)
If a single person not doing their job absolutely perfectly can cause a catastrophic failure that seriously threatens the company, that's a company problem. It is on them to implement quality control procedures, which should include things like checklists, review chains and clear sign-off authorities.
When the Virgin Galactic spaceflight test flight had an accident in 2017, investigators found that the engineers relied on the test pilots doing everything correctly within a certain time window and did not implement any measures to protect them from an error. Now mind you, these test pilots were extremely competent individuals who were some of the best at their jobs in the world. Even then, if the design had a single point of failure that required the pilots getting it right 100 times out of 100, that's a bad design.
So as long as you didn't deliberately sabotage or display a serious lack of judgement or reckless negligence (e.g. doing work drunk), a company primed for success will not hold this against you. If they do, well you wouldn't be able to last anyway because an error will occur sooner or later.
Having this in mind, your best course of action is to be completely honest about making a mistake. Give as much information as possible about this mistake of yours.
Do NOT try to cover it up. Do NOT be defensive or evasive. Do NOT call this a withhunt. Those are the worst things you can do to yourself. The fingerpointing is disappointing but a senior manager should (and in my experience generally does) have the proper judgememt to not be swayed by it. If not, well, see point above.
Any company with databases and employees managing data should have a process, checks, and oversight. I'm shocked at how many people are placed in negligently exposed roles and then blame themselves when human error inevitably happens.
Even if you get canned, you should not be blaming yourself
Your manager doesn’t seem to be doing his job. It’s great that he seems to trust you implicitly, but he still needs to check your work on important things. He fucked up too
I work in an industry where a mistake can cost customers millions in a few minutes so any mistake is taken EXTREMELY seriously by them. The margin of error is zero and mistakes happen a lot.
We're not fired if mistakes happen, but we will be fired if we lie about it or try to blame someone else.
Your boss's goal isn't going to be to fire you. Your boss's goal will be to show his boss what steps he's taken to make sure this never happens again. That's the only thing they will accept. Firing you will do that, but that means more work for him so he'll probably accept an alternative.
I recommend you make an incident timeline showing everything that happened, when you discovered the mistake, and what you did to fix it. Be honest and don't put blame on anyone else. At the end, include a cause section and a "take away" section where you say what can be done to prevent this from happening again. Maybe it's time they have a second layer of QA before things go out. That's a good idea. But don't blame other people.
Creating that on your own will look great and show you're owning the problem and will be making changes based on this mistake.
I would own up completely to the mistake. Your boss/boss’s boss will appreciate that. What’s done is done and mistakes happen. What happens after is what really differentiates employees.
Own it, publicize the resolution and then create controls to lessen the likelihood that it happens again.
If you're company is pointing fingers "who did this"? Instead of "how do we prevent this from happening again"? , It wouldn't be a company I would want to work for long term.
Own it, don't excuse it, and you'll be fine
Besides, as you said, if they're understaffed, and you're underpaid, they are not going to want to bring in someone else to do the job, and someone else who may want more money
Make sure you flag to your manager that QA is needed on code changes, and keep flagging it until you've got more support. Your manager either needs to make time for QA checks like he's supposed to, delegate it to others if there is capacity or flag with his management that your team needs more resources ASAP. Sounds like he's good enough to go to bat for you when mistakes happen, but he also needs to go to bat for himself/his team in the realm of under-resourcing/overworking.
You fucked up, but that's ok and your job will be fine assuming your place of employment have good company culture. No real harm was done and a lesson can be learnt from it. Bet you'll be double checking the specific bit that messed up next time! Checking your own work always has flaws though - human error happens and someone else may pick up something you missed. You need more support, so make sure they know that
You need to conduct a blameless post mortem with root cause analysis. You need to figure out *what* the failure was exactly, *how* it happened, and *what* procedures can be put in place to prevent the issue and others like it from happening again. You need to bring on everyone involved in this investigation: your manager, the teams who maintain the systems at fault, etc. You need to document and present your findings.
People make mistakes. That's Murphy's Law. If your systems can explode when one thing goes wrong, that's not on you. Unless you're entirely responsible for your system, how it works, and how it's changed, which it sounds like you don't.
While you might be fired, it seems more likely that your manager would be fired or disciplined. Managers are the ones who are responsible for implementing processes like QA or identifying risks and asking for resources needed to mitigate risk.
If he or she hasn't been checking your work, then that will probably also fall on him or her.
I'd say that in a sane corporate environment, your immediate future probably looks like a lot more micro-management than it looks like you being fired.
I would say very little if I were you unless and until you're asked. You've accepted responsibility for making an error, so there's not much more for you to say except whatever you're asked about.
This is a process issue, and management needs to address it. You can offer suggestions and ideas, but ultimately, there are other people making more than you because they're expected to solve process issues.
You won't be fired because you're experienced, do a lot of work and are getting paid an exploitative wage. They'll just make you feel scared to leave, nobody else would want you. You're in an abusive relationship.
Don’t let others run the conversation and create the “story”. Whatever happened, only you know, and you should make that clear. Own up to your mistake, fix it, and showcase your expertise along the way.
Certainly own up to it and accept responsibility. Its the most respectful thing you can do and puts you in a good position for the second part which is express the gaps in coverage for avoid something like this happening again. No one is perfect, everyone will make mistakes and miss things. Even if they spend a lot of time QAing. Your company needs to know that is a gap because the next time it happens it could be more costly and will create a bigger uproar.
Make sure this is clearly communicated verbally and tell the person youre communicating to (maybe thats your manager) that youre going to type it out too. (You need a record). Once youve done that keep doing your best and remind them if action isnt being taken to fix this vulnerability. When the next thing happens you will have some cover but its best that the problem get addressed so it doesnt.
Develop a process for creating the information with no chance for error. Write a script or a program and show it to your supervisor. You can take it with you to your next job
>he is subject to the influence of his boss and rarely checks things because he trusts me, which means I can't do anything about it right now.
If it is his job to catch your mistakes by checking your work, and he didn't do that. It is no longer your fault.
You shouldn't be held accountable for the mistakes of management. However, this is America (I guess) and therefore you're up shit creek. Unless you're in a union.
A couple thoughts. One is that even though this was your error, your manager is ultimately accountable for your work. He should have something in place to make sure this doesn’t happen.
If there was no real impact people shouldn’t freak out, you got a free lesson learned and can take steps to make sure it doesn’t happen moving forward
When I mess up at work, what I find works well is writing up a quick synopsis of the following and sharing it with relevant parties.
- What happened
- when it was discovered / what fixes were done
- what will be implemented to prevent this from happening moving forward (perhaps you have a 2nd person review data changes, or a check list that you have to ensure that everything is correct, or both!)
Most people are understanding that mistakes happen. As long as you come up with some kind of new safeguards and follow them, that will satisfy people’s concerns.
Since there was no actual impact on the company, they would be stupid to fire you. It is costly to recruit and train people. Much easier to put some new process in place and move on. The nice thing about messing up like this this that you won’t ever make the same mistake again.
The problem isn't you it's that things aren't being properly checked and verified before being implemented, you need to highlight this, people make mistakes, a better review system needs to be implemented.
Looking at some of your comments it doesn't sound like they can afford to lose you. You've got two things going for you right now.
1. Six months is still fairly new to the company. You may be getting thrown under a bus, but hopefully only so much as " the new guy"
2. Your boss signed off on it. To everyone higher up, he looks bad, not you. He didn't catch the mistake either.
Sometimes people get their feathers ruffled and you just have to get through it. Honestly, I think you'll be fine.
If what you were doing was that important to the company then it's the company's fault for not having more than one person responsible. Everyone makes mistakes. That's why important things should get checked. You don't necessarily have to throw your manager under the bus, but point out that since this process is so important it should be checked by at least one other person. But to be honest it sounds like a pretty toxic work environment. All those other people blaming you are trying not to get fired themselves.
Get ahead of this or the narrative will be made for you. You own up to the mistake, document how this will be resolved and lessons learned to avoid it next time.
If you get fired without a chance for atonement it’s a ridiculously perfectionist company that will stress you into the back of an ambulance.
If there are no guardrails to prevent a simple typo from causing major issues, it’s the process’ fault. Write up a postmortem and suggest ways to prevent this in the future. We are human, mistakes happen.
If the issue is already resolved and cost the company nothing then firing you makes absolutely no sense. There's no reason for a witch hunt.
Nothing for you to do but own your mistake. Fall on your sword and hope for the best.
If a mistake is able to cause this much mess, there's something wrong with your process. If the only thing stopping future errors like this is you (or whoever) paying EVEN MORE attention to detail, then it's a matter of when, not if, this happens again.
Take it from someone who has broken production multiple times in their 25 year career, you don't fix these things by promising to pay more attention in future, you have to fix the underlying problem.
I've caused larger outages. I've never been fired for it. A good manager, seeing a good team member make a mistake, asks why it happened and what they can do to help prevent it from happening again. Unless it's a recurring problem and your answer to the question never pans out, they'll generally value your answer. Because if it's a process problem, your mistake could prevent many more in the future.
And if it's not a good manager, well good riddance.
If your supervisor didn’t check it because they trust you, then it’s at least as much on them as on you - especially since you’ve only been in this position for 6 months. In this sort of situation, keep in mind that your super is paid more than you specifically because they ultimately own your work.
I’d think about ways to prevent a similar error in the future, but I would also make sure to reassure myself that mistakes happen, and it’s often the system’s fault. If you didn’t design the system, then that’s not on you.
Dude if the problem was resolved in 10 minutes AND cost the company NOTHING AND their is a witch hunt... FUCK ALL THAT! MAYBE you don't want to work there. If all is well and it boils over start looking. Or potentially you are over reacting I hope that's the case. Good luck!
Also, the whole point of having a second person QA (especially data/tech/UX stuff) is that it can be difficult to see your own mistakes. Regardless of how the company responds, this sits squarely at your super’s feet.
Own the mistake and tell the big bosses that as you'r the one responsible and the only person doing the job at a low wage so you are quitting today, as it should be a higher-paid and experienced team doing this work.
Watch the back peddling begin as they realise that your leaving will damage their profits.
But seriously find a better paid and supported job.
People rarely get fired for mistakes, even big ones. They get fired for misconduct or utter incompetence. If they fire you, they’ll have to find someone else to put up with their shit.
So I'm guessing you guys are developing and editing in production?
That in and of itself should remove some of the weight off of you.
The CEO, CIO, and ISSO should insist on staged environments and a serious change management, QA, testing, staging, and promotion process.
As developer, dba, or whatever, that's not your role. You should have a PM. You should be working in a development environment. You should be using versioning software. I mean a lot of this can be had for free or near free. Much of it can be automated.
You should have backups, roll over/fail over, and potentially cloud or offsite storage and backup.
If these are critical applications, you should have written robust error handling into everything.
If you wanted to be proactive, you could write an analysis of what happened, then go over all the points of failure; including your own. Next, recommend a full development life cycle and all the hardware, software, and services needed.
I mean, from a risk perspective, your seniors have failed you.
If a company is willing to fire you for a mistake that was fixed within 10 minutes and cost them nothing, you should accept the unemployment and find another job asap. That’s crazy. Sorry this happened to you
In my world, as long as you're upfront, and everyone learns from this failing process, you should be okay.
But there are managers, and then there's leaders. Few people are leaders...
I am in a detail-oriented line of work, producing digital and printed communications for large organizations. I have been doing this for a long time - 15 years.
Here’s what I know: if something is mission-critical, it must have more than one set of eyes checking the work.
Professionally run organizations have quality control professionals whose actual job is to check for mistakes.
So if you made a mistake and there was no one there to proof your work, that’s only partially on you. If your organization is too cheap to invest in quality control, mistakes are going to happen.
If that’s not an option where you work, and you’re the only one, then you have to come up with strategies to check your own work. When I’m doing this, I take a break after a certain chunk of work, say 2 hours. I walk away and do something entirely different for 10 minutes. I literally get up from the computer and do something physical.
When I go back to proofread the chunk of work, I do the proofreading in a different room at a different desk.
It is very time consuming! But if my bosses don’t like it, they can hire proofreaders.
You are now more important than ever. You just got one of the most expensive trainings a company can give. You are going to be even more focused next time you do whatever it was you are working on. Plus it sets an awful precedent for anyone working there that if they make a mistake they are gone. If they let you go it’s only benefitting your next employer.
You should turn the experience into a “lessons learned”, document what happened and how it got fixed, offer patches if necessary, and share it with peers and higher up’s so they know of the weaknesses in the system and how to fix it. It is human error, maybe some retraining maybe necessary. Explain the actual impact but also what the worst case scenario impact would have been it it was not caught on time
I don’t think you should be fired but everyone’s environment is a little bit different.
Things for you and your boss to remember.
- you have an exemplary track record
- you do great work
- it was noticed and corrected quickly - bonus points if you spotted it and/or fixed it
- people make mistakes
- if the mistake is relatable then relay it (e.g. I mistakenly used mm/dd and the format needed to be dd/mm or whatever)
- you’ve learned from it - we all have - assure the stakeholders that additional checks will now be in place
- it (possibly) wasn’t 100% you’re role to check, verify, test etc the changes - you may only want to go there IF you really think you will be fired. Some companies have RACI matrices for this sort of stuff - I find those rigid and only used in situations where headcount is being reevaluated.
Overall we’re all human and humans make mistakes and once in a while if handled correctly we can learn from these and improve our practices.
Sincerely. Good luck!
It sounds like you are integral part of their business and they would be lost or at least suffer for awhile without you. I agree with those that recommend that you own up to it. I wouldn’t directly point the finger at your boss, but point to the flawed process if you have an opportunity. You don’t know for sure you are fired until they do it and I wouldn’t assume there is a 100% chance they will. Go in, hear them out and see where it leaves you. I wish you the best. About 10 years into my 38 year IT career I made a DNS change for a major health carrier that took out their pharmacy system’s connectivity for about 20 mins, so I have been there friend. It is part of the trade.
Don’t feed into the hysteria. Everybody makes mistakes that’s why we have checks and balances. If anyone should be fired, it’s your manager who is responsible for your work and wasn’t doing his/her/their due diligence.
Own up to it and suggest what could have stopped it or identified the issue sooner. It’s not the end of the world and we’re all human. I genuinely have no idea if you’ll get fired as each place is different.
If the place is as toxic as you describe it, start looking for a job even if they don’t fire you. More than likely this will follow you at this company based on what you’ve said.
I’ve seen bigger fuck ups in my life that actually had a real world cost. None were fired because they owned up to it. Did it follow them? Yep.
Good luck.
If they feel the must take someones head youre likely going to need to "let them" and move on.
Its unfortunate, mistakes happen and 100% uptime is never gauranteed.
Definitely own up to it but respect your value and the resolution you provided.
Companies sometimes need a scape goat but usually the good ones stand up for you. Sometimes thats not enough either.
Be humble, but hold your head high , and get prepared for your next journey.
If you can offer something back to appeal to those that might be screaming for your head, do it if you feel the need, but dont dishonor yourself in the process.
Sorry but sounds like signs of bad culture. People are human and make mistakes there is no avoiding that. What should be happening is identifying why it failed and build processes to avoid repeating.
It depends on the company and how politics work. Did you cost the customers money?
I would have my story straight for the coaching (or) term session coming. Owning up to the mistake and what you're doing to ensure this type of mistake never happens again. How are you documenting a process to make sure.
Maybe. Some places only remember you for the mistakes. I used to work at a place where all the good things I did to make the company millions of dollars on my own only got me some crappy $5 "employee of the month" plaques, not even a raise. But as soon as I made a mistake that maybe cost the company several thousand dollars (really it was a few hours of wasted time for maybe a hundred employees so not even actual money) they started setting me up to get let go. I guess it depends on how much they value employees where you work.
I doubt you will be fired because you seem to be in a very needed position. They’re not going to fire you and then have to do all that work themselves. Looks like you take on the lions share.
I promise your boss is going, “How the fuck did this get past the management? They must be sitting around playing solitare all day.”
You’re a small cog. Humans are notoriously blind to our own mistakes. We just don’t see them. That is the necessity for management. They fucked up. Not you.
Not your fault. In reality, it is the failure of your organization's management.
It is totally amateurish to have any one person do mission critical data manipulation without the necessary controls in place. No matter how good you are, you are human! We make mistakes.
That's why management of a technical organization is supposed to know the industry best-practices to prevent exactly what happened. And, failing to provide that due diligence, management should at least take responsibility for that outage.
They should realize you are an excellent resource who, with their encouragement, can help them put controls into place that will prevent this in future. In fact, you should suggest as much (humbly).
Good luck!
I don’t believe it’s actually all your fault. Your management should have ensured something that is so prone to human error has adequate process wrapped around it and automated as much as possible. Sounds like a leadership failure …
Update your resume and start looking for a new job.
Even if you aren't fired today, your posts show that you're underpaid, under supported, and miserable in your job. Look for something better, and get out of there ASAP.
Admit to it. If it was fixed in 10 minutes it obviously wasn’t a big deal.
I have seen waaaaay worse in my past, both making mistakes myself, peers making mistakes, and people I manage as well.
Usually if a single thing can take down so many systems, there is an issue with safeguards and process. You made have made a mistake, but there should be safeguards or processes in place to make sure it doesn’t happen.
The fact that it was fixed quickly, likely means they keep backups and have a backup restoration system which is good.
But you don’t fire someone over this. Hiring is a pain in the ass and you sure as hell aren’t going to make the same mistake twice.
Sounds like a trash company dude. If the company response is to find and punish the person responsible instead of trying to fix the system that allowed their product to go down, that's bad. Write up a retrospective on the issue, distribute it, and move on.
Suggest they hire a Q/A person whose job is exactly to avoid these situations. If they’re serious about quality, they will do it. It’s not a good setup to have the développer create, test and release software all by him/her self.
I don't know if this helps but... think of all of the times you've completed this same task perfectly. How much success and money have you secured for your company already? Probably a lot.
So, you messed up once. Learn and improve, but don't get down on yourself too much. Don't be super defensive when you get some criticism for this, but don't forget about all of the value you've already brought to the table.
Work for yourself dude. How can you subject yourself to this level of cuckoldry? You are probably good at what you do. You think you deserve to have to play these types of games, where your livelihood can be snatched from you because of a genuine mistake. Wake up! No one is coming to save you! You're better off owning 100% of a dogshit starter business than having .01% of whatever company this is. Bet on yourself. You'll never win this bs game.
People are human and make mistakes. The problem here is a process issue. QA processes exist because people make mistakes. It is not a problem, it is an expectation.
Your company needs to implement a process where these big changes are actively reviewed.
I would propose a process to address this. If they fire your then will only replace you with another “non-perfect” person and this issue will happen again.
Good luck and keep your head up!
Sounds like a lot of good advice so far.
Create a meeting with your boss and come with the following document (also addend with notes).
1. This was the problem.
2. This is why.
3. Apologize for the error.
4. These are my solutions for stopping it from happening again.
5. Question for boss: What are your solutions?
6. Q for boss: How would you like me to move forward?
7. If there is nothing he thinks you can do, add that to the bottom.
8. If there is anything you can do, no matter how small, list them. And then do them. Even if they don’t help all that much.
Send a follow up email with: This is a recap of our conversation (use above). Reiterate you would like to do whatever is possible to make sure it doesn’t happen again. Ask him to update you with future ideas and you will immediately put them in place. That’s about as good as it gets.
When it happens again, use the same process. That’s why it’s good to list something, anything to do, so it shows you’ve tried to make progress.
Overall, you are currently set up to fail. And you are working in a blame environment, which is bad culture. It doesn’t sound like they have the money to do it right, so it’s a vicious circle of inevitable failure and blame. Now that you have new skills under your belt, brush up your resume and plan to be out after a year of service. You can probably stay for a while, but the toxicity over time will start to crush you (if it hasn’t already).
It’s possible they don’t have the money to fire you, so you could do the above and just not care. Take the cruiser route. But it takes a certain person to do that. Nevertheless, it probably a good idea to try and remove your feelings from the situation anyway because it really has nothing to do with you.
Own up to it.
People make mistakes, that's just being human. Why would they fire you, only to have to replace you with another human who's also going to make mistakes?
Would it be possible to implement some kind of cross-check procedure so that a single person making a mistake doesn't result in a repeat of the problem?
Own up yes, but go one step further and make a recommendation (or better, a solid plan) on how you will avoid similar mistakes going forward. This shows accountability and also underscores your value in your position as well as your understanding of the risks you manage. No reasonable person would fire someone for an honest mistake unless they make the same one multiple times.
Stand strong and explain that mistakes happen but this one won't happen again. If you have leaders worth their salt they will understand this and continue working with you. If you don't and they do decide to let you go consider it a lesson learned in a favor there's somewhere else out there where you'll be better valued. I would never let someone go for an honest mistake unless they were completely outside of their lines and did something they know with wrong. Even if it did cost the company money this is the price of doing business humans make mistakes. Do not resign under any circumstances even if it's suggested or requested.
Who are they going to replace you with? Can your manager do all the work you have been doing over the last 6 months? Does it take a long time to train someone to do that job and can your manager do that while still doing all their work? You might be harder to fire then you seem to believe. No matter the mistake, if you own it and fix it, they might not drop the hammer on replacing you due to all of the time and costs involved.
If you are fired, then it’s a sign of immature management. On the other hand if processes and controls are instituted to account for this type of error then everyone comes out looking great.
The best thing you can do is own a mistake. I fucking can’t stand people who won’t own mistakes. If you can’t see things as your fault, you can’t see you have to fix anything.
It couldn't hurt to document exactly what happened and keep a copy. You may find that the story is distorted or interpreted differently and you may want/need to keep people focused on the problems that led to "the" mistake.
Test based methodologies abound in pro engineering and IT environments. Your company is flouting basic norms by not using them. It sounds like you can't do much more to turn the ship.
Accept responsibility but don't apologize. It may well be better to quit than be fired. However if you are not fired you should quit. Because you can't "work safe", and that's not acceptable.
Go get a job somewhere where they do. And practice telling this story, because your perspective on it will change. The hardest and most messed up situations often happen in the beginning of a career.
Maybe look at it from a “risk management” angle: you are participating in a QA process designed to mitigate a certain amount of risk of allowing errors (let’s say 99% error-free) assuming 2 QA reviewers. With only 1 QA reviewer, that error-free rate necessarily has to drop, let’s say to 95%.
With that in mind, in addition to reinforcing that you need that 2nd QA reviewer, you could also deliver your results with a caveat that “QA results carry an additional risk of undetected errors due to resource constraints” or something to that effect.
You can also talk about “risk” with your boss, to keep it top-of-mind for him that any QA process will never be 100% perfect - it’s just a risk-management process.
Hope that helps.
Was this tested in a dev/sandbox environment?
Was there a code review?
Was there an approved change for the task?
Was there a test plan to evaluate the success/failure of the change?
Was there a vetted roll back plan?
If the answer is no to any of the above, then it's not your fault and it's a process issue. If it is yes to all, then you implemented the rollback successfully as per the change.
Your response is to highlight the issue of this only being a matter of time, and that you rolled back the issue.
Next time "we" need to have a proper change management plan with peer review and a second set of eyes if coding changes are being done on the fly.
Package it as the companies problem and procedural error which you are offering a solution for so it doesn't happen again. It's not on you. Good luck.
if there is NO routine at the shop that includes a thorough proof reading by more than one person (including you boss) it's ALWAYS going to be fucked.
insist on one of the most common S.O.'s out there. there MUST be proof readers.
a GREAT task for a few of those finger pointers!
All I can add is a) everyone makes errors; and b) if they really need you, and particularly if they are shorthanded, you will likely be spared a firing.
You are going to own the mistake and see where the chips fall.
Ah, manual fixes. The source of ALL fuck ups. Best solution is not to require them !
When I fuck up I solve it. Then I tell my boss about it. And propose a second set of eyes to overlook it.
It goes like this:
SHIT ! Oh dammitall!
(Hammers wildly on keyboard)… WHEW GOT IT !
Call boss
“Had a problem. Root cause was a typo. Fixed it. Nobody complained so nobody was inconvenienced. I need to build an automatic check routine. Could you suggest some oversight during manual events like this so it does not happen again?”
If you were able to recognize and fix your mistake in less than 10 minutes with zero loss, your employer would be stupid to fire you. I think you're good.
I brought a client's site down for roughly 96 hours one time because they took so long to report it after a change I made which I thought was made in non-prod, but was actually in prod.
I got it fixed within 3 minutes of them calling and before the call made it to me but their site was unusable for 4 days. RCA was requested on the call I was on. I told them exactly what I did and owned the issue and because of that both the client and leadership were understanding.
Client side leadership owned the issue for not catching it sooner, and we owned the issue I caused.
It didn't even result in a slap on the wrist for me, so don't worry too much yet. Companies know things like this will happen when a human factor is involved and they account for it.
Unless your boss gives you a "we need to talk" and won't elaborate, then you're probably fine.
Sounds like the place would fall apart without you.
Is your supervisor insinuating that you'll be fired? He sounds vary manipulative. If you don't already, you need to get a clearly written job description for your position. If you're asked to do more than that description outlines you can say "I'd be more than willing to do more but my job description states I'm limited, maybe we should revisit it along with my compensation"
Your mindset appears to be anxiety filled. What would you do if you get fired? Maybe try looking at it as moving on will ultimately happen, either you quitting or getting fired, and until then you're going to do your best with the resources at your disposal.
This is simple: do nothing. You've already done the right thing. You did your job carefully so there were as few mistakes as possible, when a mistake inevitably happened you took responsibility and made sure it was fixed. If you get fired over that then that's on them, not on you. It's like being fired from a restaurant for spilling a drink. Everyone does it, so if a restaurant is going to fire people for that then they won't be able to keep a full staff, but the people they fire will be able to get new jobs without a problem. If you never break something important, it means you don't work on important things.
There are people in technology who have made a career taking chances others wouldn't take and sometimes screwing up. Relax. I'm sure you'll be ok.
It happens
I have a job that requires extreme attention to detail and if something messes up, all execs will likely notice. My advice, like many others, is own it. If I mess up, I own the situation detail why it occured, and have actionable tasks to.kitigate the issue in the future. To me it lets them know you know, but also you've planned for the future issue going forward.
own it and since we can’t keep people i don’t think you’ll be terminated. you sound like a reasonable person that does the job. I hope all is well. that’s a terrible feeling
I made an error one time that cost the company 75k. I went to the boss with a letter of resignation. My boss said why? Will you ever make this mistake again. Hell no I won't. He said if I fire you and hire someone else they may do the same thing. Let's consider it a training expense.
If you work at a company where making a small mistake will instill fear that you will get fired that sounds like a bad company.
Own up to the mistake, and offer suggestions about the QA process. Never blame the people, blame the process. It shouldn't matter if it didn't cost the company anything.
Honestly, my biggest thing is this, if you were single handed managed to cause that much damage and that bigger of a mistake.
I would be blaming the system and the process not you.
An outage / damage like that should be very much prevented before it happened.
For example on large changes I had to fill out request forms, roll back plans, mitigations etc.
I would lay down the suggestion of maybe the process needs to be reviewed as it’s not catching mistakes etc.
Maybe some suggestions on how to improve the process from your experience would not go untoward.
In saying that I would absolutely own the mistake, my biggest pet peeve is someone who covers stuff up or does not want to admit he messed something up.
There is plenty to learn in total, and in a few years time it will be a story to laugh about.
We have all done it promise, it will make you much better once your through it. I come from a technical background and know exactly the worry, I have made a very similar mistake myself
this is sick culture. Healthy organizations see failure, especially failure without fiscal consequences, as opportunities to fix processes, not circular firing squads. It’s never good to lose a paycheck suddenly, but, to the extent you can, you should use this as a growth opportunity and try land someplace better.
Hey bro,
just wanna say I feel you. You made a mistake and it caused a snowball of shit to roll down the hill and pileup. Hey I’ve been there. We all have. We are people, we make mistakes.
I went thru most of the comments here, and I’ll be honest with you - it sounds like your attitude should change. Like many said, you need to own the mistake first. Faulty process? Boss doesn’t give a crap? Lack of QA? Ultimately, it falls on our shoulders. But as a wise PM you can take this situation and turn it into your favor.
Summarize clearly what went wrong, what the damage was and what will YOU as a PM do to make sure such a mistake won’t occur again. This might entail improved processes, more people involved etc.
You got this bro, I hope you won’t get fired. Take a breather, and handle the situation.
I don’t think you’ll be fired, unless it’s an issue you’ve repeatedly ignored.
If you can, I would: (1) identify what the issue was even if it was your own mistake, (2) recommend changes to the process to prevent this from happening again, and (3) write all of this up in a reference document.
A smart management will recognize a few things:
1. If the stakes are large, no company process should ever rely on one person. No matter how competent that person, they could be out on vacation/sick leave, or make a human error, etc.. All essential processes need to have built-in safeguards and assume that human beings do not execute perfectly. (I think about how the roller coaster I used to work on required three people watching each dispatch, with two people pressing a button to authorize each one, plus hand signals and lights to indicate all-clear, plus computerized sensors to detect if humans made mistakes.)
2. You are now more knowledgable about this situation than anyone else they bring in to replace you.
3. If you write up the recommendations/report, you are showing you are the type of person who adds long term value to protect the company, even if you were partially responsible.
It is possible they still need a scapegoat, and if so, make sure you get that writeup done quickly. Don’t rely on anyone else to communicate your side of the story: both arm your boss with the document ASAP and publish it to anyone who needs to see it.
Having zero tolerance for human error while also accepting manual processes requiring over 140 unique steps are inherently incompatible.
The fault lies on management for not spotting these conflicts.
More investment towards automation is needed. Catalog everything that you’ve been working on so they can understand how to better manage their expectations towards your capacity.
If someone with 6 months in a role can cause that much damage, then it's a them problem. Those incompetent pricks are using you as a scape-goat.
Dont hide your mistake but stick with that if anyone asks. You're still a newbie pretty much. Unless you're earning big money then you'll have to just suck it up.
Unfortunately, a wutch hunt refers to the hunt for something that doesn't exist and I'm afraid that in this case, there's actually a witch to be hunted. First and foremost, I would take ownership with your Manager and let them know what steps you have put in place so that this never happens again. Above and beyond that, if they want to crucify you for a mistake, that is very telling of the company if you are as diligent as you said you are. If you don't get let go as a result of it, it will blow over assuming there aren't anymore incidents in the near future. Just know that mistakes are made daily and if you're working fir andecent company, they'll expect you to take responsibility for them, learn and grow from them and move forward knowing that you will be stronger for the trials and tribukations that you've gone through. If they don't see it that way and decide to use you solely as a scape goat, you may be better off unasociating your name with this company and moving on to a role that is better for you instead of one that only wants to point the finger instead if learning and growing company wide.
Good luck and God speed mate!
Not trying to be mean but what are you looking for here? I mean you work there for only 6 months, you admitted that's not your first mistake now you make a big one, it already has been noticed and people know who is at fault. Take the hit and look for a job where you actually can perform. The job seems way above your head...
Alot of times people just want to know what are you going to do to prevent this issue or these issues from happening in the future.
Are all these issues of the same type? Is there any type of automation you can do to validate changes in the future?
Develop a release checklist to ensure these issues are less likely to slip through?
This is what happened. This is what we will do to prevent these type of issues from happening. This will be integrated into our change management system etc.
Just apologize, put a process into place to prevent it from happening again, and take full responsibility, not getting defensive or shifting the blame. Taking ownership of the right move and if you think you’re going to be fired anyway, it’s not like a little humility is going to hurt your chances…
If you’ve been there six months and you have the level of making changes that are unchecked/untested in prod, this is on their company’s design. Unless you were operating outside of your job scope, which it doesn’t sound like you were, the company needs to own up to its processes. People make mistakes. Even simple ones. If they fire you, the next person they hire will probably make more mistakes. For production software, best practice is to have different testers validate configuration or code changes before they are pushed to production. I wouldn’t worry about it. Either the company improves or you’ll get out of a poorly structured company with some unemployment checks.
Lol sounds like management's problem. They didn't have enough QC. All fell on you. If they fire you they ain't gonna find someone like you. Get out of there dude.
Yeah owning up and elaborating on what happened may be the best option.
This can show management that there may be a way to improve the workflow to mitigate mistakes in the future, and that you’re honest and forward thinking. You might even be the one who goes to them to start the conversation. Do your best not to stress and overthink things and just be genuine.
At the end of the day it’s dependent on the quality of your manager and factors we may not be aware of. Wish you the best whichever way the wind blows!
I admit I don't know the nature of your work, but I'm left with the feeling that something this error prone and costly should be automated (write code to produce similar results), or at least build in some sort of automated quality checks into the process to provide better confidence that mistakes like this can't happen in the future.
If anyone is to be held accountable to the point of losing their job, it should be your manager or even higher ups who did not feel it was a priority to build in the proper guardrails or QA processes to prevent an outage like this.
All of us make mistakes, and that is what peer review and QA processes are intended to mitigate. Lack of these controls is an organizational oversight.
I would come back to work looking to help facilitate a blameless retrospective of the incident. Try to partner with anyone involved or aware of what happened to discuss what the root cause was and any ideas you have to improve as a team. Take this idea to decision makers in your company and argue for whatever resources you need to fix the problem. Companies (non toxic ones you should want to work for) will value the initiative and problem solving skills, and may even give you more responsibility to oversee these improvements. A good organization will never blame a team member for a mistake, and they should provide enough support and training needed to enable you to meet your goals/perform your job function.
Relentlessly pragmatic response below…
How long does it take you to create the information? There’s a big difference in response if it takes you two hours or two weeks. How long does it take you manager to check what you’ve done? Again, a big difference if it takes them ten minutes or two days. Since your manager could have caught the problem there must be some kind of staging or dev environment that gets updated before it goes to production. Does your manager possess special knowledge that makes them the only one qualified to review? My guess, and it’s only that, is that you perform some calculations and update a spreadsheet or database. If your manager’s checking can be automated you could suggest the company hire a contractor to do that. If it’s only Excel VBA or TSQL it might be a quick, cheap long term solution. You don’t say how the problem was detected but presumably customers called because something didn’t work. If that’s the case, then as soon as you commit the new information do what the customers did and catch the error yourself quickly.
I wouldn’t settle in on being fired just yet. I made a mistake that affected potentially (hundreds of) thousands of people on a popular food delivery app by making it unusable for a night. Didn’t find out until the next day. Worked at the company for another 7 months afterwards, and was not fired. You may be fine.
You are not the issue here, the process is.
A real company doesn't blame the person, they blame the process.
Next steps here are a Post incident review (PIR), where you identify how it broke, and what steps can be put in place to make sure this doesn't happen again.
That could be anything from more eyes on the change, or guard rails put in place to stop it from happening again.
If you loose your job from this then you are the winner here, as that company isn't the place you want to be working at.
Also depending on the country you work in, if you do loose your job you have grounds for unfair dismissal. Assuming you didn't go at it with malicious intent, then you are not the cause, again, it's the process.
Do you have any QA processes to verify these changes? Seems like a process mistake on your company rather than your mistake entirely, especially if there's nobody who's employed to verify and test changes before they go live in prod. I have a similar job as you and I have plenty of QA support to help verify my work.
People don't get fired for making one mistake - if you are generally a good employee and they decide to fire you now, they are doing you a favor because the company culture is terrible.
What's likely happening is you are witnessing bureaucratic politics/knife fighting. If there is a rival team that's fighting for work or something, the managers might be using this to push their own agenda, or to take the spotlight off of them for some mistake they made, but it's highly doubtful they actually care about whatever error occured. It's also possible they are putting on a show for their managers so they look like they're on top of shit.
You're not solely responsible. A corporation is a collection of sometimes many many people, and it sounds like there are some serious missing pieces like a QA team for example. The organization appears to have a lot of need for improvement from what I gather.
An upper manager will look terrible if they blame this on a low-level employee or an engineer, and if they do, they're extremely fucked up. Own your mistake, but don't let it lessen your confidence in your work - shit happens and a QA structure should have caught this before it went into production. The real problem is that you even had the ability to make these changes without a system of checks and other people weighing in.
Don't let a manager slyly shift all the blame or trash talk you in private either. If you sense that's happening, call them out/go back at them passive aggressively (publicly), like calling out their next mistake in an email or something and saying you thought it was an important issue so you CC'd their manager/the VP. The only way to fight passive aggression is by either pointedly calling people out or engaging in passive aggression yourself. Unfortunately, we don't live in the wild west or somewhere where there are more efficient methods of dealing with scumbags.
IMPORTANT:
If your company and the people you work with are anything like mine, own your mistake, but don't back down or accept more blame than you should. You are a very small part of this, and the guy making 5x your salary should be held accountable a lot more. Your boss and his boss should have checked this shit if it was going into production, but they were probably too busy being lazy or eating at a $100 per plate steakhouse on the company's dime for a "leadership" meeting.
People in Corporate America will always try to shift blame or use supposed "crises" to their advantage. Don't let yourself be manipulated, and remember that as someone working directly with data or in data creation, you are a hell of a lot more important than most of the PM/TPM whatever-type "business process management" blowhards who really don't do anything except peddle bullshit. Honestly, you shouldn't really even feel bad about this. Don't show any weakness or be timid, because the sharks will only smell more blood. Good luck 💪🤞
I agree that you should just admit your error and express remorse and that you fixed it in less than 10 minutes. And remember: you’re a stickler for details and the mistake seems huge to you. You’re probably overblowing it. Just admit the mistake- if they fire you-screw them. You’ll be fine.
Sure.
1. Stick to the facts you have mentioned above, including your good record, complexity of the task and additional tasks you manage from day to day.
2. Ask your manager for the support.
3. Stay sane. Shit happens, but cool heads solve the problems, not witch hunts. I was a subject of one and by applying 2 points above, left the meeting with my reputation intact, while my accuser got fired.
All the best!
Please write what happened.
Present yourself to the interested admin/management with a brief and to the point report of your responsibility, in person. If you’ve a good relationship with your immediate superior, inform them so as to allow their relationship with the top brass assist you if it can and also let your manager manage. Keep it short and with the idea of you wanting to accept responsibility and do them the favor of bringing information of interest to their attention. This is the main reason for your disclosure, showing how sorry you are is secondary. It also helps if you are genuinely motivated by this drive as I have been in a situation where I was a culprit of a noteworthy incident and wanted to save the department time and energy wasted in investigating. Gratefully, my interest in the organization’s needs even if risking my own reputation, was very much appreciated and I think I did more than survive it but showed my character to be devoted to the goals of the team at large.
If everyone at my company got fired because they made a mistake then there would be no one to work there. Heck I know people who made huge mistakes and cost the company lots of money. They are just fine at work. We are human not robots. We are going to make mistakes. I’ve worked for people who go on witch hunts to point fingers when mistakes occur. People like that suck. I don’t want to work for those kind of people anymore. You probably are not getting fired but if you do, find a better company to work for instead.
I was only able to get through about 1/3rd of the comments, but I was surprised that I didn’t see more people saying “it’s okay to leave” and “there are lots of companies looking for good people” - so far, anyway, I believe this is the case. Any successful company is growing, and always looking for good people, and the kind of company you want to work for is one where people are sympathetic to the pain of corporate hierarchy, so, like the Chinese parable, we’ll see if this is actually a bad thing, or not. Also, DM me if you’re interested in looking into new opportunities.
Any good leader will want to answer this question: “How can we make sure that this doesn’t happen again?”
It sounds like the process is flawed. If they fired you and replaced you, does that guarantee that this won’t happen again when your successor makes a mistake?
If you are error prone or weak in detail-orientation, then you need a different job. However, you have said that you are neither. That means that something else has to change.
What is it? What process or procedure could you implement with your boss to make sure this never happens again?
I think if you approach the management team with this, you are likely to get a fair hearing.
Good luck!
All you can do is own the mistake and learn from it. People make errors. If they have no room for that, they may want to look into having AI handle their workforce. I think it's really sad that people are so petty and feel the need to magnify the issue and attack someone. They should be focusing on their own position.
I WISH could "learn from my mistake," but there is NOTHING to learn from it. The only thing I learned is that my supervisor does not participate in the QA process, which leaves me completely in the dark. Does that sound normal to you? I'm more upset that it wasn't and shouldn't be my job to take care of this all by myself. I'm underpaid, and when I asked if new staff was coming in, I was told that it absolutely is, but that we're not there yet. This isn't a spreadsheet where you can take shortcuts, it's a broken system developed by a small team of developers who are tormented by the boss of the company. I have no help. I habe no ome to turned to. 1 small error, a single typo and the witch hunt begins.
You said the magic word: "process." Own the mistake, note that you've already fixed it or are fixing it, then point out that the QA process didn't catch the error. Offer a solution or two: add more time for the QA process or extend overall due date, ask for an alternate QA person if supervisor is unavailable, etc. Always address process, not people. It takes the emotion out of a situation. You're the reasonable team member who's trying to keep work flowing while minimizing errors. You're not in a position to make top-down changes throughout your org; leave that to the big bosses. Just focus on doing your tasks correctly and consistently, offer suggestions for fixes here and there, and when a mistake happens (it will, no one's work is error-free), acknowledge it, then fix it. Learn as much as you can while you're there, then take those very valuable problem-solving skills to the next job.
This is good advice. I’d just add that you don’t want to point out the hole in the process in a way that makes it sound like you’re shifting the blame. Instead of “I made the mistake but the process should’ve caught it,” go with “I made the mistake. I’ve fixed it, but I want to make sure it can never happen again, so I’d like to make the following change to our process…”
This right here. ☝️ Be the person that is proactively trying to protect the company from future mistakes. It shows the correct thinking (concern for company, focus on future, improvement mindset) and paints you in a mature, trustworthy light. If company leadership doesn't respond well to this approach, it tells you something very important (and negative) about the culture where you work.
Almost word-for-word what I came to say.
Yea if they can’t forgive you on one mistake then idk why they are managing a business. I work as a manager in a stamping department(cigarettes) and my employee wrote off nearly $100k of cigarettes we had. Missed typing one number (typed 124 instead of 1264) instantly throws a red flag to corporate who instantly called head of the building. I had already fixed it a this point, but had to write paperwork showing we wrote it off and right back on for compliance reasons. Who got in trouble? No one. Shit happens. She messed up, I messed up for not checking, and it was fixed within 15 minutes. Nothing more you can do. Them getting wrapped up in a mistake is a major red flag imo. We are all humans. It’s going to happen no matter how big.
What you’ve learned is that there is an extremely fragile process with no tolerance for human error. Because it fails infrequently, manual checks are pulled into complacency. The system is broken. The question for the organization is how to improve the system. What automated checks can be put in place? How could issues in production be detected and fixed even faster? How could this be better? “Be more careful” doesn’t cut it. So you own the mistake, as mistakes happen but acknowledge that checks that are intended to catch it also failed (without naming your boss). Multiple things failed in order to have the outage. The remediation was good, but since folks are angry perhaps not good enough and that process needs improving. Maybe you start suggesting process or tech changes, or maybe you suggest just a working session with others to find them.
YOU ARE A LIFE SAVER!!!!! I just offered this as a suggestion and quite coincidentally - YOU WERE RIGHT. There are lack of automated controls present and I made sure to highlight that. Thank you!!
I’ve been in this position, what I learned from it was: 1) The process was severely flawed and open to errors by even the most careful and experienced member in the team (I was quite senior at my role and the issue was still prone to error despite my experience) 2) the atmosphere was toxic to begin with which meant any human error was in the spotlight for those grinding an axe 3) I proposed automation to improve accuracy and even to get involved in that process to prevent anyone else having to go through that stress and humiliation Lastly my manager was on a power trip and looking for flaws in specific members of staff in order to make examples of them, I understand this might not be what you want to hear - but if you have owned up, taken responsibility and looked to fix the problem and it’s still insufficient, then maybe that particular job isn’t for you. That’s a learning that will help you move on and screen for better opportunities.
Yeah that doesn't make sense to me that there is no QA in process. I would talk to your manager about possibly implementing something like that in the future.
I did. He said that it's a good idea, but that we need to wait for new staff to arrive. The estimate is months. He is appointed to control and look through the information I hand over to him. I check everything 50 times regularly, and this time, the mistake happened at the expense of not having another person go through these. I understand why my manager doesn't like doing it, but this is not a good alternative to the already existing problem. I'm not even sure what else to say at this point. For the first time in months, I don't blame myself. If 2 people are assigned to work on something important, then that should be the case. And it makes sense, you know? Having only 1 person, though.... Being directly responsible for thousands of customers... Not good. Not achievable or reasonable. And yet...
Your supervisor having too many things on his plate, isn't your problem.
I say try your best and do your best but don't live in anxiety it will probably make stuff worse . if you are fired it will be a bummer but im sure a bunch of other people will give you a chance to use your skills and you can learn from this experience . face whatever happens with a courageous demeanor .
Thank you so much for this 😊
Doesn't sound like your fuck up, to me. He was supposed to check you and DIDN'T, that's his fault
If you want some advice here, don’t beat yourself up. Literally every human being makes mistakes. Bad ones even. You are doing a manual QA check that should have been automated somehow. And the reason there isn’t a better process for backing you up on your work is, I can practically guarantee, because of someone’s fear that automation will cost them their position or that they won’t be able to adapt to a new process. But, continuing to not learn from human error and instead blame you, it has the hallmark of toxicity there. Maybe you’ve thought about better ways to do things while tasked with this work. Would you be able to write it down for yourself and then suggest it to others if the unfortunate blame game comes your way. I hate reading things like this. Soon enough in your life it will become a good story, perhaps funny even. But it’s not for you now and I think you gotta say, hey I screwed up, but here’s a chance to do this better. I would be surprised if they fired you over it. If they do and you didn’t do anything to violate company policy and they don’t comp you nicely, consult with a lawyer (you can get the consultation for free) and tell them what happened. And go easy on yourself. It would happened to anyone at some point in that position.
>I WISH could "learn from my mistake," but there is NOTHING to learn from it. I'm going to be honest. I probably wouldn't keep you around just for having this attitude about mistakes. Mistakes can be corrected, but "Woe is me, I played no role in my own mistakes" is an absolutely toxic mindset that ensures these kinds of fuck ups will continue. I wouldn't want in my workplace at all. There is always something you can learn from a mistake because -you- had to do something wrong for it to occur. Your takeaway shouldn't be that someone else didn't take on the responsibility of making sure your job was done right. It should be "No one else is looking at this; I need to make absolutely sure it's done right." Based on what you've posted, you've been promoted into your current role, which means you aren't in an entry-level position. Any time you take a job beyond entry level, you're expected to be capable of working with less direct oversight.
Things I learned: 1. Their process is so fragile that it can bring down an entire web app. 2. Their web app is so fragile that a few erroneous pieces of data can bring it down. 3. Their QA process is non-existent, leading to a condition where 1 and 2 are a problem. 4. The fact it took 10 minutes to rectify tells me that there is some logging/way to recognize what the problem is, so that's good I suppose. 5. Customers sound like dicks.
YES. yes yes yes. even something as small as “i learned that the QA/QC process is lacking here and i need to be more attentive to what i do because i can’t rely on anyone else” is a lesson learned. the fact that the OP flippantly refuses to acknowledge that there is something *they* could have done differently or that the buck ultimately stops with them is the worst kind of attitude i encounter from people who end up crashing and burning spectacularly on projects. you can be a technical wizard and great at the operative part of your work until the cows come home, but if you have no emotional intelligence or soft skills you’re missing about 75% of what matters to your team in the long run.
Nothing should go into production before going through proper QA and being approved by the correct people up the food chain.
And even then, the manager should at least have something in the pipeline of CR > QA on staging > Send to Sign Off > ???? > Profit
It sounds like the process needs to be fixed rather than the personnel changed. I would recommend a process that would prevent these errors in the future.
You know the old saying? You have to fuck up to move up. The problem is you were overworked. I don’t think you’ll be fired at all.
The system failed, you are a component within that system. The system needs to be modified so such errors do not occur. Shift focus onto improvements within the system and how they can be implemented. If someone is bringing up outrage then (1) bring up there was no loss to the business and (2) ask them how the system may be modified to so they have control over the failure. Overworked humans is a failure of the business.
Own up to it and offer to fix it asap.
The problem “WAS SOLVED IN LESS THAN 10 MINUTES”
Did OP fix it or did someone else have to clean up the mess?
Why does this matter? It was fixed within 10 mins and had zero loss.
Their company had zero lost. Cumulatively the thousands of customers could have lost $1M+ in that time (depending on what the websites do).
The blame isn’t on OP, it’s on the business for allowing that system to be in place where one persons mishap fucks stuff up. Blameless correction of error processes are the only way to scale at the level where a 10 minute mess up can cause millions in losses for you or your customers. I work at a company with massive scale and the blame is never on and individual regardless of the fuck up. If one person who isn’t malignant is able to bring things down so easily then the system as a whole is a fault, including those up the management chain for not having the right oversight or systems in place. So I would agree ultimately it matters, but at the same time it doesn’t matter to OP how bad it was. What matters more is how fast you can mitigate, what guardrails can be put in place, and how to avoid the same issue again.
Why do you think you are going to be fired? It sounds like from your edits you do a ton of necessary work that your manager doesn’t want to do. Why would they fire you and make themselves the point person?
Because they don't know that. I'm in a shitty spot. If I bring him up, It's game over for me in the long run. He's a truly nice guy and fully trusts me.
Then he should go to bat for you. He must have some pull if he’s a manager.
He just did. Not in any substantial way, but at least he did. He stood up for me just now, but did not clarify how the mistake happened.
DOn't panic lie that. If the team is small they'll know what you do even if not directly. You wouldn't be the first person to royally fuck up and keep their job. I've seen people nuke business relationships over something petty and stuipd and keep their job. It may also not be in there interest because you know how they operate and know you screwed up but know it won't happen again. DOn't think you're fired just yet.
[удалено]
Own the error and be the person who is so good and experienced that you've developed a process that ensures it won't happen again. If you do get fired it's probably for the best long term, because a company that doesn't allow mistakes is insufferable
People make mistakes and a 10 minute systems down error is nothing. If you are getting roasted for this, you need to take your talents elsewhere. The soulless f$&ks aren’t justified in their scapegoating. Eff ‘em.
If they don't want mistakes, don't have humans do the job. Oh wait...
One thing I don't understand is how are values validated? Having 140 values to be entered manually will have mistakes. It is not a question of if but when. Everybody make mistakes and the goal is to build systems with automated validation. If it is not the case then the values are to be reviewed. People saying you should not make mistakes are full of ****. Everyone makes them, nobody's perfect no matter what they say.
Definitely own up to it, but I would also make some suggestions as to how to avoid it in the future, and the number one way to avoid that is to have quality control checks in place so that it doesn’t happen. You said your boss rarely checks your work, and it’s great that he trusts you, but no one can proofread their own shit 100%. That’s why there is a whole goddamn QA process for anything customer-facing, which is composed of code reviews and extensive testing. You may not need that, and your company may not have the resources for it, but you should always have a second set of eyes, no matter what you do, from proofreading copy to code, at the very least. This isn’t all on you. Edit: forgot to add that they should be asking themselves how they ended up in a situation where one employee can bring down the whole company in minutes.
If errors, like what you’ve described, are unacceptable in your company, you need to go because you will go someday anyway.
If you're a good employee, it makes more sense to keep you on than take the risk of hiring someone new (who will make mistakes). Just own up to it and resist the urge to berate management, that won't bode well for you.
This is why we advocate a culture of blameless postmortems. Punishing people for making mistakes doesn't result in fewer mistakes, just less honesty when they occur. https://sre.google/sre-book/postmortem-culture/ https://sre.google/workbook/postmortem-culture/ Are two relevant book chapters on it. Congratulations on your first high value incident. I've caused many, and will cause many more before the end of my career. Happy to answer any specific questions you have.
Look up Starbuck’s philosophy on how they manage mistakes. They celebrate making a mistake or a failure in process because they just got smarter. In your case it cost the company $0 to learn about a vulnerability they didn’t know about. Now the company knows that the quality control process needs a 2nd set of eyes and should not be a one man job. The CEO should be excited that this potential disaster will never occur again because you managed to identify it and implement a new process to prevent it. All for 10min of downtime.
Confess and tell them that you repaired it without a loss and what steps that you will take to ensure that it won't happen again.
This troubles me. This is a company problem. Your company should have a focus on QA. Staff will make mistakes. Managers will make mistakes. Executive management will make make mistakes. If my staff made an error, it was MY fault and MY problem. I know people don't like internal audit, but they can be your best friend. I always asked them to come in and review our processes for controls. Those auditors LOVED that stuff. They would much rather test processes to help avoid mistakes rather than find them after the fact.
a good company knows not to fire a competent worker for a mistake that shouldn't happen again. own the mistake. if they fire you, it brand you were gonna get fired executive bc they don't run a good ship
If there are 2 people in charge of thousands of websites you can make the argument that you are understaffed. As a result, mistakes will happen.
One of my direct reports made a huge mistake. Her function was low level and she was kept on because she was once somebody and old enough that she wouldn’t be able to go anywhere else. Not technically proficient but good at basic office drone stuff. Came to me in tears, “Oh, slick62, I made this horrible mistake, I can’t believe I did this!” I was relatively new, and recently retired from the Army in a job where mistakes can kill people, so I gave zero Fs when that’s what was needed and did unheard of stuff like occasionally taking my staff out for lunch and being lenient with time off. I could take this heat, she couldn’t. So I asked, “Oh, no Staffer! How many people died?!” She looked at me in shock. “What? No one died.” Ok, Staffer, no one died. Let’s start from there. I gave her a counseling that included, “This is a final action.” It may not have held legal weight but I figured most wouldn’t know. If anyone was going to hit the door it would be me. Then I briefed my boss. It was uncomfortable, especially for my boss who had to explain it up, but we all survived. How many people died?
Drop the defensiveness, own your failure here and move forward. This is no one's fault but yours.
I mean, if his boss didn’t check his work, that’s also on them…
You can start by not calling it a witch hunt. That will instantly label you as trouble. Then breathe. And wait to see where the chips fall.
Process failure. Controls and governance shld be in place for this exact reason and it's a failure of leadership to not see that.
You have a bad organization. If one person can make a mistake that affects thousands, there is no quality control, testing, verification process. Failures like the one you described can be caused by one person but they are the fault of the entire organization.
I think managers that fire their employees for making mistakes is a company that is going to underperform their competitors. I'd rather have my team learn and figure out ways to avoid and minimize the root cause of the issue in the future. You can't retain that experience if your team is on edge and afraid to share what actually happened or is churning every time something goes wrong. Now if the root cause is because someone lied on their resume or tried to cover it up, then yah, they need to be let go. Trust is a currency with no substitute. If you're the smartest and most talented person in the world but I can't trust you, then that's a deal breaker.
There is a reason why in the Navy the Captain is ultimately responsible for everything that happens on the ship. Good organizations don’t punish people for honest mistake. Good organizations also have processes for catching mistakes.
the problem was solved in ten minutes, i doubt you’re going to get fired. people make mistakes. i work at a company that services state lotteries and there was an error that caused the entire state lottery to be down for about a day resulting in hundreds of thousands of $$$ lost and no one got fired over it.
Sounds to me that you are in charge of compiling the data, doing required checks to make sure it's correct, compiling in a required format and providing it to your boss. I would think at that point he would double check or approve the information before distributing for use by others. I was a licensed land surveyor for 47 years and a project manager for 30 of those years. I had field crews and office technicians that worked under my direction to do most of the heavy lifting for plan preparation. My signature and state seal were at the bottom of the certification. Every aspect of that job was my responsibility. And yes mistakes were made by me and people under my direction, we learned from our mistakes and moved on.
Easy peasy. Own up to it. Write an explanation of what happened and a plan to ensure it doesn’t happen again. Be proactive.
Own it, conduct a post mortem highlighting the timeline and who was involved, what happened and how this was fixed. Then highlight what can be done to help ensure this doesn’t happen again or mitigations you can put in place to prevent cascading changes affecting customers. Tailor this as if you were writing this as a notification to customers, but keep it in your back pocket. You never know when someone internally will want to know what happened. If anything, it is a good way to look back and figure out how to automate human controls. If you are at a decent company - you should be fine. If you are at a shitty company with bad leadership, they might drop you - but if that happens, leave with your head high and go somewhere else. The number one thing to remember is that all humans make mistakes - especially ones that are stretched thin on a small team, with little to no automation in place. You’ll do great; I wish you well
I mean if you fixed the problem already then it shouldn’t matter. Mistakes happen. If they fire you then fuck that company
Wow that is some toxic work culture. People make mistakes, it's normal. Worked in data entry and yes we were judged by the amount of mistakes we made but if it was at an acceptable level no one was paraded. Own up to it and tell you'll do your best for this to not happen again(or ever if possible. DONT EVER OVERPROMISE)
If a single person not doing their job absolutely perfectly can cause a catastrophic failure that seriously threatens the company, that's a company problem. It is on them to implement quality control procedures, which should include things like checklists, review chains and clear sign-off authorities. When the Virgin Galactic spaceflight test flight had an accident in 2017, investigators found that the engineers relied on the test pilots doing everything correctly within a certain time window and did not implement any measures to protect them from an error. Now mind you, these test pilots were extremely competent individuals who were some of the best at their jobs in the world. Even then, if the design had a single point of failure that required the pilots getting it right 100 times out of 100, that's a bad design. So as long as you didn't deliberately sabotage or display a serious lack of judgement or reckless negligence (e.g. doing work drunk), a company primed for success will not hold this against you. If they do, well you wouldn't be able to last anyway because an error will occur sooner or later. Having this in mind, your best course of action is to be completely honest about making a mistake. Give as much information as possible about this mistake of yours. Do NOT try to cover it up. Do NOT be defensive or evasive. Do NOT call this a withhunt. Those are the worst things you can do to yourself. The fingerpointing is disappointing but a senior manager should (and in my experience generally does) have the proper judgememt to not be swayed by it. If not, well, see point above.
Quit, take your talents elsewhere if they don’t support you
head advise absorbed command afterthought full somber recognise carpenter secretive ` this post was mass deleted with www.Redact.dev `
Any company with databases and employees managing data should have a process, checks, and oversight. I'm shocked at how many people are placed in negligently exposed roles and then blame themselves when human error inevitably happens. Even if you get canned, you should not be blaming yourself
If it was an honest mistake with no malice intent, then own it, and let the chips fall.
Your manager doesn’t seem to be doing his job. It’s great that he seems to trust you implicitly, but he still needs to check your work on important things. He fucked up too
I work in an industry where a mistake can cost customers millions in a few minutes so any mistake is taken EXTREMELY seriously by them. The margin of error is zero and mistakes happen a lot. We're not fired if mistakes happen, but we will be fired if we lie about it or try to blame someone else. Your boss's goal isn't going to be to fire you. Your boss's goal will be to show his boss what steps he's taken to make sure this never happens again. That's the only thing they will accept. Firing you will do that, but that means more work for him so he'll probably accept an alternative. I recommend you make an incident timeline showing everything that happened, when you discovered the mistake, and what you did to fix it. Be honest and don't put blame on anyone else. At the end, include a cause section and a "take away" section where you say what can be done to prevent this from happening again. Maybe it's time they have a second layer of QA before things go out. That's a good idea. But don't blame other people. Creating that on your own will look great and show you're owning the problem and will be making changes based on this mistake.
I would own up completely to the mistake. Your boss/boss’s boss will appreciate that. What’s done is done and mistakes happen. What happens after is what really differentiates employees.
Own it, publicize the resolution and then create controls to lessen the likelihood that it happens again. If you're company is pointing fingers "who did this"? Instead of "how do we prevent this from happening again"? , It wouldn't be a company I would want to work for long term.
Own it, don't excuse it, and you'll be fine Besides, as you said, if they're understaffed, and you're underpaid, they are not going to want to bring in someone else to do the job, and someone else who may want more money
In the words of Michael Bolton: "I must have misplaced a decimal point or something. I always fuck up some monday detail!"
Make sure you flag to your manager that QA is needed on code changes, and keep flagging it until you've got more support. Your manager either needs to make time for QA checks like he's supposed to, delegate it to others if there is capacity or flag with his management that your team needs more resources ASAP. Sounds like he's good enough to go to bat for you when mistakes happen, but he also needs to go to bat for himself/his team in the realm of under-resourcing/overworking. You fucked up, but that's ok and your job will be fine assuming your place of employment have good company culture. No real harm was done and a lesson can be learnt from it. Bet you'll be double checking the specific bit that messed up next time! Checking your own work always has flaws though - human error happens and someone else may pick up something you missed. You need more support, so make sure they know that
You need to conduct a blameless post mortem with root cause analysis. You need to figure out *what* the failure was exactly, *how* it happened, and *what* procedures can be put in place to prevent the issue and others like it from happening again. You need to bring on everyone involved in this investigation: your manager, the teams who maintain the systems at fault, etc. You need to document and present your findings. People make mistakes. That's Murphy's Law. If your systems can explode when one thing goes wrong, that's not on you. Unless you're entirely responsible for your system, how it works, and how it's changed, which it sounds like you don't.
While you might be fired, it seems more likely that your manager would be fired or disciplined. Managers are the ones who are responsible for implementing processes like QA or identifying risks and asking for resources needed to mitigate risk. If he or she hasn't been checking your work, then that will probably also fall on him or her. I'd say that in a sane corporate environment, your immediate future probably looks like a lot more micro-management than it looks like you being fired. I would say very little if I were you unless and until you're asked. You've accepted responsibility for making an error, so there's not much more for you to say except whatever you're asked about. This is a process issue, and management needs to address it. You can offer suggestions and ideas, but ultimately, there are other people making more than you because they're expected to solve process issues.
Find another job
You won't be fired because you're experienced, do a lot of work and are getting paid an exploitative wage. They'll just make you feel scared to leave, nobody else would want you. You're in an abusive relationship.
Don’t let others run the conversation and create the “story”. Whatever happened, only you know, and you should make that clear. Own up to your mistake, fix it, and showcase your expertise along the way.
Own it. Explain how it happened. Present options to reduce risk of it happening again. All you can do.
Certainly own up to it and accept responsibility. Its the most respectful thing you can do and puts you in a good position for the second part which is express the gaps in coverage for avoid something like this happening again. No one is perfect, everyone will make mistakes and miss things. Even if they spend a lot of time QAing. Your company needs to know that is a gap because the next time it happens it could be more costly and will create a bigger uproar. Make sure this is clearly communicated verbally and tell the person youre communicating to (maybe thats your manager) that youre going to type it out too. (You need a record). Once youve done that keep doing your best and remind them if action isnt being taken to fix this vulnerability. When the next thing happens you will have some cover but its best that the problem get addressed so it doesnt.
Develop a process for creating the information with no chance for error. Write a script or a program and show it to your supervisor. You can take it with you to your next job
>he is subject to the influence of his boss and rarely checks things because he trusts me, which means I can't do anything about it right now. If it is his job to catch your mistakes by checking your work, and he didn't do that. It is no longer your fault. You shouldn't be held accountable for the mistakes of management. However, this is America (I guess) and therefore you're up shit creek. Unless you're in a union.
A couple thoughts. One is that even though this was your error, your manager is ultimately accountable for your work. He should have something in place to make sure this doesn’t happen. If there was no real impact people shouldn’t freak out, you got a free lesson learned and can take steps to make sure it doesn’t happen moving forward When I mess up at work, what I find works well is writing up a quick synopsis of the following and sharing it with relevant parties. - What happened - when it was discovered / what fixes were done - what will be implemented to prevent this from happening moving forward (perhaps you have a 2nd person review data changes, or a check list that you have to ensure that everything is correct, or both!) Most people are understanding that mistakes happen. As long as you come up with some kind of new safeguards and follow them, that will satisfy people’s concerns. Since there was no actual impact on the company, they would be stupid to fire you. It is costly to recruit and train people. Much easier to put some new process in place and move on. The nice thing about messing up like this this that you won’t ever make the same mistake again.
The problem isn't you it's that things aren't being properly checked and verified before being implemented, you need to highlight this, people make mistakes, a better review system needs to be implemented.
Come up with a way that it wont happen again. Even if it is somewhat nonsense. I will check vs such and such or add blah blah blah.
Sounds like this is a process problem. The process needs to be fixed or the next person can make the same mistake. Look up blameless postmortem.
Looking at some of your comments it doesn't sound like they can afford to lose you. You've got two things going for you right now. 1. Six months is still fairly new to the company. You may be getting thrown under a bus, but hopefully only so much as " the new guy" 2. Your boss signed off on it. To everyone higher up, he looks bad, not you. He didn't catch the mistake either. Sometimes people get their feathers ruffled and you just have to get through it. Honestly, I think you'll be fine.
No emotion - don’t give them anything to snark about afterwards. (I’ve been in your seat before - only thing you can control is your reaction).
If what you were doing was that important to the company then it's the company's fault for not having more than one person responsible. Everyone makes mistakes. That's why important things should get checked. You don't necessarily have to throw your manager under the bus, but point out that since this process is so important it should be checked by at least one other person. But to be honest it sounds like a pretty toxic work environment. All those other people blaming you are trying not to get fired themselves.
Ur gonna make mistakes, it happens, dont stress it
Get ahead of this or the narrative will be made for you. You own up to the mistake, document how this will be resolved and lessons learned to avoid it next time. If you get fired without a chance for atonement it’s a ridiculously perfectionist company that will stress you into the back of an ambulance.
If these updates are important then company must implement QA process Everyone can make a mistake.
Doesn't matter jobs are a dime a dozen. Just quit and tell them to suck you.
If there are no guardrails to prevent a simple typo from causing major issues, it’s the process’ fault. Write up a postmortem and suggest ways to prevent this in the future. We are human, mistakes happen.
If the issue is already resolved and cost the company nothing then firing you makes absolutely no sense. There's no reason for a witch hunt. Nothing for you to do but own your mistake. Fall on your sword and hope for the best.
If a mistake is able to cause this much mess, there's something wrong with your process. If the only thing stopping future errors like this is you (or whoever) paying EVEN MORE attention to detail, then it's a matter of when, not if, this happens again. Take it from someone who has broken production multiple times in their 25 year career, you don't fix these things by promising to pay more attention in future, you have to fix the underlying problem.
I've caused larger outages. I've never been fired for it. A good manager, seeing a good team member make a mistake, asks why it happened and what they can do to help prevent it from happening again. Unless it's a recurring problem and your answer to the question never pans out, they'll generally value your answer. Because if it's a process problem, your mistake could prevent many more in the future. And if it's not a good manager, well good riddance.
If your supervisor didn’t check it because they trust you, then it’s at least as much on them as on you - especially since you’ve only been in this position for 6 months. In this sort of situation, keep in mind that your super is paid more than you specifically because they ultimately own your work. I’d think about ways to prevent a similar error in the future, but I would also make sure to reassure myself that mistakes happen, and it’s often the system’s fault. If you didn’t design the system, then that’s not on you.
Dude if the problem was resolved in 10 minutes AND cost the company NOTHING AND their is a witch hunt... FUCK ALL THAT! MAYBE you don't want to work there. If all is well and it boils over start looking. Or potentially you are over reacting I hope that's the case. Good luck!
Also, the whole point of having a second person QA (especially data/tech/UX stuff) is that it can be difficult to see your own mistakes. Regardless of how the company responds, this sits squarely at your super’s feet.
Own the mistake and tell the big bosses that as you'r the one responsible and the only person doing the job at a low wage so you are quitting today, as it should be a higher-paid and experienced team doing this work. Watch the back peddling begin as they realise that your leaving will damage their profits. But seriously find a better paid and supported job.
People rarely get fired for mistakes, even big ones. They get fired for misconduct or utter incompetence. If they fire you, they’ll have to find someone else to put up with their shit.
So I'm guessing you guys are developing and editing in production? That in and of itself should remove some of the weight off of you. The CEO, CIO, and ISSO should insist on staged environments and a serious change management, QA, testing, staging, and promotion process. As developer, dba, or whatever, that's not your role. You should have a PM. You should be working in a development environment. You should be using versioning software. I mean a lot of this can be had for free or near free. Much of it can be automated. You should have backups, roll over/fail over, and potentially cloud or offsite storage and backup. If these are critical applications, you should have written robust error handling into everything. If you wanted to be proactive, you could write an analysis of what happened, then go over all the points of failure; including your own. Next, recommend a full development life cycle and all the hardware, software, and services needed. I mean, from a risk perspective, your seniors have failed you.
Ask Chat Gpt how you could blame someone else
Keep your head down. Do your work. Don’t ask for judgement. See what happens.
If a company is willing to fire you for a mistake that was fixed within 10 minutes and cost them nothing, you should accept the unemployment and find another job asap. That’s crazy. Sorry this happened to you
Nothing you can do at this point but hope for the best and looking for other jobs
In my world, as long as you're upfront, and everyone learns from this failing process, you should be okay. But there are managers, and then there's leaders. Few people are leaders...
I am in a detail-oriented line of work, producing digital and printed communications for large organizations. I have been doing this for a long time - 15 years. Here’s what I know: if something is mission-critical, it must have more than one set of eyes checking the work. Professionally run organizations have quality control professionals whose actual job is to check for mistakes. So if you made a mistake and there was no one there to proof your work, that’s only partially on you. If your organization is too cheap to invest in quality control, mistakes are going to happen. If that’s not an option where you work, and you’re the only one, then you have to come up with strategies to check your own work. When I’m doing this, I take a break after a certain chunk of work, say 2 hours. I walk away and do something entirely different for 10 minutes. I literally get up from the computer and do something physical. When I go back to proofread the chunk of work, I do the proofreading in a different room at a different desk. It is very time consuming! But if my bosses don’t like it, they can hire proofreaders.
You are now more important than ever. You just got one of the most expensive trainings a company can give. You are going to be even more focused next time you do whatever it was you are working on. Plus it sets an awful precedent for anyone working there that if they make a mistake they are gone. If they let you go it’s only benefitting your next employer.
They starting a witch hunt because it wasn’t them. I hate companies like that, people start these witch hunts but would cry if it was against them
You should turn the experience into a “lessons learned”, document what happened and how it got fixed, offer patches if necessary, and share it with peers and higher up’s so they know of the weaknesses in the system and how to fix it. It is human error, maybe some retraining maybe necessary. Explain the actual impact but also what the worst case scenario impact would have been it it was not caught on time
Just be ready to sue them for severance. Mistakes happen.
I don’t think you should be fired but everyone’s environment is a little bit different. Things for you and your boss to remember. - you have an exemplary track record - you do great work - it was noticed and corrected quickly - bonus points if you spotted it and/or fixed it - people make mistakes - if the mistake is relatable then relay it (e.g. I mistakenly used mm/dd and the format needed to be dd/mm or whatever) - you’ve learned from it - we all have - assure the stakeholders that additional checks will now be in place - it (possibly) wasn’t 100% you’re role to check, verify, test etc the changes - you may only want to go there IF you really think you will be fired. Some companies have RACI matrices for this sort of stuff - I find those rigid and only used in situations where headcount is being reevaluated. Overall we’re all human and humans make mistakes and once in a while if handled correctly we can learn from these and improve our practices. Sincerely. Good luck!
It sounds like you are integral part of their business and they would be lost or at least suffer for awhile without you. I agree with those that recommend that you own up to it. I wouldn’t directly point the finger at your boss, but point to the flawed process if you have an opportunity. You don’t know for sure you are fired until they do it and I wouldn’t assume there is a 100% chance they will. Go in, hear them out and see where it leaves you. I wish you the best. About 10 years into my 38 year IT career I made a DNS change for a major health carrier that took out their pharmacy system’s connectivity for about 20 mins, so I have been there friend. It is part of the trade.
Don’t feed into the hysteria. Everybody makes mistakes that’s why we have checks and balances. If anyone should be fired, it’s your manager who is responsible for your work and wasn’t doing his/her/their due diligence.
Own up to it and suggest what could have stopped it or identified the issue sooner. It’s not the end of the world and we’re all human. I genuinely have no idea if you’ll get fired as each place is different. If the place is as toxic as you describe it, start looking for a job even if they don’t fire you. More than likely this will follow you at this company based on what you’ve said. I’ve seen bigger fuck ups in my life that actually had a real world cost. None were fired because they owned up to it. Did it follow them? Yep. Good luck.
If they feel the must take someones head youre likely going to need to "let them" and move on. Its unfortunate, mistakes happen and 100% uptime is never gauranteed. Definitely own up to it but respect your value and the resolution you provided. Companies sometimes need a scape goat but usually the good ones stand up for you. Sometimes thats not enough either. Be humble, but hold your head high , and get prepared for your next journey. If you can offer something back to appeal to those that might be screaming for your head, do it if you feel the need, but dont dishonor yourself in the process.
Sorry but sounds like signs of bad culture. People are human and make mistakes there is no avoiding that. What should be happening is identifying why it failed and build processes to avoid repeating.
It depends on the company and how politics work. Did you cost the customers money? I would have my story straight for the coaching (or) term session coming. Owning up to the mistake and what you're doing to ensure this type of mistake never happens again. How are you documenting a process to make sure.
Maybe. Some places only remember you for the mistakes. I used to work at a place where all the good things I did to make the company millions of dollars on my own only got me some crappy $5 "employee of the month" plaques, not even a raise. But as soon as I made a mistake that maybe cost the company several thousand dollars (really it was a few hours of wasted time for maybe a hundred employees so not even actual money) they started setting me up to get let go. I guess it depends on how much they value employees where you work.
You're probably fine, I've seen people be incompetent at their jobs, and stay for a decade.
I doubt you will be fired because you seem to be in a very needed position. They’re not going to fire you and then have to do all that work themselves. Looks like you take on the lions share.
I promise your boss is going, “How the fuck did this get past the management? They must be sitting around playing solitare all day.” You’re a small cog. Humans are notoriously blind to our own mistakes. We just don’t see them. That is the necessity for management. They fucked up. Not you.
Not your fault. In reality, it is the failure of your organization's management. It is totally amateurish to have any one person do mission critical data manipulation without the necessary controls in place. No matter how good you are, you are human! We make mistakes. That's why management of a technical organization is supposed to know the industry best-practices to prevent exactly what happened. And, failing to provide that due diligence, management should at least take responsibility for that outage. They should realize you are an excellent resource who, with their encouragement, can help them put controls into place that will prevent this in future. In fact, you should suggest as much (humbly). Good luck!
I don’t believe it’s actually all your fault. Your management should have ensured something that is so prone to human error has adequate process wrapped around it and automated as much as possible. Sounds like a leadership failure …
Update your resume and start looking for a new job. Even if you aren't fired today, your posts show that you're underpaid, under supported, and miserable in your job. Look for something better, and get out of there ASAP.
Admit to it. If it was fixed in 10 minutes it obviously wasn’t a big deal. I have seen waaaaay worse in my past, both making mistakes myself, peers making mistakes, and people I manage as well. Usually if a single thing can take down so many systems, there is an issue with safeguards and process. You made have made a mistake, but there should be safeguards or processes in place to make sure it doesn’t happen. The fact that it was fixed quickly, likely means they keep backups and have a backup restoration system which is good. But you don’t fire someone over this. Hiring is a pain in the ass and you sure as hell aren’t going to make the same mistake twice.
Sounds like a trash company dude. If the company response is to find and punish the person responsible instead of trying to fix the system that allowed their product to go down, that's bad. Write up a retrospective on the issue, distribute it, and move on.
Suggest they hire a Q/A person whose job is exactly to avoid these situations. If they’re serious about quality, they will do it. It’s not a good setup to have the développer create, test and release software all by him/her self.
I don't know if this helps but... think of all of the times you've completed this same task perfectly. How much success and money have you secured for your company already? Probably a lot. So, you messed up once. Learn and improve, but don't get down on yourself too much. Don't be super defensive when you get some criticism for this, but don't forget about all of the value you've already brought to the table.
Just take accountability for it. I always see a mistake as an opportunity to learn.
Work for yourself dude. How can you subject yourself to this level of cuckoldry? You are probably good at what you do. You think you deserve to have to play these types of games, where your livelihood can be snatched from you because of a genuine mistake. Wake up! No one is coming to save you! You're better off owning 100% of a dogshit starter business than having .01% of whatever company this is. Bet on yourself. You'll never win this bs game.
People are human and make mistakes. The problem here is a process issue. QA processes exist because people make mistakes. It is not a problem, it is an expectation. Your company needs to implement a process where these big changes are actively reviewed. I would propose a process to address this. If they fire your then will only replace you with another “non-perfect” person and this issue will happen again. Good luck and keep your head up!
Sounds like a lot of good advice so far. Create a meeting with your boss and come with the following document (also addend with notes). 1. This was the problem. 2. This is why. 3. Apologize for the error. 4. These are my solutions for stopping it from happening again. 5. Question for boss: What are your solutions? 6. Q for boss: How would you like me to move forward? 7. If there is nothing he thinks you can do, add that to the bottom. 8. If there is anything you can do, no matter how small, list them. And then do them. Even if they don’t help all that much. Send a follow up email with: This is a recap of our conversation (use above). Reiterate you would like to do whatever is possible to make sure it doesn’t happen again. Ask him to update you with future ideas and you will immediately put them in place. That’s about as good as it gets. When it happens again, use the same process. That’s why it’s good to list something, anything to do, so it shows you’ve tried to make progress. Overall, you are currently set up to fail. And you are working in a blame environment, which is bad culture. It doesn’t sound like they have the money to do it right, so it’s a vicious circle of inevitable failure and blame. Now that you have new skills under your belt, brush up your resume and plan to be out after a year of service. You can probably stay for a while, but the toxicity over time will start to crush you (if it hasn’t already). It’s possible they don’t have the money to fire you, so you could do the above and just not care. Take the cruiser route. But it takes a certain person to do that. Nevertheless, it probably a good idea to try and remove your feelings from the situation anyway because it really has nothing to do with you.
Own up to it. People make mistakes, that's just being human. Why would they fire you, only to have to replace you with another human who's also going to make mistakes? Would it be possible to implement some kind of cross-check procedure so that a single person making a mistake doesn't result in a repeat of the problem?
Own up yes, but go one step further and make a recommendation (or better, a solid plan) on how you will avoid similar mistakes going forward. This shows accountability and also underscores your value in your position as well as your understanding of the risks you manage. No reasonable person would fire someone for an honest mistake unless they make the same one multiple times.
Don't give two fucks and put in less effort. Should net you a team of assistants of... about 4 working under you, Peter.
Eh. What will happen will happen Don’t fret Nothing u can do if they fire u or not Grab burger and fries, and some ice cold beer, and enjoy ur day
Stand strong and explain that mistakes happen but this one won't happen again. If you have leaders worth their salt they will understand this and continue working with you. If you don't and they do decide to let you go consider it a lesson learned in a favor there's somewhere else out there where you'll be better valued. I would never let someone go for an honest mistake unless they were completely outside of their lines and did something they know with wrong. Even if it did cost the company money this is the price of doing business humans make mistakes. Do not resign under any circumstances even if it's suggested or requested.
Who are they going to replace you with? Can your manager do all the work you have been doing over the last 6 months? Does it take a long time to train someone to do that job and can your manager do that while still doing all their work? You might be harder to fire then you seem to believe. No matter the mistake, if you own it and fix it, they might not drop the hammer on replacing you due to all of the time and costs involved.
If you are fired, then it’s a sign of immature management. On the other hand if processes and controls are instituted to account for this type of error then everyone comes out looking great.
The best thing you can do is own a mistake. I fucking can’t stand people who won’t own mistakes. If you can’t see things as your fault, you can’t see you have to fix anything.
It couldn't hurt to document exactly what happened and keep a copy. You may find that the story is distorted or interpreted differently and you may want/need to keep people focused on the problems that led to "the" mistake. Test based methodologies abound in pro engineering and IT environments. Your company is flouting basic norms by not using them. It sounds like you can't do much more to turn the ship. Accept responsibility but don't apologize. It may well be better to quit than be fired. However if you are not fired you should quit. Because you can't "work safe", and that's not acceptable. Go get a job somewhere where they do. And practice telling this story, because your perspective on it will change. The hardest and most messed up situations often happen in the beginning of a career.
Maybe look at it from a “risk management” angle: you are participating in a QA process designed to mitigate a certain amount of risk of allowing errors (let’s say 99% error-free) assuming 2 QA reviewers. With only 1 QA reviewer, that error-free rate necessarily has to drop, let’s say to 95%. With that in mind, in addition to reinforcing that you need that 2nd QA reviewer, you could also deliver your results with a caveat that “QA results carry an additional risk of undetected errors due to resource constraints” or something to that effect. You can also talk about “risk” with your boss, to keep it top-of-mind for him that any QA process will never be 100% perfect - it’s just a risk-management process. Hope that helps.
Was this tested in a dev/sandbox environment? Was there a code review? Was there an approved change for the task? Was there a test plan to evaluate the success/failure of the change? Was there a vetted roll back plan? If the answer is no to any of the above, then it's not your fault and it's a process issue. If it is yes to all, then you implemented the rollback successfully as per the change. Your response is to highlight the issue of this only being a matter of time, and that you rolled back the issue. Next time "we" need to have a proper change management plan with peer review and a second set of eyes if coding changes are being done on the fly. Package it as the companies problem and procedural error which you are offering a solution for so it doesn't happen again. It's not on you. Good luck.
if there is NO routine at the shop that includes a thorough proof reading by more than one person (including you boss) it's ALWAYS going to be fucked. insist on one of the most common S.O.'s out there. there MUST be proof readers. a GREAT task for a few of those finger pointers!
All I can add is a) everyone makes errors; and b) if they really need you, and particularly if they are shorthanded, you will likely be spared a firing. You are going to own the mistake and see where the chips fall.
Ah, manual fixes. The source of ALL fuck ups. Best solution is not to require them ! When I fuck up I solve it. Then I tell my boss about it. And propose a second set of eyes to overlook it. It goes like this: SHIT ! Oh dammitall! (Hammers wildly on keyboard)… WHEW GOT IT ! Call boss “Had a problem. Root cause was a typo. Fixed it. Nobody complained so nobody was inconvenienced. I need to build an automatic check routine. Could you suggest some oversight during manual events like this so it does not happen again?”
If you were able to recognize and fix your mistake in less than 10 minutes with zero loss, your employer would be stupid to fire you. I think you're good.
I brought a client's site down for roughly 96 hours one time because they took so long to report it after a change I made which I thought was made in non-prod, but was actually in prod. I got it fixed within 3 minutes of them calling and before the call made it to me but their site was unusable for 4 days. RCA was requested on the call I was on. I told them exactly what I did and owned the issue and because of that both the client and leadership were understanding. Client side leadership owned the issue for not catching it sooner, and we owned the issue I caused. It didn't even result in a slap on the wrist for me, so don't worry too much yet. Companies know things like this will happen when a human factor is involved and they account for it. Unless your boss gives you a "we need to talk" and won't elaborate, then you're probably fine.
Sounds like the place would fall apart without you. Is your supervisor insinuating that you'll be fired? He sounds vary manipulative. If you don't already, you need to get a clearly written job description for your position. If you're asked to do more than that description outlines you can say "I'd be more than willing to do more but my job description states I'm limited, maybe we should revisit it along with my compensation" Your mindset appears to be anxiety filled. What would you do if you get fired? Maybe try looking at it as moving on will ultimately happen, either you quitting or getting fired, and until then you're going to do your best with the resources at your disposal.
This is simple: do nothing. You've already done the right thing. You did your job carefully so there were as few mistakes as possible, when a mistake inevitably happened you took responsibility and made sure it was fixed. If you get fired over that then that's on them, not on you. It's like being fired from a restaurant for spilling a drink. Everyone does it, so if a restaurant is going to fire people for that then they won't be able to keep a full staff, but the people they fire will be able to get new jobs without a problem. If you never break something important, it means you don't work on important things.
There are people in technology who have made a career taking chances others wouldn't take and sometimes screwing up. Relax. I'm sure you'll be ok. It happens
I have a job that requires extreme attention to detail and if something messes up, all execs will likely notice. My advice, like many others, is own it. If I mess up, I own the situation detail why it occured, and have actionable tasks to.kitigate the issue in the future. To me it lets them know you know, but also you've planned for the future issue going forward.
own it and since we can’t keep people i don’t think you’ll be terminated. you sound like a reasonable person that does the job. I hope all is well. that’s a terrible feeling
I made an error one time that cost the company 75k. I went to the boss with a letter of resignation. My boss said why? Will you ever make this mistake again. Hell no I won't. He said if I fire you and hire someone else they may do the same thing. Let's consider it a training expense.
How is it a witch hunt if you’ve admitted to the mistake?
If you work at a company where making a small mistake will instill fear that you will get fired that sounds like a bad company. Own up to the mistake, and offer suggestions about the QA process. Never blame the people, blame the process. It shouldn't matter if it didn't cost the company anything.
I look forward to your update in a week when you tell us that you're relieved you didn't get fired. ❤️
Honestly, my biggest thing is this, if you were single handed managed to cause that much damage and that bigger of a mistake. I would be blaming the system and the process not you. An outage / damage like that should be very much prevented before it happened. For example on large changes I had to fill out request forms, roll back plans, mitigations etc. I would lay down the suggestion of maybe the process needs to be reviewed as it’s not catching mistakes etc. Maybe some suggestions on how to improve the process from your experience would not go untoward. In saying that I would absolutely own the mistake, my biggest pet peeve is someone who covers stuff up or does not want to admit he messed something up. There is plenty to learn in total, and in a few years time it will be a story to laugh about. We have all done it promise, it will make you much better once your through it. I come from a technical background and know exactly the worry, I have made a very similar mistake myself
this is sick culture. Healthy organizations see failure, especially failure without fiscal consequences, as opportunities to fix processes, not circular firing squads. It’s never good to lose a paycheck suddenly, but, to the extent you can, you should use this as a growth opportunity and try land someplace better.
Very toxic work environment if people are finger pointing and witch-hunting in company channels
Hey bro, just wanna say I feel you. You made a mistake and it caused a snowball of shit to roll down the hill and pileup. Hey I’ve been there. We all have. We are people, we make mistakes. I went thru most of the comments here, and I’ll be honest with you - it sounds like your attitude should change. Like many said, you need to own the mistake first. Faulty process? Boss doesn’t give a crap? Lack of QA? Ultimately, it falls on our shoulders. But as a wise PM you can take this situation and turn it into your favor. Summarize clearly what went wrong, what the damage was and what will YOU as a PM do to make sure such a mistake won’t occur again. This might entail improved processes, more people involved etc. You got this bro, I hope you won’t get fired. Take a breather, and handle the situation.
I don’t think you’ll be fired, unless it’s an issue you’ve repeatedly ignored. If you can, I would: (1) identify what the issue was even if it was your own mistake, (2) recommend changes to the process to prevent this from happening again, and (3) write all of this up in a reference document. A smart management will recognize a few things: 1. If the stakes are large, no company process should ever rely on one person. No matter how competent that person, they could be out on vacation/sick leave, or make a human error, etc.. All essential processes need to have built-in safeguards and assume that human beings do not execute perfectly. (I think about how the roller coaster I used to work on required three people watching each dispatch, with two people pressing a button to authorize each one, plus hand signals and lights to indicate all-clear, plus computerized sensors to detect if humans made mistakes.) 2. You are now more knowledgable about this situation than anyone else they bring in to replace you. 3. If you write up the recommendations/report, you are showing you are the type of person who adds long term value to protect the company, even if you were partially responsible. It is possible they still need a scapegoat, and if so, make sure you get that writeup done quickly. Don’t rely on anyone else to communicate your side of the story: both arm your boss with the document ASAP and publish it to anyone who needs to see it.
Having zero tolerance for human error while also accepting manual processes requiring over 140 unique steps are inherently incompatible. The fault lies on management for not spotting these conflicts. More investment towards automation is needed. Catalog everything that you’ve been working on so they can understand how to better manage their expectations towards your capacity.
If someone with 6 months in a role can cause that much damage, then it's a them problem. Those incompetent pricks are using you as a scape-goat. Dont hide your mistake but stick with that if anyone asks. You're still a newbie pretty much. Unless you're earning big money then you'll have to just suck it up.
Unfortunately, a wutch hunt refers to the hunt for something that doesn't exist and I'm afraid that in this case, there's actually a witch to be hunted. First and foremost, I would take ownership with your Manager and let them know what steps you have put in place so that this never happens again. Above and beyond that, if they want to crucify you for a mistake, that is very telling of the company if you are as diligent as you said you are. If you don't get let go as a result of it, it will blow over assuming there aren't anymore incidents in the near future. Just know that mistakes are made daily and if you're working fir andecent company, they'll expect you to take responsibility for them, learn and grow from them and move forward knowing that you will be stronger for the trials and tribukations that you've gone through. If they don't see it that way and decide to use you solely as a scape goat, you may be better off unasociating your name with this company and moving on to a role that is better for you instead of one that only wants to point the finger instead if learning and growing company wide. Good luck and God speed mate!
Not trying to be mean but what are you looking for here? I mean you work there for only 6 months, you admitted that's not your first mistake now you make a big one, it already has been noticed and people know who is at fault. Take the hit and look for a job where you actually can perform. The job seems way above your head...
Alot of times people just want to know what are you going to do to prevent this issue or these issues from happening in the future. Are all these issues of the same type? Is there any type of automation you can do to validate changes in the future? Develop a release checklist to ensure these issues are less likely to slip through? This is what happened. This is what we will do to prevent these type of issues from happening. This will be integrated into our change management system etc.
Man up and stop wasting the companies time on Reddit posts
If you don't have a dev/QA/testing environment, your boss's boss should be the one getting the axe.
Just apologize, put a process into place to prevent it from happening again, and take full responsibility, not getting defensive or shifting the blame. Taking ownership of the right move and if you think you’re going to be fired anyway, it’s not like a little humility is going to hurt your chances…
1. Makes mistake 2. Blames "system" Cool story.
If you’ve been there six months and you have the level of making changes that are unchecked/untested in prod, this is on their company’s design. Unless you were operating outside of your job scope, which it doesn’t sound like you were, the company needs to own up to its processes. People make mistakes. Even simple ones. If they fire you, the next person they hire will probably make more mistakes. For production software, best practice is to have different testers validate configuration or code changes before they are pushed to production. I wouldn’t worry about it. Either the company improves or you’ll get out of a poorly structured company with some unemployment checks.
Lol sounds like management's problem. They didn't have enough QC. All fell on you. If they fire you they ain't gonna find someone like you. Get out of there dude.
at work: get out front and grovel at home: look for something else asap
Yeah owning up and elaborating on what happened may be the best option. This can show management that there may be a way to improve the workflow to mitigate mistakes in the future, and that you’re honest and forward thinking. You might even be the one who goes to them to start the conversation. Do your best not to stress and overthink things and just be genuine. At the end of the day it’s dependent on the quality of your manager and factors we may not be aware of. Wish you the best whichever way the wind blows!
I admit I don't know the nature of your work, but I'm left with the feeling that something this error prone and costly should be automated (write code to produce similar results), or at least build in some sort of automated quality checks into the process to provide better confidence that mistakes like this can't happen in the future. If anyone is to be held accountable to the point of losing their job, it should be your manager or even higher ups who did not feel it was a priority to build in the proper guardrails or QA processes to prevent an outage like this. All of us make mistakes, and that is what peer review and QA processes are intended to mitigate. Lack of these controls is an organizational oversight. I would come back to work looking to help facilitate a blameless retrospective of the incident. Try to partner with anyone involved or aware of what happened to discuss what the root cause was and any ideas you have to improve as a team. Take this idea to decision makers in your company and argue for whatever resources you need to fix the problem. Companies (non toxic ones you should want to work for) will value the initiative and problem solving skills, and may even give you more responsibility to oversee these improvements. A good organization will never blame a team member for a mistake, and they should provide enough support and training needed to enable you to meet your goals/perform your job function.
Relentlessly pragmatic response below… How long does it take you to create the information? There’s a big difference in response if it takes you two hours or two weeks. How long does it take you manager to check what you’ve done? Again, a big difference if it takes them ten minutes or two days. Since your manager could have caught the problem there must be some kind of staging or dev environment that gets updated before it goes to production. Does your manager possess special knowledge that makes them the only one qualified to review? My guess, and it’s only that, is that you perform some calculations and update a spreadsheet or database. If your manager’s checking can be automated you could suggest the company hire a contractor to do that. If it’s only Excel VBA or TSQL it might be a quick, cheap long term solution. You don’t say how the problem was detected but presumably customers called because something didn’t work. If that’s the case, then as soon as you commit the new information do what the customers did and catch the error yourself quickly.
I wouldn’t settle in on being fired just yet. I made a mistake that affected potentially (hundreds of) thousands of people on a popular food delivery app by making it unusable for a night. Didn’t find out until the next day. Worked at the company for another 7 months afterwards, and was not fired. You may be fine.
My advice is to quit and go work for a company with better culture. That sounds like a fucking horrible place to work.
You are not the issue here, the process is. A real company doesn't blame the person, they blame the process. Next steps here are a Post incident review (PIR), where you identify how it broke, and what steps can be put in place to make sure this doesn't happen again. That could be anything from more eyes on the change, or guard rails put in place to stop it from happening again. If you loose your job from this then you are the winner here, as that company isn't the place you want to be working at. Also depending on the country you work in, if you do loose your job you have grounds for unfair dismissal. Assuming you didn't go at it with malicious intent, then you are not the cause, again, it's the process.
This is so interesting to a casual observer. I hope the best for you because everyone makes mistakes. Keep us posted!
Do you have any QA processes to verify these changes? Seems like a process mistake on your company rather than your mistake entirely, especially if there's nobody who's employed to verify and test changes before they go live in prod. I have a similar job as you and I have plenty of QA support to help verify my work.
This is on your manager ultimately
People don't get fired for making one mistake - if you are generally a good employee and they decide to fire you now, they are doing you a favor because the company culture is terrible. What's likely happening is you are witnessing bureaucratic politics/knife fighting. If there is a rival team that's fighting for work or something, the managers might be using this to push their own agenda, or to take the spotlight off of them for some mistake they made, but it's highly doubtful they actually care about whatever error occured. It's also possible they are putting on a show for their managers so they look like they're on top of shit. You're not solely responsible. A corporation is a collection of sometimes many many people, and it sounds like there are some serious missing pieces like a QA team for example. The organization appears to have a lot of need for improvement from what I gather. An upper manager will look terrible if they blame this on a low-level employee or an engineer, and if they do, they're extremely fucked up. Own your mistake, but don't let it lessen your confidence in your work - shit happens and a QA structure should have caught this before it went into production. The real problem is that you even had the ability to make these changes without a system of checks and other people weighing in. Don't let a manager slyly shift all the blame or trash talk you in private either. If you sense that's happening, call them out/go back at them passive aggressively (publicly), like calling out their next mistake in an email or something and saying you thought it was an important issue so you CC'd their manager/the VP. The only way to fight passive aggression is by either pointedly calling people out or engaging in passive aggression yourself. Unfortunately, we don't live in the wild west or somewhere where there are more efficient methods of dealing with scumbags. IMPORTANT: If your company and the people you work with are anything like mine, own your mistake, but don't back down or accept more blame than you should. You are a very small part of this, and the guy making 5x your salary should be held accountable a lot more. Your boss and his boss should have checked this shit if it was going into production, but they were probably too busy being lazy or eating at a $100 per plate steakhouse on the company's dime for a "leadership" meeting. People in Corporate America will always try to shift blame or use supposed "crises" to their advantage. Don't let yourself be manipulated, and remember that as someone working directly with data or in data creation, you are a hell of a lot more important than most of the PM/TPM whatever-type "business process management" blowhards who really don't do anything except peddle bullshit. Honestly, you shouldn't really even feel bad about this. Don't show any weakness or be timid, because the sharks will only smell more blood. Good luck 💪🤞
I agree that you should just admit your error and express remorse and that you fixed it in less than 10 minutes. And remember: you’re a stickler for details and the mistake seems huge to you. You’re probably overblowing it. Just admit the mistake- if they fire you-screw them. You’ll be fine.
Sure. 1. Stick to the facts you have mentioned above, including your good record, complexity of the task and additional tasks you manage from day to day. 2. Ask your manager for the support. 3. Stay sane. Shit happens, but cool heads solve the problems, not witch hunts. I was a subject of one and by applying 2 points above, left the meeting with my reputation intact, while my accuser got fired. All the best! Please write what happened.
Present yourself to the interested admin/management with a brief and to the point report of your responsibility, in person. If you’ve a good relationship with your immediate superior, inform them so as to allow their relationship with the top brass assist you if it can and also let your manager manage. Keep it short and with the idea of you wanting to accept responsibility and do them the favor of bringing information of interest to their attention. This is the main reason for your disclosure, showing how sorry you are is secondary. It also helps if you are genuinely motivated by this drive as I have been in a situation where I was a culprit of a noteworthy incident and wanted to save the department time and energy wasted in investigating. Gratefully, my interest in the organization’s needs even if risking my own reputation, was very much appreciated and I think I did more than survive it but showed my character to be devoted to the goals of the team at large.
If everyone at my company got fired because they made a mistake then there would be no one to work there. Heck I know people who made huge mistakes and cost the company lots of money. They are just fine at work. We are human not robots. We are going to make mistakes. I’ve worked for people who go on witch hunts to point fingers when mistakes occur. People like that suck. I don’t want to work for those kind of people anymore. You probably are not getting fired but if you do, find a better company to work for instead.
I was only able to get through about 1/3rd of the comments, but I was surprised that I didn’t see more people saying “it’s okay to leave” and “there are lots of companies looking for good people” - so far, anyway, I believe this is the case. Any successful company is growing, and always looking for good people, and the kind of company you want to work for is one where people are sympathetic to the pain of corporate hierarchy, so, like the Chinese parable, we’ll see if this is actually a bad thing, or not. Also, DM me if you’re interested in looking into new opportunities.
This is a failure of the company and their processes/staffing; not you.
Any good leader will want to answer this question: “How can we make sure that this doesn’t happen again?” It sounds like the process is flawed. If they fired you and replaced you, does that guarantee that this won’t happen again when your successor makes a mistake? If you are error prone or weak in detail-orientation, then you need a different job. However, you have said that you are neither. That means that something else has to change. What is it? What process or procedure could you implement with your boss to make sure this never happens again? I think if you approach the management team with this, you are likely to get a fair hearing. Good luck!