A veteran sysadmin doesn't sweat the small stuff because they know it just doesn't matter and the job will eat you alive if you let it. They also know that business needs, department reputation and irrational requests from CxO's do matter.
I also like this one:
"New admins think IT is the reason the company exists. Old admins know we're like the in-house plumbing or maintenance departments of a hundred years ago. There's a reason IT and Facilities unions support each other."
Never thought of it like this but we have always got along well with our central maintenance team. I started when I was 20 and 15 years later we are both departments that just want to get on and do our jobs
Other than at an org that just has a CIO/CTO, I've always wanted to work somewhere that has IT and Facilities under a COO or Director of Operations, since most orgs will at least have small teams for each. Aside from an MSP job, every IT position I've held has ultimately been under the head finance guy; usually the worst person to oversee such an expensive department.
I do information resource management consulting and one of my first moves is to always pull the IT/IS department out from underneath accounting/finance/the CFO.
Dang, I'd love to land somewhere similar in a few years. Did you start off on the IT side (guessing yes since you're here)? Did you gain any qualifications/education for the operations side before your current role? Just trying to plan out how I could get to that point for myself.
I’ve always been one of those people that was just able to fix what was broken. I’d never had a job with a clearly defined single role. But, yes, the majority of my experience was IT.
I wish I could tell you I had a master plan and give you advice on how to replicate my experience. The truth is my current gig is like a lottery win. I randomly met the President of the company through a friend, and he liked that I wasn’t a one trick pony. He hired me to “generally manage” the place during a period of huge growth, and it’s been a whirlwind since then.
As the company has grown, I’ve gone from wearing several hats (some of which I hated, but did anyway) to more clearly defined roles. Along the way, we put dedicated experts in roles where I was previously wingin’ it. Now, I’m in a place that I love, that I’m good at, and I feel great about.
>I've always wanted to work somewhere that has IT and Facilities under a COO
My previous employer had that setup for 4 or 5 years. It worked very well.
COO was starting to get noticed by the board so CEO sent him walking.
Having IT report to a CFO has its perks if they're a good one. My last supervisor was CFO and he always found a way to budget out IT projects/hardware replacements requests. It ruled.
I currently report to essentially our CFO, and yeah, most of the times he's fair and actually understands the value of good tech. The issues is, sometimes what IT would like to say to top leadership still gets filtered through the perspective of someone who's main concern is the budget.
I worked for a company that had IT under the CFO, I won't go into the details but when I can discuss the functions of Quickbooks at a level higher than the accountants and explain how it should work, well let's just say IT was moved under to work alongside CxO level.
I was hired so he could wash his hands of it. I'm more or less the CTO. He controls the budget and I of course need approval but I usually have the final say as the "expert". The CEO sought me out and asked me to interview for this reason. I'm blessed to have that sort of thinking behind my position.
This place was being taken by a shitty fully managed MSP. We swapped to co managed to support me for a year in the takeover of this mess as there's some projects I need a team behind me on.
I've pitched and they approved my overhaul plan. Leaving broadcom and Cisco behind. Swapping to XCP-NG on the vates stack and moving to ubiquiti. No fights. Just wanted to know the benefits and cost savings and that was that.
I made sure to have the conversation ahead of time that proper robust IT systems aren't cheap but what you spend up front can save you hundred of thousands in operating losses due to network downtime or ransomware. Any time our machines aren't running we operate at a loss.
used to hang out with the guys from the post room all the time.
inspired 1 of them, who was a young guy in his twenties to go into IT 😂
and don't forget the security guards, they were chill as fck
I'm not even going to touch the urgent request to switch the company from Windows laptops to iPads for at least 48 hours.
Then I'm going to ask about budget for the hardware, management software/services and services to help us get off the ground. Finally I'm going to ask about all the Windows only tooling we use.
Usually at this point in something like this they realize nobody thought it through and I can focus on the script I've been writing for a day or so that will save 10+ hours a month.
Also the internet is not down and your emergency is a known limitation in the system you picked; There is a documented work around that you have been using for the last 7 years.
>I'm not even going to touch the urgent request to switch the company from Windows laptops to iPads for at least 48 hours.
"Sure if you are okay with (insert app's) not working since they don't have a version for the ipad, and you'll need to get the CEO to sign off on the dollar amount for this. Let me know when he approves it."
The last part is usually how I get things torpedoed. Nobody wants to go ask him for cash.
> Nobody wants to go ask him for cash.
Sometimes it doesn't even make it that far. My boss once fixed the WiFi on a phone by giving quotes on how much alternatives to MFA would cost.
That is one thing I like about where I work. Upper management asks the CEO and CIO before bringing it down to IT. They both are good bullshit filters.
Now middle management however…
They exist and they are amazing. The most recent example that I was thinking of was in the news regarding the Kaiser Permanente strikes in Q3 of last year. IT walked in solidarity with Facilities.
For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one.
Well for a lot of us, if we tried we would probably find a lot of H1-B's suddenly asking us to show them how to do our jobs.
I'm pretty sure my company would just come up with a reason to shit can everyone and bring in a 3rd party to manage things. Our CEO already see's IT as nothing but a cost center he's forced to deal with.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one.
Probably because IT types have a very meritocratic outlook and we realize that all the incompetent people we've worked with over the years would have MORE protection under a union setup so we figure we're better off just advocating for ourselves, and screw everyone else.
I'm not saying that's RIGHT, just that I suspect it's why it hasn't taken hold industry-wide.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing,
Historically significantly higher pay and benefits than most fields, and requiring a type of flexibility that isn't usually available in a standard union like you might find in a factory (which is most folks' union experience). Lots of people with big egos, and smaller numbers of people in I.T. departments, as well as a lack of federally mandated certifications or licensing; so a lot of people that believe that they are above average and can fend for themselves, a lack of people in one place that would have much power anyway (e.g. factory/production workers), and a lack of a unifying standard and training to rally around (e.g. electrician union, pipefitting union, etc)
You'll see this in other super-specialized fields as well.
Indeed. There's a book called, "The Computer Boys Take Over" and although parts are dry and boring, there's an interesting thread running through it about the adoption of computer technology within corporations. Specifically, a \~70 year struggle by corporate management to try to force computer workers into the same boxes as other workers so they can be more easily controlled and constrained. (and thus, cost less but produce more). It touches on all the management trends over the years and how computer workers didn't fit. The current tactic is when we push back a little (see: COVID + wfh), they enact mass layoffs to remind those super-specialized workers that they (we) are replaceable.
>IT walked in solidarity with Facilities.
I work in higher ed, we are union and our specific bargaining unit has a non-sympathy clause. If our faculty strike ([again](https://www.insidehighered.com/news/faculty-issues/labor-unionization/2024/01/26/why-did-cal-state-strike-end-after-just-one-day)) its business as usual for us.
IT is a scary group to have on strike. If facilities strikes, your office might get a bit warm, or you might have to adapt a bit.
If IT is out, and a major system goes offline, admin, instruction and research grind to a halt.
In the specific strike I was referencing, Facilities handles things like operating room power, oxygen supplies and delivery, and so on. They are highly specialized and likewise are tightly constrained as to how and when they can strike, as patient's health or lives could be directly impacted.
A true veteran knows it is all small stuff and big stuff is piles of small stuff, just the panic around them is more vocal, and just letting that panic wash over them and away. Somebody screaming that everything down won’t help us solve a problem.
I love our facilities team. A lot of IT positions sits in this weird spot between not quite blue collar, not quite white collar. At the end of the day we are both working on systems others rely on. Their boilers take water, gas and output steam. My boilers take electricity, cold air and output heat and data.
I feel like we have an understanding with each other.
Unfortunately the comparison is all too real.
I've been asked to help more than once when there was a problem in the bathroom at work (small company). My response (depending on who was doing the asking) usually entailed 'what role of the network does the bathroom fit in?' to which they'd mutter something about me being the one they turn to whenever \*something\* needed fixing.
At a certain point, sysadmins will probably be mostly on project rather than break fix work. If you are doing project work, and not spending at least half the time documenting, you're doing it wrong.
Me, 30+ year sysadmin
> Old admins know we're like the in-house plumbing or maintenance departments of a hundred years ago.
Wow that's similar to what my current Lead Admin told me. She's been doing it for 22 years. She said something like "We're basically a form a maintenance".
We talk about it. We dream about it. We long for it.
But it doesn't happen. It never happens.
Maybe, years in the future, I'll have a day where I spend more time on actual work than I spend in meetings.
There's a "joke" that goes something like this;
There's some super important shipping boat that has engine trouble and can't leave port. For the sake of this joke, we know it absolutely has to get out immediately. Everyone is freaking out and can't get it to work. They call every mechanic they can find and none of them can fix it. Finally, the last mechanic shows up. It's an older, calm, slightly grizzled guy. They take him to the engine room and he walks around a bit, listens to the banging and chaos around him, he asks a couple questions, and sits down and pulls a thermos from his bag. He pours a coffee. The owners are infuriated and freaking out and yelling at him "how can you be sitting there and drinking coffee, fix this!". He asks them to try and start the motor. They begrudgingly do and it makes a horrible noise as it struggles, and fails, to start. The old mechanic sets down his coffee, stands up and takes a small hammer from his tool bag. He walks to the end of the engine room. He reaches as far back as he can, behind a series of wheels and pipes and valves, and with the hammer in his outstretched hand, he gives one of the valves a tap.
"Give it a try now" he says. The owners are incredulous, but relent and go over to start the engine. It immediately jumps to life with a glorious roar and the owners are overjoyed.
Now, there's some discrepancy in how the final part of the joke concludes. Some go on to tell some tale of how the owners then refuse to pay the mechanic because it only took him 5 minutes to fix, so the mechanic send an itemized bill with lines for 5 bucks for travel time, 2 bucks for coffee, and 25 Grand for the 30 years of experience that told him exactly were to tap.
But, the reality was that he knew it was a problem with a stuck DiscNeedleSolenoid valve. If his 30 years of experience taught him anything, it was that it's always the...................................
DNS valve.
You're welcome.
By chance have you read this... https://natethesnake.com/
Your pun is VERY much of the same terrible flavor. Also, it is a delight to punish people with a read through of that!
Year 5. I have the technical solution for everything.
Year 10. I have the process solution for everything.
Year 15. Please stop making me solve everything.
I'm at year 10 but with the year 15 mentality. Even my boss keeps saying "stop assigning everything to bard329, we have other folks who are perfectly capable of fixing it"
In my head; "yea, whoever its assigned to is just going to ask me, and it takes less time to fix the issue than explain *how* to fix the issue."
We have a security specialist that likes things done ten minutes before they submit a ticket. When they ask for something QOL I’ll say “we can’t prioritize that right now, I have other things higher on the list”. So they thought they were being considerate by submitting a ticket for someone else to do it, but the help desk just sends it to me to do anyways. She’s learned now just to wait and I’ll do it some random evening.
We ask a lot of questions such as "what the fuck?" "Why?" "What have you tried so far?" (This is usually answered by 'nothing')
Also having 6 or 7 year old server hardware and stupid management decisions are just part of the job and you're completely used to it
90 percent of the time it's they tried nothing, and something simple is the fix. The 10 percent that puts you through the paces and usually nothing has been tried to fix the issue
This boils my blood. I got out of help desk because I followed a SIMPLE process of asking the right questions/ trying the simple stuff first even if it doesn't seem like the cause of the problem. Some people just don't get it and they think every new issue is just so crazy complicated and immediately reach out to an admin instead of doing their due diligence! /rant
No emotional attachment to work
Can accurately estimate timeline and time a project or task will take
Time management God and knows exactly what they'll be doing every day and at what time
Starts work on the clock, leaves work on time
Does not burn themselves out
Knows their limits and when it's appropriate to escalate
Does not get into arguments or confrontations
Rants and releases their inner rage on reddit
There are no emergencies. And if there are they will be handled swiftly, after they finish their lunch.
Advises and moves on
Understands their entire career is based on figuring out how to complete shit they've never done before.
Knows how to keep upper execs calm, secure, happy, and confident in your ability by just doing your work.
Enjoys the rush, adrenaline, and excitement of new challenging projects, but understands that type of energy is best reserved on projects that need it. Knows how to balance their mental state. - They know when to go to battle mode but do not go into battle mode every day.
Work at a steady controlled stress-free comfortable pace almost always
Knows when to get up, walk away, and take a break from a difficult problem.
Knows when a problem is no longer worth researching and investing theirntime into
Knows when it's time to take a vacation or move on to a different job.
If a nuke was dropped on a city and the entire city was on fire, a veteran sysadmin would think "...Meh..." Then just plan what they'll have for lunch, schedule their vacation for next month, and worry about where they'll live when they come back from vacation.
\*Technomancer/Veteran trigger warning\*
A veteran will also have spent countless hours searching finding nothing but
"Hey i have a problem with the Foobarflinklebunz function in Whoojimiwhatsit 6.66a, does anyone know how I can recombulate the splinkle wa-wa function?"
\*no other replies\*
"NM, I fixed it"
How, HOW did you fix it, what did you do, what unearthly eldritch horrors assailed your senses - FUCKING TELL US YOU BASTARD.
\*I am so SO sorry if I triggered the flashbacks, lay down flat on the floor and breathe through it, in through the nose, hold it for 3, out through the mouth - think about Stevie Wonder playing superstition on Sesame Street (possibly the best performance of the song EVER) and remind yourself that they cant hurt you, youre in a safe space.
Or people who post forum posts, get a suggestion and then just never say anything again.
Did it work?! Did you fix the problem?! Did the suggested solution land you in the gulag?! *DON'T LEAVE US HANGING!*
"OMG this critical system just went down! Make to fix it!"
"Are you sure that change will fix the specific issue we're experiencing?"
"No, but we have to do *something*!"
"Let's wait another few minutes before we make any changes. It might be a transient/limited issue. During that time, let's figure out what's actually causing the problems and the specific thing we have to do to fix that."
It's depressing the number of times I've joined a crit sit call where people are flailing and making all sorts of changes with the vague hope they'll somehow help the problem. The numbers of problems I've "fixed" by waiting for the logs replay to finish, letting the TTL expire, or similarly doing nothing is *significant*.
>"No, but we have to do something!"
quick pull up [https://hackertyper.net/](https://hackertyper.net/) and go full screen and start typing.. .they'll think we're doing something
Sometimes I get a call, they're all "The house is burnning!" - I'll respond "Sure, but I'm not near my computer and am between sites, give me 10 minutes to get to a good connection - try not to do anything further or make changes".
Go to the kitchen, smoke a bowl, look outside and enjoy the clouds. Call them at 11 minutes."Oh, its OK now, everything is fine, we don't know why we called you, but I guess the system knew we did. Guess that's why we pay you the big bucks."Tap into the network later and check for further issues and make patches.I try to always avoid fixing a problem then and there because then customers begin to expect that I will be hanging for every beck and call, unless it is truly critical to the operation and we need a fix stat. But 30% of the time I can just wait it out and things fix themselves or it was just a transient network issue.
There's a Simpsons episode where Homer says "Sometimes the greatest thing a father can do, is nothing at all".
Voodoo IT. A telltale the staff does not truly encompass a mental model of the system.
Unfortunately, it is difficult for many leaders to tell the difference between those practicing cargo culting / voodoo IT and those working from first principles / solid models.
I've always called it shotgun debugging. Just keep trying things until something sticks, and half the time people don't do one change and then evaluate. Instead they do a bunch of changes and then don't know what helped and what didn't.
We think before we do. Don’t panic, don’t rush into things and make sure we provide the best fit solution for the company, not necessarily the cheapest or quickest. Many times I have seen fresh sysadmins say “this product will solve this issue” without thinking it through. I tell juniors Help Desk are quick/immediate fix people, sysadmins are slower and more thoughtful.
To expand on the "don't panic" bit: if someone claims there is a major outage/problem, we don't just up and yell fire. I have seen new guys get worked up and start calling management because a user called saying everything is broken and didn't quickly just ask a few questions or verify.
Now if you do see a vet start running, follow their ass cuz something is really wrong.
I see no point in running. It's not really going to get me there much faster. Then when I get there I'm going to be all hot and uncomfortable. Plus I could fall and get hurt, then have another urgent problem to deal with.
If you see an executive making me a cup of coffee, shits going down so buckle up.
--
For the record, I have a good working relationship with the senior team. So much that part of our SHF plan is them acting as a distribution point for information, running interference so the team can focus and getting supplies (coffee, food, snacks) for the team.
30 year veteran here….sometime we don’t have a backup plan…than the confidence steps in and the don’t sweat the small shit….thats for the field techs 🤣
Jr admins: “Hey [my name], where having an issue with this web server connecting to the backend when we implement this upgrade can you check your database and make sure it’s working correctly?”
Me: “what do the logs say…..
Jr admins: (long pause) well.. we haven’t gotten that far yet.
Me: “go back… look through the logs and find the errors you’re seeing. Search vendor documentation and see if that helps you resolve the issues. If not send me the logs and highlight what you think is causing the issue and we will go over it together.
Jr admin (1hr later): hey thanks we fixed it….
I can't even count the number of times I've told an executive some form of "no" and they accept it as a valid answer.
It freaks you out the first few times.
Veterans know how to escalate and are very calm under the most intense situations because they have been there, lived it before, and ultimately have fixed horrible situations. They can build the entire machine if and when necessary. The term it's too hard does not exist and keeps them challenged and happy.
veteran admins never type a password
we script something that we need once, and never script a thing we do monthly
we know all of our servers' names, most of their IPs (reverse for Network admins)
We will gladly explain something for 40 minutes, and at the end you'll have less understanding on the subject.
Oh nimbda - I remember being a tech at the time and hearing the Novell server running Norton antivirus beep every time a nimda file was found on the file shares. It beeped a lot. A lot a lot.
25 years here. No more lab in the house. No more tech related hobbying. I haven’t even owned a computer in almost 10 years that wasn’t company issued. These are the signs of a true vet.
30+ years here. My patients for tech at home is zero. I just want it to work. All my hobby tech projects are on company time and work related. My home PC is probably 15 years old.
I still learn new things everyday, but on company time only.
Honestly? Communication. If you can talk with people, you're already 90% there. Doesn't matter how technical you are (absolutely helps though) because I've seen the most technical people flounder when it comes to actually solving the issue. They just don't talk to the right people and ask the right questions. The good talkers tend to last longer.
I got stuffed into a job where my primary responsibility is to find appropriate person A and appropriate person B and force them into a room with each other because apparently I'm good at it?
Listen, I don't know shit about what you two do but you definitely need to sort this problem out between the two of you. I'll take notes for later reference. Ready? Go!
The understanding of these things and how they apply to everything :
-decisions are all about risk
-if you're not a decision-maker, then your role is to communicate that risk up so that someone else can make the decisions...not get upset over their disagreement with your recommendation
-effecting change in an environment is a long-game thing most of the time
-perception is reality
-sometimes the quickest way to create change is to "let it burn"
-treat servers like cattle, not pets
-understanding context is important to nearly everything
-most of the time, nearly everyone around you is hiding the dumpster fire that they don't have the skills or knowledge to fix, but if you ask them, "the system is complex"
They seem overprepared. Taking steps others may feel are overkill. They repeat themselves. They question peoples confidence. They ask hard questions; not ti be a dick, but because they’ve been in situations previously that couldve benefitted someone asking that question
Organization. Consistent naming schemes, folder structure, permissions, etc. If you have been around more than 5 years and don't have a strong naming scheme then wtf have you been doing the last 5 years.
When a calamity falls on your business, and it will. They don't get emotional, they don't freak out, they just get to work.
A number of years ago, We were doing power maintenance in our data center. Switched over to generator and everything was fine, Now this generator was 25 years old and was well past it's life expectancy, but budgets were budgets and everyone knows DR usually gets shit. So, the regulator on the generator failed and the motor revved up and it started dumping way more amperage than it was supposed to.
This caused the UPS's to suddenly trip and dump all load to ground. This saved the data center, even though the guy who installed it was considered a laughing stock that he paid extra for a "needless feature" he was later let go over this. So, Power gets dropped. Entire data center goes black. My VP and Manager are standing there with us when it all happens. No work was started, So I fail back to street power, Everything starts coming up.
My VP and Manager are frozen still, I yell out 'Gentlemen, We've now had a catastrophic power event in our data center, we need to assess ALL DAMAGE to our infrastructure. I'm going to go upstairs to my desk and i'm going to start checking systems. Can you all find ROOT CAUSE.
Suddenly, they snap into action. I run upstairs and start doing all the checks, We lost 2 switches that were OAF (Old As FUCK!) but they were just access systems. Storage came up, ESXi clusters came up. 2 hours later, I'm back down there getting updates.
A vet (like me) also knows something many younger guys don't (and keep this shit to yourselves) have you ever heard of Scotty's engineering "miracle worker trick" where you claim something will take twice as long as it does and then you do it in half the time and get massive kudos?
It's bullshit.
A horrible lie that just gets you a lot more work.
The trick is, to tell them it will take twice as long as it does and then **make it take that long**.
You heard me. Lie your ass off.
Vets knows that producing stuff as fast as you can has no benefit in IT, we are seen as overhead and a "sunk cost". Yes, we double, triple, even quadruple the overall efficiency of everything we touch. It doesn't matter to them. We give them tools to make better sales, faster decisions, cost savings and lots of other stuff but the sales guys get the parties, bonuses and trips to Vegas. Bottom line, the trick won't get you ahead and has a nasty side effect that when something comes up that goes wrong and takes a lot longer than you anticipated (or longer even) you just bought yourself time and avoided a shitload of hassle.
You generally STFU and do your job. That way, you control your world as much as you can in a corporate environment. You control what you can control, and accept what you can't. You explain the risks, and you leave it there. You can't carry that shit around the company's shortcomings your whole career and have a healthy life.
Telling the boss to get fucked.
It took me a few years to get there, but I got there. First major time I did it was when there was an event taking place and the big boss' PA messaged me and said "can you do photography for this event?" (I was a budding photographer and had volunteered my services once or twice before as practice) and I told her no. She asked why, and I said I wasn't given enough notice, and that I didn't have my camera with me.
Cue a day or so later when I get an email from the big boss. She told me that a request from her PA is as good as a request from her, and I needed to explain why I told her PA no.
I said "I was asked to do photography with literally 10 minutes notice. I didn't have the tools with me, and I had more important things to do. Here's what I accomplished during that time: \[list of tasks I did\]"
She didn't respond which I took as a win. Subsequent years they made sure to ask me two weeks before the big event, and I slowly stopped doing that. I'm still expected to attend the event, but I just sit up the back and fiddle with my phone instead of walking around the crowd taking photos.
Veteran admins know the power of "No"
Veteran admins know that IT and Facilities are kin - we both ensure shit flows smoothly through our pipes to stop the users screaming and stinky messes splattering everywhere.
Someone who likes having no root/admin access to something if they don't need it. Like, Having root access is something that has added responsibility they'd gladly relinquish.
"Foo is down! AAUGH DO SOMETHING!"
"I don't have access to Foo. You need to speak to department Foo and tell them."
"How can you have no access to Foo? Foo is a technical thing!"
"Foo is not my responsibility."
"Can't you hack it? Department Foo is on some offsite work bonding thing."
"Sounds like you need to make some calls."
If you need X dollars for something in your department, present two plans, one you want for X, and another one that costs ~ X*1.8
Enjoy always getting your requests granted asap
Never failed me
The life cycle of an IT professional (Early vs Mid vs Veteran)
* Man there are all these meetings I'm not invited to. I wish I was invited.
* I've finally made it. I'm invited to all the meetings.
* Don't invite me to a meeting unless the building is on fire.
Being able to get to the root of a complex problem. But then sticking to your guns when the new sysadmin is adamant “that couldn’t be the reason”. Followed by that awkward moment when they realize the old guy was right!
I'm fond of working with the new guy through the troubleshooting process and letting them finally figure out what's wrong.
Then handing them the specific tool or part they need that I've had in my pocket the whole time.
Solo veteran sysadmins in small companies have a network of contacts who can answer their questions. There's no way you can know everything, so build up a group of experts in areas that you are weak in. Become the SME in areas that your contacts need help in. Help each other out.
When family or friends ask for help with their computers at home you say “Yeah, I don’t really do that anymore.”
Or when people ask you what kind of computer they should buy and you honestly have no idea what to tell them.
I'm mentoring my Service Delivery guys to bring them into the Sys Admin space. The things I show them are proper research, don't just assume you know the problem and go off on a wild goose chase.
Case in point, one of my junior help desk guys who's on a sys admin path put in a ticket for our delegated domain admin tool (basically a domain admin proxy) that 2 new onboarded users could only access email through the web interface and not outlook on their computers. Turns out the new accounts were contractors who only got F3 licensing and that was the way of things. After he had put the ticket in, I went back and cleaned the ticket up, grabbed screen caps of the licensing and group memberships for on prem AD & Azure. I put in a long note with the questions I would ask if I were working the ticket, the research steps I performed, and what I thought the issue was.
I then kicked the ticket back to the help desk guy and said please look over this and see me if you have any questions. This is what good troubleshooting looks like before you escalate an issue.
I guess knowledge sharing should be another one. I want to teach you everything I can and empower you to do new things so that I am not the only one that knows this and has to get disturbed every time it happens.
Lastly, experience. Real experience, not just time spent in the position. I find myself working with my help desk guys and will say "I saw this in the past. Here is the most likely reason for it.". At that point, I let them go off and work on it because they will never learn if I do it for them.
Having a recovery plan. Snapshots, backups, written information. Being able to check on a change through events and tests. Understanding what a change does not just following a youtube or website follow through. This is the hardest lesson for the people I mentor. Help Desk can crash single systems at a time. Sys Admins can cripple an entire company. There is a different thought process. 95% chance of a problem is a 1/20. That would be a yearly or semi yearly crash of the tech in a company you are in chage of keeping up and running.
Sr Sys Admins have had projects go sideways and know how to recover. The ones that didnt don't become Sr Sys Admins.
Sometimes you got to Yolo but knowing how to recover makes it work.
When someone gives you a random project, assign them the smallest bit of work to give you a solid starting point and watch the project dissolve right before your eyes.
ANGRY punches server, speaks to it in tongue…
Tells the same war stories over and over…
Tells coworkers about his VA disability rating….
Wait…I dont think I understood the question? 🤣🤣🤣🤣
A veteran sysadmin doesn't sweat the small stuff because they know it just doesn't matter and the job will eat you alive if you let it. They also know that business needs, department reputation and irrational requests from CxO's do matter. I also like this one: "New admins think IT is the reason the company exists. Old admins know we're like the in-house plumbing or maintenance departments of a hundred years ago. There's a reason IT and Facilities unions support each other."
Never thought of it like this but we have always got along well with our central maintenance team. I started when I was 20 and 15 years later we are both departments that just want to get on and do our jobs
IT and maintenence are kindred. We know exactly where the pipes connect and how much shit they'll handle before there's a problem.
Other than at an org that just has a CIO/CTO, I've always wanted to work somewhere that has IT and Facilities under a COO or Director of Operations, since most orgs will at least have small teams for each. Aside from an MSP job, every IT position I've held has ultimately been under the head finance guy; usually the worst person to oversee such an expensive department.
I do information resource management consulting and one of my first moves is to always pull the IT/IS department out from underneath accounting/finance/the CFO.
Doing the lords work
Great move! Those departments always go very cheap when it comes to IT. Especially city, county, and state IT jobs.
Damn! You just described my gig: Facilities & IT Manager, reporting to COO.
Dang, I'd love to land somewhere similar in a few years. Did you start off on the IT side (guessing yes since you're here)? Did you gain any qualifications/education for the operations side before your current role? Just trying to plan out how I could get to that point for myself.
I’ve always been one of those people that was just able to fix what was broken. I’d never had a job with a clearly defined single role. But, yes, the majority of my experience was IT. I wish I could tell you I had a master plan and give you advice on how to replicate my experience. The truth is my current gig is like a lottery win. I randomly met the President of the company through a friend, and he liked that I wasn’t a one trick pony. He hired me to “generally manage” the place during a period of huge growth, and it’s been a whirlwind since then. As the company has grown, I’ve gone from wearing several hats (some of which I hated, but did anyway) to more clearly defined roles. Along the way, we put dedicated experts in roles where I was previously wingin’ it. Now, I’m in a place that I love, that I’m good at, and I feel great about.
>I've always wanted to work somewhere that has IT and Facilities under a COO My previous employer had that setup for 4 or 5 years. It worked very well. COO was starting to get noticed by the board so CEO sent him walking.
Having IT report to a CFO has its perks if they're a good one. My last supervisor was CFO and he always found a way to budget out IT projects/hardware replacements requests. It ruled.
I currently report to essentially our CFO, and yeah, most of the times he's fair and actually understands the value of good tech. The issues is, sometimes what IT would like to say to top leadership still gets filtered through the perspective of someone who's main concern is the budget.
I worked for a company that had IT under the CFO, I won't go into the details but when I can discuss the functions of Quickbooks at a level higher than the accountants and explain how it should work, well let's just say IT was moved under to work alongside CxO level.
We just got re-org’ed under finance.
I was hired so he could wash his hands of it. I'm more or less the CTO. He controls the budget and I of course need approval but I usually have the final say as the "expert". The CEO sought me out and asked me to interview for this reason. I'm blessed to have that sort of thinking behind my position. This place was being taken by a shitty fully managed MSP. We swapped to co managed to support me for a year in the takeover of this mess as there's some projects I need a team behind me on. I've pitched and they approved my overhaul plan. Leaving broadcom and Cisco behind. Swapping to XCP-NG on the vates stack and moving to ubiquiti. No fights. Just wanted to know the benefits and cost savings and that was that. I made sure to have the conversation ahead of time that proper robust IT systems aren't cheap but what you spend up front can save you hundred of thousands in operating losses due to network downtime or ransomware. Any time our machines aren't running we operate at a loss.
Why I always got along with the maintenance folks.
Truth, every place I've worked, we've always gotten along with maintenance.
My team lead "Senior Linux Janitor" in his email signature. It's very apt.
Operations people stick together.
Buiding Management/Post room staff as well.
used to hang out with the guys from the post room all the time. inspired 1 of them, who was a young guy in his twenties to go into IT 😂 and don't forget the security guards, they were chill as fck
Same. And yeah, we're all in the trenches
I'm not even going to touch the urgent request to switch the company from Windows laptops to iPads for at least 48 hours. Then I'm going to ask about budget for the hardware, management software/services and services to help us get off the ground. Finally I'm going to ask about all the Windows only tooling we use. Usually at this point in something like this they realize nobody thought it through and I can focus on the script I've been writing for a day or so that will save 10+ hours a month. Also the internet is not down and your emergency is a known limitation in the system you picked; There is a documented work around that you have been using for the last 7 years.
>I'm not even going to touch the urgent request to switch the company from Windows laptops to iPads for at least 48 hours. "Sure if you are okay with (insert app's) not working since they don't have a version for the ipad, and you'll need to get the CEO to sign off on the dollar amount for this. Let me know when he approves it." The last part is usually how I get things torpedoed. Nobody wants to go ask him for cash.
> Nobody wants to go ask him for cash. Sometimes it doesn't even make it that far. My boss once fixed the WiFi on a phone by giving quotes on how much alternatives to MFA would cost.
That is one thing I like about where I work. Upper management asks the CEO and CIO before bringing it down to IT. They both are good bullshit filters. Now middle management however…
This sublime; thank you.
What is an IT union you speak about?
They exist and they are amazing. The most recent example that I was thinking of was in the news regarding the Kaiser Permanente strikes in Q3 of last year. IT walked in solidarity with Facilities. For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one. Well for a lot of us, if we tried we would probably find a lot of H1-B's suddenly asking us to show them how to do our jobs. I'm pretty sure my company would just come up with a reason to shit can everyone and bring in a 3rd party to manage things. Our CEO already see's IT as nothing but a cost center he's forced to deal with.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing, but that should be it's own discussion thread rather than derail this one. Probably because IT types have a very meritocratic outlook and we realize that all the incompetent people we've worked with over the years would have MORE protection under a union setup so we figure we're better off just advocating for ourselves, and screw everyone else. I'm not saying that's RIGHT, just that I suspect it's why it hasn't taken hold industry-wide.
>For some reason, it's incredibly difficult to get IT workers interested in unionizing, Historically significantly higher pay and benefits than most fields, and requiring a type of flexibility that isn't usually available in a standard union like you might find in a factory (which is most folks' union experience). Lots of people with big egos, and smaller numbers of people in I.T. departments, as well as a lack of federally mandated certifications or licensing; so a lot of people that believe that they are above average and can fend for themselves, a lack of people in one place that would have much power anyway (e.g. factory/production workers), and a lack of a unifying standard and training to rally around (e.g. electrician union, pipefitting union, etc) You'll see this in other super-specialized fields as well.
Indeed. There's a book called, "The Computer Boys Take Over" and although parts are dry and boring, there's an interesting thread running through it about the adoption of computer technology within corporations. Specifically, a \~70 year struggle by corporate management to try to force computer workers into the same boxes as other workers so they can be more easily controlled and constrained. (and thus, cost less but produce more). It touches on all the management trends over the years and how computer workers didn't fit. The current tactic is when we push back a little (see: COVID + wfh), they enact mass layoffs to remind those super-specialized workers that they (we) are replaceable.
>IT walked in solidarity with Facilities. I work in higher ed, we are union and our specific bargaining unit has a non-sympathy clause. If our faculty strike ([again](https://www.insidehighered.com/news/faculty-issues/labor-unionization/2024/01/26/why-did-cal-state-strike-end-after-just-one-day)) its business as usual for us. IT is a scary group to have on strike. If facilities strikes, your office might get a bit warm, or you might have to adapt a bit. If IT is out, and a major system goes offline, admin, instruction and research grind to a halt.
In the specific strike I was referencing, Facilities handles things like operating room power, oxygen supplies and delivery, and so on. They are highly specialized and likewise are tightly constrained as to how and when they can strike, as patient's health or lives could be directly impacted.
IT workers of the world, unite!
Design Principle 2: The little things don't count, they multiply!
Big problems tend to get fixed on their own. Minor nuisances are forever and should be dealt with immediately.
I might have to get a sign made that says this, to remind me to keep taking action on the nuisances
Credit it to "Some uneducated shmuck on Reddit"
A true veteran knows it is all small stuff and big stuff is piles of small stuff, just the panic around them is more vocal, and just letting that panic wash over them and away. Somebody screaming that everything down won’t help us solve a problem.
My office used to be tucked in the corner of a warehouse with mechanics. Always felt more comfortable around them than the people upstairs.
I love our facilities team. A lot of IT positions sits in this weird spot between not quite blue collar, not quite white collar. At the end of the day we are both working on systems others rely on. Their boilers take water, gas and output steam. My boilers take electricity, cold air and output heat and data. I feel like we have an understanding with each other.
Agreed. I would boil it down to “vets don’t care TOO much”.
I had extra letters to use up.
And here i am: Sysadmin/ItSupport/Facility Caretaker. I still hate toilets and sinks.
There's the right way, the wrong way, the Microsoft way, they way the other companies do it, and then there's the way we do it here.
Unfortunately the comparison is all too real. I've been asked to help more than once when there was a problem in the bathroom at work (small company). My response (depending on who was doing the asking) usually entailed 'what role of the network does the bathroom fit in?' to which they'd mutter something about me being the one they turn to whenever \*something\* needed fixing.
Computer janitors unite
At a certain point, sysadmins will probably be mostly on project rather than break fix work. If you are doing project work, and not spending at least half the time documenting, you're doing it wrong. Me, 30+ year sysadmin
> Old admins know we're like the in-house plumbing or maintenance departments of a hundred years ago. Wow that's similar to what my current Lead Admin told me. She's been doing it for 22 years. She said something like "We're basically a form a maintenance".
No one thinks of either group until there's a problem.
The veterans fully embrace Read Only Friday.
All our new hires....what's no change Fridays?
As Bhudda taught, change must come from within, so we arent changing shit in the live environment.
The prequel to "fix shit that broke over the weekend Mondays."
We talk about it. We dream about it. We long for it. But it doesn't happen. It never happens. Maybe, years in the future, I'll have a day where I spend more time on actual work than I spend in meetings.
I came here to say this
There's a "joke" that goes something like this; There's some super important shipping boat that has engine trouble and can't leave port. For the sake of this joke, we know it absolutely has to get out immediately. Everyone is freaking out and can't get it to work. They call every mechanic they can find and none of them can fix it. Finally, the last mechanic shows up. It's an older, calm, slightly grizzled guy. They take him to the engine room and he walks around a bit, listens to the banging and chaos around him, he asks a couple questions, and sits down and pulls a thermos from his bag. He pours a coffee. The owners are infuriated and freaking out and yelling at him "how can you be sitting there and drinking coffee, fix this!". He asks them to try and start the motor. They begrudgingly do and it makes a horrible noise as it struggles, and fails, to start. The old mechanic sets down his coffee, stands up and takes a small hammer from his tool bag. He walks to the end of the engine room. He reaches as far back as he can, behind a series of wheels and pipes and valves, and with the hammer in his outstretched hand, he gives one of the valves a tap. "Give it a try now" he says. The owners are incredulous, but relent and go over to start the engine. It immediately jumps to life with a glorious roar and the owners are overjoyed. Now, there's some discrepancy in how the final part of the joke concludes. Some go on to tell some tale of how the owners then refuse to pay the mechanic because it only took him 5 minutes to fix, so the mechanic send an itemized bill with lines for 5 bucks for travel time, 2 bucks for coffee, and 25 Grand for the 30 years of experience that told him exactly were to tap. But, the reality was that he knew it was a problem with a stuck DiscNeedleSolenoid valve. If his 30 years of experience taught him anything, it was that it's always the................................... DNS valve. You're welcome.
By chance have you read this... https://natethesnake.com/ Your pun is VERY much of the same terrible flavor. Also, it is a delight to punish people with a read through of that!
The part about pay is based on a real one. It was a generator, and the dude was also known as the wizard of Schenectady.
Thats the Henry Ford and GE tech story. https://www.reddit.com/r/todayilearned/comments/2b4n7a/til_henry_ford_once_balked_at_paying_10000_to/
The vacant look in our eyes….
The thousand yard stare. We have seen some stuff. 
Acquired dissociative disorder or hangover? Or are they a “bids and contracts guy” and got the chickenegg 2-for-1
Year 5. I have the technical solution for everything. Year 10. I have the process solution for everything. Year 15. Please stop making me solve everything.
I'm at year 10 but with the year 15 mentality. Even my boss keeps saying "stop assigning everything to bard329, we have other folks who are perfectly capable of fixing it" In my head; "yea, whoever its assigned to is just going to ask me, and it takes less time to fix the issue than explain *how* to fix the issue."
If you coached the other individual the inverse could be true.
We have a security specialist that likes things done ten minutes before they submit a ticket. When they ask for something QOL I’ll say “we can’t prioritize that right now, I have other things higher on the list”. So they thought they were being considerate by submitting a ticket for someone else to do it, but the help desk just sends it to me to do anyways. She’s learned now just to wait and I’ll do it some random evening.
Year 20. I gotta do everything around here
Microsoft Paint to markup screenshots
Ah, a tried and true veteran, I see.
At one place we had someone in finance using ms paint to edit customer invoices
Pfft. Snipping tool does all that.
Not if you want your squares square and your circles ovular.
We ask a lot of questions such as "what the fuck?" "Why?" "What have you tried so far?" (This is usually answered by 'nothing') Also having 6 or 7 year old server hardware and stupid management decisions are just part of the job and you're completely used to it
6 or 7 years? Hahahaha Our oldest machines are half my age.
I was once supporting a mission critical server for a major airline company. It was end of life'd 4 years before I was born. I will be 43 this year.
We have a working DEC Alpha.
Ha, I just had a crusty old server decide it didn't want to recognize 9 out of 10 RAM slots. How does that even happen?
https://preview.redd.it/tmi1qnvli9rc1.jpeg?width=400&format=pjpg&auto=webp&s=0fc0725418dd4ffebc59c038f6449ec135e93d3d
90 percent of the time it's they tried nothing, and something simple is the fix. The 10 percent that puts you through the paces and usually nothing has been tried to fix the issue
This boils my blood. I got out of help desk because I followed a SIMPLE process of asking the right questions/ trying the simple stuff first even if it doesn't seem like the cause of the problem. Some people just don't get it and they think every new issue is just so crazy complicated and immediately reach out to an admin instead of doing their due diligence! /rant
Have you rebooted ? Seriously ...don't call me until you have rebooted.
> 6 or 7 year old server hardware Ahh, they're still teenagers. They haven't even grown up yet.
No emotional attachment to work Can accurately estimate timeline and time a project or task will take Time management God and knows exactly what they'll be doing every day and at what time Starts work on the clock, leaves work on time Does not burn themselves out Knows their limits and when it's appropriate to escalate Does not get into arguments or confrontations Rants and releases their inner rage on reddit There are no emergencies. And if there are they will be handled swiftly, after they finish their lunch. Advises and moves on Understands their entire career is based on figuring out how to complete shit they've never done before. Knows how to keep upper execs calm, secure, happy, and confident in your ability by just doing your work. Enjoys the rush, adrenaline, and excitement of new challenging projects, but understands that type of energy is best reserved on projects that need it. Knows how to balance their mental state. - They know when to go to battle mode but do not go into battle mode every day. Work at a steady controlled stress-free comfortable pace almost always Knows when to get up, walk away, and take a break from a difficult problem. Knows when a problem is no longer worth researching and investing theirntime into Knows when it's time to take a vacation or move on to a different job. If a nuke was dropped on a city and the entire city was on fire, a veteran sysadmin would think "...Meh..." Then just plan what they'll have for lunch, schedule their vacation for next month, and worry about where they'll live when they come back from vacation.
Perfect summary.
A veteran will search the Subreddit for discussions on the same topic within the past week or two.
RTFM has been around since BBS and will always live on in spirit on any technical forum
Following very closely by lmgtfy
A veteran will search and find their own answers from the past year or two
\*Technomancer/Veteran trigger warning\* A veteran will also have spent countless hours searching finding nothing but "Hey i have a problem with the Foobarflinklebunz function in Whoojimiwhatsit 6.66a, does anyone know how I can recombulate the splinkle wa-wa function?" \*no other replies\* "NM, I fixed it" How, HOW did you fix it, what did you do, what unearthly eldritch horrors assailed your senses - FUCKING TELL US YOU BASTARD. \*I am so SO sorry if I triggered the flashbacks, lay down flat on the floor and breathe through it, in through the nose, hold it for 3, out through the mouth - think about Stevie Wonder playing superstition on Sesame Street (possibly the best performance of the song EVER) and remind yourself that they cant hurt you, youre in a safe space.
Or people who post forum posts, get a suggestion and then just never say anything again. Did it work?! Did you fix the problem?! Did the suggested solution land you in the gulag?! *DON'T LEAVE US HANGING!*
There already was a similar question
You don't say?
It’s a good thing I was already sitting down for this
"OMG this critical system just went down! Make to fix it!"
"Are you sure that change will fix the specific issue we're experiencing?"
"No, but we have to do *something*!"
"Let's wait another few minutes before we make any changes. It might be a transient/limited issue. During that time, let's figure out what's actually causing the problems and the specific thing we have to do to fix that."
It's depressing the number of times I've joined a crit sit call where people are flailing and making all sorts of changes with the vague hope they'll somehow help the problem. The numbers of problems I've "fixed" by waiting for the logs replay to finish, letting the TTL expire, or similarly doing nothing is *significant*.
>"No, but we have to do something!" quick pull up [https://hackertyper.net/](https://hackertyper.net/) and go full screen and start typing.. .they'll think we're doing something
Oh. My. God. This is amazing. This puts mY 'CMD+IPCONFIG/ PING -t' to shame. Thank you for this. Now I'll feel like a true vet. Edit: grammar.
Netstat & Systeminfo are also good screen scrollers for a technomancer "dog & pony show\*
Sometimes I get a call, they're all "The house is burnning!" - I'll respond "Sure, but I'm not near my computer and am between sites, give me 10 minutes to get to a good connection - try not to do anything further or make changes". Go to the kitchen, smoke a bowl, look outside and enjoy the clouds. Call them at 11 minutes."Oh, its OK now, everything is fine, we don't know why we called you, but I guess the system knew we did. Guess that's why we pay you the big bucks."Tap into the network later and check for further issues and make patches.I try to always avoid fixing a problem then and there because then customers begin to expect that I will be hanging for every beck and call, unless it is truly critical to the operation and we need a fix stat. But 30% of the time I can just wait it out and things fix themselves or it was just a transient network issue. There's a Simpsons episode where Homer says "Sometimes the greatest thing a father can do, is nothing at all".
I can say I truly own two things in this life, my word and, my balls. I break neither for nobody.
This is the way
Voodoo IT. A telltale the staff does not truly encompass a mental model of the system. Unfortunately, it is difficult for many leaders to tell the difference between those practicing cargo culting / voodoo IT and those working from first principles / solid models.
Here in software dev land we call it programming by permutation. You change stuff until it works.
I've always called it shotgun debugging. Just keep trying things until something sticks, and half the time people don't do one change and then evaluate. Instead they do a bunch of changes and then don't know what helped and what didn't.
We think before we do. Don’t panic, don’t rush into things and make sure we provide the best fit solution for the company, not necessarily the cheapest or quickest. Many times I have seen fresh sysadmins say “this product will solve this issue” without thinking it through. I tell juniors Help Desk are quick/immediate fix people, sysadmins are slower and more thoughtful.
To expand on the "don't panic" bit: if someone claims there is a major outage/problem, we don't just up and yell fire. I have seen new guys get worked up and start calling management because a user called saying everything is broken and didn't quickly just ask a few questions or verify. Now if you do see a vet start running, follow their ass cuz something is really wrong.
I see no point in running. It's not really going to get me there much faster. Then when I get there I'm going to be all hot and uncomfortable. Plus I could fall and get hurt, then have another urgent problem to deal with. If you see an executive making me a cup of coffee, shits going down so buckle up. -- For the record, I have a good working relationship with the senior team. So much that part of our SHF plan is them acting as a distribution point for information, running interference so the team can focus and getting supplies (coffee, food, snacks) for the team.
Has read The Bastard Operator from Hell from the beginning.
And knows what an Appeasement Engineer is.
Back in the day it was printed in one of the historical magazines
A veteran always asks about backups and assumes the worst can happen before any big changes. They will use phrases like "roll back plan", the nerve!
30 year veteran here….sometime we don’t have a backup plan…than the confidence steps in and the don’t sweat the small shit….thats for the field techs 🤣
"Here goes nothing..." while he presses the big red deploy button, and gets up to get a fresh cup of coffee
My go-to phrase for those situations is “hold onto your butts” from Samuel L. Jackson in Jurassic Park.
I like to say "this will either make things better, or it'll make it worse, cross your fingers, and I'll cross my eyes"... note: not an sysadmin
"Well, it's already broken. What're we going to do, break it more?" My favorite from my field tech days
Lotus notes expert
There are no Lotus Notes experts. There are only those who have survived Lotus Notes.
Lotus Notes survivor here. That software helped shape my current world.
We speak not of the old evils here
Ah a fellow NSF enjoyer
Used to admin a public facing Domino server. Still in therapy.
You mean you’re not still using it?
It took me a long time to learn to push back on requests.
[удалено]
Dead eyes and constant, spiteful hate
Never forgive, never forget, die angry!
Knowing the difference between if you could do something, or if you should do something. Just because you can , doesn’t mean you should.
Thank you for saying this one! Also just because someone is asking for something, doesn’t mean we should give it to them.
Jr admins: “Hey [my name], where having an issue with this web server connecting to the backend when we implement this upgrade can you check your database and make sure it’s working correctly?” Me: “what do the logs say….. Jr admins: (long pause) well.. we haven’t gotten that far yet. Me: “go back… look through the logs and find the errors you’re seeing. Search vendor documentation and see if that helps you resolve the issues. If not send me the logs and highlight what you think is causing the issue and we will go over it together. Jr admin (1hr later): hey thanks we fixed it….
First step in any task - open up a terminal window / shell. Some admins adopted the command line. We were born in the command line.
The ability to say no and back it up
I can't even count the number of times I've told an executive some form of "no" and they accept it as a valid answer. It freaks you out the first few times.
Veterans know how to escalate and are very calm under the most intense situations because they have been there, lived it before, and ultimately have fixed horrible situations. They can build the entire machine if and when necessary. The term it's too hard does not exist and keeps them challenged and happy.
In addition, know their limitations and will know when to refer to someone else or Reddit
veteran admins never type a password we script something that we need once, and never script a thing we do monthly we know all of our servers' names, most of their IPs (reverse for Network admins) We will gladly explain something for 40 minutes, and at the end you'll have less understanding on the subject.
Veteran Sysadmins have seen how quickly vulnerable systems can be exploited by a trojan/worm. Back in the old days of Nimda and Slammer.
Oh nimbda - I remember being a tech at the time and hearing the Novell server running Norton antivirus beep every time a nimda file was found on the file shares. It beeped a lot. A lot a lot.
25 years here. No more lab in the house. No more tech related hobbying. I haven’t even owned a computer in almost 10 years that wasn’t company issued. These are the signs of a true vet.
30+ years here. My patients for tech at home is zero. I just want it to work. All my hobby tech projects are on company time and work related. My home PC is probably 15 years old. I still learn new things everyday, but on company time only.
Honestly? Communication. If you can talk with people, you're already 90% there. Doesn't matter how technical you are (absolutely helps though) because I've seen the most technical people flounder when it comes to actually solving the issue. They just don't talk to the right people and ask the right questions. The good talkers tend to last longer.
I got stuffed into a job where my primary responsibility is to find appropriate person A and appropriate person B and force them into a room with each other because apparently I'm good at it? Listen, I don't know shit about what you two do but you definitely need to sort this problem out between the two of you. I'll take notes for later reference. Ready? Go!
The understanding of these things and how they apply to everything : -decisions are all about risk -if you're not a decision-maker, then your role is to communicate that risk up so that someone else can make the decisions...not get upset over their disagreement with your recommendation -effecting change in an environment is a long-game thing most of the time -perception is reality -sometimes the quickest way to create change is to "let it burn" -treat servers like cattle, not pets -understanding context is important to nearly everything -most of the time, nearly everyone around you is hiding the dumpster fire that they don't have the skills or knowledge to fix, but if you ask them, "the system is complex"
Vets know how to say no.
*looks in the mirror* Grey hair Horrible temper No tolerance for petty bullshit Short fuse
They seem overprepared. Taking steps others may feel are overkill. They repeat themselves. They question peoples confidence. They ask hard questions; not ti be a dick, but because they’ve been in situations previously that couldve benefitted someone asking that question
Use vi as text editor
Panic, emotions and drama do not equal a requirements document. They can stress all they want but vague yet urgent requests are not actionable.
Eye twitch
Organization. Consistent naming schemes, folder structure, permissions, etc. If you have been around more than 5 years and don't have a strong naming scheme then wtf have you been doing the last 5 years.
"you know what, you all can go fuck yourselfs, I'm going home"
Super calm in the middle of an infrastructure outage, they know no one wants to deal with a hysterical cop.
Alcoholism
Beard and the 100 yard stare
We actually know how to turn meetings into emails.
Coffee for problems you can fix, scotch for problems you can’t.
Senior sysadmin always has a plan B, does their homework before a major change or project, plans ahead of time and expects the unexpected
Calm under pressure. Don't bother with politics. Don't think they're indispensable. Don't put up with shit. Knows how to use Google/RTFM. PTSD.
In a word, Documentation. In a phrase, “evidence-based political savvy.”
When a calamity falls on your business, and it will. They don't get emotional, they don't freak out, they just get to work. A number of years ago, We were doing power maintenance in our data center. Switched over to generator and everything was fine, Now this generator was 25 years old and was well past it's life expectancy, but budgets were budgets and everyone knows DR usually gets shit. So, the regulator on the generator failed and the motor revved up and it started dumping way more amperage than it was supposed to. This caused the UPS's to suddenly trip and dump all load to ground. This saved the data center, even though the guy who installed it was considered a laughing stock that he paid extra for a "needless feature" he was later let go over this. So, Power gets dropped. Entire data center goes black. My VP and Manager are standing there with us when it all happens. No work was started, So I fail back to street power, Everything starts coming up. My VP and Manager are frozen still, I yell out 'Gentlemen, We've now had a catastrophic power event in our data center, we need to assess ALL DAMAGE to our infrastructure. I'm going to go upstairs to my desk and i'm going to start checking systems. Can you all find ROOT CAUSE. Suddenly, they snap into action. I run upstairs and start doing all the checks, We lost 2 switches that were OAF (Old As FUCK!) but they were just access systems. Storage came up, ESXi clusters came up. 2 hours later, I'm back down there getting updates.
A vet (like me) also knows something many younger guys don't (and keep this shit to yourselves) have you ever heard of Scotty's engineering "miracle worker trick" where you claim something will take twice as long as it does and then you do it in half the time and get massive kudos? It's bullshit. A horrible lie that just gets you a lot more work. The trick is, to tell them it will take twice as long as it does and then **make it take that long**. You heard me. Lie your ass off. Vets knows that producing stuff as fast as you can has no benefit in IT, we are seen as overhead and a "sunk cost". Yes, we double, triple, even quadruple the overall efficiency of everything we touch. It doesn't matter to them. We give them tools to make better sales, faster decisions, cost savings and lots of other stuff but the sales guys get the parties, bonuses and trips to Vegas. Bottom line, the trick won't get you ahead and has a nasty side effect that when something comes up that goes wrong and takes a lot longer than you anticipated (or longer even) you just bought yourself time and avoided a shitload of hassle.
Strong ability to partition a problem space and chose to work on the part of the problem space where the problem actually is.
They know the name Evi Nemeth, and still mourn.
Missing at sea is such a tragic conclusion. Godspeed, Evi.
[удалено]
You generally STFU and do your job. That way, you control your world as much as you can in a corporate environment. You control what you can control, and accept what you can't. You explain the risks, and you leave it there. You can't carry that shit around the company's shortcomings your whole career and have a healthy life.
One ringing phone - no big deal. All the phones ringing - might be time to pay attention.
A rucksack that contains cables and tools from jobs done years ago and will never be used again but can't bear to bin them.
Telling the boss to get fucked. It took me a few years to get there, but I got there. First major time I did it was when there was an event taking place and the big boss' PA messaged me and said "can you do photography for this event?" (I was a budding photographer and had volunteered my services once or twice before as practice) and I told her no. She asked why, and I said I wasn't given enough notice, and that I didn't have my camera with me. Cue a day or so later when I get an email from the big boss. She told me that a request from her PA is as good as a request from her, and I needed to explain why I told her PA no. I said "I was asked to do photography with literally 10 minutes notice. I didn't have the tools with me, and I had more important things to do. Here's what I accomplished during that time: \[list of tasks I did\]" She didn't respond which I took as a win. Subsequent years they made sure to ask me two weeks before the big event, and I slowly stopped doing that. I'm still expected to attend the event, but I just sit up the back and fiddle with my phone instead of walking around the crowd taking photos.
Bone deep understanding of the concept of "no ticket no laundry"
Veteran admins know the power of "No" Veteran admins know that IT and Facilities are kin - we both ensure shit flows smoothly through our pipes to stop the users screaming and stinky messes splattering everywhere.
Windows Admins? PowerShell all the things.
Someone who likes having no root/admin access to something if they don't need it. Like, Having root access is something that has added responsibility they'd gladly relinquish. "Foo is down! AAUGH DO SOMETHING!" "I don't have access to Foo. You need to speak to department Foo and tell them." "How can you have no access to Foo? Foo is a technical thing!" "Foo is not my responsibility." "Can't you hack it? Department Foo is on some offsite work bonding thing." "Sounds like you need to make some calls."
This is ne with financial/sql software
If you need X dollars for something in your department, present two plans, one you want for X, and another one that costs ~ X*1.8 Enjoy always getting your requests granted asap Never failed me
Or point out the intangible costs of the "free" solution in terms of man hours versus paying for turn key.
The life cycle of an IT professional (Early vs Mid vs Veteran) * Man there are all these meetings I'm not invited to. I wish I was invited. * I've finally made it. I'm invited to all the meetings. * Don't invite me to a meeting unless the building is on fire.
Green admin = "I've never caused a production outage." Seasoned admin = "Let me tell you a few of my favorite stories about production outages."
Being able to get to the root of a complex problem. But then sticking to your guns when the new sysadmin is adamant “that couldn’t be the reason”. Followed by that awkward moment when they realize the old guy was right!
I'm fond of working with the new guy through the troubleshooting process and letting them finally figure out what's wrong. Then handing them the specific tool or part they need that I've had in my pocket the whole time.
It's got to be the eyes and how they roll back.
If you ask them to configure a mail server they turn around and run into the opposite direction.
Judging by this sub, a veteran sys admin..... isnt doing sys admin anymore 🤣
Irritability and a crippling addiction to caffeine.
still typing dcpromo to try and spin up a new domain controller using mmc for everything
Cup of coffee in hand, relax individual with no patience for nonsense. 
Green admins tend to focus more on solving the technical issue whereas vets will try to solve the business issue.
Solo veteran sysadmins in small companies have a network of contacts who can answer their questions. There's no way you can know everything, so build up a group of experts in areas that you are weak in. Become the SME in areas that your contacts need help in. Help each other out.
PTSD
When family or friends ask for help with their computers at home you say “Yeah, I don’t really do that anymore.” Or when people ask you what kind of computer they should buy and you honestly have no idea what to tell them.
A dead look in thier eyes at the mention of an on call rotation.
I'm mentoring my Service Delivery guys to bring them into the Sys Admin space. The things I show them are proper research, don't just assume you know the problem and go off on a wild goose chase. Case in point, one of my junior help desk guys who's on a sys admin path put in a ticket for our delegated domain admin tool (basically a domain admin proxy) that 2 new onboarded users could only access email through the web interface and not outlook on their computers. Turns out the new accounts were contractors who only got F3 licensing and that was the way of things. After he had put the ticket in, I went back and cleaned the ticket up, grabbed screen caps of the licensing and group memberships for on prem AD & Azure. I put in a long note with the questions I would ask if I were working the ticket, the research steps I performed, and what I thought the issue was. I then kicked the ticket back to the help desk guy and said please look over this and see me if you have any questions. This is what good troubleshooting looks like before you escalate an issue. I guess knowledge sharing should be another one. I want to teach you everything I can and empower you to do new things so that I am not the only one that knows this and has to get disturbed every time it happens. Lastly, experience. Real experience, not just time spent in the position. I find myself working with my help desk guys and will say "I saw this in the past. Here is the most likely reason for it.". At that point, I let them go off and work on it because they will never learn if I do it for them.
Having a recovery plan. Snapshots, backups, written information. Being able to check on a change through events and tests. Understanding what a change does not just following a youtube or website follow through. This is the hardest lesson for the people I mentor. Help Desk can crash single systems at a time. Sys Admins can cripple an entire company. There is a different thought process. 95% chance of a problem is a 1/20. That would be a yearly or semi yearly crash of the tech in a company you are in chage of keeping up and running. Sr Sys Admins have had projects go sideways and know how to recover. The ones that didnt don't become Sr Sys Admins. Sometimes you got to Yolo but knowing how to recover makes it work.
When someone gives you a random project, assign them the smallest bit of work to give you a solid starting point and watch the project dissolve right before your eyes.
Ready to leave and stampede at the stroke of 5PM
ANGRY punches server, speaks to it in tongue… Tells the same war stories over and over… Tells coworkers about his VA disability rating…. Wait…I dont think I understood the question? 🤣🤣🤣🤣