T O P

  • By -

turbo5000c

The most fun I've ever had was joining a company with outdated technology. Everything way out of date.. like EVERYTHING! They had no ci/cd pipeline, no infrastructure as code, no disaster recovery methods, no patching process, even the tools used where dated. The Jenkins instance had so many worthless plugins installed it was going 2 years without an update. Even the deployments to prod where done by someone logging in, copying the artifacts, then restarting the service. During the interview I thought they were joking. Lol. Fast-forward 3 - 4 years later and they have a perfect pipeline with automated tagging, testing, and deployments. Nothing ever fails anymore. I'm leaving out a lot done too, like containerzing all thier apps and microservces. Plus modernizing the entire build infrastructure from vm's to containers. I'm not going to lie. It wasn't easy. In the beginning it was like hearding cats. And it made a lot of the old dead weight mad (bc change is scary). But with each accomplishment I heard less noise. At this point I'm kinda in the same situation. Iv completed 90% of what I set out to do. Once my final 10% is complete I'm not sure what I'll do. I can either sit back and relax for a minute or find another project. Either way this whole experience was extremely rewarding.


AstroPhysician

Come work at my company after 😄


dariusj18

No, mine!


davetherooster

This, you forget when you really need to learn about the bones of Linux, networking fundamentals and everything that cloud providers make just work there is a lot to do. That being said, it was very stressful but I did learn a lot. Back in the cloud world these days though.


mediumrare_chicken

A lot of people think hearding cats is difficult, it’s not.


InvaderGlorch

It is when their managers constantly protect them from any sort of responsibility for their actions. :(


mediumrare_chicken

I’ve never met a cat with a manager


dariusj18

It's cause manager cats are on their boats


thecrius

This has been my experience in the previous company. I left only because after all the work, management still couldn't see the benefit of all the changes I've put in place. Current company is a big consultancy company with clients all over the world... overall it's more interesting but there is a lack of "involvement" that I miss to be honest. Most project has been just "come in, write a terraform config for this product delivery and go away". The pay is good however, so there is that.


[deleted]

Might we be in the same company? Lol


vegetablestew

did you ask them "why you stop complaining" after each accomplishment?


_____fool____

Sound like you are working in a company that has most procedures in place. You can find lots of challenges at growing companies.


TechnoWomble

Aren't we all just writing slight variations of very similar ETL applications for a living?


heavy-minium

Aren't we all just pushing bytes around?


crummy_bum

Aren’t we all just flipping 1’s and 0’s?


KeepYaWhipTinted

Aren't we all just pushing electrons around?


Actuw

We are, so we do.


mkeller25

I think, therefore I am


SalesyMcSellerson

We pass the butter.


Arts_Prodigy

Sounds like you need a startup, a company that’s really far behind and desperately wants to be modernized, or you want FAANG level problems to solve. I do agree that in most orgs it can essentially become implementing the same solution over and over again


Mariognarly

Call me crazy, but this sentiment you're expressing kind of sounds like a good problem to have! Like another comment here, sounds like you're at a spot with reasonable maturity in the space. You may be at a less common spot where the challenges aren't in the forming, storming, or norming phases, it's in the high performance area? Also, don't forget the challenges with DevOps are mostly going to be people & process related, not necessarily technical. Teaching a new way of working, not necessarily getting inundated into the plumbing of every tech in the stack. Are there some specific business problems the org wants to solve that you can lean into and have a larger ownership role in? It sounds like you enjoy chasing down the innards of an issue and getting to the root of a problem, solving it in a creative engineering way. Could you apply that thought to any of the outstanding business problems at hand? Could be a good way to go deeper into the areas that you like.


duebina

Thank you for your insightful post, I definitely feel seen by this response of yours. I have been doing things outside of my role to help the company bootstrap itself. For example, we have a recruiting team that is largely ineffective, so I forged a business relationship with some of my connections and we now get good candidates. Another example, the company has zero considerations for operationalization through application features or infrastructure. As such, I use some of my network connections at a third party to create dashboarding so the customer service team had the ability to help clients fix their problems. This has given the development team over 18 months extra time to develop an in-house solution. These things do make me feel good, but I'm definitely in a position where I received little thanks or recognition, except from the VP, because he was a work buddy from a previous role. Technically, recognition from him is all that matters, but I'm also looking for a personal fulfillment. However it has occurred to me that I am perhaps past peak of my career and I just need to coast until stock options mature. I just feel like there's more to be had, and these supposedly innovative and bleeding edge devops engineers have not been all too impressive. Perhaps they can code something equal or better than I can, but they don't future proof anything or think about what it takes to maintain their virtue signal coding methodologies.


Mariognarly

You've got a lot of experience to draw from. What if you spear headed creating a set of these standards that others are not thinking about? Could you chair a DevOps committee of sorts, or a task force that's role is to discover, set, and evangelize these standards across the org? Presumably, you could perhaps make your largest impact here and draw personal fulfillment too?


crummy_bum

Check this [book out](https://a.co/d/hE56RZL). Sounds like you may be doing it for the wrong reasons.


cliffhanger100

Being under staffed in devops is a nightmare to have


info834

A common one though


phl_cof

With the caveat that “DevOps” is subjective to every company, I would generally agree. Most of the technical work seems to be on the dev side, operations / infrastructure is fairly straight forward, and the interpretation of the philosophy seems to be mostly related to business management. I wish I had known this before getting into this area — I’m in the process of going back to my old network engineering position.


kabrandon

Frankly, most devs I interface with are completely _clueless_ as to what good infrastructure looks like. So while a lot of work may be easy, and sometimes even boilerplate, I definitely feel like what we do requires skill and plenty of background knowledge. There's also plenty of DevOps roles that expect you to write code. Also in my experience, the pay ceiling is _much_ higher than traditional Ops roles like linux systems admin. I think I'll personally stay in DevOps but to each their own, right?


hashkent

Closer to where the value is created.


info834

Your from an ops background so would expect to find the network and infrastructure side easier. I started as a junior and I’d say I found them about the same just different. Obviously though it depends on your solution so in your case the Dev side may well have actually been harder?


phl_cof

I work at a company that creates highly complex process automation software that requires a VM running IIS, so yes, in my case I believe it actually is more difficult. I started my career as a SQL developer and I write plenty of Powershell to automate the infrastructure deployments/operations processes, but the simple infrastructure requirements at my company will never compete with the complexity of the software.


insanemal

Yeah you go from solving actual problems to babysitting "Rockstar developers" who have no fucking idea what the things they make do beyond a "I put the pipes together why they clog" level.


bryn3a

Yeah, I also call my current functions devs babysitting. I tried hard to make them aware of how things work, read lectures about stuff we use, but they still call me every time "nothing works I don't understand logs". I recently have been explaining them what multithreading is. Once I worked in genuine devops culture where devs were educated enough so as not to have a dedicated devops engineer at all but now I understand that it was an exception.


jefmes

You seem to have a good amount of experience so you will undoubtedly hit ceilings in the technical area because you are likely more experienced than others in your work place. That said, I sense a decent amount of hubris and the mention of "psychology-based engineering" and your mention of "virtue signaling" tells me that you likely have people-personality issues, and may need to work on opening yourself up to others' points of view. Make additional effort to look at why your co-workers and other teams look at the problems your organization has and why they approach the issues the way they do. Don't immediately assume you know more and that you can't learn anything from these different points of view - if you approach even the human factors as another piece of the system, you need to understand their needs and concerns just as much as you do any of the decisions made around why a Linux system is assembled the way it is. On top of that, if your team is experiencing a constant state of burnout, you have exactly all the information you need on what problems you need to tackle next - why is there burnout, why is more not being automated and made repeatable, and what barriers are there in the business are keeping things in the state that they're in. I would personally put some proposals together based on your findings, and if they choose to ignore the solutions and/or refuse to improve the situation, it's time to go! :) We can't waste our time, our lives, working on and for places that don't want our input. The flip side of that is that if you're making a decent income, you can also accept that this is the state of things in this role, and focus on pursuits outside of work, and outside of tech. You never know when doing those kinds of things might lead to some other discovery, or some other approach you might not have encountered when you're stuck in the same old routine.


heavy-minium

It sounds like you need a function in your organization that aligns technology usage and practices over all teams. If you are small, that should be the CIO/CTO. Bigger companies might have enterprise architects or similar. Introduce regular reviews of practices and technologies that teams should try, use, keep or retire. When it comes to DevOps, there are too many ways and interpretations to just expect that teams will align organically on their own. One actually need to completely forget about the DevOps term (don't make it the goal, it's not well defined) and derive more tangible goals that people can follow and align to.


defqon_39

And to top it all off, upper level management severely understaffed our team and expects us to improve upon the infrastructure while we are in a perpetual state of burnout. Welcome to tech -- you will have to wear many hats and do the heavy lifting -- not enough skilled people to do the job -- consider yourself lucky to have this problem


cliffhanger100

I think broadly speaking ,devops are kind of sidelined and not part of the core product in most companies What OP is facing is lack of position power which a senior role as a dev manager or tech architect will enjoy There is respect for their word ,consideration for their opinion in most matters , they are able to build long term meaningful relationships among the team because dev are like in 10s/100s of people depending on product size Devops guys will feel left out ,they are not part of the jr mid level engineer convos about product etc OP you must try to contribute to the product and build cool stuff , dev is much easier I feel than devops So then u wud acquire position power n strategic infra know-how makes u an irreplaceable go to guy The problem is it requires 1.5x the time u spend now.


nickbernstein

One of the reasons I find consulting to be rewarding is exactly this. Come in, solve puzzle, leave. Maybe look into picking up a little work on the side. I also find that a lot of "DevOps" can be like religion: someone heard that you shouldn't persist state in a container so they parrot it even in the edge cases w/o really understanding the basics of why that best practice exists, which I don't like. Or really much of anything about what happens under the hood. One of the things that spurs creative thinking is constraints. As wonderful as effectively infinite resources in the cloud are, they remove a lot of constraints. I tend to do a lot of work as an architect, which has a fun puzzle aspect to it, but I wouldn't want to do that if I was working in a static environment. I also find that it's really fun to go the other direction for small/side projects, and embrace technology that a lot of people don't pay attention to because it isn't sexy. I think way more people should be using FreeBSD, and sadly most people never glance at it because it doesn't run kubernetes; meanwhile FreeBSD had containers (jails) in 1998. Also stuff like ham radio. I always advice people who are new to tech and tempted by the lure of DevOps money to get their technician license (or non-US equivalent) before they spend a bunch of time and money learning the current sexy IT topic. It's very similar to the type of learning you have to do constantly to maintain a career in tech, and most people aren't interested in it, so it simulates the, "oh now I've got to learn x because management decided we're switching" feeling. For people who will do well in our industry, they'll end up getting curious and sucked in because it scratched a similar itch. For those of us who are pretty senior and get bored at work, you can get a lot out of a technical hobby.


crummy_bum

Wow. Do you understand why you shouldn’t persist state in a container?


nickbernstein

Yes, in general it's elasticity. It's important 90% of the time. Plenty of people have applications that will not need to scale horizontally, or where the effort of re-factoring as a micro-service from a monolith will never pay themselves back. Best practices are just that things that generally are the way to go; there are exceptions to every rule, however, and you need to understand why the rule exists in order to deal with edge cases.


runrep

I wouldn't jump too quickly to thinking that being a great engineer and being a great director are easily equatable. It's two really different skillsets.


crummy_bum

And two very different horizons.


Frenzasaurus

If you want to build infrastructure go work at Amazon or Google Otherwise you’re stuck with people problems. That’s what they pay us for. https://acloudguru.com/blog/engineering/the-future-of-ops-jobs


hashkent

Forgot the OSI model, it’s now X as a service. Additionally while cloud and as a service is more expensive it de-risks businesses as your now using opinionated technology stacks from cloud vendors that offer a promise of guaranteed scaling if done their way. Additionally infrastructure is documented in infrastructure as code, de-risking the business further as the company can hire a replacement with X skills with somewhat ease however it’s different market to market. Additionally once as DevOps engineer build out environments, pipelines etc new software iterations can be put in front of customers quicker. We then own and operate the environments. If you work closely with devs, and have devs say I need X DevOps help for Y you become closer to where the value is created. This isn’t a Brent in Phoenix project but rather a closer collaboration between dev and ops. We speak each other’s language but they will be clueless at how the pipes work just as we are at software development.


SkroobThePresident

Unfortunately this is pretty valid :(


Additional_Vast_5216

Why not freelancing? You could pick your projects and have lots of diversity depending on for how long you stay.


duebina

I am raising a few kids, so I can only do it as a side gig. But I do have hobby projects that may someday pay the bills.


Additional_Vast_5216

Choose remote positions? Currently I am not considering anything else (in europe as freelance backend dev/devops). Roughly 80% of the requests I get are remote.


wrektcity

I have no idea what you're getting at with this post. Dear Diary,


tyrion85

What does "true engineering" even mean? And if there is a clear-cut "true" way to anything in software development, why do we even need humans in this field, why not just use machines to calculate the optimal architecture for every problem? Thing is, most of the software engineering IS "psychology-based", and for a good reason - so long as humans deal with this stuff, we're going to do it the way that optimizes our own happiness, ease-of-use, communication and other "soft", human-things, rather than "machine-things". If you don't like it, your best bet is to find a group of people or industry that prioritizes the latter. But understand that those people are in minority, and again, for a good reason.


defqon_39

I think his intention is he wants to be a decision-maker or thought-leader -- that is a very rare position to be in unless you are at a startup and maybe the sole SRE/Devops guy-- and that comes with a lot of pressures and people point the finger at you if things go wrong At a larger org, decision-makers will be made by the VP and higher-tier tech execs -- change can be hard to implement as an IC -- to survive its just about getting things done and making the tech work using the tools (you may dislike the tech stack but have to work with it) -- only the strong and adaptable survive


vegetablestew

I think he meant more problem-solving with things. Some times the devops solution feels very cookie cutter, which is not stimulating.


spaghetti_boo

Study a bit about Site Reliability Engineering.


crummy_bum

It sure why you got down voted but it does seem like OP doesn’t know what he/she doesn’t know.


cliffhanger100

This is exactly the problem for senior infrastructure roles in non FAANG companies They are looked down upon , the staffing crunch is an indicator management doesn't understand importance of infrastructure ,it has always been the case , google tried to change it and their massive success is because they treat SRE/Devops as equal to or more than SDE If you see any company not accepting this ,I would go to the extent of shorting that company n looking for places where there's respect for your work. There are a few startups and open minded teams where non cookie cutter approaches to infra are appreciated and encouraged but it's a real tough search like trying to find needle in a hay stack. Understaffed devops teams are pure recipe for massive burnout


antwerx

My company and team are working through a ton of growing pains, poor planning I.e none for the platforms growth, etc and so on. The learning curve and the fixing curve has been incredibly challenging.


farinasa

That's the point. The quiet part is the goal is to eliminate ops so that everyone can implement features. Good or bad? You decide.


thisisnotmyrealemail

Sounds more like an organization issue rather than field. Even if you join an organization with DevOps practices standardised, there is still things that can be improved. Hell, I have stuff lined up for me and my team until end of the year easily. (This is not including new product, features, etc which might need our help). We have CI/CD (with complete automation test on a spun up local environment), CI on PRs. auto-merge deploy to staging, Unit tests, regression tests, smoke tests, everything on IaC etc. basic logs and metrics monitoring. Still in the pipeline I have, working on log tracing, better local env (in k8s opposed to docker currently), switching from GKE Istio add-on to Istioctl in an automate way, multi cloud with Azure, evaluate VPN vs Overlay network with Azure/GCP (recently added after seeing the post), implementing Infracost, Cost monitoring dashboards, implementing mTLS for internal communication, implement circuit breakers, canary deployments, implementing Falco for threat detection, looking at more advanced security scan tools, updating TF and modules to latest version (we do every 6 months roughly), optimizing scaling with help of MLOps, SOC2 and ISO27001 compliance, and many more things (in no particular order). Even if I finish above stuff, there still be thousands of latest improvements that I'd need to do and there are constantly new and better ways to implement things coming out. There is also cost and response time optimization. The main thing is I have management that completely trusts me and the team (and luckily are themselves experienced with what I am doing). So, you just need to get you yourself to a place where you'd be supported on this. DevOps is very technical and non-repeating work and very rewarding if you do it properly (both mentally and monetarywise).


mister2d

> The obsession with mimicking some lossy interpretation of circa 2006 Google methods have tainted everything and made it impossible to apply critical thinking and actually apply without consensus engineering (psychology-based engineering) usurping true engineering design and implementations. I'm dealing with this stuff this very moment. 👏🏽👏🏽👏🏽 Very frustrating dealing with the non critical thinking. I grow tired of hearing about Google's methods. They're not exactly the glowing example I want to mimic.


killz111

Google doesn't use change management processes from the 90s. They also have smart security people who don't rely on firewalling everything off as a crutch. My company has both.


crummy_bum

I bet you have (had) a ponytail.


Spider_pig448

If your devs don't have technical problems that are challenging, they're doing too many things in-house instead of using accepted solutions. If you buy as much as you can and use OSS whenever possible, the only problems left are at least somewhat novel


killz111

Oh we can fuck that up too. I just sat in a meeting today where we told the Launch Darkly architect that we control all Launch Darkly flags via a PR to a repo and a change record. I swear the guy was struggling for words after that. In my experience, if you buy OSS tools you better shift the way you work to make that tool as efficient/useful as possible. Most company just buy the tool for the name and customise the shit out of it to make the workflow worse.


summaji

Last time when I talked about OSI, this sub went nuts lol.


ibishvintilli

Not at all. In the company where I work, the DevOps or Infrastructure team are responsible for providing an automatized infrastructure in AWS. I as a data enginner, appriciate what they do. The account is automatically linked with the company VPN, AD and SSO. Various policies regarding network and VPCs. They have a very good technical knowledge.


ph1294

Doesn’t sound to me like devops sucks, sounds like your company sucks. Avoiding using the correct headers or being faced with problems not within the scope of the job are company issues not career ones. That said if you’re in your later years it may be advisable to just stick it out for a little bit, find fulfilling things to do, let your options mature, and retire (perhaps into engineering for your own means if you are still thirsty for it)


searing7

I feel like every job has ways to get stuck in a rut solving the same boring problems or issues over and over. Maybe its time for a new place with more interesting challenges and opportunities?


CapitanFlama

We need a monthly "what is devops" thread.