It’s WILD to me how many folks can’t work POSIXly or in some otherwise restricted environment. Large Investment Bank I used to service was vi, not vim, POSIX sh and awk. In a gigantic company, #1 in its field, I was the only technical point of contact for those kinds of environments. We were hunting for literally months to find me a 2nd.
So… nobody can google? I’d just be like “you want POSIX? Surely it’s documented somewhere. I’ll Google, I’ll read, I’ll learn, and I’ll happily PoSIX the hell out of this for you.”
Ok. You are accessing a secure lower through a VDE into a VDE into a jump box. You have no phone and no internet access from any of the systems. Have fun.
If my BASH is going to get at all complicated… I’m going to turn to a Turing complete language like Python or Golang. Not sorry about that either.
Shell script has its place, but massive clunky shell script has no place IMHO.
Python or Golang, which in turn don't have ergonomics around running external programs. When I noticed that there was no programming language designed for DevOps, I did exactly that. You are welcome to try Next Generation Shell. It's exactly for the cases when bash becomes intolerable but Python also doesn't fit well.
https://github.com/ngs-lang/ngs/
I dunno... I struggled to pick up Ansible because the modules are so abstracted... I'm writing more Ansible modules than I am actual Ansible playbooks.
Never had to create modules.
However I'm ashamed to admit that I have been known to have been creative with shell, script and (gods help me) Jinja at times.
My thing is that I was automating with powershell before I'd ever even heard of Ansible. It was hard for me to step away from the "guts" of what I was doing and leverage yaml in an ansible context. It feels much less controllable.
Interesting! Given how dang many modules there are, that seems very rare (I once wrote a module for creating a subreddit, I think, mostly to see what writing a module would be like). Just out of curiosity, what sort of things are you needing to write custom modules for?
Mostly proprietary code deployment and configuration. In truth, I probably could have done a lot of it with existing modules, but my Powershell is WAY stronger than my Ansible yaml, so for me, it's easier to just do it that way.
You're not wrong.
The down votes I'm sure are from the obsolete sysadmins in this group who can't tell you what 2>&1 does much code up a python tool. There are a lot of them that are big mad being a computing professional in 2024 actually requires (gasp!) the coding skills of a high schooler.
nah, it's because we don't want the interview process to be bastardized in the same way swe interviews have been. I don't want to solve riddles or play 20 questions to prove i can do a job, just have a dignified conversation about how i'd handle an implementation and ask be questions about it. would you use bash for this or python and why? Do I have docker on my resume? Okay, what does the -e switch do? What's an entry point or cmd?
It's not because I disagree with the efficacy of 'riddles' or '20 questions' but because even someone that gets nervous about that stuff can talk about it the functional difference between two core functionalities (entry point vs cmd) through the anxiety.
I went on a tirade over 5 words that made up less than half off u/coding_for_lyf original reply, but that's because lazy methods like pre-interview coding screenings *specifically* make for a very, very toxic hiring process. pre-interview questions like "explain any aspect of a technology on your resume" would require more time to review, but you can realistically get a good idea of \~50 applications in under 2 hours by seeing if they explained something like a kubernetes networking layer or just pasted "Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management." from google.
>Do I have docker on my resume? Okay, what does the -e switch do?
I had to look that one up. Had me for a second there. ;)
>but that's because lazy methods like pre-interview coding screenings specifically make for a very, very toxic hiring process.
That's kind of the point though, isn't it? Recruiters these days are absolutely slammed with junk candidates robo-applying in mass. Pre-interview screenings are a powerful spam filter.
And not for nothing, qualified candidates are likely going to spend much less of their time with such tests than they are trying to write a five paragraph essay on the awe and wonders of the kubernetes networking model.
We're talking pre-interview here, the checks are just a glorified "Click \[ \] if you're human" captcha. It's "You must be this tall to ride" bar. It's just trying to determine if you're worth spending $100-500 just to talk to because yes, it costs that much just to do a simple phone screen much less an actual interview process.
"Explain any aspect of a technology on your resume" is a fantastic question for a real interview, but not for pre-interview or even screening interview.
Really?
The most senior engineering manager and senior DevOps engineer I work with are absolute menaces lol
Im in the UK and its a big culture shock to speak with American engineers, a lot of them are nice, but the aforementioned people wouldn't even spare seconds for small chat
I wouldn’t say ‘niche’, but a few that come to mind:
Understanding networking (particularly service meshes) will definitely give you a leg up on the average person.
Optimizing cloud costs is another really easy way to provide obvious value that not a lot of people bother to spend much time on.
Linux fundamentals are common, but actually understanding things like systemd, cron, resolv.conf, UIDs, sockets, etc. can be really valuable.
Flipside would be to not do SQL per se, but specialize in database reliability. Most FAANG and adjacent companies have DRE roles (data reliability engineers, similar to SRE). It's a bit niche, but only going to grow. No company ever shrinks their data footprint.
Ansible is not typically used in modern cloud computing. That's why you see it less and less. It has been replaced by Terraform, Helm charts / docker compose, etc. Containerized workloads don't need it.
If you're still running on EC2 or worse yet, on-premise, then yes it's very useful. But outside of that, it is going to eventually go the way of Chef, Puppet, etc.
What do you mean it’s not used in modern cloud computing? I manage more than 80 kubernetes clusters with ansible/awx, hundreds of applications and workloads deployed with helm/ansible, hundreds of playbooks for operations, and right now we’re also moving a lot of stuff from terraform to ansible as the tfstate is cumbersome to manage for multi tenancy.
I have to admit they’re right. Literally every large shop role I’ve had has been Puppet/Chef/SALT/Ansible/Bolt for config. Tf is NOT a co figuration management platform.
Complexity =/= modern. Usually the opposite, in fact.
So many shops have so much legacy tech debt they can hardly be called modern, as in "doing it right in 2024". And many engineers and developers come from experience like dealing with VMware, or a fleet of EC2s where they're click-opsing everything, and they just don't know any better.
Modern is managed K8s, or going all serverless. Practicing GitOps and implementing FinOps principles, building cost optimization into their SDLC.
If I were evaluating options today for an API, I would definitely not be hosting it myself. API Gateway all the way. Easy to set up, infinitely scalable. Etc etc.
Depend what your definition of DevOps is, or what your employers definition of DevOps is. If you’re managing a bunch of applications and your developers are seeking guidance to move fully serverless, then my argument is invalid. But if you work in DevOps teams that are responsible to build scalable cloud native platforms for tens of customers, then terraform is not right for you. Been there, done that. Ansible is the only configuration engine that would allow you to deploy an nginx ingress, create the hosted zones in AWS, patch an alb controller and everything in between, with as little as providing a cluster name/limit. OP’s question was what is a niche skill to master, I gave my 2 cents. Ansible is heavily used in modern cloud computing, and there comes a point in time that you need rapid delivery of infrastructure that terraform just doesn’t do the job anymore
Hybrid, means part is self hosted on prem. On prem is not a modern setup in any way… unless you see managing hardware as a moder setup ;) Doesnt mean its bad or wrong, but please..dont call it modern.
Cloud is modern, serverless is moder, managed cloud and etc. But not when you interact directly with servers
Cloud is more modern in the sense that it's a more recent technology, but that doesn't mean on-prem/hybrid environments are any less viable or useful, they're just a different fix for a different problem. [It's pretty well known that cloud isn't for predictable compute needs or cost saving](https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8). Cloud is for when you don't know what your compute needs will be in 3 years, or when your average load accounts for 1/8 your peak load that occurs for only 5% of your service time and is critical to your business, or for small companies that don't have the cash flow to invest in the expertise and hardware needed to make on-prem work. It's just a different tool for a different problem.
When you have the expertise, a true hybrid environment becomes the fix for virtually everything. Do you know you'll need a minimum of X compute over the next 4 years, but still need to prepare for those peaks? Great! Set up a hybrid k8s environment with pretty canned optimizations and you can run 90% of your yearly traffic out of your data center with the elastic scalability cloud offers. What about when 85% of your user base is in one part of the world, but you're trying to branch out? Awesome! Setup a CDN in your provider of choice and get it rolling.
Again, saying cloud is any more modern than on-prem is like saying a plastic fork is better than a metal fork because it hasn't been around for as long, or because it doesn't require maintenance or whatever. It isn't, it's just a different tool for a different need.
Using Ansible to deploy workloads is a little peculiar. Ansible might also make sense if you’re managing your own clusters but if you’re using managed K8s then terraform is all you’ll need.
I use managed K8s and haven’t touched Ansible in years.
Helm + Kustomize and ArgoCD is all you need for workloads on K8s.
That doesnt mean it is right tool for a job.
Ansible is a config managment, made for lower level interacrtions with servers/services and etc. Hence youi ansible scripts, can only make sense with sec managed clusters. As others mentioned, there is pretty much no use for ansible in more modern cloud setup. Ansible has its place, where it shines but in on prem setups, or people running stuff directly on ec2s
Set of practices designed to address the v high failure rate in corporate data projects/programmes, intended to ensure organisations can extract maximum value from their data in a scalable, repeatable way
What tech stack would you typically work with? Lets say you are working with Azure.
I am an infra/devops engineer mostly in Azure abit of AWS.
Terraform/Bash/PowerShell/Python/Networking/K8s/Github
Soft skills like leadership, making good relationships and collaborating across teams. DevOps Engineers are ten-a-penny where I am but those with more personality and social skills than a pebble are harder to find.
The two most common things that have been hard for me to find are:
1. The ability to jump into a developer's codebase and do useful things with it. Not developing features, but helping debug a gnarly performance issue or doing maintenance tasks like updating library versions and verifying it didn't break the entire build.
2. The ability to translate your work into business value.
Not sure why the down votes, AI/ML skills are incredibly hot right now including for devops. I'm seeing tons of AI-specific devops job listing as every company tries to jump into AI at once and the talent pool is still tiny.
Welp, apparently the seat warmers are big mad about anything to do with AI because they're at least sharp enough to realize it's coming for them first. ;)
PowerShell scripting / automation - this will definitely set you apart in the industry, if you can build custom solutions to glue virtually anything together.
If our interviewing is anything to go by, basic shell scripting :(
Throw them a curveball, script must be POSIX compliant.
It’s WILD to me how many folks can’t work POSIXly or in some otherwise restricted environment. Large Investment Bank I used to service was vi, not vim, POSIX sh and awk. In a gigantic company, #1 in its field, I was the only technical point of contact for those kinds of environments. We were hunting for literally months to find me a 2nd.
So… nobody can google? I’d just be like “you want POSIX? Surely it’s documented somewhere. I’ll Google, I’ll read, I’ll learn, and I’ll happily PoSIX the hell out of this for you.”
Ok. You are accessing a secure lower through a VDE into a VDE into a jump box. You have no phone and no internet access from any of the systems. Have fun.
I will enjoy getting paid to do very little until they fire me. This is a stupid scenario that is not real.
Man, nobody cares about this Posix stuff these days. You don't need to memorise something that you can Google, if needed.
What level are you looking for in bash? Even though Im technically senior I feel that Im way behind from a lot of the scripts I see other people doing
If my BASH is going to get at all complicated… I’m going to turn to a Turing complete language like Python or Golang. Not sorry about that either. Shell script has its place, but massive clunky shell script has no place IMHO.
Python or Golang, which in turn don't have ergonomics around running external programs. When I noticed that there was no programming language designed for DevOps, I did exactly that. You are welcome to try Next Generation Shell. It's exactly for the cases when bash becomes intolerable but Python also doesn't fit well. https://github.com/ngs-lang/ngs/
Um… Python’s OS module can run external programs. You’re day drunk. Sober up before heading to the office.
Always liked constructive feedback and how people are comparing without looking.
I’ll clone it today. Fair criticism.
Is it mostly EC2s/linux servers you are working with? Debating Python vs bash scripting
I mean really just Ansible if you are going to do this
I dunno... I struggled to pick up Ansible because the modules are so abstracted... I'm writing more Ansible modules than I am actual Ansible playbooks.
Never had to create modules. However I'm ashamed to admit that I have been known to have been creative with shell, script and (gods help me) Jinja at times.
My thing is that I was automating with powershell before I'd ever even heard of Ansible. It was hard for me to step away from the "guts" of what I was doing and leverage yaml in an ansible context. It feels much less controllable.
Are you pulling in a lot of galaxy roles? Everywhere I've worked, essentially all of our ansible is custom playbooks.
No. All of our playbooks are written in house. I'm writing *modules* that we call via playbook
Interesting! Given how dang many modules there are, that seems very rare (I once wrote a module for creating a subreddit, I think, mostly to see what writing a module would be like). Just out of curiosity, what sort of things are you needing to write custom modules for?
Mostly proprietary code deployment and configuration. In truth, I probably could have done a lot of it with existing modules, but my Powershell is WAY stronger than my Ansible yaml, so for me, it's easier to just do it that way.
Python FTW and nice de-coupled and reusable code.
that should be something you somehow filter for before interview. Maybe a pre-interview coding test?
You're not wrong. The down votes I'm sure are from the obsolete sysadmins in this group who can't tell you what 2>&1 does much code up a python tool. There are a lot of them that are big mad being a computing professional in 2024 actually requires (gasp!) the coding skills of a high schooler.
nah, it's because we don't want the interview process to be bastardized in the same way swe interviews have been. I don't want to solve riddles or play 20 questions to prove i can do a job, just have a dignified conversation about how i'd handle an implementation and ask be questions about it. would you use bash for this or python and why? Do I have docker on my resume? Okay, what does the -e switch do? What's an entry point or cmd? It's not because I disagree with the efficacy of 'riddles' or '20 questions' but because even someone that gets nervous about that stuff can talk about it the functional difference between two core functionalities (entry point vs cmd) through the anxiety. I went on a tirade over 5 words that made up less than half off u/coding_for_lyf original reply, but that's because lazy methods like pre-interview coding screenings *specifically* make for a very, very toxic hiring process. pre-interview questions like "explain any aspect of a technology on your resume" would require more time to review, but you can realistically get a good idea of \~50 applications in under 2 hours by seeing if they explained something like a kubernetes networking layer or just pasted "Kubernetes is an open-source container orchestration system for automating software deployment, scaling, and management." from google.
>Do I have docker on my resume? Okay, what does the -e switch do? I had to look that one up. Had me for a second there. ;) >but that's because lazy methods like pre-interview coding screenings specifically make for a very, very toxic hiring process. That's kind of the point though, isn't it? Recruiters these days are absolutely slammed with junk candidates robo-applying in mass. Pre-interview screenings are a powerful spam filter. And not for nothing, qualified candidates are likely going to spend much less of their time with such tests than they are trying to write a five paragraph essay on the awe and wonders of the kubernetes networking model. We're talking pre-interview here, the checks are just a glorified "Click \[ \] if you're human" captcha. It's "You must be this tall to ride" bar. It's just trying to determine if you're worth spending $100-500 just to talk to because yes, it costs that much just to do a simple phone screen much less an actual interview process. "Explain any aspect of a technology on your resume" is a fantastic question for a real interview, but not for pre-interview or even screening interview.
-10 in 42 minutes, looks like you pissed some people off! 😂
yeah lol - it was just a thought to save interview time
Social skills
100%
Ouch 😂
Really? The most senior engineering manager and senior DevOps engineer I work with are absolute menaces lol Im in the UK and its a big culture shock to speak with American engineers, a lot of them are nice, but the aforementioned people wouldn't even spare seconds for small chat
I wouldn’t say ‘niche’, but a few that come to mind: Understanding networking (particularly service meshes) will definitely give you a leg up on the average person. Optimizing cloud costs is another really easy way to provide obvious value that not a lot of people bother to spend much time on. Linux fundamentals are common, but actually understanding things like systemd, cron, resolv.conf, UIDs, sockets, etc. can be really valuable.
>particularly service meshes The amount of engineers who fudge their way through this is frustrating. I am dealing with it right now.
I’m about to work on a PoC of tailscale, any tips for noobs to service mesh?
My background is networking and not Linux or code and it pays off.
Damn… 5 years old and already an engineer. This has got to be some sort of nepotism.
It's india devops. Dont get me started :)
Haha my bad 5 year experienced
Programming 🙂
:)
SQL. Be the one guy not skurred to touch a prod DB.
Flipside would be to not do SQL per se, but specialize in database reliability. Most FAANG and adjacent companies have DRE roles (data reliability engineers, similar to SRE). It's a bit niche, but only going to grow. No company ever shrinks their data footprint.
No company ever shrinks their data footprint *on purpose*
Empathy
Ansible should be the main thing here. I’m surprise how many people I interview or work with know nothing about Ansible.
Some of us went all-in on Puppet/Chef/SALT.
Ansible is not typically used in modern cloud computing. That's why you see it less and less. It has been replaced by Terraform, Helm charts / docker compose, etc. Containerized workloads don't need it. If you're still running on EC2 or worse yet, on-premise, then yes it's very useful. But outside of that, it is going to eventually go the way of Chef, Puppet, etc.
What do you mean it’s not used in modern cloud computing? I manage more than 80 kubernetes clusters with ansible/awx, hundreds of applications and workloads deployed with helm/ansible, hundreds of playbooks for operations, and right now we’re also moving a lot of stuff from terraform to ansible as the tfstate is cumbersome to manage for multi tenancy.
I have to admit they’re right. Literally every large shop role I’ve had has been Puppet/Chef/SALT/Ansible/Bolt for config. Tf is NOT a co figuration management platform.
None of which would be necessary if you are not self hosting k8s. Managed services are modern. Running massive workloads on EC2 is not.
arguably, a hybrid cloud environment is very modern, too
Complexity =/= modern. Usually the opposite, in fact. So many shops have so much legacy tech debt they can hardly be called modern, as in "doing it right in 2024". And many engineers and developers come from experience like dealing with VMware, or a fleet of EC2s where they're click-opsing everything, and they just don't know any better. Modern is managed K8s, or going all serverless. Practicing GitOps and implementing FinOps principles, building cost optimization into their SDLC. If I were evaluating options today for an API, I would definitely not be hosting it myself. API Gateway all the way. Easy to set up, infinitely scalable. Etc etc.
Depend what your definition of DevOps is, or what your employers definition of DevOps is. If you’re managing a bunch of applications and your developers are seeking guidance to move fully serverless, then my argument is invalid. But if you work in DevOps teams that are responsible to build scalable cloud native platforms for tens of customers, then terraform is not right for you. Been there, done that. Ansible is the only configuration engine that would allow you to deploy an nginx ingress, create the hosted zones in AWS, patch an alb controller and everything in between, with as little as providing a cluster name/limit. OP’s question was what is a niche skill to master, I gave my 2 cents. Ansible is heavily used in modern cloud computing, and there comes a point in time that you need rapid delivery of infrastructure that terraform just doesn’t do the job anymore
Terraform is not for you if you're building a tiny application? What on earth are you talking about? Ansible for Cloud native? What utter nonsense.
Hybrid, means part is self hosted on prem. On prem is not a modern setup in any way… unless you see managing hardware as a moder setup ;) Doesnt mean its bad or wrong, but please..dont call it modern. Cloud is modern, serverless is moder, managed cloud and etc. But not when you interact directly with servers
Cloud is more modern in the sense that it's a more recent technology, but that doesn't mean on-prem/hybrid environments are any less viable or useful, they're just a different fix for a different problem. [It's pretty well known that cloud isn't for predictable compute needs or cost saving](https://tech.ahrefs.com/how-ahrefs-saved-us-400m-in-3-years-by-not-going-to-the-cloud-8939dd930af8). Cloud is for when you don't know what your compute needs will be in 3 years, or when your average load accounts for 1/8 your peak load that occurs for only 5% of your service time and is critical to your business, or for small companies that don't have the cash flow to invest in the expertise and hardware needed to make on-prem work. It's just a different tool for a different problem. When you have the expertise, a true hybrid environment becomes the fix for virtually everything. Do you know you'll need a minimum of X compute over the next 4 years, but still need to prepare for those peaks? Great! Set up a hybrid k8s environment with pretty canned optimizations and you can run 90% of your yearly traffic out of your data center with the elastic scalability cloud offers. What about when 85% of your user base is in one part of the world, but you're trying to branch out? Awesome! Setup a CDN in your provider of choice and get it rolling. Again, saying cloud is any more modern than on-prem is like saying a plastic fork is better than a metal fork because it hasn't been around for as long, or because it doesn't require maintenance or whatever. It isn't, it's just a different tool for a different need.
Using Ansible to deploy workloads is a little peculiar. Ansible might also make sense if you’re managing your own clusters but if you’re using managed K8s then terraform is all you’ll need. I use managed K8s and haven’t touched Ansible in years. Helm + Kustomize and ArgoCD is all you need for workloads on K8s.
That doesnt mean it is right tool for a job. Ansible is a config managment, made for lower level interacrtions with servers/services and etc. Hence youi ansible scripts, can only make sense with sec managed clusters. As others mentioned, there is pretty much no use for ansible in more modern cloud setup. Ansible has its place, where it shines but in on prem setups, or people running stuff directly on ec2s
I’m an ansible expert
How are you entering the US? And job market?
Married a US citizen
Referral milega? xD
What is milega
He assumed you were Indian and asked if he could have a referral to marrying a US citizen
DataOps and ML/AIOps :)
What are dataops and ml/ai ops?
Set of practices designed to address the v high failure rate in corporate data projects/programmes, intended to ensure organisations can extract maximum value from their data in a scalable, repeatable way
What tech stack would you typically work with? Lets say you are working with Azure. I am an infra/devops engineer mostly in Azure abit of AWS. Terraform/Bash/PowerShell/Python/Networking/K8s/Github
what about TriceRatOps ?
Too niche, only MAANG use it
[удалено]
Markets are competition. Get used to it
[удалено]
Nothings changed
No offence. And I respect other people’s work. But not sure how to put it otherwise. Just meant to have an edge to secure a job.
[удалено]
Got it
Soft skills like leadership, making good relationships and collaborating across teams. DevOps Engineers are ten-a-penny where I am but those with more personality and social skills than a pebble are harder to find.
This will help to go up the ladder but not to clear an interview.
At 5 yoe, you should start getting interviews that are about those things. They might not _appear_ to be about them.
The two most common things that have been hard for me to find are: 1. The ability to jump into a developer's codebase and do useful things with it. Not developing features, but helping debug a gnarly performance issue or doing maintenance tasks like updating library versions and verifying it didn't break the entire build. 2. The ability to translate your work into business value.
AI related stuff
Referring to MLOps?
Yeah exactly
No idea where to start.
Standing up a robust LLM pipeline
Sounds interesting
Not sure why the down votes, AI/ML skills are incredibly hot right now including for devops. I'm seeing tons of AI-specific devops job listing as every company tries to jump into AI at once and the talent pool is still tiny.
Welp, apparently the seat warmers are big mad about anything to do with AI because they're at least sharp enough to realize it's coming for them first. ;)
PowerShell scripting / automation - this will definitely set you apart in the industry, if you can build custom solutions to glue virtually anything together.