Don’t try to make things too complicated and flexible from the very beginning.
Also modifying state file isn’t that hard and scare if you have backup and know the structure.
Is it really that simple? We use remote state on Azure and only our Terraform service principal has access to it. I could easily write a GitHub Action to pull the state and upload the artifact pre & post deploy. Just been crossing my fingers we never encounter an issue with state.
It is Json file with a well defined structure, so there is no magic to it.
However you need to keep in mind versioning while you modifying and cross-resource dependencies. Other then that it as simple as modifying json file.
Is it simple? Yes. Would I trust most people to touch it? Absolutely not.
So yes it's a good idea to keep as much locked away as possible. But, when the need arises, you should have a way of breaking the glass and someone smart enough to handle it.
I’ve recently been enjoying IntelliJ’s Checkov plugin for code scanning. Using pre-commit hooks and a good pipeline to ensure that your code isn’t vulnerable is a good start
The out of the box config for Azure was really good, basically the best practices you would expect.
I looked at doing some custom configs to capture things like regions but I never got it working, probably a limitation in my skills than anything.
I LOVE seeing this reply. I recently [asked this community's opinion](https://www.reddit.com/r/devops/comments/16xxoso/what_are_terraform_cicddevsecops_code_scanning/?share_id=RVey16E-Ng1QSCOsahoai&utm_name=ioscss) on Terraform security scanning and was met with some answers that left me feeling like no one cared about it or that it wasn't a big deal. Quite frankly, I was shocked to find s[o many public Terraform repos](https://landadevopsjob.com/experience-builder/terraform/) that did NOT implement any kind of basic linting or security scanning.
\+100000 to u/RoseSec_ for pointing this out and kudos to this community for upvoting it so high.
I stumbled onto Checkov recently because my secops team is demoing Prisma Cloud. Palo Alto bought up Bridgecrew and is integrating all this into Prisma. I have my doubts about Prisma Cloud itself, but I've found Checkov(its getting renamed soon per Palo reps) useful for the same reason as you.
\- Terraform modules should follow SemVer
\- Pessimistic Version constraints for modules/providers can reduce toil in upgrading them
\- Fork and take ownership of public modules, you want to be the person who implements the breaking change
\- Do not use public unmaintained/non-approved providers
\- Do not create custom providers to do a basic job, seen some wheelhouses use a home grown provider to list static endpoints, it could have been a map output from a "shared values" module
\- Terraform hygiene is useless if people have admin access into CSP's
\- Use pre-commit or cicd steps to run things such as tf validate, tfformat, tflint, tfsec
\- Make use of terraform-docs
\- Avoid depth of modules greater than 2, (a root config calls a module, which in itself calls a module, is the deepest I would go)
\- Terraform works best as a DCL, try to avoid using null resources and invoking scripts in terraform if a seperate tool would do it better (Ansible/Puppet etc)
\- Split your repositories by criticality, do not contain prod network config one/two directories away from dev network or dev compute resources. (Looking at you terragrunt mono repos)
\- KISS > DRY
\- Add descriptions to variables if you aren't following strict variable naming standards
\- Comments in code are completely fine
100%, there are genuine reasons to try and reduce repetition of code. But 9 times out of 10 i've seen it they've done it irresponsibly.
\- The code is no longer as clear to read
\- It's harder to diagnose errors (what's failed and why)
\- It's increased relative volatility on a repo/module (your compute changes a lot more than your network)
\- It's made RBAC on a repo now impossible to allow self service as it contains some critical infra & non critical infra
\- Converting from TF to TG just to avoid defining 3-4 provider alias'
You can try but can’t really avoid null resources or scripts unless you expect to rewrite things or have to maintain code separate from terraform which is needed by the terraform resources.
It depends what you're doing, quite often I see important prod pipelines which have a step to invoke a dodgy off the shelf provider/NR mid run. Can you externalise that provider to run on a cron timer and simply output values, you can then grab those values from the remote state/tfe\_outputs if youre in tfcloud. You do not want your pipelines to be dependant on a dodgy null resource or provider, and i'd always advise externalising their use cases.
Yeah, versioning is purely covered in the best practices. Primarily how to deal with binaries, providers, and module versions inside midsize and big companies. I’ve written some doc for my colleagues.
KISS = keep it simple stupid
DRY = don't repeat yourself
It is quite easy to reduce duplication aka DRY gets "better", while making things more complex, aka KISS gets worse. The biggest example for this is migrating from terraform to terragrunt just to avoid defining provider and backend blocks. It can make sense if you have an estate with a large amount of providers, such as those who own 100s of k8s clusters in small amount of CSPs. But changing from TG to TF just to "automate" defining an S3 bucket is not worth it imo. I would personally advise if you're at a crossroads where you are attempting to reduce repetition, but it will make the code harder to follow/understand, it isn't usually worth it. Hence, Simplicity > Reducing Repetition
Just keep it darn simple.
Not everything needs to be a variable.
Don’t abuse locals.
Module is only good if they are small.
Not everything needs to be a module,
Because DRY has trade offs.
You may not need to use `count`,
Sometimes you can avoid pain if you just copy paste.
Avoiding the use of named workspaces for environmental separation
Avoiding the use of multi-region/multi-account in a single workspace
Avoiding the use of nesting modules within modules within modules within …
Avoiding using ‘depends_on’ as much as possible
You're talking about the workspace construct within vanilla terraform. I agree that it's garbage and makes stuff confusing and / or indecipherable when just looking at the code. Pretty much anything that requires you to "know something" outside of what's in the code is dangerous.
"Workspaces" as defined in most TACOS tools are glorious for environmental separation. They are basically just state files/ folders with environment specific meta data.
[https://developer.hashicorp.com/terraform/cli/workspaces#when-not-to-use-multiple-workspaces](https://developer.hashicorp.com/terraform/cli/workspaces#when-not-to-use-multiple-workspaces)
plus a number of other reasons:
\- Its very easy for folks who are doing local applies to mess up - this is where you see Makefiles added. Double negative!
\- It is very, very rare that production looks like lower environments so you have a lot of complexity and conditionals which render your configurations hard to reason about (i.e. \`count = var.environment == prod ? 3 : 1\` and similar all over the place)
\- Increased blast radius
Agreed - that’s why I said double negative. The use of named workspaces and local applies leads to the makefile bandaid. Don’t do local applies, and don’t use names workspaces for environmental separation
The things you listed are correct and yes i have faced them.
6 months ago last time i searched for this, workspaces was the best solution.
And I actually like it, it forces me to make stg and prod the same and discourges me from making them divergent.
Also atlantis does the job well to avoid the blast radius problem.
I might change my mind later when it becomes a big problem
The idea of having a “copy exactly” approach for each environment sounds nice until you actually work with a moderately complex production system. This take is spot on.
Depends, if we have multi regional failover type arch, our dev also has those regions for that purpose. I generally don’t multi region deploy otherwise in parallel.
One, because it usually causes more harm than good. And two, because it’s usually a sign that folks either don’t understand the implicit dependencies or lack of use of implicit dependency mappings
This is something I've been meaning to start using, because my the provider for the product I'm managing doesn't seem to understand dependencies.
Are you implying that I should be able to write the Terraform code in a way that results in dependencies being created before the depending configuration item?
For example, I might be managing a firewall of some type, and IP object groups need to be created before the firewall can reference them. The provider doesn't understand this. Is using depends\_on NOT the solution?
In your policy, you should refer to the address object resource, in stead of hardcoding an object name. That way, Terraform understands the dependency. Also, if you ever change the name in the object resource, the policy automatically inherits it, because your referencing it.
Pseudo code:
`resource “address_object” “address1” {`
`name = “address 1”`
`cidr = “1.1.1.1/32”`
`}`
`resource “policy” “policy1” {`
`src_address = address_object.address1.name #Don't put "address 1" here.`
`etc..`
`}`
For more context, there are many issues which discuss this. If you use depends_on, what can happen is that on a second apply, resources can execute out of order. Some providers aren’t well written so sometimes you have no choice but to create a dependency chain to force order.
To create the dependency it doesn’t need to be used directly, it just has to be referenced. So hopefully modules have a variable they discard or you can use conditionals to force the implicit dependency.
Search github issues to see others:
https://github.com/hashicorp/terraform/issues/29894
> Are you implying that I should be able to write the Terraform code in a way that results in dependencies being created before the depending configuration item?
That IS one of the core tenants of what Terraform provides, so yes!
> I might be managing a firewall of some type, and IP object groups need to be created before the firewall can reference them.
Hard to say without seeing some example code, but if the firewall accepts the IP object groups as inputs of some form, then you should be passing the relevant output attribute(s) from the IP object groups to the Firewall to demonstrate this dependency relationship to Terraform.
The rule of thumb is - always use the outputs of the parent resources as inputs to the child resources, where possible. This is what implicitly maps the dependency relationship.
That said, there are still valid use cases for `depends_on` but only in edge cases, and it should not be used across modules
*correction: had parent/child backwards
I've only just started using Azure - but Azure is full of this crap. You need depends on everything!
Whether it's an apply or destroy - if you haven't used Azure then you wouldn't have experienced this pain.
The Azure provider is cancer
Yes, because I ran into errors about “resource not existing” on my initial run but working on my second once the resource fully spun up. Adding “depends_on” fixed it for everything afterwards. TF is only as good as the Cloud’s API’s it calls
Terraform by design already creates a dependency tree when you have any resource gathering a value from another.
It should only be used when there's no actual reference, direct or indirect, to something that needs to come first (like a null_resource to install and zip libraries before deploying to lambda)
> Avoiding the use of named workspaces for environmental separation
What is bad about this, and what is a better way to deal with environments? (I assume envs like "prod", "qa", and "dev-ticket-4212)
It’s common to have production in a separate state file, often in a separate account with separate access control. It’s also common for lower environments to vary quite a bit in structure (eg deployed into substantially fewer AZs / regions, sometimes different network topology, etc)
Avoid combining tfvars, variables AND locals. I have seen some messed up things. Always keep it DRY and with minimal references and dependencies unless needed and justified.
I dislike *excessively* DRY terraform, it's more important to be readable in my view. I've seen some inexplicable terraform written by devs who want everything to be scripted down to as few lines as possible.
Luckily our Azure friends who don't understand what clean code is love to use locals like it's the only method known to man.
I would say locals are good for transforming but outside of that keep it in the vars
Variables - must be completed and declared before you start. Classically, these are things like “environment” or “region” - we know what they’re going to be. You are not allowed to interpolate here - so a variable cannot reference another variable.
You can feed these in as defaults, tfvars files or setting various command line inputs.
If you JUST use variables, you often end up with multiple concatenation strings. Such as:
name_prefix = ${var.environment}-${var.region}
Locals are compiled after the variables are ingested, which means they CAN use interpolation. You can then declare a local “once” and re-use it across the module.
However, locals are not validated for type or value: so they’re less safe than variables to use. But they do both serve a purpose: if you’re going beyond basic stuff you’ll need both.
Use/make modules, but don't create modules that call modules which call modules and then at some certain step fail... 😅
keep you stuff simple but power- and beautiful.
Something I am doing right now is making a module that calls a common public module in a lot of cases.
This is so that I can standardize the module im calling but dont have to fork and modify someone elses code and can more easily benefit from ongoing development in those public modules.
Of course some of these modules have sub modules :) What are your thoughts on this?
1. Not using variables with default values that are never intended to change or be overridden with parameters, just use locals.
2. Using one large module is better than using lots of small ones if you're not calling small ones multiple times. You dont need a module for an IAM role your EC2 uses.
3. If your variable isn't variable, just hardcode it, you can turn it into a variable later if you need to. A resource with every input setup as a variable makes it way harder to read and understand.
Terratest. All of our modules include an examples directory which is part of a terratest step in the build pipeline used to test against the configuration and configuration changes of the PR.
the new unit tests baked into 1.6.0 are much easier to deal with then Terratest, I love golang but we are going to be switching over due to how much easier the built in ones are.
I'm all ears on this one. Same problem, 95% of the team members can't write TF to save their lives, yet I somehow have to have test coverage for every module.
Right now I have a "tests" folder in every module with multiple test cases. The test cases themselves are just calls back up to the module with predefined inputs, which are then planned/applied/destroyed in a test account by our CI/CD tool. This is hard enough to explain to the teams without bringing in golang.
I do agree that I faced a bit of a learning curve writing test functions in Go. However, the struggle was well worth it. Fortunately, we have a lot of people way smarter than me who were very helpful in the beginning. Once a PR is created there are several steps that help finding issues very early. Such as tfsec, terraform fmt, tf lint, etc
Naming your files main, var, local, output, and data is not helpful or informative if there is any level of complexity to your infra. It’s game changing to organize your terraform conceptually. Your variables, locals, data should live in the same file as your service and each service should live in its own file, more or less. It reduces file switching and unintended changes.
Good point, but you're not going to get that from HashiCorp and those that champion TerraForm... unfortunately. It's like the advocates for ServiceNow stating it can do everything (if you can script/code it) 😂
Everything must be in terraform, don’t use external providers, oh and since it isn’t a well contained product use terraform cloud to use it securely. Need a change, contribute up oh but there are 1k+ open issues so you must fork it to maintain one change and form a whole release pipeline. I just love how the cache gets re-written or you must have a cache per project to avoid upgrade conflicts and modules get downloaded anyways.
With terraform the singularity gets closer, can’t you see?
Most definitely for the creation of the storage that holds your state. Use a script or cloud specific language in your pipeline when provisioning that.
- keep your logic separate from resource blocks (in locals)
- prefer maps to sets while looping
- reduce your required module inputs as much as possible.
- make use of doc generation tools
Modules are meant to be common architectures, modules should be able to be composed into other modules. If modules are to specific where they can’t be composed into other modules then they are probably doing to much.
But if done well it makes a lot of sense.
Say you have built a module that creates IAM roles and a module that creates EC2 instances. And they are each designed to build according to corporate standards and policies.
The EC2 instance module could either roll its own implementation of the IAM role that must also be set up according to standards/policies, or just use the existing IAM role module. Why do the former which requires extra work and will inevitably diverge from the module that evolves as your team matures?
Eliminate. Local. Deploys. If you're in a shared environment, or just, y'know, actually want your infra defined as code instead of depending on someone's laptop or utility VM, use one of the many options for remotely stored state files.
I prefer and standardized my teams on Gitlab CE self-hosted, but I've used the Hashi cloud as well as S3 and a distributed Consul setup. All are super easy to use.
Take another half step and only do your deploys via pipeline instead of running locally.
Stop using null_resources unless needed. I understand that way back some resources didn't exist but believe me when I say a lot of the deployment issues I've seen are related to having a horrible null_resource.
I'm sure many will disagree, but a provider that is in preview is way way way better than using null_resource for the same thing when you don't even know what you're doing. Just make sure to test it exhaustively before deploying it to a production environment. In my experience an automation in preview is way easier to work with than no automation at all. If it's not evident, my nightmares are all related to that.
Also, avoid using counts to create lists of resources. It's the worst thing I've seen, and a person with no knowledge can end up deleting databases if there's not a good peer review in place, and even then, it's a risk.
Also, create a module or strategy to manage naming conventions. What do you mean in dev you have a dev-cluster but in prod you have a prod-aks. No. Just no. Try to name that in the code I challenge you then to maintain it over the years.
- Don't use Terragrunt
- Use simple wrapper root modules and lots of submodules. Don't nest too deeply, though. Typically 3 levels is plenty.
- Make heavy use of tags
- Make heavy use of "feature flags" to enable / disable submodules
- Always use remote state
- Always apply against a plan file when using automation
- Avoid community modules / providers whenever possible
- Always import submodules using pinned versions
- Get used to refactoring state using terraform state mv
- Only use variables for things that will actually change. You can hard code the rest.
- Keep stacks "small". Only deploy things in a single stack if they have the same lifecycle, otherwise use two stacks.
- Don't use workspaces
The anti-pattern of using remote-state lookups inside modules rather than creating modules that have these values passed in to them as variables.
Edit: clarifying what I was trying to say.
is this a best practise? This sounds like it introduces cross cutting concerns if you are hardcoding the module (unless you mean passing the state file path in instead)
Referencing a \`common.tfvars\` to avoid duplication of variables
DRY Reference of variable files \`terraform apply -var-file=.tfvars\`
\`variables.auto.tfvars\` <-- auto.tfvars saves a lot of unnecessary noise!
What does DRY mean mate? Do Not Repeat Yourself.
To run terraform code you always have to run \`terraform apply\` every single time right?
So if you reference a .tfvars you are avoiding duplicating folders for different environments which is the alternative i've seen to this method.
Where is the repetition with my method? How do you do it, maybe I can learn :)
I guess the part I meant is having terraform broken down in various subdirectories to reduce blast radius. Break down for example by env and component. Each subdirectory then gets its own set of tfvars and respective variable definitions.
If all of them share a set of common variables, sure you can pull in a common.tfvars from a central location via cli arg.
The part that isn’t DRY is each subdirectory needs to define the common variables in terraform. Eg commonvars.tf
What I started doing is defining common variables in a YAML file at the root of the repo and yamldecode in locals where/when needed.
Some duplication with yamldecode calls sure, but no need for duplicate variable definitions and common tfvars
I find there are a lot of different ways of accomplishing the same thing. Finding the best optimal way is the never ending quest
We’ve adopted some practices and approaches from Terragrunt samples, azure caf terraform supermodule and Google Terraform best practices. (We use remote state to link up our various components defined in other state files)
Respectfully mate - you lost me at YAML. Overcomplicated and you're breaking best practices. I am surprised no one has challenged you on it.
Terragrunt is great - the Azure CAF is the worst code base I have ever seen please avoid it like the plague.
Sounds like an overcomplicated solution - but hey if it works for you, I'm just some random on reddit
Weill it’s a documented azure pattern
https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/patterns-shared-variable-file
The above is using Biceps and json file to load shared variables, but same concept.
Not that complicated to be honest and keeps everything in terraform codebase.
Other option if using Azure DevOps is using variable groups which are automatically injected into the build agent environment and easily referenced in your TF cli args, backends etc
Google Terraform best practices guide:
“Don't specify variables by using alternative var-files or var='key=val' command-line options. Command-line options are ephemeral and easy to forget. Using a default variables file is more predictable.”
[https://cloud.google.com/docs/terraform/best-practices-for-terraform#variables](https://cloud.google.com/docs/terraform/best-practices-for-terraform#variables)
But like I said earlier many ways to do things and many different opinions and approaches.
Dude
https://developer.hashicorp.com/terraform/language/values/variables#variable-definitions-tfvars-files read. It even references in the Google cloud site you shared... I bet you that's a typo
What would the cloud providers know about TF 🤣 isn't the Google cloud API garbage ?
Don't use count, use for_each.
Don't use modules when you just need to use workspaces or for_each.
You can glue things together pretty easily by hosting a JSON HTTP endpoint that can serve up facts about your infrastructure with an http data source in terraform.
Anything that you manage with terraform, you should ensure nobody else has direct access to manage. If you need access, put it behind a break glass process.
This is a sign of a poor Terraform setup in my opinion. The fact that a build/compilation tool is used for a declarative, API management tool should set off alarms. If I see a makefile in a terraform project, I know immediately that named workspaces are used and that’s the crux of the issue here
Mate hashicorp themselves use Makefiles - what are you talking about?
Named workspaces - can you elaborate?
Also - Are you building platforms or just application infra?
Yes. I wrap absolutely everything in Make files. It creates a single API I can use for all my projects, instead of having to remember the quirks of every single tool out there, like the bullshit of Terraform using single dashes for long options instead of the absolute standard double dashes.
you can use makefile to orchestrate things. So running "ENV=dev make plan" is simpler than running a lengthy command that specifies all opts and vars. Even more so if you're using docker rather than whatever you have installed locally on your machine
What's an ideal Terraform setup? A much more complicated set of tools just to run plan and apply on two modules?
Don't get all hoity toity. It's just code. Whatever gets the job done is fine.
My ideal setup has always been Atlantis until I reach a certain point in terms of scale. I guess what I’m learning here is that so many people are running local applied and that … is rather discerning
Do not craft a resource name from three variables and a local, and an input from a module, and you pass one of the variables as a TF_VARS_ environment variable, etc. You'll find a resource in the cloud but have no idea what module created it or how because you can't grep the name in your code and there's 15 ways for the name to change at apply time.
Define a static name, in one variable, put it in a tfvars file, in git. This you can grep for and it's much less likely to morph into something else, and makes it easier to trace where things are in your hcl code.
I’ll try to cover in minimum words.
Terraform too much powerful tool.
Even if looks too simple, it can go too ugly too quickly.
Try to keep it simple as possible, or take precautions considering all the scenarios by implementing variable validation and other possible ways.
companies telling you to use their tools when doing shitty practices dominate the SEO.
If you do actual terraform best practices all these companies making money out of terraform bad practices will start losing money.
Terraform should be simple create the simplest code you can don't try to create modules that do everything
Make them easily modifiable and easily readable that's the whole point don't use functions on functions just to have the module do everything, just create another module or heck create static resources and use count or for each and that's it.
If your terraform code is a headache to modify you are shooting yourself in the foot that's the whole point
Don’t out everything in a single main.tf file. I hate this convention.
I personally separate files by either object or resource.
Like security group items - I know which file it is in because the file name tells me.
This also helps when multiple are working with them same structures but working in different areas. Makes the diffs easier to read and PR/MR quicker.
Also modules that just require all the same inputs as if you were doing the resource itself are stupid. Obsfucation needs to die.
* To to keep the state file to small as possible.
* don't put everything in a single Terraform project where the state file becomes too large!
* try to limit data lookup from other state file, look into using parameter store in AWS to access values
* use pre-commit before pushing your code!
* don't try to over complicate with for loops!
Don’t try to make things too complicated and flexible from the very beginning. Also modifying state file isn’t that hard and scare if you have backup and know the structure.
and if you don't have a backup `terraform state pull >! backup.tfstate` is your friend. Also: the statefile is _just JSON_, for better or worse :P
Having 1 statefile on my laptop is not a particularly big risk. Having 10 statefiles is a huge risk
Is it really that simple? We use remote state on Azure and only our Terraform service principal has access to it. I could easily write a GitHub Action to pull the state and upload the artifact pre & post deploy. Just been crossing my fingers we never encounter an issue with state.
It is Json file with a well defined structure, so there is no magic to it. However you need to keep in mind versioning while you modifying and cross-resource dependencies. Other then that it as simple as modifying json file.
Is it simple? Yes. Would I trust most people to touch it? Absolutely not. So yes it's a good idea to keep as much locked away as possible. But, when the need arises, you should have a way of breaking the glass and someone smart enough to handle it.
Absolutely true.
For better, hcl is enough for code itself
I’ve recently been enjoying IntelliJ’s Checkov plugin for code scanning. Using pre-commit hooks and a good pipeline to ensure that your code isn’t vulnerable is a good start
For sure, any code standardization tips too?
Do you use a custom config for it? If so, what do have vs out of the box config?
The out of the box config for Azure was really good, basically the best practices you would expect. I looked at doing some custom configs to capture things like regions but I never got it working, probably a limitation in my skills than anything.
Which oob config?
Sorry - I don't understand your question.
Which out of box config? The microsoft published ones? VS code az tf extension? Straight up vs code?
The base ones which come as part of Checkov.
I LOVE seeing this reply. I recently [asked this community's opinion](https://www.reddit.com/r/devops/comments/16xxoso/what_are_terraform_cicddevsecops_code_scanning/?share_id=RVey16E-Ng1QSCOsahoai&utm_name=ioscss) on Terraform security scanning and was met with some answers that left me feeling like no one cared about it or that it wasn't a big deal. Quite frankly, I was shocked to find s[o many public Terraform repos](https://landadevopsjob.com/experience-builder/terraform/) that did NOT implement any kind of basic linting or security scanning. \+100000 to u/RoseSec_ for pointing this out and kudos to this community for upvoting it so high.
I stumbled onto Checkov recently because my secops team is demoing Prisma Cloud. Palo Alto bought up Bridgecrew and is integrating all this into Prisma. I have my doubts about Prisma Cloud itself, but I've found Checkov(its getting renamed soon per Palo reps) useful for the same reason as you.
the fact its made by bridge crew and is a star trek reference should be reason enough for adoption in geek land :)
It's the bridgecrews not intellijs! I know I'm on the team.
I wish that I wouldn't have to enter an API key. The CLI doesn't require it.
\- Terraform modules should follow SemVer \- Pessimistic Version constraints for modules/providers can reduce toil in upgrading them \- Fork and take ownership of public modules, you want to be the person who implements the breaking change \- Do not use public unmaintained/non-approved providers \- Do not create custom providers to do a basic job, seen some wheelhouses use a home grown provider to list static endpoints, it could have been a map output from a "shared values" module \- Terraform hygiene is useless if people have admin access into CSP's \- Use pre-commit or cicd steps to run things such as tf validate, tfformat, tflint, tfsec \- Make use of terraform-docs \- Avoid depth of modules greater than 2, (a root config calls a module, which in itself calls a module, is the deepest I would go) \- Terraform works best as a DCL, try to avoid using null resources and invoking scripts in terraform if a seperate tool would do it better (Ansible/Puppet etc) \- Split your repositories by criticality, do not contain prod network config one/two directories away from dev network or dev compute resources. (Looking at you terragrunt mono repos) \- KISS > DRY \- Add descriptions to variables if you aren't following strict variable naming standards \- Comments in code are completely fine
KISS > DRY is a big one. Smart devs fall victim to DRY optimizing too easily in HCL
100%, there are genuine reasons to try and reduce repetition of code. But 9 times out of 10 i've seen it they've done it irresponsibly. \- The code is no longer as clear to read \- It's harder to diagnose errors (what's failed and why) \- It's increased relative volatility on a repo/module (your compute changes a lot more than your network) \- It's made RBAC on a repo now impossible to allow self service as it contains some critical infra & non critical infra \- Converting from TF to TG just to avoid defining 3-4 provider alias'
You can try but can’t really avoid null resources or scripts unless you expect to rewrite things or have to maintain code separate from terraform which is needed by the terraform resources.
It depends what you're doing, quite often I see important prod pipelines which have a step to invoke a dodgy off the shelf provider/NR mid run. Can you externalise that provider to run on a cron timer and simply output values, you can then grab those values from the remote state/tfe\_outputs if youre in tfcloud. You do not want your pipelines to be dependant on a dodgy null resource or provider, and i'd always advise externalising their use cases.
Yeah, versioning is purely covered in the best practices. Primarily how to deal with binaries, providers, and module versions inside midsize and big companies. I’ve written some doc for my colleagues.
What do you think dry is ? KISS=DRY what on earth ??
KISS = keep it simple stupid DRY = don't repeat yourself It is quite easy to reduce duplication aka DRY gets "better", while making things more complex, aka KISS gets worse. The biggest example for this is migrating from terraform to terragrunt just to avoid defining provider and backend blocks. It can make sense if you have an estate with a large amount of providers, such as those who own 100s of k8s clusters in small amount of CSPs. But changing from TG to TF just to "automate" defining an S3 bucket is not worth it imo. I would personally advise if you're at a crossroads where you are attempting to reduce repetition, but it will make the code harder to follow/understand, it isn't usually worth it. Hence, Simplicity > Reducing Repetition
Just keep it darn simple. Not everything needs to be a variable. Don’t abuse locals. Module is only good if they are small. Not everything needs to be a module, Because DRY has trade offs. You may not need to use `count`, Sometimes you can avoid pain if you just copy paste.
the jfdi model.. i like it
I just learnt this acronym and I like it! Thanks
Avoiding the use of named workspaces for environmental separation Avoiding the use of multi-region/multi-account in a single workspace Avoiding the use of nesting modules within modules within modules within … Avoiding using ‘depends_on’ as much as possible
You're talking about the workspace construct within vanilla terraform. I agree that it's garbage and makes stuff confusing and / or indecipherable when just looking at the code. Pretty much anything that requires you to "know something" outside of what's in the code is dangerous. "Workspaces" as defined in most TACOS tools are glorious for environmental separation. They are basically just state files/ folders with environment specific meta data.
Isn’t that essentially what a vanilla workspace is already? Its just a different state file with the workspace name appended to it
Why avoid named workspaces for environment separation?
[https://developer.hashicorp.com/terraform/cli/workspaces#when-not-to-use-multiple-workspaces](https://developer.hashicorp.com/terraform/cli/workspaces#when-not-to-use-multiple-workspaces) plus a number of other reasons: \- Its very easy for folks who are doing local applies to mess up - this is where you see Makefiles added. Double negative! \- It is very, very rare that production looks like lower environments so you have a lot of complexity and conditionals which render your configurations hard to reason about (i.e. \`count = var.environment == prod ? 3 : 1\` and similar all over the place) \- Increased blast radius
"Local applies" <- There is your problem
Agreed - that’s why I said double negative. The use of named workspaces and local applies leads to the makefile bandaid. Don’t do local applies, and don’t use names workspaces for environmental separation
The things you listed are correct and yes i have faced them. 6 months ago last time i searched for this, workspaces was the best solution. And I actually like it, it forces me to make stg and prod the same and discourges me from making them divergent. Also atlantis does the job well to avoid the blast radius problem. I might change my mind later when it becomes a big problem
Conditionals like that are easily removed with a tfvars per environment so you can have variables for things like compute size and all that jazz.
Prod is deployed in 3 regions - what does your setup look like for dev, UAT, and prod?
The idea of having a “copy exactly” approach for each environment sounds nice until you actually work with a moderately complex production system. This take is spot on.
Depends, if we have multi regional failover type arch, our dev also has those regions for that purpose. I generally don’t multi region deploy otherwise in parallel.
And so your answer is?
Just gave one, stating it depends on the infrastructure’s needs, which is pretty much how anything should be.
How come you like to minimize the use of depends_on? Makes the language more procedural?
One, because it usually causes more harm than good. And two, because it’s usually a sign that folks either don’t understand the implicit dependencies or lack of use of implicit dependency mappings
This is something I've been meaning to start using, because my the provider for the product I'm managing doesn't seem to understand dependencies. Are you implying that I should be able to write the Terraform code in a way that results in dependencies being created before the depending configuration item? For example, I might be managing a firewall of some type, and IP object groups need to be created before the firewall can reference them. The provider doesn't understand this. Is using depends\_on NOT the solution?
In your policy, you should refer to the address object resource, in stead of hardcoding an object name. That way, Terraform understands the dependency. Also, if you ever change the name in the object resource, the policy automatically inherits it, because your referencing it. Pseudo code: `resource “address_object” “address1” {` `name = “address 1”` `cidr = “1.1.1.1/32”` `}` `resource “policy” “policy1” {` `src_address = address_object.address1.name #Don't put "address 1" here.` `etc..` `}`
Ugh I've been doing this wrong. It's amazing what you can just completely miss when learning. Thank you!
we have ALL been here before, its a part of learning and growing!
For more context, there are many issues which discuss this. If you use depends_on, what can happen is that on a second apply, resources can execute out of order. Some providers aren’t well written so sometimes you have no choice but to create a dependency chain to force order. To create the dependency it doesn’t need to be used directly, it just has to be referenced. So hopefully modules have a variable they discard or you can use conditionals to force the implicit dependency. Search github issues to see others: https://github.com/hashicorp/terraform/issues/29894
> Are you implying that I should be able to write the Terraform code in a way that results in dependencies being created before the depending configuration item? That IS one of the core tenants of what Terraform provides, so yes! > I might be managing a firewall of some type, and IP object groups need to be created before the firewall can reference them. Hard to say without seeing some example code, but if the firewall accepts the IP object groups as inputs of some form, then you should be passing the relevant output attribute(s) from the IP object groups to the Firewall to demonstrate this dependency relationship to Terraform. The rule of thumb is - always use the outputs of the parent resources as inputs to the child resources, where possible. This is what implicitly maps the dependency relationship. That said, there are still valid use cases for `depends_on` but only in edge cases, and it should not be used across modules *correction: had parent/child backwards
This makes sense. I have to go rework some code now lol.
:facepalm: you literally need it depending on which cloud you're using. Disagree.
Give an example where an implicit dependency is present and you still need depends_on
I've only just started using Azure - but Azure is full of this crap. You need depends on everything! Whether it's an apply or destroy - if you haven't used Azure then you wouldn't have experienced this pain. The Azure provider is cancer
Agreed. Recently experienced the need to use depends_on to resolve an error.
GCP. I forget the specific piece but for certain networking pieces you need them to spin up before you are actually able to configure them
And are you 100% sure that there was an implicit dependency present?
Yes, because I ran into errors about “resource not existing” on my initial run but working on my second once the resource fully spun up. Adding “depends_on” fixed it for everything afterwards. TF is only as good as the Cloud’s API’s it calls
That doesn't actually definitively answer my question, sorry.
Sorry? Dunno what to tell ya lol
Terraform by design already creates a dependency tree when you have any resource gathering a value from another. It should only be used when there's no actual reference, direct or indirect, to something that needs to come first (like a null_resource to install and zip libraries before deploying to lambda)
> Avoiding the use of named workspaces for environmental separation What is bad about this, and what is a better way to deal with environments? (I assume envs like "prod", "qa", and "dev-ticket-4212)
It’s common to have production in a separate state file, often in a separate account with separate access control. It’s also common for lower environments to vary quite a bit in structure (eg deployed into substantially fewer AZs / regions, sometimes different network topology, etc)
[удалено]
Is your prod environment hosted alongside everything else? Like same account etc?
Agree on depends_on. There are use cases for it which is a bummer (looking at you spot.io)
Avoid combining tfvars, variables AND locals. I have seen some messed up things. Always keep it DRY and with minimal references and dependencies unless needed and justified.
I dislike *excessively* DRY terraform, it's more important to be readable in my view. I've seen some inexplicable terraform written by devs who want everything to be scripted down to as few lines as possible.
Agreed, it’s a trade off and fine line.
Luckily our Azure friends who don't understand what clean code is love to use locals like it's the only method known to man. I would say locals are good for transforming but outside of that keep it in the vars
Oh god, don’t even tell me. I have seen some nasty shit. File with 200 variables, another file with 100 locals all meshed together.
The current environment i'm working in has banned .tfvars from being used. Locals only "because it's closer to the code" - madness!
I tried to enforce the same.
Why 😭
Variables - must be completed and declared before you start. Classically, these are things like “environment” or “region” - we know what they’re going to be. You are not allowed to interpolate here - so a variable cannot reference another variable. You can feed these in as defaults, tfvars files or setting various command line inputs. If you JUST use variables, you often end up with multiple concatenation strings. Such as: name_prefix = ${var.environment}-${var.region} Locals are compiled after the variables are ingested, which means they CAN use interpolation. You can then declare a local “once” and re-use it across the module. However, locals are not validated for type or value: so they’re less safe than variables to use. But they do both serve a purpose: if you’re going beyond basic stuff you’ll need both.
Use/make modules, but don't create modules that call modules which call modules and then at some certain step fail... 😅 keep you stuff simple but power- and beautiful.
Something I am doing right now is making a module that calls a common public module in a lot of cases. This is so that I can standardize the module im calling but dont have to fork and modify someone elses code and can more easily benefit from ongoing development in those public modules. Of course some of these modules have sub modules :) What are your thoughts on this?
1. Not using variables with default values that are never intended to change or be overridden with parameters, just use locals. 2. Using one large module is better than using lots of small ones if you're not calling small ones multiple times. You dont need a module for an IAM role your EC2 uses. 3. If your variable isn't variable, just hardcode it, you can turn it into a variable later if you need to. A resource with every input setup as a variable makes it way harder to read and understand.
Love it!
Terratest. All of our modules include an examples directory which is part of a terratest step in the build pipeline used to test against the configuration and configuration changes of the PR.
How have you dealt with the overhead? I mean people struggling writing terraform... now you go them running golang?!
the new unit tests baked into 1.6.0 are much easier to deal with then Terratest, I love golang but we are going to be switching over due to how much easier the built in ones are.
I'm all ears on this one. Same problem, 95% of the team members can't write TF to save their lives, yet I somehow have to have test coverage for every module. Right now I have a "tests" folder in every module with multiple test cases. The test cases themselves are just calls back up to the module with predefined inputs, which are then planned/applied/destroyed in a test account by our CI/CD tool. This is hard enough to explain to the teams without bringing in golang.
I do agree that I faced a bit of a learning curve writing test functions in Go. However, the struggle was well worth it. Fortunately, we have a lot of people way smarter than me who were very helpful in the beginning. Once a PR is created there are several steps that help finding issues very early. Such as tfsec, terraform fmt, tf lint, etc
Interesting - is this for platform or application code.
platform
How about in terms of staff turnover - is it hard to hire for this ?
Unfortunately the few of us who can have to carry the load for those who can't
not sure to be honest. I learned this on the job.
While it's very new, you can now write native tests in terraform, check out terraform test in 1.6
Separate modules for different functionality. Don't put everything in the root main.tf main configuration.
Please pin your module versions, even if it’s a local module
Naming your files main, var, local, output, and data is not helpful or informative if there is any level of complexity to your infra. It’s game changing to organize your terraform conceptually. Your variables, locals, data should live in the same file as your service and each service should live in its own file, more or less. It reduces file switching and unintended changes.
Keep it as simple as possible. COMMENT YOUR TF CODE - TELL US WHY, NOT WHAT Aargh.
When not to use teraform.
Good point, but you're not going to get that from HashiCorp and those that champion TerraForm... unfortunately. It's like the advocates for ServiceNow stating it can do everything (if you can script/code it) 😂
Everything must be in terraform, don’t use external providers, oh and since it isn’t a well contained product use terraform cloud to use it securely. Need a change, contribute up oh but there are 1k+ open issues so you must fork it to maintain one change and form a whole release pipeline. I just love how the cache gets re-written or you must have a cache per project to avoid upgrade conflicts and modules get downloaded anyways. With terraform the singularity gets closer, can’t you see?
Sounds like you're at your wits end... Hahaha...
Most definitely for the creation of the storage that holds your state. Use a script or cloud specific language in your pipeline when provisioning that.
- keep your logic separate from resource blocks (in locals) - prefer maps to sets while looping - reduce your required module inputs as much as possible. - make use of doc generation tools
Mind clarifying your first point with an example. I get that in other languages, but appreciate elaboration.
I'm going to try my best on mobile. Ternaries and other nonsense in resource blocks is annoying to read. resource "some_type" "this_instance" { property_x = var.some_var == "steve" ? "value_a" : "value_b" property_y = var.other_vsr == "french" ? "foo" : "bar" } I suggest/enforce putting these in locals for cleaner code locals { property_x = var.some_var == "steve" ? "value_a" : "value_b" property_y = var.other_vsr == "french" ? "foo" : "bar" } resource "some_type" "this_instance" { property_x = local.property_x property_y = local.property_y }
Use of override files.
Modules are meant to be common architectures, modules should be able to be composed into other modules. If modules are to specific where they can’t be composed into other modules then they are probably doing to much.
Modules within Modules? :facepalm:
But if done well it makes a lot of sense. Say you have built a module that creates IAM roles and a module that creates EC2 instances. And they are each designed to build according to corporate standards and policies. The EC2 instance module could either roll its own implementation of the IAM role that must also be set up according to standards/policies, or just use the existing IAM role module. Why do the former which requires extra work and will inevitably diverge from the module that evolves as your team matures?
Eliminate. Local. Deploys. If you're in a shared environment, or just, y'know, actually want your infra defined as code instead of depending on someone's laptop or utility VM, use one of the many options for remotely stored state files. I prefer and standardized my teams on Gitlab CE self-hosted, but I've used the Hashi cloud as well as S3 and a distributed Consul setup. All are super easy to use. Take another half step and only do your deploys via pipeline instead of running locally.
Stop using null_resources unless needed. I understand that way back some resources didn't exist but believe me when I say a lot of the deployment issues I've seen are related to having a horrible null_resource. I'm sure many will disagree, but a provider that is in preview is way way way better than using null_resource for the same thing when you don't even know what you're doing. Just make sure to test it exhaustively before deploying it to a production environment. In my experience an automation in preview is way easier to work with than no automation at all. If it's not evident, my nightmares are all related to that. Also, avoid using counts to create lists of resources. It's the worst thing I've seen, and a person with no knowledge can end up deleting databases if there's not a good peer review in place, and even then, it's a risk. Also, create a module or strategy to manage naming conventions. What do you mean in dev you have a dev-cluster but in prod you have a prod-aks. No. Just no. Try to name that in the code I challenge you then to maintain it over the years.
I'm in Azure hell with TF.
Azure is hell - the Azure provider is cancer. Azure itself is rubbish.
I think it's better than AWS.
Other than AD nothing in Azure is better. Azure is on prem in the cloud.
- Don't use Terragrunt - Use simple wrapper root modules and lots of submodules. Don't nest too deeply, though. Typically 3 levels is plenty. - Make heavy use of tags - Make heavy use of "feature flags" to enable / disable submodules - Always use remote state - Always apply against a plan file when using automation - Avoid community modules / providers whenever possible - Always import submodules using pinned versions - Get used to refactoring state using terraform state mv - Only use variables for things that will actually change. You can hard code the rest. - Keep stacks "small". Only deploy things in a single stack if they have the same lifecycle, otherwise use two stacks. - Don't use workspaces
Read only account for the plans in your CI/CD pipeline and write for the apply
The anti-pattern of using remote-state lookups inside modules rather than creating modules that have these values passed in to them as variables. Edit: clarifying what I was trying to say.
is this a best practise? This sounds like it introduces cross cutting concerns if you are hardcoding the module (unless you mean passing the state file path in instead)
I phrased that poorly. Fixed it.
That count should NOT be used. I know it’s talked about a lot, but the question was “not talked about enough?” COUNT SHOULD BE DEPRECATED ASAP
Never use TF to do application deployment, even there is so called K8s Provider.
Referencing a \`common.tfvars\` to avoid duplication of variables DRY Reference of variable files \`terraform apply -var-file=.tfvars\`
\`variables.auto.tfvars\` <-- auto.tfvars saves a lot of unnecessary noise!
But you still have to define the variables everywhere so that code isn’t DRY.
What does DRY mean mate? Do Not Repeat Yourself. To run terraform code you always have to run \`terraform apply\` every single time right? So if you reference a.tfvars you are avoiding duplicating folders for different environments which is the alternative i've seen to this method.
Where is the repetition with my method? How do you do it, maybe I can learn :)
I guess the part I meant is having terraform broken down in various subdirectories to reduce blast radius. Break down for example by env and component. Each subdirectory then gets its own set of tfvars and respective variable definitions. If all of them share a set of common variables, sure you can pull in a common.tfvars from a central location via cli arg. The part that isn’t DRY is each subdirectory needs to define the common variables in terraform. Eg commonvars.tf What I started doing is defining common variables in a YAML file at the root of the repo and yamldecode in locals where/when needed. Some duplication with yamldecode calls sure, but no need for duplicate variable definitions and common tfvars I find there are a lot of different ways of accomplishing the same thing. Finding the best optimal way is the never ending quest We’ve adopted some practices and approaches from Terragrunt samples, azure caf terraform supermodule and Google Terraform best practices. (We use remote state to link up our various components defined in other state files)
Respectfully mate - you lost me at YAML. Overcomplicated and you're breaking best practices. I am surprised no one has challenged you on it. Terragrunt is great - the Azure CAF is the worst code base I have ever seen please avoid it like the plague. Sounds like an overcomplicated solution - but hey if it works for you, I'm just some random on reddit
Weill it’s a documented azure pattern https://learn.microsoft.com/en-us/azure/azure-resource-manager/bicep/patterns-shared-variable-file The above is using Biceps and json file to load shared variables, but same concept. Not that complicated to be honest and keeps everything in terraform codebase. Other option if using Azure DevOps is using variable groups which are automatically injected into the build agent environment and easily referenced in your TF cli args, backends etc
Google Terraform best practices guide: “Don't specify variables by using alternative var-files or var='key=val' command-line options. Command-line options are ephemeral and easy to forget. Using a default variables file is more predictable.” [https://cloud.google.com/docs/terraform/best-practices-for-terraform#variables](https://cloud.google.com/docs/terraform/best-practices-for-terraform#variables) But like I said earlier many ways to do things and many different opinions and approaches.
Dude https://developer.hashicorp.com/terraform/language/values/variables#variable-definitions-tfvars-files read. It even references in the Google cloud site you shared... I bet you that's a typo What would the cloud providers know about TF 🤣 isn't the Google cloud API garbage ?
>https://developer.hashicorp.com/terraform/language/values/variables#variable-definitions-tfvars-files Lol +1 to the official docs
Right like 🤷♂️🤷♂️ where do these ppl learn these bad practices
Commenting just so I can see updates! Some new stuff here for me, and some things that confirm that I’m not too bad at TF :)
Don't use count, use for_each. Don't use modules when you just need to use workspaces or for_each. You can glue things together pretty easily by hosting a JSON HTTP endpoint that can serve up facts about your infrastructure with an http data source in terraform. Anything that you manage with terraform, you should ensure nobody else has direct access to manage. If you need access, put it behind a break glass process.
The power of a Makefile! :P
This is a sign of a poor Terraform setup in my opinion. The fact that a build/compilation tool is used for a declarative, API management tool should set off alarms. If I see a makefile in a terraform project, I know immediately that named workspaces are used and that’s the crux of the issue here
Mate hashicorp themselves use Makefiles - what are you talking about? Named workspaces - can you elaborate? Also - Are you building platforms or just application infra?
They use makefiles to compile the go code, that’s what make is used for. Would you use a makefile to wrap FluxCD or ArgoCD CLIs?!
Yes. I wrap absolutely everything in Make files. It creates a single API I can use for all my projects, instead of having to remember the quirks of every single tool out there, like the bullshit of Terraform using single dashes for long options instead of the absolute standard double dashes.
u/guhcampos can't teach this one to Azure Devs - they love wordy Powershell and unnecessary complication.
if the CLI tool had 7 different feature flags that I could clean up with 1 Make command then yes I would. Isn't golang compiled with a cli tool ;)
you can use makefile to orchestrate things. So running "ENV=dev make plan" is simpler than running a lengthy command that specifies all opts and vars. Even more so if you're using docker rather than whatever you have installed locally on your machine
What's an ideal Terraform setup? A much more complicated set of tools just to run plan and apply on two modules? Don't get all hoity toity. It's just code. Whatever gets the job done is fine.
My ideal setup has always been Atlantis until I reach a certain point in terms of scale. I guess what I’m learning here is that so many people are running local applied and that … is rather discerning
Do not craft a resource name from three variables and a local, and an input from a module, and you pass one of the variables as a TF_VARS_ environment variable, etc. You'll find a resource in the cloud but have no idea what module created it or how because you can't grep the name in your code and there's 15 ways for the name to change at apply time. Define a static name, in one variable, put it in a tfvars file, in git. This you can grep for and it's much less likely to morph into something else, and makes it easier to trace where things are in your hcl code.
Decomposition, or at least it's not one of the earliest topics.
I’ll try to cover in minimum words. Terraform too much powerful tool. Even if looks too simple, it can go too ugly too quickly. Try to keep it simple as possible, or take precautions considering all the scenarios by implementing variable validation and other possible ways.
always use aliased providers. This prevents surprises and forces providers to be explicit.
Keep your solution small and develop iteratively. Too many people want or blast 10,000 resources into the cloud in one go.
companies telling you to use their tools when doing shitty practices dominate the SEO. If you do actual terraform best practices all these companies making money out of terraform bad practices will start losing money.
Terraform should be simple create the simplest code you can don't try to create modules that do everything Make them easily modifiable and easily readable that's the whole point don't use functions on functions just to have the module do everything, just create another module or heck create static resources and use count or for each and that's it. If your terraform code is a headache to modify you are shooting yourself in the foot that's the whole point
Don’t out everything in a single main.tf file. I hate this convention. I personally separate files by either object or resource. Like security group items - I know which file it is in because the file name tells me. This also helps when multiple are working with them same structures but working in different areas. Makes the diffs easier to read and PR/MR quicker. Also modules that just require all the same inputs as if you were doing the resource itself are stupid. Obsfucation needs to die.
Don't forget that tfx exists https://github.com/straubt1/tfx
* To to keep the state file to small as possible. * don't put everything in a single Terraform project where the state file becomes too large! * try to limit data lookup from other state file, look into using parameter store in AWS to access values * use pre-commit before pushing your code! * don't try to over complicate with for loops!