This is the 7th part in the series of blog posts on managing the Azure DevOps using Terraform. You can find the series index here. Although this is the one of the part in the series, this can also be a completely independent post in itself. In this post, we’ll be using a Git repository to store the Terraform code files and discuss the best practices around it. This aligns with one of the practices in the Infrastructure as Code (IaC) framework and somewhat aligns with what is now a days known as GitOps framework. We’ll not go into details of if GitOps is suitable for Terraform or not into this post.

Why to use a separate Repository

This question asks whether we should be hosting a separate source code repository to host Infrastructure as Code (for Chef, Terraform, Ansible etc.) files. Some people go as far as asking if Terraform files are to be considered as code files or is to be considered as a config files. This is because, all what Terraform code, is storing configuration about your Infrastructure. And real code, in their minds, is written using languages such as Java, Python, Golang etc. Although what is considered as programming is debatable, let’s just stick to the original premise of asking if we should be storing Terraform files in separate repository or should store them alongside the code written for business functionality.

Although, it seems like more work and more management to maintain separate repositories for the Terraform files and for the business functionality code, it comes with lot of benefits. For example, over the time, you would find that number of code changes required to maintain or update infrastructure is very less as number of code changes required to maintain or update business functionality. Storing them alongside, also makes it more prone to any unwanted or unintended changes. You would not want to deploy IaC files again and again, every time business functionality code changes, and be inadvertently deploy any unwanted or unintended changes. Also, you would not want to run around developers, Dev and QA leads, to try to judge the code changes required for the Infrastructure and approve/review changes. More appropriately, this code needs to be checked by fellow members of the operations team or ones having good knowledge on the Infrastructure side.

So it is always considered a best practice to store IaC files (whether Terraform, AzureARM, Cloudformation, Ansible etc) in their own repository. However this is only a guideline and not a mandate. Feel free to follow what makes best sense to meet your requirements.

Create a hosted Git Repository

Even if you consider yourselves as a lone wolf, its still advisable to use a Git repository due to numerous benefits offered by it. However most of the developers or operations folks work for some organization, or in a team and as such they need to share code between them from time to time. So they need a common place where they can host their git repository. This is place where hosted git services (both on-premise or cloud versions wherever applicable) like BitBucket, GitLab, Azure Repos, etc play a critical role. For the sake of this post, we’ll be using Azure Repos.

Head over to Azure DevOps organization, open a project, navigate to Repos section and create new repository by giving it a name:

Initialize local Git Repository

For this, navigate to the directory where we have stored Terraform files and run git init command. This will tell git to initialize the directory as a git repository in your local machine. Optionally, run a git status command to verify the status:

Add Azure Repo as remote repository

We need to let the git repository know, how to find the remote azure repository we created. For this we can use command git remote add origin followed by the url of the remote repository. After this, running git fetch --all will query all information from remote repository and pull down the remote state:

We can also use command git pull origin master to fetch and merge all commits in the remote master branch to the local repository.

Create a .gitignore file to avoid checking sensitive files

Beside the code written by us, if you are following along from previous posts, you may notice that our Terraform directory consists of several other files, such as below:

• .terraform/ – this directory contains the providers pulled down to local. It will be re-created when terraform init is run in a new environment.
• terraform.tfstate and terraform.tfstate.backup – these files contain the terraform state specific of a specific environment and do not need to be preserved in a repository. These files may contain sensitive information stored in plain text from the previous deployment.
• terraform.tfvars – may contain secrets (usernames, password, ip addresses, etc) about a specific environment

We should take steps to avoid check-in of such files. For this, we can use a .gitignore file and mention the extension of such files:

	# local .terraform directories
	**/.terraform/*
	# .tfstate files
	*.tfstate
	*.tfstate.*
	# .tfvars files
	*.tfvars

view raw terraform-general.gitignore hosted with ❤ by GitHub

In above, * represents a wildcard character and would match any valid sequence of characters and ** represents any matching directory path. You can use above pattern to mention any other such files, which you need to ignore, depending upon your requirements such as .bat, etc. Here is a good collection of general purpose .gitignore files for the common languages.

Commit and Sync the changes

To view what files we are going to check-in, we can use git status command. If we are satisfied, run the command git add . to add all files to git staging area. Using git commit -m "commit message" will commit our changes in the local copy of the git repository.

To push the commit to the remote repository, we can use the git push origin <remote-branch> command:

Note that even though we have pushed our changes to master branch directly, you would not do that when working in a team. You would be pushing your code to branch created specifically for your changes and then raise a pull request with the release branch (which is generally master branch). After this, you need to have your code reviewed and approved by your fellow team members and leads and merge it to the release branch.

Attach terraform plan output to pull requests

One of the best practices is to attach the output of terraform plan to your pull request as well. It creates an opportunity for team members to review each other’s work. This allows the team to ask questions, evaluate risks, and catch mistakes before any potentially harmful changes are made. You can save the output of the plan by using -out parameter name followed by the file to store the output. Some of the CI service providers like Atlantis, Terraform Cloud, etc. would automatically run this command and attach the output to the pull requests, saving you some time.

Use one and only one release branch

I cannot express in enough words, how important it is to use only one release branch to deploy to the target environment. Since we are not checking in terraform state files, when someone is running the deployment, there is no way for terraform to know the ideal current state of the environment. In particular, if two team members are deploying the same code to the same environment, but from different branches, you’ll run into conflicts that locking can’t prevent.

To understand this, let’s say developers A and B created a branch from master branch. Let’s A modified the code and increases the number of EC2 instances to 2 because he’s observing high traffic and deployed it. Also checked-in his changes. Meanwhile, B only added a tag to the EC2 instance but since he downloaded his code before A did, his code still have 1 instance, instead of 2. So if he runs terraform apply, he is about to undo A’s change made to the environment!!

However, if the deployments are made to the common release branch say master, he’ll get a conflict before he can check-in his changes for deployment. There he’ll get the opportunity to review and make appropriate modifications to his code.