Set up a Terraform S3 backend with this video tutorial
State is one of the most important concepts in Terraform. Storing state in a remote backend is a critical step in making your infrastructure more robust and fault-tolerant.
In this tutorial, learn the benefits of using Terraform remote state compared to local state, how it can make infrastructure configuration more stable and how to configure it in an AWS S3 backend.
The importance of maintaining your state files in a backend
State makes Terraform configuration idempotent -- meaning that if you reapply the same Terraform configuration over and over, only the changes in the configuration will be applied.
This makes Terraform more suited for large infrastructure deployments than other tools like AWS Command Line Interface. With tools like AWS CLI, creating and modifying resources requires separate commands, each specially crafted to make a specific change.
Instead, Terraform allows you to store each created resource in a state file. That state file is then a requirement for each time someone needs to change the infrastructure, either one developer or a team of engineers. If the state file is damaged, incomplete or unavailable, Terraform won't recognize any of the resources in the account. This renders them unmanaged or orphaned resources.
If your state file is on a local machine, like a developer's laptop, only that developer can manage infrastructure, and only as long as the file is preserved. Instead, using a remote state file allows multiple developers to share it, store it in a fault-tolerant way, and even version and revert it if something catastrophic happens to the contents of the state file.
Having unmanaged resources in an account can also be a huge problem for compliance and introduce vulnerabilities into your infrastructure. For instance, if you patch a vulnerability in your infrastructure and apply it using Terraform, and there are some unmanaged resources in your account that should have received that patch but didn't, this leaves you vulnerable to an exploit that you thought you had fixed.
AWS S3 isn't the only kind of backend that Terraform remote state can use. There are different ways to configure remote state using any cloud provider, including Azure Blob Storage and Google Cloud Storage. There are also several vendor-neutral backends available, such as a PostgreSQL database, Kubernetes Secrets and REST API.
00:03
Hello, and welcome to this tutorial on how to use Terraform remote state file. So let's start out here, I'm going to demonstrate this in AWS, but there are several other different Terraform state or Terraform backends that you can use. What I've got here is an AWS EC2 instance. This is going to create a T3 micro with the tag 'hello world' as the name. So let's go ahead and create this. I'll run Terraform init first. And then I'll go ahead and run Terraform apply right afterwards.
00:38
So before I actually hit yes, and apply this, I want to go ahead and point out that a new file has been created in this directory, and that is the terraform.tfstate file. And that's important because if you're working on this as a developer on a team, or just on your own project, this file is the only copy of the state if you're not storing it in some kind of backend. So you can't share it between your teammates. And if anything happens to it, it can impact your resources. So let's go ahead and take a look at that; I'll go ahead and apply this instance.
01:19
And now that's populated my terraform.tfstate file here. So if I rerun the Terraform apply command, Terraform sees that my infrastructure already matches the configuration that I want. So I don't need to make any changes, or Terraform thinks it doesn't need to make any changes rather. And I can confirm that there's only one instance running right now with the AWS EC2 command line.
01:49
As I mentioned, if something happens to this tf.state file, it can compromise the integrity of your infrastructure. So let's go ahead and delete this. And now I'm going to rerun the Terraform apply command again.
02:04
So Terraform doesn't know that there's an instance there already, because it thinks that there's a blank Terraform state, because we deleted the previous one. So let's go ahead and apply this one more time. And it's added a new instance here. But if I use the AWS CLI to list those instances again, you see that there's two instances now -- the one we created before our state file got damaged or lost or destroyed, and the one we just created.
02:33
So if I run a Terraform destroy now, I still have unmanaged resources in AWS, because it only destroyed the one that was in the state file. I've still got one running EC2 instance left, which I don't want. I'm going to pause the video here, clean up my account, and then we'll talk a little bit about how we can prevent this with remote backends.
03:00
All right, I'm back here in my workspace, I've cleaned up everything in the account. So here's my describe instances command -- we can see I don't have any running instances in this account. Right now, I've left main.tf alone, but I've added a new file here called backend.tf. Backend.tf here has a couple of things. It has a back-end block labeled S3, and it specifies a bucket, a key and a region. So let me actually go into my AWS account. I'm going to check the buckets I have with the AWS CLI. And you see I've got a few others here for other projects I've worked on. But I've also got that TechTarget state bucket that is specified there. That'll become important in just a minute, we'll come back to that.
03:46
So next, I'm going to go ahead and run Terraform init and Terraform apply again. I've created one EC2 instance, same as last time, which I can verify with the EC2 CLI command. But this time, we didn't get a terraform.tfstate file that was created on our local machine. And that's because instead, that file got created in the state bucket. So let me go ahead and verify that with AWS S3 ls. And I'm going to list the bucket name that I specified as well as the key to that state file. So that's the state file there; let's actually go ahead and read it as well so we can just make sure it looks like we'd expect a state file to look.
04:31
Alright, so this is what I'd expect my state file to look like, just making sure that everything is the way it's supposed to be. Let's go ahead and clear the screen. And I'll delete my .Terraform files here and rerun Terraform init and Terraform apply just to see if I can recreate the same behavior we had before we moved the state file to the remote backend.
04:57
Alright, this time everything ran from scratch. We ran init, we ran apply. But instead of recreating the instance, like it did last time, it's gone and detected that we already have this instance in our Terraform remote state, and therefore no infrastructure changes were necessary. And we can go ahead and confirm that there's still only one instance running with the EC2 describe-instances command.
05:24
So in this video, you learned how to set up a Terraform remote state file, what the benefits are of using remote state versus local state, and why that's important when you're developing infrastructure in a team. Thank you all for watching. Take care.