Terraform is a very powerful tool used in DevOps practices to maintain Infrastructure as Code and it uses a state file. But what does it mean?
Imagine that you are an interior designer and you have an apartment to decorate. You have an assistant working for you to buy furniture and place it in the apartment. He has an inventory that he’ll keep updated every time he adds, moves or removes something from the apartment, while you design your project in a digital 3D model of the apartment.
In this case the apartment is your platform, the assistant is Terraform, your 3D model of the apartment is your Terraform script and the assistant’s inventory is the Terraform state file.
You first receive the apartment completely empty so you add a bunch of furniture to your 3D model and send the assistant to buy it all.
terraform init (= “hey assistant come work for me”)
terraform plan (= “look at my 3D model and compare it with your inventory”)
terraform apply (= ”yes do all that”)
He looks at the inventory and sees it empty: the apartment still has nothing in it. He delivers all what requested and updates the inventory list.
Plan: 30 to add, 0 to change, 0 to destroy.
Now you need to add one more lamp. All you do is go to your 3D model and add the lamp. You send your project again to the assistant asking him to make sure the apartment looks exactly like that. You’re very busy and have no time to tell him what’s changed.
He looks at your project and compares it to the inventory. He notices that you added a lamp and he’ll deliver it.
Plan: 1 to add, 0 to change, 0 to destroy.
You change your mind about the location of the lamp. You first placed it in the living room but now you want it in the bedroom. You move it in your 3D model and ‘apply it’ to make the assistant aware that you want him to compare it to his inventory. He notices the difference between the project and the inventory and changes the lamp’s location.
Plan: 0 to add, 1 to change, 0 to destroy.
You need to change the colour of the cushion. You change it in the 3D model and run ‘apply’. The assistant compares the project to the inventory and notices that the colour of the cushion has changed. He goes and replaces the cushion. In this case he’ll have to ‘destroy’ the previous cushion and ‘add’ a new one as the colour is not something he can change on the existing object.
Plan: 1 to add, 0 to change, 1 to destroy.
You want to remove a chair? Remove it from the 3D model. Same story as above, the assistant will notice the difference and will go ‘destroy’ one chair.
Plan: 0 to add, 0 to change, 1 to destroy.
The apartment is broken into!
The burglars took the TV. You get to know about it and you send the unchanged 3D model to the assistant because the TV is on it and you want him to add one. However, he doesn’t know about the burglary. As the inventory still has the TV listed on it, he won’t do anything because the project matches the inventory.
No changes. Infrastructure is up-to-date.
You realise your mistake so you tell him that the TV is no longer there and he needs to remove it from the inventory.
terraform state rm apartment.living_room.TV
Then you send the project again and this time he knows that the TV is missing so he’ll deliver one again.
Plan: 1 to add, 0 to change, 0 to destroy.
The owners of the apartment show up and place a statue in the apartment.
The statue is not in your 3D model, so if you send it to the assistant, he’ll compare it to his inventory. The statue is not in the 3D model nor the inventory, so he won’t do anything.
You add the statue in your 3D model because the owners tell you about it. You send the project to your assistant and he thinks he has to add the statue now, because it’s not in his inventory. He goes and buys a statue but when he shows up at the apartment the statue is already there.
Error 409: statue already exists
You’re not sure what happened, so you ask him to see his inventory in order to help him out.
terraform state list
or
terraform show
Our guy is not really smart so unless you tell him to add the statue to his inventory, he’ll keep trying to deliver a statue and fail. Let’s tell him to add it to his inventory because it’s already there.
terraform import apartment.hallway.statue
Now he knows the statue is there and he’ll add it to his inventory. Next time he won’t try and deliver the statue even if he didn’t personally add it in the first place.
The above are examples of manual changes made in the platform without Terraform knowing about it. The Terraform state file will get out of sync and will need a fix.
A colleague of ours jumps in and updates the 3D model but has a different assistant who doesn’t have the latest inventory file
If you change location of the state file, misplace it, lose it or have it local, like our assistant kept it for himself, you’ll run into problems when working on the same project with others. New state files get created every time the command ‘terraform init’ doesn’t find one and they’ll end up being empty and out of sync with the current state of the platform.
In this case the new assistant will think that this is a completely new project. He’ll create an empty inventory and go to the apartment with a copy of all the furniture already existing and fail with a bunch of 409 errors, but he’ll deliver that one thing, let’s say a plant, that our colleague added, because that’s the only thing that he could do.
Now when my assistant comes back to work, he’ll notice the change on the 3D model but he doesn’t know that the plant was already delivered by our colleague’s assistant, because they’re not sharing the inventory file. His inventory doesn’t have it, so he’ll go and try to add it. Now he’s the one who will fail.
A common mistake at this point, especially in a test scenario, is to try and ‘terraform destroy’ the plant with the original assistant, or even worse ‘terraform destroy’ everything, thinking the plant will go too and then we recreate everything with the updated 3D model. Let me tell you what would happen.
terraform destroy
Our original assistant checks his inventory and goes to remove everything listed on it. He doesn’t know if the owners moved in, in the meantime, and have added something to the apartment which shouldn’t be taken away by him. He doesn’t remember what was there before he started working on it. We don’t want him to remove the kitchen and the bathroom which were already in the empty apartment. He won’t take risks. He’ll take only and exactly what he added.
Do you want to guess? The plant is still there.
Now his inventory is empty. Let’s tell him to re-deliver everything shown on our 3D model which now includes the plant, hoping for his inventory to get synced.
I’ll give you a second to guess it:
Error 409: plant already exists
How do we fix it? Well, let’s make our assistants talk and merge their inventories, shall we?
cd assistant1
terraform state mv -state-out=../assistant2.inventory apartment.kitchen.plant apartment.kitchen.plant
All the commands in this blog have clearly been modified to apply to the real life scenario example. For exact syntax of terraform commands I suggest to check the documentation on the terraform website.
Following the above sync of the state files, now our original assistant will know about the plant that our colleague added. In this case assistant 2 will have an empty inventory again, but why don’t we prevent this above situation from happening again?
If the inventory was in a central location, the new assistant would have been able to compare the 3D model to the inventory and pick up the work from his colleague.
The best practice is to store the terraform state file in a remote bucket on the cloud platform and make sure your code points out to it. We’ll add a note in our 3D model to tell the assistants that the inventory is at the office, in a drawer. Now all our colleagues can pick up the 3D model and make changes. As long as they have the latest copy of it, it will tell whoever the assistant is, where to find the inventory and pick up the work from.
There can be many other situations you might find yourself in with Terraform and there are many more commands I did not include in this blog, but I hope my real life scenario example can shed some light on the state file functionality and how to understand when and why it gets messy and out of sync.
In my first role as a DevOps Engineer I was known as the one that destroys everything. I would get into messy situations with the state file and just run ‘terraform destroy’ hoping it would fix it all. Turn it off and on again, right?
Not really… but making mistakes is the best way to learn! I hope I could help preventing you getting into a messy situation with Terraform state files, but if you do, trust that you’ll get out.
Just read the docs and don’t destroy everything, ok?
Don’t hesitate to get in touch with me via LinkedIn at /debora-piu.
