Edit Terraform configuration files programmatically with Python
Question:
I am trying to edit Terraform configuration files with Python. I am parsing Terraform files (.tf) using python hcl2 library which returns a python dictionary. I want to add new key/value pairs or change some values in the dictionary. Directly writing to the file is not a good practice since the returned python dictionary is not in Hashicorp Configuration Language format. Also there can be multiple configuration files like variables.tf etc. which are linked together. Should I implement my own serializer which converts python dictionary to terraform configuration file or is there an easier way to do it?
Answers:
The python-hcl2 library implements a parser for the HCL syntax, but it doesn’t have a serializer, and its API is designed to drop all of the HCL specifics and retain only a basic Python data structure, so it doesn’t seem to retain enough information to surgically modify the input without losing details such as comments, ordering of attributes, etc.
At the time I’m writing this, the only HCL implementation that explicitly supports updating existing configuration files in-place is the Go package hclwrite. It allows callers to load in arbitrary HCL source, surgically modify parts of it, and then re-serialize that updated version with only minor whitespace normalization to the unchanged parts of the input.
In principle it would be possible to port hclwrite to Python, or to implement a serializer from a dictionary like python-hcl2 returns if you are not concerned with preserving unchanged input, but both of these seem like a significant project.
If you do decide to do it, one part that warrants careful attention is serialization of strings into HCL syntax, because the required escaping isn’t exactly the same as any other language. You might wish to refer to the escapeQuotedStringLit function from hclwrite to see all of the cases to handle, so you can potentially implement compatible logic in Python.
As @mark-b mentioned, Terraform supports json. So once you have imported your hcl2 file with the python-hcl2 library, you can modify the data structure internally and then dump it with json.dump() to a file with .tf.json extension.
I just ran this script to have my main.tf change from a GCS backend to a local one.
import hcl2
import json
with open('main.tf', 'r') as file:
dict = hcl2.load(file)
dict['terraform'][0]['backend'][0]= {
'local' : { 'path' : 'default.tfstate' }
}
with open('main.tf.json', 'w') as file:
json.dump(dict, file, indent=4)
After running it, delete the main.tf file.
I am trying to edit Terraform configuration files with Python. I am parsing Terraform files (.tf) using python hcl2 library which returns a python dictionary. I want to add new key/value pairs or change some values in the dictionary. Directly writing to the file is not a good practice since the returned python dictionary is not in Hashicorp Configuration Language format. Also there can be multiple configuration files like variables.tf etc. which are linked together. Should I implement my own serializer which converts python dictionary to terraform configuration file or is there an easier way to do it?
The python-hcl2 library implements a parser for the HCL syntax, but it doesn’t have a serializer, and its API is designed to drop all of the HCL specifics and retain only a basic Python data structure, so it doesn’t seem to retain enough information to surgically modify the input without losing details such as comments, ordering of attributes, etc.
At the time I’m writing this, the only HCL implementation that explicitly supports updating existing configuration files in-place is the Go package hclwrite. It allows callers to load in arbitrary HCL source, surgically modify parts of it, and then re-serialize that updated version with only minor whitespace normalization to the unchanged parts of the input.
In principle it would be possible to port hclwrite to Python, or to implement a serializer from a dictionary like python-hcl2 returns if you are not concerned with preserving unchanged input, but both of these seem like a significant project.
If you do decide to do it, one part that warrants careful attention is serialization of strings into HCL syntax, because the required escaping isn’t exactly the same as any other language. You might wish to refer to the escapeQuotedStringLit function from hclwrite to see all of the cases to handle, so you can potentially implement compatible logic in Python.
As @mark-b mentioned, Terraform supports json. So once you have imported your hcl2 file with the python-hcl2 library, you can modify the data structure internally and then dump it with json.dump() to a file with .tf.json extension.
I just ran this script to have my main.tf change from a GCS backend to a local one.
import hcl2
import json
with open('main.tf', 'r') as file:
dict = hcl2.load(file)
dict['terraform'][0]['backend'][0]= {
'local' : { 'path' : 'default.tfstate' }
}
with open('main.tf.json', 'w') as file:
json.dump(dict, file, indent=4)
After running it, delete the main.tf file.