A tool to bring existing Azure resources under Terraform's management

Azure Terrafy

A tool to bring your existing Azure resources under the management of Terraform.

Goal

Azure Terrafy imports the resources inside a resource group, which are supported by the Terraform provider, into the Terraform state, and generates the valid Terraform configuration. Both the Terraform state and configuration should be consistent with the resources' remote state, i.e., terraform plan shows no diff. The user then is able to use Terraform to manage these resources.

Install

From Release

Precompiled binaries for Windows, OS X, Linux are available at Releases.

Note: The release is in the format of .tar.gz, Windows users might want to have 7zip installed to extract the files.

From Go toolchain

go install github.com/Azure/aztfy@latest

Usage

Follow the authentication guide from the Terraform AzureRM provider to authenticate to Azure. The simplist way is to install and login via the Azure CLI.

Then you can go ahead and run aztfy:

aztfy [option] <resource group name>

  -o string
        Specify output dir. Default is a dir under the user cache dir, which is named after the resource group name
  -v    Print version

The tool will then list all the resources resides in the specified resource group.

For each resource, aztfy will ask the user to input the Terraform resource type and name for each Azure resource in the form of <resource type>.<resource name> (e.g. azurerm_linux_virtual_machine.example). Users can press r to see the possible resource type for the selected import item, though this is not guaranteed to be 100% accurate.

In some cases, there are Azure resources that have no corresponding Terraform resource (e.g. due to lacks of Terraform support), or some resource might be created as a side effect of provisioning another resource (e.g. the Disk resource is created automatically when provisioning a VM). In these cases, you can skip these resources without typing anything.

After getting the input from user, aztfy will run terraform import under the hood to import each resource. Then it will run terraform add -from-state to generate the Terraform template for each imported resource. Whereas there are kinds of limitations causing the output of terraform add to be an invalid Terraform template in most cases. aztfy will leverage extra knowledge from the provider (which is generated from the provider codebase) to further manipulate the template, to make it pass the Terraform validations against the provider.

As the last step, aztfy will leverage the ARM template to inject dependencies between each resource. This makes the generated Terraform template to be useful.

Demo

asciicast

Limitation

Some Azure resources are modeled differently in AzureRM provider, which means there might be N:M mapping between the Azure resources and the Terraform resources.

For example, the azurerm_lb_backend_address_pool_address is actually a property of azurerm_lb_backend_address_pool, whilst in the AzureRM provider, it has its own resource and a synthetic resource ID as /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group1/providers/Microsoft.Network/loadBalancers/loadBalancer1/backendAddressPools/backendAddressPool1/addresses/address1.

Another popular case is that in the AzureRM provider, there are a bunch of "association" resources, e.g. the azurerm_network_interface_security_group_association. These "association" resources represent the association relationship between two Terraform resources (in this case they are azurerm_network_interface and azurerm_network_security_group). They also have some synthetic resource ID, e.g. /subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/mygroup1/providers/microsoft.network/networkInterfaces/example|/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/group1/providers/Microsoft.Network/networkSecurityGroups/group1.

Currently, this tool only works on the assumption that there is 1:1 mapping between Azure resources and the Terraform resources.

How to develop with vscode

vs pre requiries

  1. Install Go extension from "Go Team at Google"

  2. Install dependencies when ask in the editor.

  3. Build without optimision
    go build -gcflags=all="-N -l" (To run, in context of the folder)

  4. Add some code in the main.go to stop the init.

var reader = bufio.NewReader(os.Stdin)
os.Setenv("AZTFY_DEBUG", "true")
os.Setenv("AZTFY_MOCK_CLIENT", "false")
os.Setenv("AZTFY_LOGFILE", "aztfylogs.log")

log.Println("Main hitted")

input, _ := reader.ReadString('\n')
value := strings.TrimSpace(input)
fmt.Printf("input: %v", value)
  1. Run app
./aztfy rg-my-demo
  1. Get pid of the app

    • Linux : pgrep aztfy
    • Windows : Task manager / tab detail
  2. Update launch setting processId with pid (Sample in folder .vscode\launch.json)

  3. launch debug session

  4. Press enter

Owner
Microsoft Azure
APIs, SDKs and open source projects from Microsoft Azure
Microsoft Azure
Comments
  • Issue with importing blobs / containers

    Issue with importing blobs / containers

    I get this following error when I try to provide the https:// url for the container

    Error: invalid resource id: id should start with "/"

    When I try to import as part of resource group import, I get this error:

    Failed to import https://xxxxyyyyyyy.blob.core.windows.net/%5Cazure-webjobs-secrets as azurerm_storage_container.res-35: exit status 1

    Error: retrieving Container "\azure-webjobs-secrets" (Account "xxxxyyyyyyy" / Resource Group "rg-xxxxxxx"): containers.Client#GetProperties: Failure responding to request: StatusCode=400 -- Original Error: autorest/azure: Service returned an error. Status=400 Code="InvalidUri" Message="The requested URI does not represent any resource on the server.\nRequestId:2993a3c4-101e-00a8-3d2f-cd69dd000000\nTime:2022-09-20T20:31:14.7346835Z"

  • Exit Code 1

    Exit Code 1

    Unable to copy exact text, but as executing against a RG, getting exit code 1 cannot connect to terraform registry; afterwards when trying to aztfy specific RG, starts to list resources and skips to command line on both interactive and --non. all other RGs in subscription work, only 2 (out of 8) specifically this occurs with. Nothing I can notice in common or out of place...

  • Aztfy Passes All Performance Tests

    Aztfy Passes All Performance Tests

    Let's define a set of performance tests to get understanding of general benchmarks and time it would take a user to import various sets of resource groups. We want to make sure that nothing is exceedingly slow while importing infrastructure for the user.

  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    Hello Team,

    I'm getting below error when I tried to run aztfy command to import all resources inside a resource group. this resource group has around 500 resources inside it.

    Azure Terrafy

    panic: runtime error: invalid memory address or nil pointer dereference [signal 0xc0000005 code=0x0 addr=0x0 pc=0x1660809]

                                                                                                                         goroutine 176 [running]:
                                                                                                                                                 github.com/magodo/azlist/azlist.listDirectChildResource({0x2c4b110, 0xc000038100}, 0xc003199780, 0x10000c0032c7290?, {{0x2c4f818?, 0xc003458cd0?}, 0xc00273bce0?})
                                                                        github.com/magodo/[email protected]/azlist/azlist.go:347 +0x129
                                                                                                                                                               github.com/magodo/azlist/azlist.ListChildResource.func2()
                                                                                                                                                                                                                                github.com/magodo/[email protected]/azlist/azlist.go:285 +0x47
                                                                         github.com/magodo/workerpool.(*workPool).Run.func1(0x0?)
                                                                                                                                        github.com/magodo/[email protected]/workerpool.go:75 +0x99
                                                                                                                                                                                                                              created by github.com/magodo/workerpool.(*workPool).Run
                                                github.com/magodo/[email protected]/workerpool.go:66 +0x5d
    
  • Azure Government Use

    Azure Government Use

    Hello - I love this tool but if I try to run it Azure Gov it errors out because https://management.azure.com needs to be https://management.usgovcloudapi.net/ when exporting the templates (POST https://management.azure.com/subscriptions/subscriptionID/resourcegroups/RG-Name/exportTemplate)

    I know this may well be by design but please imagine what it's like to be dumped with undefined infra in GovCloud you need to get into TF fast.

    Don't suppose there is anyway around this?

  • CLI changes

    CLI changes

    Note all changes detailed here are to the -rg --resource-group mode of aztfy.

    • ~Remove -batch flag and instead default to batch behavior of running non-interactively~ Delayed
    • ~add a -interactive -i to access the UI for those who would still want to use it.~ Delayed
    • [x] Change --batch -b to ~--quiet -q~ --non-interactive/-n. (#242)
    • [x] tag for --hcl-only to only import the HCL files. (#246)
    • [x] add a -generate -g tag to just generate a mapping file based on the specified resource group. (#240)
    • [x] New subcommand map/mapping-file to consume the mapping file, and removing the original -m option from rg and query subcommands. (#241)
    • [x] Makes all subcommands consistent about interactive/non-interactive modes (#243)

    Though not directly CLI related, within interactive mode having an option to export just HCL and keep "w" as export HCL and state is also a feature change.

  • Documentation on -m/--batch option.

    Documentation on -m/--batch option.

    Hello,

    I have a populated .aztfyResourceMapping.json file, how do I structure the command to use this file in batch mode?

    Do I include the resource group and use --batch? i.e aztfy rg 'rg_name' --batch ".aztfyResourceMapping.json Or do I use -m i.e aztfy rg 'rg_name' --m ".aztfyResourceMapping.json" Or do I exclude i.e aztfy ".aztfyResourceMapping.json"

    None of these options are working for me and I would prefer to use the non interactive mode as I had to cull a few items in the mapping.

    Thanks.

  • aztfy skips storage accounts

    aztfy skips storage accounts

    When running aztfy on a resource group that only has a storage account, the aztfy command skips over the storage account. No reason given for why it's being skipped, it just lists all the resources and lists (Skip) with each of them.

    ` Azure Terrafy

     EQ_Azure_RG_Mgmt01_CostMgt
    
    7 items
    

    │ /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd │ (Skip)

    /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd/blobServices/de
    (Skip)
    
    /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd/fileServices/de
    (Skip)
    
    /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd/queueServices/d
    (Skip)
    
    /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd/tableServices/d
    (Skip)
    
    /subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG/providers/Microsoft.Storage/storageAccounts/myteststoracc538asd/blobServices/de
    (Skip)
    
    💡/subscriptions/xxx-yyy-zzz/resourceGroups/Test_RG
    azurerm_resource_group.res-6
    

    `

  • New option: `--append` to allow importing to existing workspace

    New option: `--append` to allow importing to existing workspace

    For local backend, aztfy will by default ensure the output directory is empty at the very begining. This is to avoid any conflicts happen for existing user files, including the terraform configuration, provider configuration, the state file, etc. As a result, aztfy generates a pretty new workspace for users.

    One limitation of doing so is users can't import resources to existing state file via aztfy. To support this scenario, this PR adds a new option --append. This option will make aztfy skip the empty guarantee for the output directory. If the output directory is empty, then it has no effect. Otherwise, it will ensure the provider setting (create a file for it if not exists). Then it proceeds the following steps.

    This means if the output directory has an active Terraform workspace, i.e. there exists a state file, any resource imported by the aztfy will be imported into that state file. Especially, the file generated by aztfy in this case will be named differently than normal, where each file will has .aztfy suffix before the extension (e.g. main.aztfy.tf), to avoid potential file name conflicts.

    • Fix #135
    • Fix #132

    Test

    ❯ AZTFY_E2E=1 go test -timeout=30m -v -run=TestMutateOutputDir ./internal/test
    === RUN   TestMutateOutputDir
    Initializing...
    List resources...
    Import resources...
    Importing /subscriptions/67a9759d-d099-4aa8-8675-e6cfd669c3f4/resourceGroups/aztfy-rg-budmjjug1 as azurerm_resource_group.round1_0
    Generating Terraform configurations...
    Initializing...
    List resources...
    Import resources...
    Importing /subscriptions/67a9759d-d099-4aa8-8675-e6cfd669c3f4/resourceGroups/aztfy-rg-budmjjug2 as azurerm_resource_group.round2_0
    Generating Terraform configurations...
    --- PASS: TestMutateOutputDir (133.90s)
    PASS
    ok      github.com/Azure/aztfy/internal/test    134.036s
    
  • Odd issue while exporting the resource group

    Odd issue while exporting the resource group

    Not sure if this is an issue with my local setup, but Azure CLI seems to be working fine, so writing here:

    I have logged in using az login and then set the subscription context with az account set --subscription="id here". I have confirmed that the context is correct after this one with az account show, so I assume that Azure CLI is fine.

    Then I try to run aztfy resourcegroupname, and it tries to fetch the resources, and I get this error message:

    exporting arm template of resource group resourcegroupname: DefaultAzureCredential: failed to acquire a token. Attempted credentials: EnvironmentCredential: missing environment variable AZURE_TENANT_ID ManagedIdentityCredential: IMDS token request timed out AzureCLICredential: exec: "cmd.exe": executable file not found in %PATH%

    Is there a way to run this in debug mode? I assume some REST API calls drop with tenant ID missing, but Azure CLI seems to be okay.

    E: Also, rgp ARM export works with Azure CLI fine in the same session.

  • aztfty unable to export resource  group more that 200 templates

    aztfty unable to export resource group more that 200 templates

    Hello,

    i have following error when doing an import :

    aztfy XXXX exporting arm template of resource group XXXXX: resources.GroupsClient#ExportTemplate: Failure sending request: StatusCode=0 -- Original
    Error: autorest/azure: Service returned an error. Status= Code="ExportTemplateResourcesLimitExceeded" Message="Export template is not supported for resource groups that have more than '200' resources." <<<

    Please help. Thank you

  • Log output to disk?

    Log output to disk?

    TL/DR: Can we log to disk as well as to screen? If so? How? If not? When?

    I'd like to be able to save the output of the initial aztfy run and the subsequent import run to file (csv or tsv please) so I can more easily deal with any snags encountered

    I'd rather not have to go through each screen of the output copying each page one-at-a-time ( not that annoying when you are dealing with a resource group with a small amount of resources in it - but when you are talking several hundred resources identified in a resource group? That can get pretty old really fast)

    If I'm missing something obvious? Please let me know as it will make my life easier - if this functionality isn't available in the latest build? Or not in the next release? Might it please be added? Especially for the output of ""w" import" -trying to deal with those snags is a lot of fun - as you need to access each error individually - and if you have a lot of snags to deal with? It makes for a LOT of unnecessary grunt work

    Thanks for considering this

    -=A=-

    .

  • New option `--module-path` to support appending to nested module

    New option `--module-path` to support appending to nested module

    This PR adds a new option --module-path, which accepts a path to the module (e.g. "module1.module2") where the resources will be imported and config generated. Note that only modules whose "source" is local path is supported. By default, it is the root module.

    This option can only be used in append mode.

    Meanwhile, in order to support the module, we've made related changes in depending repos:

    • https://github.com/magodo/tfmerge/commit/f52e46d03402690329b93689632a48106ef7f4b2
    • https://github.com/magodo/tfadd/commit/27c06e79a0a2a73bb90672fc95ce1f574733f6d1
  • [Feature] Create `azurerm_key_vault_secret` with value stored as `sensitive` variable in `terraform.tfvars` file

    [Feature] Create `azurerm_key_vault_secret` with value stored as `sensitive` variable in `terraform.tfvars` file

    At this time the secrets from Azure Key Vault are imported into the main.tf file in plain text.

    It would be good to create the Terraform resource for azurerm_key_vault_secret with a corresponding sensitive variable that has the value appended to a variables file (e.g. terraform.tfvars).

    When importing massive amounts of secrets, this would make it easy to add the produced Terraform code to a Git repo with appropriate .gitignore for *.tfvars files.


    A great tool in any case. Thanks team! Much appreciated 🙏

  • subnet delegation wants a name with a slightly different case (serverFarms vs serverfarms)

    subnet delegation wants a name with a slightly different case (serverFarms vs serverfarms)

    1rst of all great project and work.

    I just installed fresh (github.com/Azure/aztfy@latest) and processed a resource group with a subnet with app service vnet integration and it produced the hcl:

    resource "azurerm_subnet" "res-2" { ... delegation { name = "delegation" service_delegation { actions = ["Microsoft.Network/virtualNetworks/subnets/action"] name = "Microsoft.Web/serverfarms" } } ...

    terraform validate gives an error stating that that the name "Microsoft.Web/serverfarms" is not expected. In the long list of expected is "Microsoft.Web/serverFarms". NOTE: the capital F.

    The resource template does indeed use lowercase "Microsoft.Web/serverfarms".

    I can manually change the HCL but not sure if/how this should be addressed going forward with aztfy.

    Let me know if I can provide any more info or details.

  • Ubuntu install documentation improvement

    Ubuntu install documentation improvement

    Minor improvements to Ubuntu installation documentation section:

    • Changed Import the Microsoft repository key command to be run with sudo (this is required, and simply adding sudo in front doesn't work due to redirect)
    • Added line to retrieve Ubuntu version from from lsb-release file instead of manual adjustment of variable value
  • Issue: Error occurred unmarshalling JSON

    Issue: Error occurred unmarshalling JSON

    Hello, I have no error, but no results! Using version 0.9. After executing a basic

    aztfy rg RESOURCE_GROUP_NAME

    The process goes normally, a list of resources is generated, and I choose import. It takes some minutes and the progress bar goes up to 100%, then goes back to the resources list. After exit, no error is shown, but no state files are saved!

    I quickly created a new RG from scratch and the command goes fine in same azure subscription, creating the tfstate file.

    How can I get details of Aztfy execution on that RG? Thanks!

Terraform Provider for Azure (Resource Manager)Terraform Provider for Azure (Resource Manager)
Terraform Provider for Azure (Resource Manager)Terraform Provider for Azure (Resource Manager)

Terraform Provider for Azure (Resource Manager) Version 2.x of the AzureRM Provider requires Terraform 0.12.x and later, but 1.0 is recommended. Terra

Oct 16, 2021
azqlite is a lightweight wrapper around Azure's SDK to interact with the Azure Storage Queue service in a simpler and more idiomatic way.

azqlite azqlite is a lightweight wrapper around github.com/Azure/azure-storage-queue-go to interact with the Azure Storage Queue service in a simpler

Mar 12, 2022
Simple tool to move Azure resources based on Terraform state

aztfmove Simple tool to move Azure resources based on Terraform state Goal It is sometimes inevitable to move Azure resources to a new subscription or

Dec 29, 2022
A helm v3 plugin to adopt existing k8s resources into a new generated helm chart

helm-adopt Overview helm-adopt is a helm plugin to adopt existing k8s resources into a new generated helm chart, the idea behind the plugin was inspir

Dec 15, 2022
dockin ops is a project used to handle the exec request for kubernetes under supervision
dockin ops is a project used to handle the exec request for kubernetes under supervision

Dockin Ops - Dockin Operation service English | 中文 Dockin operation and maintenance management system is a safe operation and maintenance management s

Aug 12, 2022
OpenYurt - Extending your native Kubernetes to edge(project under CNCF)
OpenYurt - Extending your native Kubernetes to edge(project under CNCF)

openyurtio/openyurt English | įŽ€äŊ“中文 What is NEW! Latest Release: September 26th, 2021. OpenYurt v0.5.0. Please check the CHANGELOG for details. First R

Jan 7, 2023
Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd data, and intelligent diagnosis.
Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd data, and intelligent diagnosis.

Kstone 中文 Kstone is an etcd management platform, providing cluster management, monitoring, backup, inspection, data migration, visual viewing of etcd

Dec 27, 2022
Azure Kubernetes Service (AKS) advanced networking (CNI) address space calculator.

aksip Azure Kubernetes Service (AKS) advanced networking (CNI) address space calculator. Download Download the the latest version from the releases pa

Dec 23, 2022
Kubernetes operator for the Azure DevOps pipe-line agents

adoagent-operator Kubernetes operator for the Azure DevOps pipe-line agents init.sh #!/bin/bash # docker and github repo username export USERNAME='ba

Nov 11, 2021
Kubernetes operator for the Azure DevOps self-hosted pipe-line agent.

Kubernetes operator for the Azure DevOps self-hosted pipe-line agent. The operator adds an extra layer of configuration on top of the default images like: proxy settings, pool settings and auth keys.

Sep 1, 2022
Azure translation SDK For Golang

azure-translate Azure translation SDK Example key := "YOUR-SUBSCRIPTION-KEY" location := "global" client := translate.NewClient(key, location) result

Mar 15, 2022
Sample Hello World Pulumi Program for Azure

Overview This is a standard hello world style Pulumi program for Azure straight from the Pulumi docs. This is a sample repo used to test Pulumi's Auto

Jan 13, 2022
This sample shows how to host multiple Azure functions in Golang.
This sample shows how to host multiple Azure functions in Golang.

azure-function-custom-handler-with-golang This sample shows how to host multiple Azure functions in Golang. To learn more about this sample please che

Dec 26, 2022
How to get a Go / Golang app using the Gin web framework running natively on Windows Azure App Service WITHOUT using a Docker container

Go on Azure App Service View the running app -> https://go-azure-appservice.azurewebsites.net ?? This is an example repo of how to get a Go / Golang a

Nov 28, 2022
A simple go tool, that calculates the allocated resources from all nodes matching the label selector.

kube-allocated-resources This is a simple go tool, that calculates the allocated resources from all nodes matching the label selector. Build Build on

Jan 12, 2022
OpenAPI Terraform Provider that configures itself at runtime with the resources exposed by the service provider (defined in a swagger file)
OpenAPI Terraform Provider that configures itself at runtime with the resources exposed by the service provider (defined in a swagger file)

Terraform Provider OpenAPI This terraform provider aims to minimise as much as possible the efforts needed from service providers to create and mainta

Dec 26, 2022
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod
A kubernetes plugin which enables dynamically add or remove GPU resources for a running Pod

GPU Mounter GPU Mounter is a kubernetes plugin which enables add or remove GPU resources for running Pods. This Introduction(In Chinese) is recommende

Jan 5, 2023
A curated list of awesome Kubernetes tools and resources.

Awesome Kubernetes Resources A curated list of awesome Kubernetes tools and resources. Inspired by awesome list and donnemartin/awesome-aws. The Fiery

Jan 2, 2023
A cli that exposes your local resources to kubernetes
A cli that exposes your local resources to kubernetes

ktunnel Expose your local resources to kubernetes ?? Table of Contents About Getting Started Usage Documentation Contributing Authors Acknowledgments

Jan 7, 2023