LTF is a minimal, transparent Terraform wrapper. It makes Terraform projects easier to work with.

LTF

Status: alpha

LTF is a minimal, transparent Terraform wrapper. It makes Terraform projects easier to work with.

In standard Terraform projects, the *.tf files are typically duplicated in each environment, with minor differences for the backend configuration. Every environment directory contains nearly identical *.tf files and *.tfvars files. It requires some effort to maintain all of these environments and keep them consistent. Changes take longer because they involve more files.

terraform
├── dev
│   ├── dev.auto.tfvars
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
├── qa
│   ├── qa.auto.tfvars
│   ├── main.tf
│   ├── outputs.tf
│   └── variables.tf
└── live
    ├── blue
    │   ├── live.blue.auto.tfvars
    │   ├── main.tf
    │   ├── outputs.tf
    │   └── variables.tf
    └── green
        ├── live.green.auto.tfvars
        ├── main.tf
        ├── outputs.tf
        └── variables.tf

Using LTF, the *.tf files are shared between all environments. A single *.tfbackend file is used to configure the backend for all environments, making use of variables. Environment directories only contain what is unique about the environment: the *.tfvars file. Maintenance is easier and environments are consistent by default. Changes take less time because they involve fewer files.

ltf
├── auto.tfbackend
├── main.tf
├── outputs.tf
├── variables.tf
├── dev
│   └── dev.auto.tfvars
├── qa
│   └── qa.auto.tfvars
└── live
    ├── blue
    │   └── live.blue.auto.tfvars
    └── green
        └── live.green.auto.tfvars

Using LTF is very easy. It avoids tedious command line arguments. Change to an environment directory and use ltf just like you would normally use terraform in a simple, single-directory Terraform project.

$ cd dev
$ ltf init
$ ltf plan
$ ltf apply
$ cd ../live/green
$ ltf init
$ ltf plan
$ ltf apply -target=random_id.this

Why use LTF?

LTF only does a few things:

  • It finds and uses a parent directory as the configuration directory.
  • It finds and uses tfvars and tfbackend files from the current and parent directories.
  • It supports variables in tfbackend files.
  • It supports hooks to run custom scripts before and after Terraform.

LTF is good because:

  • It avoids tedious command line arguments, so it's quick and easy to use.
  • It is a transparent wrapper, so all standard Terraform commands and arguments can be used.
  • It is released as a single binary, so installation is easy.
  • It keeps your Terraform configuration DRY using only a simple directory structure.
  • It only does a few things, so there's not much to learn.
  • It runs Terraform in the configuration directory, so there are no temporary files or build/cache directories to complicate things.

LTF is purposefully simple and feature-light, so it doesn't do everything:

Installation

Install manually

Download the appropriate binary for your system from the releases page, move it to your PATH, and make it executable.

Install using bin

bin manages binary files downloaded from different sources. Run the following to install the latest version of LTF:

bin install github.com/raymondbutcher/ltf

Usage

Run ltf instead of terraform. All standard Terraform commands and arguments can be used.

Example:

$ ltf init
$ ltf plan
$ ltf apply -target=random_id.this

How LTF works

LTF searches the directory tree for a Terraform configuration directory, tfvars files, and tfbackend files, and passes their details to Terraform.

When LTF finds no *.tf or *.tf.json files in the current directory, it does the following:

  • Finds the closest parent directory containing *.tf or *.tf.json files, then adds -chdir=$dir to the Terraform command line arguments, to make Terraform use it as the configuration directory.
  • Sets the TF_DATA_DIR environment variable to make Terraform use the .terraform directory inside the current directory instead of the configuration directory.

When running ltf init, it does the following:

  • Finds *.tfbackend files in the current directory and parent directories, stopping at the configuration directory, then updates the TF_CLI_ARGS_init environment variable to contain -backend-config=$attribute for each attribute.
    • The use of Terraform variables in *.tfbackend files is supported.

It always does the following:

  • Finds *.tfvars and *.tfvars.json files in the current directory and parent directories, stopping at the configuration directory, then sets the TF_VAR_name environment variable for each variable.
    • Terraform's precedence rules are followed when finding variables, with the additional rule that variables in subdirectories take precendence over variables in parent directories.
    • If any tfvars files exist in the configuration directory, variables from those files take precedence over any environment variables. LTF raises an error if it tries to set an environment variable that would be ignored by Terraform. This can be avoided by using variable defaults instead of tfvars files, or by moving the tfvars files into a subdirectory.
  • Runs hook scripts before and after Terraform.

Hooks

LTF also supports hook scripts defined in ltf.yaml. It looks for this file in the current directory or any parent directory. Hook scripts are just Bash scripts; they can contain multiple lines, and they can even export environment variables. Environment variables will persist to subsequent hooks and to the Terraform command.

Hooks can be configured to run before specific Terraform commands, and/or after they have completed successfully, and/or after they have failed.

Schema

hooks:
  $name: # the name of the hook
    before: # (optional) run the script before these commands
      - terraform # the hook will always run
      - terraform $subcommand # the hook will only run before this subcommand
    after: [] # (optional) run the script after these commands finish successfully
    failed: [] # (optional) run the script after these commands have failed
    script: $script # bash script to run

Example: running commands

hooks:
  greetings:
    before:
      - terraform
    script: |
      echo "Hello, this is a hook!"
      echo "The date is $(date -I)"

Example: Setting environment variables

hooks:
  TF_VAR_hook:
    before:
      - terraform apply
      - terraform plan
    script: export TF_VAR_hook=hello
Comments
  • Change how variables are passed to Terraform

    Change how variables are passed to Terraform

    LTF currently finds *.tfvars and *.tfvars.json in the directory tree and updates the environment variables TF_CLI_ARGS_plan and apply to contain -var-file=$name for each file.

    I am wondering if it would be better to do this:

    • for each directory up to and including the configuration directory:
      • parse any *.tfvars and *.tfvar.json files
      • also parse *.tf and *.tf.json files for variable defaults
      • for each parsed variable:
        • export TF_VAR_$name=$value

    It should also handle CLI args (-var and -var-file) and environment variables like TF_CLI_ARGS and TF_CLI_ARGS_plan etc.

    This would allow the hooks to easily read any of the variables, and also to alter variables before Terraform runs. It might also make it easier to create additional wrapper tools that don't need to understand the CLI args and can just check TF_VAR_name values.

    One ugly part about this idea is related to variable precedence. If all of the variables are sent using environment variables, then any tfvars files in the configuration directory will take precedence. Perhaps LTF could know which variables are affected by this and raise an error (read all variables from files in the config dir and CLI args, and raise an error if any TF_VAR_name environment values don't match).

  • Search directory tree for tfvars and tfbackend

    Search directory tree for tfvars and tfbackend

    It currently only looks in the current directory. It should look at parent directories, stopping at and including the configuration directory.

    Update example and the tree in the docs accordingly.

  • Support variables in backend configuration

    Support variables in backend configuration

    Proposed idea

    Feature:

    • Dynamic backends:
      • Allow use of variables in backend configuration blocks.

    LTF solves the issue of Terraform not supporting input variables in the backend configuration block. Note that LTF only adds support for input variables. It does not support accessing local values, nor data sources.

    When LTF runs, it does the following:

    • Reads the backend block from the Terraform configuration.
    • Renders the backend block using HCL using Terraform variables.
    • Passes each line from the rendered backend block to Terraform using the -backend-config= command line argument, which takes precedence over the values in the file.

    Work required

    • Parse configuration
    • Load tfvars like Terraform does (parse files, understand CLI arguments involving variables, understand environment variables like TF_CLI_ARGS_plan=-var-file and TF_VAR_name=value)
    • Render HCL
    • Pass -backend-config args.

    This seems feasible because it's written in Go and we have access to the Terraform source code. It's also feasible in Python, Pretf does all of this. It might be difficult in other languages.

  • Support variables in tfbackend files

    Support variables in tfbackend files

    Now that LTF understands variables (#19) it might be easy to support variables in tfbackend files. This would reduce the number of files needed in a project.

  • Add version command

    Add version command

    When running ltf version it should print the LTF version before it runs terraform version. It should automatically use the current git tag, so probably needs to do something in the github action.

  • Improve command matching in hooks

    Improve command matching in hooks

    It currently matches on terraform and terraform $subcommand.

    Could use regular expressions, but -chdir and other arguments (which can be in any order) makes this awkward. So some more custom matching seems better.

    Ideas:

    # always
    terraform
    
    # any subcommand
    terraform *
    
    # no subcommand
    terraform !
    
    # subcommands
    terraform !init|plan|apply # all except for init, plan and apply
    terraform plan|apply
    
    # flags
    terraform -target
    terraform !-target
    

    Needs more thought and refinement.

  • Support custom hooks

    Support custom hooks

    Use cases:

    1. Before plan and apply, run SOPS to get decrypted YAML, base64 encode it, turn the output into a TF_VAR_secrets environment variable, and then include that in the environment when Terraform runs. BUT: can't this be done with a module that uses a data source to run SOPS?
    2. Post to Slack when someone is running Terraform

    Some ideas for implementation:

    1. Search parent dirs for ltf.yaml which allows running commands at certain points such as before-terraform, before-apply, after-apply, after-terraform. It would need flags for exporting environment variables, whether it should run even after Terraform errors, etc.
    2. Search parent dirs for ltf.yaml which allows specifying a hook script which receives command line arguments containing the hook event details.
    3. Embed a language like Starlark or Lua. Search parent dirs for ltf.bzl or ltf.starlark (what file extension is it?) or ltf.lua and call that.

    Possible hooks:

    1. When LTF starts (but what's the use case for this?)
    2. Before and after each of Terraform's 22 commands, or any command, or no command.

    Other ideas for ltf.yaml besides hooks:

    1. Logging/verbosity level.

    If there are any YAML settings, it's preferable for them to also be settable with environment variables (e.g. LTF_HOOK_COMMAND or LTF_BEFORE_PLAN).

  • Add release pipeline

    Add release pipeline

    • Add GitHub actions to build and release binaries
      • Can probably copy what I did in remake
    • Update README with installation instructions
    • Try out https://github.com/marwanhawari/stew and maybe add instructions
    • Try out https://github.com/marcosnils/bin and maybe add instructions
    • Maybe create asdf plugin and add instructions

    Possible instructions:

    LTF is released as a single binary. Download and add it to your PATH. Make it executable if needed.

  • Manage secrets

    Manage secrets

    There are many different ways to deal with secrets. Different cloud providers have different services, and they can't all be supported directly.

    SOPS is a pretty good option, but it might be best to run it with hooks, or with a data source that runs SOPS, or with the SOPS Terraform Provider. It probably shouldn't be supported/included directly in LTF. Just having examples might be enough.

    It could be interesting to build a CLI/TUI program for managing KMS secrets directly in Terraform code (in tfvars or tf files) and then access them via aws_kms_secrets data source. This could be a separate program, then you'd create a hook to run it.

    Inspiration:

  • Other ideas

    Other ideas

    Ideas for later or never

    • Creating backend resources.
      • Different backends have different methods for creation so this tool could only support some, if it supports any at all.
      • For S3/DynamoDB it might be better to create a separate tool or easy to use CloudFormation template that can be used to set up the Terraform project.
    • Managing secrets
      • For SOPS, I'm not sure.
        • Consider adding a hooks system to run commands like SOPS and use the results as Terraform variables or environment variables.
        • Or have first-class support for running SOPS.
        • Or include the SOPS library and use it directly.
        • Or use the Terraform Provider
    • Creating backend resources and managing secrets in AWS.
      • Create S3/DynamoDB/KMS resources with CloudFormation.
      • Create built-in CLI and TUI interfaces for managing KMS secrets that get committed to git.
      • Also consider the aws_kms_secrets data source.
    • Remote modules, e.g. Terragrunt.
      • Probably will not implement because the project structure is too different.
  • Allow hooks to prevent running Terraform

    Allow hooks to prevent running Terraform

    This would allow hooks to create custom commands like ltf sops or ltf plan-all that don't just run Terraform.

    Ideas:

    • Hooks update the LTF_TERRAFORM_CMD environment variable, then LTF uses that.
      • This was suggested in #7
    • Export some other special env var which LTF checks for and then stops.
      • Set LTF_EXIT=n to skip the Terraform command but still run remaining hooks.
      • Set LTF_EXIT_NOW=n to exit immediately after the current hook.

    Worse ideas:

    • An extra field in the YAML file (not flexible, you might want to do this only sometimes and use the script to figure out when).
  • Provide useful functions to use in hook scripts

    Provide useful functions to use in hook scripts

    Some ideas:

    ltf_find

    Finds a file in any of the relevant directories that LTF has found (dirs = all directories between cwd and chdir).

    Use cases:

    • Use ltf_find secrets.yaml to find the secrets file in the current directory, or a parent directory, and then run:
      • export TF_VAR_secrets=$(sops --decrypt $file | base64)
  • Provide useful environment variables for hooks

    Provide useful environment variables for hooks

    Some ideas:

    LTF_TERRAFORM_CMD

    The Terraform command that is going to run (or has run, if it's an after/failed hook).

    Use cases:

    • Post the command to Slack or some logging system.
    • Alter it to run sops instead, when you run ltf sops. This would only work if LTF reads back these variables after running a hook script and then updates the command accordingly.

    LTF_TERRAFORM_DIR

    Where Terraform will be running, AKA the chdir directory or cwd if there is none.

    Use cases:

    • Read/write files in this directory (but why).
    • Write variables to this directory and delete afterwards (probably should use env vars instead).

    LTF_YAML_DIR

    Where ltf.yaml lives, probably the top level of the repo/project.

    Use cases:

    • Use scripts or other files in this directory (e.g. a hooks directory in the top level of the project, next to the YAML file).
Related tags
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.
Hassle-free minimal CI/CD for git repositories with docker or docker-compose projects.

GIT-PIPE Hassle-free minimal CI/CD for git repos for docker-based projects. Features: zero configuration for repos by default automatic encrypted back

Sep 23, 2022
Clones github projects into ~/Projects/github/{org}/{repo}

Tidy clone Github cli extension (gh extension) to clone repos into ~/Projects/github/{org}/{repo} on the local filesystem Install gh extension install

Jan 19, 2022
Terraform-equinix-migration-tool - Tool to migrate code from Equinix Metal terraform provider to Equinix terraform provider

Equinix Terraform Provider Migration Tool This tool targets a terraform working

Feb 15, 2022
Terraform-in-Terraform: Execute Modules directly from the Terraform Registry

Terraform-In-Terraform Provider This provider allows running Terraform in Terraform. This might seem insane but there are some edge cases where it com

Dec 25, 2022
jsPolicy - Easier & Faster Kubernetes Policies using JavaScript or TypeScript
jsPolicy - Easier & Faster Kubernetes Policies using JavaScript or TypeScript

Website • Getting Started Guide • Documentation • Blog • Twitter • Slack jsPolicy - Easier & Faster Kubernetes Policies using JavaScript or TypeScript

Dec 30, 2022
Kubeswitch - Easier way to switch your kubernetes context

Switch Kubectl Context Easier way to switch your kubernetes context Set PATH Dow

Jun 17, 2022
Stuff to make standing up sigstore (esp. for testing) easier for e2e/integration testing.
Stuff to make standing up sigstore (esp. for testing) easier for e2e/integration testing.

sigstore-scaffolding This repository contains scaffolding to make standing up a full sigstore stack easier and automatable. Our focus is on running on

Dec 27, 2022
Utility to make kubeseal --raw a bit easier.

ks Utility to make kubeseal --raw a bit easier. Building GOOS=windows GOARCH=amd64 go build -o ks-windows-amd64.exe ks.go GOOS=windows GOARCH=386 go b

Aug 19, 2022
A kubectl plugin for easier query and operate k8s cluster.
A kubectl plugin for easier query and operate k8s cluster.

kube-query A kubectl plug-in that makes it easier to query and manipulate K8S clusters. (what is kubectl plug-in ?) Kube-query support some resource s

Jun 9, 2022
Terraform utility provider for constructing bash scripts that use data from a Terraform module

Terraform Bash Provider This is a Terraform utility provider which aims to robustly generate Bash scripts which refer to data that originated in Terra

Sep 6, 2022
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)
Terraform provider to help with various AWS automation tasks (mostly all that stuff we cannot accomplish with the official AWS terraform provider)

terraform-provider-awsutils Terraform provider for performing various tasks that cannot be performed with the official AWS Terraform Provider from Has

Dec 8, 2022
Terraform Provider for Azure (Resource Manager)Terraform Provider for Azure (Resource Manager)
Terraform Provider for Azure (Resource Manager)Terraform Provider for Azure (Resource Manager)

Terraform Provider for Azure (Resource Manager) Version 2.x of the AzureRM Provider requires Terraform 0.12.x and later, but 1.0 is recommended. Terra

Oct 16, 2021
Quick start repository for creating a Terraform provider using terraform-plugin-framework

Terraform Provider Scaffolding (Terraform Plugin Framework) This template repository is built on the Terraform Plugin Framework. The template reposito

Dec 15, 2022
Terraform-provider-mailcow - Terraform provider for Mailcow

Terraform Provider Scaffolding (Terraform Plugin SDK) This template repository i

Dec 31, 2021
Terraform-provider-buddy - Terraform Buddy provider For golang

Terraform Provider for Buddy Documentation Requirements Terraform >= 1.0.11 Go >

Jan 5, 2022
Terraform-provider-vercel - Terraform Vercel Provider With Golang

Vercel Terraform Provider Website: https://www.terraform.io Documentation: https

Dec 14, 2022
Terraform-grafana-dashboard - Grafana dashboard Terraform module

terraform-grafana-dashboard terraform-grafana-dashboard for project Requirements

May 2, 2022
Puccini-terraform - Enable TOSCA for Terraform using Puccini

(work in progress) TOSCA for Terraform Enable TOSCA for Terraform using Puccini.

Jun 27, 2022