A Guide to Open-Source IaC Testing

Introduction

Over the past several years, Infrastructure-as-Code (IaC) platforms, such as Terraform, CloudFormation, and Kubernetes, have rapidly gained traction as the preferred mechanism to provision and manage cloud infrastructure. And for good reason.

It wasn’t that long ago that ClickOps was the dominant approach for cloud management. Everyone is sympathetic to the need for agility when there is a business-critical change required. “Just log onto the console” can seem like a perfectly justifiable action. It usually is…until it isn’t.

Collectively, as an industry, we’ve wised up to the fact that it is almost impossible to manage infrastructure at scale with a high degree of security and reliability without a soul-crushing load of red tape and change management. Moving cloud configuration to IaC managed in source control is the best option we have.

However, there are challenges that come with this shift to IaC. Terraform and CloudFormation can have idiosyncrasies in implementation, and usage is not standardized. One org may execute Terraform from Terraform Cloud, while another may use OSS Terraform executed in CI, while still another may run from developer laptops. It feels like application code in the ’00s when CI and test automation were still novel through much of the industry.

If modern agile software development taught us anything, it’s that the ROI on continuous integration and test automation is almost always positive, and the return usually comes faster than you expect. IaC is no different: any automated testing for your IaC is better than none. It may be a confusing field of options, but that isn’t a good reason to pretend the problems of cloud configuration are not all around you.

Although manual code reviews can capture some of the issues, much like application source code reviews, these errors are hard to catch with human review alone. Nobody really enjoys reviewing Terraform pull requests. That’s why it’s important to use static analysis to identify security issues and fix them before they are deployed into production.

But where to start? Static analysis tools for IaC are relatively new. At Soluble, we looked at the various open-source IaC static analysis tools available, and here is our assessment of them.

Terrascan

Terrascan leverages Open Policy Agent (OPA)as the underlying assessment engine. It provides a wide variety of out-of-the-box policies such as the CIS Benchmark. You can use OPA to create custom policies using the Rego query language.

• Pros: It leverages OPA, which is an industry standard, widely used in Kubernetes.
• Cons: Using Terrascan requires manual detection of Terraform, CloudFormation, and Kubernetes files to run the appropriate checks. Scans were difficult to read. Some may find Rego, OPA’s query language a barrier to adoption, even though it is an industry standard.

Checkov

Checkov lets you set policies in your GitHub repository, and share the version-controlled policies with your teams so they can scan their IaC against them. It is written in Python, which makes it very approachable. Rule extensions are also implemented in Python. It has good static analysis capabilities.

• Pros: It automatically runs all types of checks without manual invocation of any corresponding checks, and it covers most of the CIS, HIPAA checks. Adding a new check is comparatively easy when compared to using other tools, albeit requiring Python development.
• Cons: It’s written in Python, so it is not as easy to deploy as other tools written in Go, which can be distributed as a static binary. If you’re committed to using OPA, you may not want to use Checkov.

TFLint

TFLint is a Terraform linter that is focused on syntactical correctness and best practices. It does not focus on security. It has a framework where features are provided by plugins. The key features are as follows: Find possible errors (like illegal instance types) for Major Cloud providers (AWS/Azure/GCP). Warn about deprecated syntax, unused declarations. Enforce best practices, naming conventions.

• Pros: Basic Terraform linter.
• Cons: Does not solve security and compliance issues.

Tfsec

Tfsec is a static analysis security scanner that developers can use for checking Terraform code. It uses static analysis and integration with the official Hashicorp HCL parser to detect security issues. It’s designed to run locally and in your CI/CD pipelines.

• Pros: Vendor-neutral open-source. It has a readable output and is easy to deploy. It has relatively good detection and a small wiki of remediation instructions. Adding custom checks is comparatively easy.
• Cons: It only works for Terraform; it doesn’t support CloudFormation or Kubernetes manifests. It has fewer out-of-box checks compared to the other tools. You will need to implement a number of checks to get good coverage against your compliance standards, such as CIS, HIPAA, etc.

Sentinel

Sentinel is Hashicorp’s policy-as-code assessment tool designed to help Terraform users safely create and manage infrastructure. Hashicorp offers it noting that the rapid adoption of Terraform “means less experienced users could make significant mistakes that impact business operations.” It limits exposure by codifying business and regulatory policies to ensure infrastructure changes are safe, and it is designed to work across the Hashicorp suite of enterprise products.

• Pros: Works well with other Hashicorp products, including Vault, Nomad, and Consul.
• Cons: Does not work for CloudFormation or Kubernetes. Not open-source. Uses its own rule language which may not be to taste.

KICS

KICS is an open-source solution for static analysis of IaC created by Checkmarx. This makes sense for Checkmarx customers. If you are using Checkmarx for static analysis security testing, you could use their IaC scanner as well. Like Terrascan, it uses OPA as the underlying rules engine, which uses the Rego language. It is written in Go and is distributed as a static binary.

• Pros: Works with Checkmarx.
• Cons: Terraform HCL parser not as robust as other tools. OSS community support is uncertain compared to other tools.

Cfn-Lint

Cfn-lint, also known as cfn-python-lint, is an open-source AWS CloudFormation linter. It is sponsored by AWS, but is not an AWS product. Like TFLint, it is not intended to address security configuration items. Instead, it focuses on syntax and best practices.

• Pros: Wide adoption.
• Cons: Little security focus. Implemented in Python, which can make distribution challenging in some environments as compared to tools that are distributed as static binaries.

Cfn_Nag

Cfn_nag is an open-source assessment tool that addresses some security best practices. It is written in Ruby and has broad usage.

• Pros: cfn_nag is the OG of IaC security tools. It has wide usage and has more of a security focus than cfn-lint.
• Cons: Not as popular as some other tools listed. Distribution as Ruby gem can be challenging as compared to tools that compile down to a static binary.

Kubelinter

Kubelinter is a static analysis tool that checks Kubernetes YAML files and Helm charts to ensure the applications represented in them adhere to best practices.

• Pros: Static binary
• Cons: Limited to checking Kubernetes configurations. Rules must be compiled into kubelinter. There is no runtime extensibility as there is with most of the other options here.

Conclusion

Here’s a rundown of our results.

Overall, Checkov is one of the most polished tools with a good open-source community. However, without integration with the companion commercial offering, it does not really help developers do much more than identify problems, which may be of limited utility.

TFSec is very promising, at least for Terraform users, particularly because it has a vibrant open-source community and is not strongly vendor-aligned.

However, it is still hard to know where to start, what tools to choose, and how to implement them effectively.

The most important conclusion to draw from this is that any automated testing for your Infrastructure-as-Code is better than none. It may be a confusing field of options, but that isn’t a good reason to pretend the problems of cloud configuration are not all around you. If you run your code through any of these tools, you might be surprised by what you find.