Crossplane is great, but what about critical infrastructure?

Crossplane, a promising tool from CNCF, uses the power of Kubernetes to manage an organization's entire infrastructure. By providing continuous reconciliation and declarative state management, it aims to streamline infrastructure provisioning and management. However, when it comes to critical infrastructure, there are still important factors to consider.

What is Crossplane?

Crossplane is a control plane that runs inside Kubernetes.

A control plane, as a concept/paradigm, refers to the idea of a service that watches a declared state and makes sure that a system's actual state reflects that of the declared. In other words, a control plane reconciles a system’s current state to match the desired.

Crossplane is often used to provision and manage cloud resources. It promises that the resources running in the cloud provider are in sync with the state declared in Kubernetes.

For example, a developer can declare a database as a Kubernetes manifest (YAML) and apply it to the Kubernetes cluster with Crossplane installed. Crossplane will then start syncing the declared state with the database in the cloud provider. To provision a database resource on AWS, a developer would have to create the following:

Crossplane will read the above  RDSInstance and create a database with those specifications on AWS. If there is a change in the manifest, Crossplane will reconcile the state and update the database accordingly. If there are manual changes to the database instance from the AWS management console, Crossplane will automatically revert those changes to match the state declared in Kubernetes.

Using Crossplane for infrastructure management makes it possible to provision resources on multiple cloud providers simultaneously, which can be beneficial because every cloud provider has a different offering. Currently, Crossplane supports AWS, GCP, and Microsoft Azure as cloud providers. A DigitalOcean provider is also in active development.

Why Crossplane over Terraform?

When it comes to Infrastructure as Code (IaC) there are many great tools on the market, with Terraform being the most popular. Crossplane and Terraform try to solve the same problems by letting you describe your complete infrastructure as code, but Crossplane has some advantages over Terraform:

• Crossplane easily integrates with GitOps workflows.
• Crossplane will automatically correct drift.

• Crossplane doesn’t need a state to be stored.

Configuration drift with Terraform can be avoided by using an automated tool, e.g., Atlantis, or a script that simply runs terraform apply regularly (like the tf-controller for flux).

Doing this essentially creates a system that works like a control plane. So, instead of using a tool like Terraform and wrapping it in some automation tool/script, it may be better to use a control plane-based tool built to solve exactly that.

Crossplane only allows configuration drift if you intentionally pause the reconciliation loop; otherwise, Crossplane will continuously sync the state as long as it’s running. If someone makes a change manually to the cloud resource in the cloud provider’s UI, Crossplane will revert the changes to the state declared in Kubernetes. So, if you want to make changes to the cloud resource, it must be done through Crossplane.

Furthermore, if you manage your Kubernetes resources with ArgoCD or FluxCD, you can check in your Crossplane resources into Git and manage the entire infrastructure configuration from within a Git repository. This ensures an audit trail in Git and enables your team to manage infrastructure changes through pull requests (or other Git processes your team might already use).

A notable downside to Terraform is its state, which can be lost and corrupted, leading to complications if you use it to manage your entire infrastructure. Additionally, you must store the state in a remote location with the correct access control; otherwise, the whole team won’t be able to access it, etc., which can be troublesome and time-consuming to set up.

Terraform looks at three entities when applying changes — your local Terraform files, the Terraform state, and the actual state in the cloud provider. If the state in the cloud provider divagates from the stored state, this can cause complications.

In contrast, Crossplane only looks at the declared resources and what’s running in the cloud provider. There is no mutable state that it needs to worry about.

Just like Terraform, Crossplane uses the concept of providers. Crossplane-providers work similarly to Terraform-providers. Service providers can create a plugin that integrates with Crossplane, giving the user the ability to provision external resources on their infrastructure. It is now up to the service provider to manage and ensure that the state running on their infrastructure matches the desired state declared in the Kubernetes cluster.

Why Terraform over Crossplane?

We’ve highlighted the upsides of Crossplane; now let’s look at what it lacks in comparison.

One of the biggest drawbacks of using Crossplane is that there’s no option to preview changes before they’re applied.

With Terraform, the developer can run terraform plan to see a preview of the changes before committing to the new configuration. Crossplane has no such feature, meaning you cannot preview the resources it will create/modify/delete. Developers can only apply the manifests and hope they did so correctly.

Let’s say you rename your  RDSInstance as in the example above. Crossplane may delete the existing database and recreate one with the new name. This depends on the implementation of the provider and what kind of safeguards you’ve set up, making Crossplane risky for critical infrastructure. This is also described in a GitHub Issue.

Testing your Crossplane changes before merging into production

You could limit the risk of applying the wrong configuration by testing it in a staging environment. However, it’s important to note that even though the staging environment is supposed to be as close to the production environment as possible, it will never be identical.

There’s always a risk of breaking the production environment with the new configuration. You have to be careful when using Crossplane for your critical infrastructure because you have no “plan” step where you can preview your changes before they go into production.

Even though you can revert your changes in Git and return to the old state, it will not resurrect/recreate a deleted production database. It will, however, create a pristine new one (not exactly a fall-back mechanism).

The future of Crossplane

It is uncertain if Crossplane will have a preview-feature or "dry-run"-feature (where it runs the new configuration but without changing anything) in the future. At this point, this feature has been discussed for over 2 years. I imagine, preview features being inherently difficult to make for control planes as it’s not obvious when in the process, a developer would review the changes.

Crossplane vs. Terraform

While Crossplane offers several advantages over Terraform and similar tools, it still falls short in managing critical infrastructure due to its lack of a dry-run/plan feature.

Terraform's terraform plan command allows developers to review and validate changes that occur before committing to a new configuration. This feature provides an additional layer of safety and helps prevent unintended changes.

In contrast, Crossplane doesn’t offer a preview feature. Without the ability to assess and verify changes in advance, there’s a greater risk of errors and disruptions to the production environment.

Don’t get me wrong, I think Crossplane is a great project, and I am a big fan, but if I ran a company, I would be hesitant to use Crossplane for my critical core infrastructure.