HTTPS (the DevOps way) for web applications behind the firewall

The problem

For web applications, communicating over SSL/HTTPS is a must. Put simply, unencrypted traffic over HTTP is a security risk.

This is nothing new, and it is hard to find a public web service in production that is not secured with SSL. Non-production environments are a completely different story, however.

As a DevOps consultant, I’ve lost count of the number of times I have seen non-production, dev, test, QA, and staging environments running plain HTTP. Generally speaking, I think the main reasons for this are:

• These environments are often behind a firewall and not exposed to the external world. Therefore developers don't really feel the need to acquire a certificate as they feel safe on an intranet/LAN.
• The process of getting and renewing a certificate is often time consuming when Let’s Encrypt cannot be used.
• The process is often manual or centralized in a different organization which does not fit with development processes driven by CI/CD.

Ask any security expert and they will tell you that such networks are compromised all the time, and unencrypted traffic is vulnerable. That feeling of safety is false.

Development teams who practice CI/CD work at speed – release cycles are frequent for both old and new deliveries, and so waiting on the creation of a certificate is not an option. That new service is going to be deployed to the internal environments, just not with SSL/HTTPS.

In CI/CD the deployment process is automated, and having automated deployments and tests break while waiting for certificate renewal is not an option.

The solution

The best way to ensure that any web service is running over HTTPS with a valid certificate is to make it as painless as possible – via automation. When you deploy a web service it should just happen. And a DevOps approach is automated and repeatable.

Let’s Encrypt made this possible for public web services using the Automated Certificate Management Environment (ACME) protocol.

But that doesn’t help much when it comes to non-public, non-production environments. To use Let’s Encrypt you will need to have the firewall open somewhere. A common example could look like the following diagram:

This requires that you:

• Use the DNS-01 challenge.
• Have a DNS provider that supports the ACME protocol.
• Have published the domain (even if it is not reachable).
• Have a mechanism to distribute the certificates. 
• Have an opening in the firewall somewhere.

This solution is not optimal for a number of obvious reasons.

Luckily for us, Smallstep provides a certificate authority (CA) that supports the ACME protocol. It is open source, lightweight, and provides multiple ways to provision and manage certificates. 

The Smallstep CA was built for DevOps and Modern Systems. It can be packaged into a container, run on a K8's cluster, and automated with provisioning tools such as Chef, Ansible, Puppet, etc. So it is possible to get that Let’s Encrypt experience and have automated handling of certificates.

A diagram of this solution would look something like this:

• The HTTP-01 challenge is simpler. 
• There is no need to involve a DNS provider.
• There is no need to open the firewall.
• There is no need to publish domains. 
• You do not need to implement and maintain a mechanism to distribute the certificates as there are many existing ACME clients that can be leveraged. To name a few, Traefik, ACME.sh, Certbot and Smallstep’s CLI.
• A lot of organizations already use and distribute internally signed certificates. The Smallstep CA allows you to import the root certificates.

 This is a much cleaner solution. 

The demo

I have created a repository with a simple demo implementation using Smallstep’s CA with the behind the firewall solution. 

I dockerized the CA and created a docker-compose file to easily spin it up. There is an entrypoint script to initialize the CA with some values defined in a .env file.

I then dockerized and created docker-compose files for two proxies (Nginx and Traefik) to demonstrate how to get certificate generation and renewal for a simple “Hello World!” web service.

I chose Traefik because it has built-in support for ACME. You just need to configure the compose file with the proper values (CA, challenge type, etc) and Traefik will handle everything, including certificate renewal.

I chose Nginx because it does not have built-in support for ACME. Instead I baked in Smallstep’s CLI, which supports certificate generation and renewal, into the Nginx image and created an entrypoint script to get the certificate generated and run the renewal daemon. The Nginx configuration just references the certificates that the Smallstep ACME client generates and renews.

To try it out, clone the repository and follow the Quick Start guide. But the basic process is as follows:

1. Fill out some environment variables in the .env file.
2. Update your host file.
3. Spin up the CA container, get the fingerprint, and add it to the .env file.
4. Add the root certificate generated by the CA when the container is spun up to your host’s trust store. This is something that would normally be handled by ITOPs
5. Spin up Traefik or Nginx to see the certificate generation.

Here is a screencast demonstrating the setup. It shows the spinning up of the certificate authority, adding the generated root and intermediate certs to the host’s trust store, and spinning up Traefik and Nginx to proxy a simple “Hello World!” web service.

There are some things to take note of:

• When bringing the CA container up, it is initialized in the entry point and the root and intermediate certs are generated.
• Smallstep’s CA supports multiple types of provisioners with the default provisioner type being JWK. But the entry point adds an ACME provisioner type, as that is what we want to use.
• Both the Traefik and the Nginx images have Smallstep’s CLI baked in. The CA serves traffic over HTTPS. Therefore client containers need to have the root and intermediate certs added to the container’s trust store. The CLI is used to do this in the container's entry point.
• I use Smallstep’s CLI locally to add the certificates to my computer's trust store. It packs them into a .pem file for me, so I don’t need to do it manually.
• The containers use environment variables for configuration. This is not really 

secure and a file-based approach should be used for anything besides a demo.

Summary

The demo proves that it is possible to automate the handling of SSL certificates when Let’s Encrypt cannot be used due to firewall restrictions. 

In real life, operations should own and manage the certificate authority (CA). They are responsible for distributing the root and intermediate certificates to the needed hosts on the network anyway. 

This blog does not address the management of the CA itself either, but rather focuses on how to use it to get that Let’s Encrypt experience for development teams. 

I would encourage operations teams to talk to the Smallstep guys about what else can be done with the CA and dig into the details of managing the CA. Providing something like this really supports the DevOps approach of breaking down the barriers between development and operations; dare I say it even supports DevSecOps? 

The benefits from a development, operations and security perspective are fairly obvious:

• Out-of-the-box encrypted traffic for web services. Security smiles :-)
• Automated SSL certificate management and removal of manual certificate handling processes. Operations smiles :-)
• More stable environments that look more like production at the pace of development needs. Development smiles :-)