Mitigate against tampering attacks - Secure your software delivery chain

On top of that, it was revealed yesterday that the PHP source code repository has also been compromised. If left unnoticed, all servers running PHP could have been targeted once they updated the PHP version. To be clear, the compromised commits were found before they hit the production version of PHP – so do not worry if you are running PHP servers. However, it keenly illustrates the importance of security in your software delivery chain.

Indeed, both of these stories highlight the fact that source code leaks may no longer be our primary security breach concern anymore.

Security is a very complex matter. This blog's aim is not to Fix Everything™, but to focus on the risks we have in our software delivery chain and give examples on how to mitigate against them.

Securing your chain with the CIA triad

The CIA triad can be used when assessing risk. The three elements that make up the CIA triad are:

• Confidentiality –: protection and secrecy of information. This applies to software algorithms, business processes, and end-user data. Lost secrets may impact a business’ competitive advantages, cause loss of end-user confidence, or even legal actions (e.g. GDPR).

• Integrity – tamper protection of our information. This applies to both software artifacts and data, including end-user data. Lost integrity may result in our software platform being open for hackers,  thereby also causing loss of confidentiality on a wider scale.

• Availability –: the availability of the service of our software. This applies to how our infrastructure and applications are designed, and how it might stand up against malicious attacks. It also affects our processes – for example, are we using sound DevOps processes that guarantee a short time-to-recovery in case of downtime, where downtime may be caused by ourselves (e.g. software updates) or external events (e.g. disaster recovery).

Protecting the software delivery chain

To make this guide tangible, we are going to illustrate the flow from developer commit to artifact in production.

In this situation, the developer will prepare a commit and push it to the Git repository. After this, the CI/CD pipeline will trigger and deploy a new version of the artifact in production. 

In the example above, the transportation of data is secured by SSH tunnels – but a malicious attacker can still plant commits in the Git repository if they have access to the repository without getting noticed.

Adding a security layer in our software delivery chain

The way we work with Git impacts the confidentiality and integrity of our software delivery chain. Typically the following are used by developers to secure Git operations:

• An SSH key is used to 1) encrypt our connection to the Git server through an SSH tunnel and 2) to authenticate us towards the Git server and authorize our actions.
• HTTPS towards a GitWeb UI. Although the TLS protocol used in HTTPS is secure, it is not uncommon for corporate PCs to have custom root Certificate Authorities – i.e. the HTTPS encryption protecting the Git UI may not be end-to-end, but have a corporate man in the middle.

We can conclude that the confidentiality is secured if we are using the SSH protocol to talk to our Git repository. SSH tunnels provide local confidentiality, only since the scope of the tunnels are not end-to-end, but point-to-point in the chain. The integrity of our source code is not secured however, and needs a new step to be added: cryptographic signing of your commits.

GPG keys are used to sign Git commits and tags. A signed Git object certifies that the commit/tag was done by someone with access to the private signing key – and therefore ensures the integrity of each and every commit in the repository.

To get started using GPG with GitHub, create and link your GPG key to your account. You can find out how to do this by looking at  the documentation on github.com. GitLab also supports the use of GPG keys, as does Bitbucket Server/DC. Bitbucket Cloud unfortunately does not, however 

Your Git client does already support working with GPG, you can read up on how to do it on the Git client’s home page. 

After creating your signed GPG key, you can make Git sign all subsequent commits and tags automatically by associating your key in the Git configuration.

$git config --global user.signingkey 0A46826A

Git will now use your key by default to sign tags and commits if you want.

Note: HTTPS cannot be used to secure integrity. The UI allows us to do most Git operations, except signing commits since it does not have access to our private signing key.

The new secured software delivery chain

In this section we introduce the workflow for a developer who uses a CI pipeline for building container images to use in a production environment. Here, we are assuming the developer signs commits.

This workflow also illustrates the use of SSH keys for accessing Git, i.e. we see SSH tunnels between the developer and Git, and similarly between Git and the pipeline. Git signatures provide strong end-to-end integrity, which is important since loss of integrity could also result in loss of confidentiality. Git SSH keys for accessing Git only provides local confidentiality.

The CI pipeline builds container images from Git source and should validate Git signatures to establish trust between the developer and pipeline. The CI pipeline must also sign the built container image to ensure integrity between the pipeline and production environment through the container image registry (see Docker Content Trust). Therefore our CI/CD system is a trust bridge between source code and binary artifact.

In this workflow, the CI pipeline may be a weak spot, since it performs complicated operations under control of the source application using a potentially wide variety of tools. This means the CI pipeline has a large attack surface. This weakness is amplified by the fact that the CI pipeline bridges trust from the developer-signed commits to signed images validated for production. If this trust bridging is compromised – for example by tricking the CI pipeline to build from source which is not properly signed – end-to-end integrity is lost.

To mitigate this, we should use an external agent to perform a secondary verification of Git signatures. This check should be done by an agent that is not directly related or impacted by the CI system – i.e. we assume it to be very unlikely to be compromised simultaneously with the CI system.

The external signature checker architecture is illustrated below:

How to verify your commits

Michael has made a Git signature checker that can be used both in CI pipelines and the external Git signature checker. It can be found here and it's open source.

The checker needs a directory with public keys for all signing keys that should be trusted. You can export your public Git GPG key as follows, but note that you need all public keys that are considered valid keys for signing:

$ gpg --armor --export user@example.com > user-pub-key.asc

Next, run the signature checker with a Git repository and your list of trusted public keys as follows:

$ docker run --rm -v $PWD/public-keys:/public-keys:ro -v $PWD/.git:/repo:ro michaelvl/git-signature-checker

Alternatively, to use an existing GPG keyring (potentially with trust assigned to keys), use the following:

$ docker run --rm -v $HOME/.gnupg:/gnupghome:ro -v $PWD/.git:/repo:ro michaelvl/git-signature-checker --git-dir /repo --keyring /gnupghome --minimum-trust ULTIMATE

By default, the checker will validate all commits on the current revision of the given repository. Use the argument --revision-range for checking alternative references and sub-trees.

Note: at the time of writing, GitHub displays the commits of the repository as unverified. This is because the key used to sign the commits have expired. Having eternal keys is dangerous from a security perspective, but having too short-lived keys can lead to mistrust of validly signed commits.

A similar situation arises when employees leave - you should revoke their signing key, but the signatures they made in the past should remain trusted. In general, evaluate the validity of a signature based on the status of the key when the signature was made.

A note on public keys

This tool does not validate the public keys imported through the path given by --public-keys. Trust between keys and required trust level can be defined by using --keyring to use a specific GPG keyring with keys with defined trust and --minimum-trust can be used to specify the minimum key trust level required for a successful signature validation.

Obviously, strong attention to key management is a prerequisite for validating Git integrity with signatures.

Caveats with GitHub merges through the Web UI

If you use the GitHub Web UI to merge PRs, your merges will be signed with a GitHub public key, not your own key. This breaks end-to-end trust, because we cannot know who accessed the UI –possibly someone using a leaked username/password. Therefore, you should always use 2FA on the Web UI. 

You may consider merging PRs yourself. If so, you will need to include the GitHub public key in the list of trusted signing keys. The key can be found here:

curl https://github.com/web-flow.gpg

This issue also applies to the other Git vendors mentioned in the “Adding a security layer in our software delivery chain” section.

Note: If you import the keys of the server as trusted, you will no longer be able to regard your Git repository server as uncompromisable. The traceability for merging is lost, and added commits through the UI of the server will be regarded as trusted.

Allowing only fast-forward merges could give you the best of both worlds, as the Git server will not alternate or add any commits. This requires that you are keeping your PRs up to date with the master all the time.

Best practice

We recommend using a hardware token to store your GPG certificates. In this way, you can mitigate against your key being stolen, even if your computer gets compromised. One such key is the YubiKey.

Keys stored on YubiKey are non-exportable (as opposed to file-based keys that are stored on disk) and are convenient for everyday use. Instead of having to remember and enter passphrases to unlock SSH/GPG keys, YubiKey needs only a physical touch after being unlocked with a PIN. All signing and encryption operations happen on the card, rather than in OS memory.

For a more in-depth guide, we found the descriptions in this GitHub repo to be well written. Please read this with care, as we have not done a security audit on the information provided.

Final notes

If you are running GitOps and Kubernetes, there are several ways to integrate integrity checking in the infrastructure.

For example, if the SRE signs all commits, the GitOps agent can verify Git commit signatures before applying changes to Kubernetes. This provides end-to-end integrity in the delivery of infrastructure changes. 

To validate images, we could use reproducible builds. However, this is rather complicated in practice and not without its own problems. For example, the pipeline doing the secondary build must not be susceptible to the same attacks as the primary pipeline.

Secure traceability in your software delivery chain

When breaches occur, we tend to focus on the theft of intellectual property in the form of source code. But we instead need to shift our focus over to securing traceability in our software delivery chain to avoid attacks like the one targeted at SolarWinds.

Signing and verifying each commit mitigates the risk of a malicious attacker inserting exploits in your software without you knowing it, thereby compromising your software integrity.

Doing this right requires new tools and the belief that everyone is knowledgeable around the use of said tools. The alternative has shown itself to be devastating to a point where we must ask ourselves if we are willing to take that risk.