Setting up Gradle build cache nodes in Azure Kubernetes Service (AKS)

As a build engineer, I encountered several challenges with Gradle build caching that demanded a simple, cost-effective alternative to Gradle Enterprise and provided me with:

•  A reduction in build times through efficient caching.

•  Ease of scaling with project growth.

•  Build consistency across environments.

•  A budget-friendly solution.

• Full control over my caching infrastructure.

Please note that the outlined implementation deployed to AKS was suited to our organization (Eficode) but may be easily adjusted to another platform.

What is a Gradle build cache?

A Gradle build cache is a powerful feature that stores and reuses the output of previous builds. It significantly reduces the time for subsequent builds, which is especially beneficial in CI/CD environments where multiple occur.

Project overview

Our project provides a robust and scalable solution for deploying Gradle build cache nodes in AKS. You can find it open-sourced here. Key components include:

• A Kubernetes StatefulSet with two replicas of the build cache node.

• A Kubernetes Service to expose the cache.

• An Ingress Controller for external access.

• An AzureFile for persistent storage.

Prerequisites

To implement this solution, you'll need:

• An Azure Kubernetes Service cluster.

• Kubernetes CLI (kubectl).

• Azure CLI (az).

• Docker CLI.

Implementation steps

We chose AzureFile for persistent volume as we found this the simplest solution for an AKS. You may wish to consider an alternative depending on the needs of your project, team, or organization.

Setting up Azure file share

We started by creating an Azure file share to provide persistent storage for our cache nodes. This involved generating a storage account, a file share, and setting up the necessary Kubernetes secrets.

Next, we deployed the StatefulSet and service using Kubernetes manifests. The StatefulSet ensured that our cache nodes had stable network identities and persistent storage.

kubectl apply -f gradle-cache-stateful-set.yamlkubectl apply -f service.yaml

Configuring Ingress

We set up an Ingress resource to manage external access to our cache nodes. This allowed our developers to interact with the cache from outside the Kubernetes cluster. In the example, we used: webapprouting.kubernetes.azure.com as the Ingress class for simplicity, as we did not have strict requirements on how to access the cache nodes, whereas you might prefer a proper HTTPS server.

To restrict/configure user access, you can provide a custom configuration to «config-dir»/config.yaml at the node, but this comes with some challenges.

We wanted a preconfigured user access from the beginning with the config.yaml file set as a secret. If you want to mount a secret or a config map as the build cache node config file, it must be writable.

It is possible to mount secrets or config maps as read-write, but this requires disabling a feature gate and, as such, will be global, which is inadvisable.

For salted hashed password generation, we used this command:

docker run --interactive --tty gradle/build-cache-node:19.0 hashFor this config, we made a Kubernetes secret:

Accessing the build cache node

For accessing the cache server, find the external IP of your Ingress:

Then, you can simply navigate to it from your browser: http://<external-ip>:5071

Configure your Gradle project with the cache node

Alternative storage solutions

While we chose Azure file share for our implementation, there are a number of alternative storage solutions that might be suitable for different environments and or requirements. For example, Amazon Elastic File System if your team uses AWS and Google Cloud Filestore for GKE as well as many cloud provider agnostic options like Rook, Longhorn, OpenEBS, and so on.

When choosing an alternative, consider the following factors to suit your cloud provider or on-premise infrastructure:

• Performance requirements.

• Scalability needs.

• Data replication and backup features.

• Cost.

• Ease of management and integration with existing tools.

For our project, Azure file share provided the right balance of simplicity, performance, and integration with our Azure-based infrastructure. Implementing a Gradle build cache node in AKS significantly improved our build times and overall CI/CD efficiency. However, each solution has its strengths, and the best choice will ultimately depend on your specific use case and environment.

By leveraging Kubernetes StatefulSets and Azure's robust cloud infrastructure, we were able to create a scalable, persistent, and secure solution for our build-cache needs.

I hope this blog post serves as a useful reference for your team to optimize Gradle builds in a Kubernetes environment. Feel free to check out the full implementation details in our GitHub repository.