Kubernetes on Windows

As of Kubernetes 1.5, Windows containers support was introduced in an alpha release. With the beta version just around the corner, we put the Windows support to the test. Is it mature enough for production environments?

Container solutions, like Docker, have become a popular software development and deployment trend, that enable developers to easily ship applications “anywhere,” without worrying about environment setup. Kubernetes(aka k8s) helps us to manage the lifecycles of containerized applications by providing an orchestration framework (i.e. auto-scaling, health checking and recovery).

Another current trend in software development is microservices, where applications are split into smaller, lightweight, distributed (containerized) services. Since containers stemmed from Linux, Docker (and consequently k8s) initially supported only Linux. However, whether we like it or not, there are still many applications (legacy or new) that are Windows specific. Furthermore, in microservices, some services might need to run in a windows environment. This made Microsoft team up with Docker Inc. to introduce windows containers with the release of Windows Server 2016.

Windows containers still need to run on Windows hosts. So, if you want to utilize the power of k8s for managing container applications running partially (or fully) on Windows, then you will need to have Windows nodes in your k8s cluster. Consequently, a Windows special interest group (SIG) was formed to provide support for adding Windows minions (nodes) to a Kubernetes clusters. In Kubernetes 1.5 , Windows support was in the alpha release, with the roadmap targeting a beta release by the time Kubernetes 1.8 is out. The million dollar question, though, is whether it is mature enough for production environments?

The alpha release comes with some limitations, as stated in its announcement. First, Windows does not support network namespaces, which means containers in the same pod cannot communicate over localhost. This restricts the number of containers per pod to only one. Additionally, DNS capabilities were not fully implemented, and UDP was not supported inside a container.

Testing the alpha release on AWS with Kops

We tested the alpha release of the Windows support for k8s on a cluster created by Kops on AWS. Kops makes it easy to deploy and manage a k8s cluster on AWS. Kops supports creating a Linux-based cluster (for both master and minions) which means you have to manually add Windows minions to the cluster by following the guide provided on k8s. The guide takes you through configuring the networking, building the kubelet and kube-proxy binaries for Windows, and starting both of them. Once you have the Windows minions successfully joined to your k8s cluster, you can tell the k8s scheduler to schedule specific pods (logical group of containers) to the Windows minions using NodeSelector.

The alpha release approach when used with Kops on AWS faces multiple challenges which makes it not well suited for production use:

• Manually fiddling with the cloud resources of your k8s cluster (e.g. launching Windows EC2 instances inside the cluster’s VPC) creates dependencies which your cluster management tool (e.g, Kops) is not aware of. This may affect the operations of your cluster management tool. For example, trying to delete the cluster with Kops after you have manually added Windows minions will fail due to unknown dependencies (between the AWS cloud resources) to Kops.
• The setup process suggested in the official guide is fragile and the network setup can be easily messed up. That’s not all -, you will also need to guess the Pod CIDR range that will be allocated to your Windows minion by the master before it is actually assigned. This is not very nice!
• As Windows support in k8s is relatively new, it is still going through some changes as it matures. For example, the SIG group is now adopting a new approach to add the windows minions to a k8s cluster which uses OVN for networking. Although this guide uses OVN to take care of the networking, it still involves several steps for setting up and configuring OVS/OVN (not only on the Windows minions, but also on the master and Linux minions). This renders the OVN approach unusable alongside k8s management tools such as Kops.
• Also, Windows Server 2017 (due later this year) will bring some container networking improvements which promise to improve Windows support on k8s clusters.

Verdict

As it stands, Windows support in k8s clusters is maturing but it’s not ready for production environments, especially if you want to use a cluster management tool. So what to do if you still need to have Windows minions in your Kubernetes clusters? You have two options:

1. Either, move your clusters to Azure cloud and use its recently added Azure Container Service feature allowing you to deploy Kubernetes clusters with Windows minions.
2. Or, if you just need the Windows minions to complement existing applications deployed in your k8s cluster (e.g, hosting Windows slaves for Jenkins), launch your Windows instances (VMs) outside of your k8s clusters (on a separate VPC) and let the applications deployed on your k8s cluster use these Windows instances. We will demonstrate this approach in an upcoming post where we deploy Jenkins master on a k8s cluster with Windows slaves connecting to the master from another VPC.

The past year has seen rapid developments in the evolution of Windows support around Docker and k8s, and Microsoft (and its partners) seem determined to take it further. We shall keep an eye on this space over the next few months as Windows Server 2017 gets released and Windows support in k8s matures.