A major Danish national TV station and subscription-based streaming platform changes its infrastructure and ways of working to meet the demands of a modern-day streaming service.

The challenges: Managing load spikes and minimizing risks when making infrastructure changes

Originally, with only a single person focusing on DevOps and infrastructure within the streaming platform backend team, TV 2 Denmark struggled to recruit new talent. This team, which develops the API, one of the most critical systems for the streaming platform to function, needed immediate outside help to add a broad range of new skills.

They need to be able to handle a massive immediate increase in load on the video streaming platform. When a popular TV program starts, many viewers use the platform at the same time. And when the program ends, viewers are prompted to “watch the next episode now.” So it is crucial that the platform is reliable, can handle the load, and scale rapidly.

"What you don’t want is that a minor network change, causes the whole platform to fail,” said Martin Villumsen, Staff engineer at TV 2 Denmark.

A major problem was lurking on the horizon. They had ended up with a Terraform monolith. New changes often caused side effects to their infrastructure because of complex coupling. Continuing to add code to the monolith was unsustainable and would only create more technical debt and maintenance problems in the future. So they wanted to minimize risks when making changes to the infrastructure.

The solution: A new architecture for the cloud infrastructure, and knowledge sharing

Having spoken to various consultancies, TV 2 chose Eficode because they gave the best impression and seemed to have the right personalities and approach for the work ahead.

Breaking down the Terraform monolith and improving code quality

One of the first tasks was refactoring Terraform code and setting up infrastructure in AWS. The Terraform monolith was re-architectured from the ground up into loosely coupled modules and tiers. This massively reduced the blast radius and made it much easier to introduce changes without side effects.

Eficode also introduced a tool called Atlantis as part of a newly created pull request flow. This reduced manual work and allowed developers to see the actual effects of changes before they got applied.

And to improve quality and developer experience in general, Eficode consultants introduced more automation and tests on the Jenkins pipeline. They also helped increase the quality of common Jenkins Shared Libraries.

Sharing knowledge daily and giving hands-on training

As the team at TV 2 grew, and new practices were spreading, Eficode also provided training courses in Docker, Kubernetes and Git.

To ensure daily work went smoothly, and to provide those extra resources and skills, Eficode consultants worked shoulder to shoulder with the TV 2 team, sharing their knowledge along the way. They worked on both strategic and tactical tasks. For example:

  • Whenever there was an incident, Eficode helped in debugging — finding out what went wrong and fixing it.
  • Since load patterns were a challenge to spot failures early, Eficode defined and set up a tool called Locust for distributed load testing.

“When we bring in consultants at TV 2, we want them to be part of the team. That worked well with the Eficode consultants. They are also good at proactively communicating and keeping us in the loop. Plus they also communicate and align internally, avoiding double-work and inefficiency,” says Martin Villumsen, Staff Engineer at TV 2 Denmark.

Eficode is currently helping another TV 2 team migrate to AWS, using the same architecture as the backend team.

The results: More visibility, confidence and skills for the TV 2 teams

Although there are no metrics in place to measure the exact impact of Eficode’s work, there are many clear advantages that the TV 2 teams enjoy every day.

With the help of Eficode consultants, TV 2 now has a well-defined microservice architecture for how they want to implement infrastructure. When any change has to be made to the infrastructure, they can minimize the blast radius. This successful approach has set a good example and has later been copied by other teams around the organization.

“We see fewer incidents triggered by accident when we try to deploy,” says Martin Villumsen.

TV 2 now has a whole library of test suites and quality gates, so they immediately see if they break something when making changes. This has been very valuable for the developers.

TV 2 brought in external experts to bridge their personnel and knowledge gaps. Not only did this solve the most pressing challenges, but by working side by side with the experts, their own expanding team quickly grew their knowledge in all the relevant areas.

The future: Changing ways of working by establishing an internal platform

Currently, TV 2 is fundamentally changing the way they work by building a common, in-house platform. Different teams and their workloads will be migrated to this new platform. Everybody at TV 2 is onboard — from management to the teams. They have complete “buy-in” (a luxury in many organizations) — even excitement. So the ultimate challenge is “where do we even begin?”.

Eficode is helping to build this internal platform step by step to ensure teams will actually use it. And to give the platform team clear purpose and direction, the consultants have held workshops and interviews with stakeholders from across the organization. These conversations also gave important insight into how mature each team was in areas such as cloud. The results today provide the platform team with a solid ground to move forward.