An in-depth analysis of 19 DevOps assessments

The unexamined life is not worth living. A DevOps assessment gives a company a clear view of where their DevOps adoption currently stands, and what actions they need to take, in what order, to get where they want to be. 

This analysis is based on the DevOps assessments conducted between 2016-2018 of 19 Finnish companies from the following industries: media, construction, heavy industry, software, telecommunications, electricity, medical, and healthcare/pension.

The DevOps assessments were conducted as part of Eficode's DevOps Development Plan service. 

Firstly, what is a DevOps assessment?

A DevOps assessment results in a DevOps Development Plan of prioritized action points for the company in question. An assessment is a wholesale evaluation of the company's DevOps capabilities. It analyses the tech in use, and includes a series of interviews of the development team and leadership, as well as an investigation of processes. The end result is a tailored roadmap for the teams that were analysed and top line advice for the company's management. 

In general, some of the DevOps pillars which are assessed include: 

Organization

• DevOps is all about bridging silos. In an ideal scenario, the organisational structure allows all areas of the development organisation to collaborate because teams are cross-functional and able to work autonomously.

Automation

• Automation reduces human interaction by defining everything as code. This lets the scripts and tools take over repetitive parts of software creation, while people can concentrate on creating value. Automated processes can also convert customer feedback, development metrics, quality assurance and business value data etc. into meaningful dashboards.

Processes

• Software development methods are optimised and waste is minimal. Development and deployment flow smoothly. Upgrading new versions as well as rolling back to an old one is easy.

Culture

• Experimentation encourages new ways of working and there is a low threshold for suggesting new ideas. Communication with management is open and stress free, and every mistake is seen as a learning opportunity.

Each of these categories covers a large number of practices, such as immutable server environments, automated quality assurance, taking end to end responsibility for the product and bringing down the wall of confusion between different teams, to name a few.

Some of them have been widely used in the IT-industry for decades and some didn’t exist until a handful of years ago. The best case scenario is when a practice is rooted deeply into everyday activities.

How widely were DevOps practices adopted in the 19 Finnish companies?

My research showed that even the practices that are considered to be obvious by some, like version control and accessible documentation, are not used everywhere.

An overview of the 19 DevOps assessments 

Figure 1: An overview of the analysed DevOps assessments

It's evident that there were great differences in DevOps maturity amongst the companies assessed. (See maturity scores in the last two columns). 

The most prevalent DevOps practices

The most used practices include version control (in 12 out of 19 companies) and scrum practices (in 10 out of 19 companies). Both of these practices have been a major part of modern software development for even longer than the concept of DevOps has existed (the word DevOps was coined in 2009).

Semi-popular DevOps-specific practices observed include some form of a continuous integration platform which is triggered by version control commits, and a culture of experimentation (both were noticed in roughly one third of companies).

A gap in DevOps practice: quality assurance

Automating quality assurance and building quality in through quality gates and coding conventions is a really important step. It allows for the speed and quality that DevOps promises. 

Still, I found that only a few organisations actually used these practices. Most of the testing and code validation was still manual (and thus prone to human error).

The most observations were made in the QA category 

Figure 2: Total code group counts in each code maturity model category. Suggestions, yellow: those made by Eficode during the assessment. On track, green: issues that were found to be on track during the assessment. Current, blue: this is the problem category and the on track category added together, denoting things that are currently being worked on.

Here you can see that the most observations were made in the QA category. 

But following the assessments, the least action was taken in the QA category

The graph below doesn't concern the assessments themselves. Rather, it is based off interviews conducted by me well after the assessments. It shows what aspects of the assessments were actioned. Green means the action was completed fully. 

Figure 3: Amount of suggestions implemented in each maturity model category (based on interviews conducted post-assessment after some time had passed). Fully and partially implemented suggestions are somewhat interchangeable as some interviewees argued that "nothing is ever fully ready", refusing to mark implementations as fully done.

This graph analyses the action that have been changed post-assessment. Which categories has the organization focussed on after receiving their DevOps Development Plan and Roadmap?

As you can see, despite the fact that we gave the most suggestions in the QA category, the QA category has seen the least progress out of all the areas.

This may be because other categories are easier to implement. I drew the conclusion that there were more easy win actions on the other categories, whereas the QA category included suggestions which need a lot of resources to implement. However, given how important quality is to software is across the board, one could say this conclusion is worrying.

Wholesale DevOps transformations do not consist solely of quick wins and ad-hoc solutions, but need to keep the big picture in mind. That’s the only way to reap the rewards of DevOps as a part of your organization’s day-to-day activities.

Differences between industries 

I also did a comparison between organizations in the software industry versus organizations operating in other industries.

Figure 4: The average number of problem codes per industry. The 'codes' mentioned on the y axis refer to a data analysis method called open coding/axial coding which breaks data apart and finds recurring concepts, helping to draw relationships between concepts and categories. 

Here we see software companies performing worse on the QA front than other industries. When your main industry is software, you focus on delivering as fast as possible, and that could mean QA considerations fall by the wayside. Test practices do not necessarily develop at the same pace as other factors ramping up speed.

On the other hand, in healthcare or mining industries for example, you don’t need to update your software as frequently. Products have longer lifecycles, which means you have time to develop better quality gates and practices.

We also see a discrepancy in the environments and release category. Software companies are ahead here, because in order to create stable and fast-moving software you need a solid base, which is gained through a stable environment and release cycles. If a company still feels that their software is a secondary product, then they may be making do with legacy infrastructure.

DevOps as a whole is not as common as its pieces

The DevOps assessments show that DevOps as a whole concept is not as common as its pieces. It’s unclear whether companies are aiming for a full-blown DevOps transformation which involves in-depth cultural collaboration across an organization, or choosing ad-hoc incremental changes instead.

Taking a single tool into use won’t lead to the full promise of DevOps. However, and this may be stating the obvious: when making small changes constantly, the efficiency of software development will steadily improve.

Why did companies have a DevOps assessment?

The reasons organisations had for assessing their DevOps practices varied. The most common reason given is to do with scaling up while upping speed and quality. The CTO of one of the companies spoke of how the organization needed to grow, and they wanted to make sure their current software development practices would scale.

What is it like to undergo a DevOps assessment?

“The audit left a good aftertaste and gave momentum to the team’s workflow. We gained what we were looking for, which was a viewpoint into our practices. It cemented the faith that our way of doing things was heading in the right direction.” - Product Manager, assessed company

All of the interviewed company representatives felt like the DevOps assessments were beneficial.

The goal of the assessments is to provide an honest and truthful overview of the assessed organization's situation.

“When the results came in and were reviewed, they really were on the "brutally honest" level. The results were not romanticised at all which allowed us to understand that the organisation is in early stages of good software development practices." - CTO, assessed company

In general, DevOps practices are becoming more popular and all of the 19 Finnish companies I spoke to found DevOps beneficial. It was encouraging to see that the recommended actions were being taken by the assessed companies. 

This piece is based on a master’s thesis I wrote in 2018.