Kung Fu DevOps - Learning to overcome complacency

From the hallowed halls of martial arts styles (mostly Kung Fu), I'm proud to have trained with various masters - each giving me unique insight and perspective in helping me look at things from many angles.

Much like when you switch companies and see how the same thing can be done in very different ways, observing martial arts from a different perspective allows you to both deepen your own style and get a grasp of its shortcomings. Over the years, the parallels between martial arts and my work have been fun and interesting in seeing how one has benefitted from the other.

I read about a major outage in a well-known site the other day and noted what results from the Root Cause Analysis (RCA) found. Basically, they had failed to thoroughly test their basic setup, recovery and backup processes because they knew they worked - despite knowing they had the issues in the first place. It was this story that reminded me about something in martial arts that might help others understand why basic processes are so important.

Understanding the martial arts styles

There are a growing number of styles in martial arts that do not attend open-style fighting tournaments to compete with others. Some don’t even go to fighting tournaments at all. The reasons behind this can usually be put into two categories:

1. "We do not want to fight. We just like the movement and the sports."
   
   This is perfectly fine. After all, you shouldn’t have to fight if you don’t want to. These styles and clubs usually have a very good understanding of their lack of fighting skills and often know more about the philosophical side of martial arts. Thumbs up from me - keep having fun!
2. "We are too deadly to fight. We would annihilate our opponents." 

This usually comes from styles with high lethality at one point in the distant past and is rightfully mentioned to establish boundaries with training. No using deadly techniques in a free fight. But over time, these styles deteriorate into little more than martial arts-based dances, which cannot compete in a free fight with other styles.

So why even bring up these points if this blog is about connecting DevOps and Kung Fu together? Well, as the saying goes:

“Assumption is the mother of all suckiness.”

When you assume that you’re too good to check on your skills, you WILL fall behind. While the rest of the world changes daily (with others constantly adjusting their abilities to compete), the ones who assume they’re too good for the shifting tide to stay in one place.

Martial arts enthusiasts know that training is crucial to maintain skill level. When attention shifts away from honing technique and delivering hits, it's easy for fighters to forget how the basic moves like blocking are actually done correctly - which has a snowball effect since their partners never learn how best to punch either! Over time this can create an amalgamation of movements based on those centuries-old dangerous techniques, yet not effective in actual combat - simply because they've been executed incorrectly or are no longer used by other styles.

It may seem innocuous but incorrect execution allows more mistakes – what goes around truly does come back around!

Mastering the DevOps style

So now you may be thinking - how does all this relate to my work?

Regarding software, you can't assume that things will work forever without proper maintenance. Over time, the requirements and capabilities of the products and libraries you use to power your own tools will change - meaning what you built on top must change as well. If we don’t regularly stay on top of what’s happening within our sector (or keep those skills sharp), we not only do we risk the chance for system breakdowns but also miss out on new opportunities or solutions that would be beneficial for business growth. Trust nothing until it's verified.

It's easy to forget the world changes quickly, and processes that worked in the past can now cause major disruptions if not updated regularly. Even when an outage isn't caused by outdated assumptions, you're still building on shaky ground - unless your findings are up-to-date.

Like how martial artists must practice their skills regularly, a company must test their basic IT processes to see if they are still good enough. Sure, companies can’t just step into the ring and fight. But Chaos Monkeys serve the same purpose.

Remember:

• Do not assume backups and recoveries always work.
• Do not assume your load balancing works when a node fails. 
• Do not assume your Terraform can re-create your whole environment.
• Do not assume your code works just fine after a year without changes.
• Do not assume you are compliant.
• Do not assume your site is secure.

Test. Test. Test.