What broken bikes teach us about resilience and security in DevOps

Darren (0:00:07): Real security is like an onion that you have layers and layers that you just keep peeling back and then you start to cry. 

Marc (0:00:21): Welcome to DevOps Sauna Season 4, the podcast where technology meets culture and security is the bridge that connects them. We are back in the sauna. I like the idea of Löyly. Löyly is what I'm, of course, missing the pronunciation a bit, but in Finland, that is the steam of life. It is pouring water onto the hot stones on the sauna and receiving the good feeling of the steam, the Löyly, and something about the, you know, the peace and goodwill there. How's your peace and goodwill going, Darren? 

Darren (0:01:11): I don't know how to answer that. It's been a rough couple of days, but, you know, starting a weekend soon or a longish weekend with the, I mean, obviously not when this episode goes out, but we're about to have a bank holiday. So it's good to have some time to rest. How about you? Have you been enjoying the sauna over the weekend? 

Marc (0:01:32): Indeed. And this is a nice time in Finland because we have after a long and dark winter, we have a lot of sunshine and we have some warmth and outdoor activities I've started. And one of my friends called me and we were talking about something else and he said, do I know anyone that could help him remove a bolt? And I kind of laughed and thought to myself, it's like, well, it's your lucky day. So I said, what happened? And he said that he has a, he's a bicycle guy. So he rides a bicycle all over the place all the time. It's his primary mode of transportation. And he has this nice high end, I think titanium framed bicycle, and he managed to remove all of the edges that are used by the wrench in order to remove the front axle. At which point he buried a drill bit into the hole that remained, a very hard cobalt steel drill bit into the hole that remained leaving it pretty much impossible for him to imagine how to remove. So I've said, is it still rideable? He said, sure. And the problem was he had his winter tire still stuck on the front of the bicycle. So I said, well, ride it over and I'll get the bolt out. So when he arrived, sure enough, there was round where there used to be a hole to put a wrench. There was a round area with a drill bit firmly stuck down inside. So I proceeded to remove the bolt. It took about an hour and one of the lessons along the way, he fought me basically every step, he wanted to know, why don't we just drill it out? And I said, well, you had that chance already, but now you've broken a very hard steel drill bit down in the hole that you would want me to be drilling from. So what I did is I used a grinder and I basically ground off everything I could from the business end. And then I went to the other hollow end of the axle and I pounded something in and I put liquids, a wrench penetrating oil to get down into the threads on both ends. I applied heat. I applied more oil. I applied more heat. I applied the hammer and started pounding down into the hollow end, a hardened bit of steel tool. And then I applied the heat, a wrench and a hammer to tap on the wrench. And after putting all of these things into play, then pretty much on the hour, the bolt became free and we backed it out the wrong way through the front of the bicycle and we're able to install the new one. And he was on his way. 

Darren (0:04:12): So now a sensible thing to do, I guess, would be for us to try and take any kind of lesson from this inside technology, because while I'm sure there are a few people who are having trouble with their bike axles. And if you are, feel free to contact Marc. He is the guy who will get your bike axle off, but you do kind of bring up a couple of interesting ideas. It's kind of unrelated, but there's been this news going around about the Akira ransomware. And so what's been happening, it's like any ransomware in that it gets connected to systems and then it encrypts things and tries to extort money. And it comes down to redundancy because your friend was relying on one tool, the wrench, and then trying to drill. He had a couple of things. He didn't really have the redundancy to achieve a desired result. And that's what we're seeing here with ransomware, in my opinion, because ransomware is as frightening as your backup policy. And his backup policy was to get a drill bit stuck into the axle, which is, I guess, problematic. And that's the thing we're seeing. We're seeing a lot of people fall victim to Akira ransomware because their backup policies are lacking. So I feel like if you have the redundancy in place, like if you have backups stored, if you have these ready to access without issue, separated logically, separated physically, then that redundancy just allows you to laugh at ransomware and be back in business inside days instead of weeks or months or never, as it has been in some cases. 

Marc (0:05:57): So if I think about the bicycle and I think about me versus my friend, I understood the materials that were at play, the steel going into aluminum. Generally, when you're installing something like this, you need to put some lubricant on the thread so you'll be able to get them back out again. He failed to do that. He admitted later. I understood that you needed to have multiple mechanical advantages in order to be able to free something that was stuck through, you know, a singular means, his singular little wrench. So the heat, the liquid, the lever of the wrench that I had placed in and the hammer that was causing, you know, quick blows in order to have really quick, small turns on the stock bolt. So if I think in terms of security, to me, one of the things that I understand is its perimeters. So you can start with least privilege and work your way outwards. Or I tend to think in terms of, you know, have you secured the infrastructure? So if we’re talking about, you know, we did our April Fool's podcast on the servers being locked in the basement. You know, but if literally, if you know where your machines are, that's one layer. And then you can break out into perimeter after perimeter, going into the network, the server application. 

Darren (0:07:26): Yeah, basically. There's this quote I like, I'm not sure who said it first, but it says that real security is like an onion that you have layers and layers that you just keep peeling back and then you start to cry. And that's the kind of mentality behind defense in depth. So you have this, yeah, you have at the core, your deepest level, you have like your protected assets, maybe like a database or a Git repository. You have your code. Then you have the application that's used to access that, the computer that runs the application, the network that connects to that application, the physical access to that network. And then things like external policies, procedures, that kind of thing. So you really have this layer. And obviously the security controls are different with each layer. On the database level, you're looking at encryption. If you go to the application level, you're starting to look at things like firewalls. Well, actually on the application level, you're probably looking more like access control lists. And then on the computer level, you'll have firewalls, and the network level, you'll have access controls like wifi, this sort of thing. So it's all about having these multiple layers and actually one of the biggest failings in security currently, and it has been this way for 15 years, is people get to perimeter security. They'll get to like, let's say the network layer, because everyone's interacting over the internet. And they'll get to that layer and they'll secure that layer. And then they'll say, well, this is fine. This is okay. We've kept the bad people out. And I think there's a lesson to be learned in museum security and that they don't try to keep people out, if what literature I've read is to be believed. But they try and keep people in after they have made an attempt. And obviously we can't keep people in, but we have to look further inside than just this perimeter, because that's where our threats sit. That's what people haven't quite understood is that 90% of attacks, I think, are a result of user error. And most of this user error comes from inside because that's your users. They're interacting. That's that right there. So I don't know. I feel like if we're going to take something from your bike story, we take this model that looks a lot like the OSI networking model for security. 

Marc (0:09:52): I completely agree. And you give me another idea here, which is that sometimes, you know, the person that rides the bicycle is not necessarily the best person that maintains the bicycle. And sometimes also having a different approach helps as well. So when I think about security personnel in IT, I oftentimes think of two different, very different types. One, I think is that people that understand very, very explicitly all of the specifications, all of the standards, all of the models, they're very detail oriented, and they make sure that all of the boxes are checked. And, you know, audits can go very smoothly when you have a person like this in charge of your security. And then I see a different type where people that are just very, very, very curious. And I think, Darren, you remind me of this archetype, people that will absolutely not stop until they understand the problem. They understand the roots. They have found the little hole in the firewall or metaphorical firewall. They found the little hole in the application that could be exploited. And this curiosity is what drives them in order to make sure that the security is as tight as it possibly can be. 

Darren (0:11:11): Yeah, I think there's a kind of mentality in cybersecurity that, although on the kind way to put it, a hacker is just a person who is burdened by curiosity, who is just interested. And the less positive way to spin it is a hacker is just a person who spends way too much time obsessing over a system that they probably shouldn't. And that's what it boils down to. I do agree. Like, you have these two types, and I do think both types are important. I think of our, in Eficode, our security manager and how he's a very process oriented person and how easy he makes our security audits. So we have this, I say this definite divide, and yeah, it's really all about this curiosity. So I do think having different people working on the bike is a good idea. I'm not sure how metaphorical we're going to take this episode, but by your metaphor, if I had my hands on the bike, basically it would be laid out on the floor in different parts. And your friend probably wouldn't be able to ride it off, but I'd likely know how it works by the end of it. 

Marc (0:12:25): Yeah, I could think of that. Or if I put this in a bit more positive forward terms, after having assembled the bicycle, you would have noticed that the bearing on the other wheel needed some attention. 

Darren (0:12:38): Perhaps. But there is this methodology in security that testing turns destructive. So while it's entirely possible, I may have noticed that bearing, it's also possible half the spokes would have been melted. Because as soon as you said you had a like stuck axle, my initial thought was, why not use a torch to like melt it off? If the bolt is liquid, it can no longer hold its place. I feel like if I had have taken that approach, then actually you mentioned it might have been a titanium frame. So perhaps it would have been okay. But I wonder if I would have melted part of the winter wheel at least. 

Marc (0:13:19): And there was some paint involved. So rather than using a gas burning torch, I used an electric heat gun, but we're in the right idea. But there's furthering this metaphor a little bit as well. I think there's a stigma in many areas of IT and security being one that if you are a security person responsible for your organization's security, of course, you may order, you know, outside things like pen testing and things like this. But there may be a stigma against getting others to come and investigate your security where anything that anyone finds would be actually helping to strengthen the security. But do you think, Darren, there is a stigma against allowing outsiders to try to penetrate your security forces when we all know that this actually would benefit your organization's security? 

Darren (0:14:12): There is a huge stigma about it in any kind of testing, any kind of war game. If it's tabletop simulated, if it's actual testing, there's always this idea that if they find something that you have failed and it's one of the dumbest things you can do that I have done myself that everyone falls victim to, especially because security teams tend to run quite light. They don't tend to have an overinvestment. They don't tend to have a huge number of people. So you can quite often end up with gaps and then you develop this tunnel vision. And that's the important thing about bringing these kind of third parties in. So you clear this tunnel vision. You have someone who has a wider look, a wider view to be able to look at things you're doing and say, okay, here is a gap I've missed. And there's actually a kind of actual tangible thing you can do in these situations. And that is make sure you sit on the same side of the table as your auditor. If you have two auditors and two people, make sure the auditors are not sat on one side and the people being audited and not sat on the other, because that isn't an audit. That's an interrogation. What you want to do is you want one person from each party on either side of the table. This way, you were basically broken down a barrier between the auditor and the audited. And this way you're all attacking the problem from the same side. And it's the easiest way to actually get people working together. Just sit them on the same side of the desk. It's also a great life hack if you think your boss is going to yell at you. When you enter the office, sit on the same side of the table, he won't have the table between you. He'll feel more defensive and you can get away with just about anything. 

Marc (0:16:09): I like this idea. And, you know, one of the things with, you know, I've participated in quite a few audits over the years and what I have been schooled in so many different ways about, well, when you go into the audit, don't offer anything that the auditor didn't ask about. Or if the auditor asks you something, you're not prepared. Say, you know, can we come back to this later? So that you're not kind of dancing on your feet. Whereas when I am being a participant in an audit, I come in swinging and telling them everything that I can about every question that they have and kind of try in the other direction to overwhelm them with understanding and allow them to pick out the things that they think are the most important and then they typically come back and say, “Hey, you know, you have a lot of good things going on and here's a few areas where you can improve.” And sometimes those are not necessarily even things that the average person would have gotten an answer for, but because I'm more forthcoming, I tend to find more of those. And I think that this is… Yeah, go ahead. 

Darren (0:17:21): I do think if you are more forthcoming, then the results will typically be more useful. 

Marc (0:17:26): Yes. 

Darren (0:17:26): And this is the thing. People see audits as a pass/fail thing. And in some cases they are, if we're looking at things like, but I mean, even when we come to things like the ISO standard, they're not pass/fail. What happens is if you have a major deviation on, like, so say the ISO 27001 audits, you actually end up with something like a month to fix that deviation before it even gets acknowledged. So the best way to approach audits is always to come out swinging. It's just a lot of people can't because they are afraid because there's a stigma about failing audits. There's a stigma about being vulnerable, even to people who are friendly, people who are here to make us feel like they're not judging us. But I think that actually makes for a really good auditor, if they can break down that stigma and it just helps things flow a lot more easily. 

Marc (0:18:25): You know, you kind of allude to the imposter syndrome that all of us in IT generally still carry around. But one of the things that I've said, and you actually made a really nice connection for me. Over the years, I've said many times that there's two jobs that I think everyone in the world should experience. One of them is the person in the shopping mall or the supermarket that says, “Hey, would you like to try? Hey, would you like to… Hey, may I tell you about?” Because what this teaches, you know, trying to sell something on the street or in the shopping mall or the supermarket. And what this teaches is it teaches that with volume, there's a certain conversion rate that you're practically always going to get. That the person will surprise you that you didn't think is going to buy, it's probably going to be the one to buy. So keep trying and to understand that, you know, success can be measured in very, very small steps that are very far apart from one another. And then being a software developer goes the other direction where the best software developers in the world produce bugs all the time. And the best software developers actually find a lot of these in their own testing and create unit tests, module integration, whatever. They create test automation in order to prevent that happening again, and then find that something completely unexpected has happened. And there's yet another bug. So software teaches us that no matter who you are, you will be making mistakes at a pretty regular rate. And when you take these things and you put them together, you realize that the more communication that you attempt, the more that you will understand how you can succeed. And asking people for help, asking people to look at something, try to even, you know, tell them about the thing that you're working on. And on the software side, everything that we do will have mistakes. So why not understand that going forward and say, OK, it's fine to have someone else come and look at my system and help me fix the mistakes that I couldn't see? 

Darren (0:20:27): It's so crucial to learn how to fail. That's the thing, because failure, if you don't understand how to acknowledge it and how to navigate it, is the thing that will take the wind out of your sails faster than anything else. And I think there's actually probably a little bit more of this in IT, because I think it's fair to say, having once been to a set of schools, that schooling is not all that complicated. And so there's a lot of people in a lot of fields, and I will say IT specifically, who have done extremely well in primary and secondary school. And then they get to university and that's when they start learning how to fail. And in my opinion, that might even be a little bit late, because I think you need to learn how to take these things as early as possible. That may be just me, because then you end up with this kind of thing that leads to imposter syndrome, where you fail at something that you feel should be obvious. And because you haven't realized that repeated failure is the only way we get anywhere, then you have this problem where you fail and you stop. And in software development, that doesn't work. As you say, you patch one bug and you end up with 15. And in security, that doesn't work, because 95% of something like penetration testing is failing. It's like, hey, I'll test this. Oh, it didn't work. I'll test it again. Oh, it didn't work. And to be able to do that, you have to have failure kind of as a mentality saying, I know I'm going to fail. It's okay that I'm going to fail because all of the failures will teach me something. 

Marc (0:22:05): And, you know, when I bring this full circle there and the thing that kind of makes me think about perimeters, for example, you know, you peel the onion until you cry. It's like when a bad actor is determined, there's a fair chance that if they're determined enough, they will keep trying and trying and trying and trying and failing and failing and failing and failing until they succeed. But good security and risk management is about deterrence. And it's about being the one that is the hardest to hack or the hardest to penetrate and setting up good security, having audits, having good transparency towards those audits, inviting penetration testing, inviting others to try to hack their way into your system are all good faith ways in order to reduce that risk. 

Darren (0:22:55): It's actually something I've been advocating for a while. There’s this one concept that everyone thinks is terrible in security, which is security through obscurity. And that's kind of true. If obscurity is your only security control, it's a terrible idea. But if it is used alongside a security profile with multiple tools, when you have multiple mechanical advantages, then you are basically able to use it as a high pass filter. Like you move your SSH port from 22 to 1022, all of a sudden you cut down on 89% of your incoming attack traffic. Like everything is a tool. And if you overly rely on one tool, you end up stripping your bulk so that you can't get your spanner around it anymore. You have to have other means of working with security. And if you just have that one tool, everything you start to see will look like a bolt that you try and unscrew with force. And the same is true in software development. If you just have one approach, all you're going to end up doing is driving that one approach as far as it can go. It's why all the good developers are learning multiple languages. They'll learn something high and low level. It's why people do scripting as well as like actual software. It's why you want a diverse portfolio. 

Marc (0:24:25): Did you just dig at our scripting friends and colleagues by calling what they do not actual software? 

Darren (0:24:31): Well, if anyone has seen my coding efforts, they would know that I could never be confused as an actual coder, even accidentally. The best thing I do is a little bit of Python. So I'm mostly bash scripts. And I personally don't think it's actual software to be writing bash scripts. And I think a lot of my Python doesn't really go into the area of actual software, but I'm sure there'll be some people that argue against that. 

Marc (0:25:00): I think you gave an absolutely brilliant summary already, Darren, that keeping in mind multiple mechanical advantages when looking at solving problems, both in security, in software, in, you know, mechanics, I think, is a really key thing to understand. And this flexibility and curiosity of approach is something that, you know, when you can't solve the problem, if you keep working with different approaches sooner or later, you're going to find the way through. And then you'll move through that one and move on to the next one. 

Darren (0:25:36): I think that's really the only way you'll progress. That mean you demonstrate that in real life along with the bike. If you exhaust your tools, it's okay to ask someone else to help you with this. Find someone with more experience with the specific problem and push it forward in the ways that you can. 

Marc (0:25:54): I love it. One more Löyly? 

Darren (0:25:56): One more? Yeah, why not? 

Marc (0:25:59): So I was thinking in terms of testing as well, when we had this discussion and there's a testing pyramid. And I hope that some of our listeners are familiar with the testing pyramid and testing pyramid is that generally speaking, you spend the most effort or you have the greatest volume or the base of the pyramid is unit testing. And then you have the next layer is integration testing. And then the final layer on top is into in testing. And sometimes you can divide this into a further steps where you might have unit component, or you might have smoke test integration system. There might be modules in there that might be manual even, or exploratory types, but I think there's a similar metaphor here to our multiple mechanical advantages. 

Darren (0:26:58): Yeah, I would think so. It all comes down to two aspects. I would say you have compartmentalization and prioritization. So the reason we want the unit tests to be the larger chunk of testing is because they are being tested against far smaller amounts of the whole unit. Like if we look at the top of the pyramid, we have end to end testing. So we're testing the entire thing. If we just run end to end testing, obviously we have so much more to test from any kind of automation period. We want tests. And it comes down to something I feel we've discussed a lot before, which is the idea of rapid feedback. So by putting the emphasis on unit testing, what we do is we basically increase the frequency of our feedback with regards to this test. And this is all, in my opinion, the testing period is for to ensure that the frequency and the feedback that you're getting is as rapid and as useful as possible. 

Marc (0:28:03): Absolutely. And I've seen a trend lately with some of my customers going towards essentially putting everything in the fast feedback lane these days. You know, we talked a lot over the years about, you know, some release cycles used to be, you know, months or, you know, or worse sometimes. But we've often talked about, you know, fast feedback being something in the number of minutes, maybe double digit minutes is okay. And then a couple of rounds of larger feedback during the day. And then for a long time, for maybe more than, gosh, 20 years, nightly build feedback has been to me the longest acceptable testing round. If you can't test it overnight with test automation, then you're simply doing it wrong. There are always going to be ways. And I don't care if you're embed it. I don't care if you're burning CDs or DVDs and shipping those. I don't care if it's software as a service or on-prem. You should be able to run test automation on most systems that give a good release candidate capability overnight. But the trend that I've been seeing now has been towards getting a lot of more of the testing into the fast feedback lane and saying, if it takes three hours for your change to get into trunk based development, into main, into the trunk, that's okay. As long as we are putting into place the unit module type of testing, integration testing that's necessary in order to validate your software and have a reasonable release candidate a couple of times a day, every day. 

Darren (0:29:41): I think that's a sensible approach, personally. I think it does beg the question, what projects are you working on that are still burning things to CDs and DVDs? 

Marc (0:29:51): There are certain products in the world that are run in places that are not online. And the most convenient way to deliver those is still physical media. And those are still physical medias that can be found in basements worldwide. 

Darren (0:30:08): Fair enough. There are some interesting challenges and I'm thinking we should actually talk about it at some point for these kind of remote developments, especially in like industries where, well, let's talk about it another time, but I think it's quite an interesting challenge to tackle. But yeah, back on the testing subject, what do you think is being sacrificed if we are moving everything more towards a fast feedback bridge? 

Marc (0:30:37): The fast feedback itself. One way that I look at this, I think trunk based development from a cognitive load perspective can be pretty hard to beat in a lot of ways. It helps people understand that they really need to break their changes down into smaller atomic changes. And then, you know, the counter force to this is getting development teams to solve large and complicated problems in novel and innovative ways. So there's a rock and a hard place or some, some very different driving forces here. But I still like the idea that as a developer, I can get feedback in a number of minutes on my change. And when I see this in the context of low level software, middleware and application level software, or let's even say API and application, it starts to get a little bit more interesting because, you know, you change the API and one application works and the other application does not. And the application that does not work gets the bug for the changed API. And sometimes traditionally that can be some time down the line before that type of bug would be found. So getting fast feedback that allows the developer to understand that their change works in a certain context. And then still during that same day, being able to get the feedback that, okay, all of the applications that depend on my API are still working three hours later. I have a reasonable chance of recovering that cognitive context or the state of mind that I had when I made that change compared to some time ago where it could be tomorrow or some days from now. But still this fast 15 minutes, does the smoke stay in the device when my software change runs? It has been something that to me is still valid. So if I were doing this today, of course I would support the development community as long as it survives under an overnight situation. But I like the idea of being able to get something very, very quickly into a production test state.

Darren (0:32:47): It comes back to something you said during the episode we talked about stress that obtaining and maintaining a flow state, a state where productivity is actually in flow, where creativity is in flow. And yeah, I think long build times, long testing times has been probably the largest killer of such flow states for developers.

 Marc (0:33:10): Yep. And bringing it back to, there's certainly an advantage towards getting the state or getting the result of your software while you still have that cognitive state going. So what have we learned today, Darren? So when your bike axle is stuck? 

Darren (0:33:29): Yeah, yeah. You pretty much just call Marc. He'll sort it out for you. 

Marc (0:33:33): All right. 

Darren (0:33:34): And if not, I have a torch to melt the axle. It should be fine. 

Marc (0:33:37): Excellent. So Akira ransomware? 

Darren (0:33:41): The Akira ransomware. Yeah. So basically have robust backups in place because ransomware stops being scary when you can restore. 

Marc (0:33:51): Akira ransomware. Okay. Redundancy. 

Darren (0:33:56): Always good. Relying too heavily on one tool is problematic and will not get you the range of results you need. 

Marc (0:34:04): Perimeter defense. 

Darren (0:34:06): Is a poor thing to be doing in 2024, really. Most of your threats are going to come from inside your perimeter, just because mistakes can happen and people are human. 

Marc (0:34:18): Excellent. Testing. 

Darren (0:34:20): Make sure it's done. Try and get your feedback as rapidly as possible to stay in that flow state and break it down into a small chunks as you can. 

Marc (0:34:30): All right. And mechanical advantages. 

Darren (0:34:34): Maintain as many as you possibly can, because if you don't have a spanner, an angle grinder will be your best friend. 

Marc (0:34:40): Brilliant. All right. Thank you for this gregarious conversation, Darren. 

Darren (0:34:46): It was fun as always, Marc. 

Marc (0:34:48): It certainly was. And hey, we'll see you next time back in the sauna. Thank you. And goodbye. We'll now tell you a little bit about who we are. Hi, I'm Marc Dillon, lead consultant at Eficode in the advisory and coaching team, and I specialize in enterprise transformations. 

Darren (0:35:08): Hey, I'm Darren Richardson, security architect at Eficode, and I work to ensure the security of our managed services offerings. 

Marc (0:35:15): If you like what you hear, please like, rate, and subscribe on your favorite podcast platform. It means the world to us.