Repatriation in DevOps

Darren (00:06): You don't take the weekend off from taking care of your cat, do you? So why would you leave your server alone for the whole weekend and just expect things to be running fine Monday morning when you get back to work?

Marc (00:21): Welcome to DevOps Sauna Season 4, the podcast where technology meets culture and security is the bridge that connects them. Welcome back to the DevOps Sauna. Hello, Darren. How are you today?

Darren (00:43): Hey, Marc. I'm doing pretty well.

Marc (00:45): I think that we need to talk about something that has been brewing for quite some time and we haven't really brought to our listeners' attention yet, but that is a word called repatriation. What does this mean to you, Darren, in the DevOps context?

Darren (01:03): Yeah, repatriation is kind of a revisit to our roots. So a movement that in time where we go to things like physical bare metal servers and when we start talking about things like manual deployments over automation, where we really bring it back to the roots of development and operations and kind of spreading those out.

Marc (01:26): And I think the important thing to understand here is that computing has been around for more than 70 years now. And most of the time that we have spent in computing and computer science, building applications, doing deployments, it's been bare metal with manual deployment.

Darren (01:48): Yeah, yeah. And this shift over to these kind of automated architectures, these cloud services, they're starting to feel like they've been kind of a swing and a miss where they're not quite serving people in the way that they expect them to. So, for example, if we start looking at it from my perspective, obviously, we start looking at security, and we're starting to see these restrictions where we start to need to control everything. And for that, the cloud makes things extremely difficult. So yeah, if we talk about compliance, then what we have is this kind of painful situation where we have to waste time getting these data processing agreements. We have to manage GDPR requirements, and all these things just take time away from what we should be doing, which is obviously focusing on development.

Marc (02:38): Absolutely. And this control aspect, I mean, you know, can you have more control than locking something up in your basement? Or you know, you look at a bank vault, it's like, you know, where the money is, is in the bank. And it's so much, you look at so much fraud in the world and online crime and you think, but you know, how often do you hear about, you know, physical bank robberies anymore? They're almost non-existent.

Darren (03:05): That's why you end up with currencies, for example, like the US dollar backed by all the gold in Fort Knox. That's like a sense of security added to these things that they can't just keep creating them. So we have this kind of this security that's added by creating these vaults or these locking spaces or keeping everything as tightly controlled as we possibly can.

Marc (03:27): You know, pulling all the gold out of Fort Knox and just flying it all over the place doesn't really sound like a smart idea to me.

Darren (03:34): Yeah, I fully agree.

Marc (03:36): And when we think of deployments and we think of all of the things that can go wrong with automated deployment and, you know, all the tremendous amount of trust that we place in, you know, getting software onto a server, understanding that it works and then moving on to the next one and verifying that that works, you know, I think that there's an awful lot to be said about the practices of, you know, before we had intranet, we had sneakernet and you know, we would literally burn things onto a CD and sometimes it was rolling it onto a tape. And, you know, like if we think about, you know, Linux tools, we still are using tape archiver store Tar.

Darren (04:19): There's this kind of idea that what's happening is as you go to any specific server that you're working with, you'll go through a number of hops over the network devices. And each of these is a hop controlled by some device that's just out of your purview. And encryption will take us so far, but with the dawn of quantum computing, it's entirely possible we'll start seeing breaches in TLS 1.3 as we go forward, it's going to be in such a case that the encryption standard is lacking. And then you just have all these uncontrolled spaces. So yeah, I think what we're going to see is maybe CDs, but then you kind of have like an immutable thing. So we might end up on 3.5 inch floppy disks, and you know, then you have the weight of your software. That's always a satisfying feeling. Just have a stack of 26 floppy disks that contain your days coding and actually physically inserting them into the drive one by one. Because there's no sense of tactile feel when it comes to SSH, when it comes to automated deployment. And it's always more satisfying to be switching those disks out.

Marc (05:28): Especially, you know, over some silly synthetic haptics or something that we could be putting into place. But I mean, after all, Darren, is your software not at least worth its weight in gold?

Darren (05:39): Yep. It should be worth its weight in gold or at least in magnetic storage.

Marc (05:44): At least. And I think about, you know, all of the places that there is no cloud, ladies and gentlemen, those who have not yet decided, our listeners, there is no cloud. It's somebody else's computer. So essentially you're taking the most valuable and important keystones of your business and paying to give them a way to run on someone else's computer.

Darren (06:05): And think of all the advantages you lose in that situation. I'm sure you're as familiar with, for example, server rooms as I am.

Marc (06:12): Absolutely.

Darren (06:13): And server rooms were just such a great place to be spending your time.

Marc (06:19): Yeah, I absolutely loved it. And, you know, coming, I haven't been in Finland my whole life, I came from much warmer climates and, you know, the coolest place to be would always be in the server room.

Darren (06:31): Yeah. I mean, the days get kind of warm here in Finland and just hanging out in those server rooms, that was definitely the best way to kill time. And also the prospect, because obviously we all have to deal with supervisors and we all have to hide from them from time to time.

Marc (06:46): Absolutely.

Darren (06:47): And you could often make it so that the supervisor didn't even have a key to the server room, which just gave you a nice little sanctuary where you could hide. For example, if you didn't want to do your job if you were behind on several deadlines, then you could just kind of chill in the server room and let yourself relax. You could let the stress out, enjoy a cold drink on a warm day.

Marc (07:08): You know, and many things are not taken into account today. For example, are you maintaining the temperature of the servers at the right temperature for the software that you're running? You know, Linux is certainly cooler than any Windows-based stuff.

Darren (07:23): Well, I certainly think that way, but let’s see who’d agree with me, but I don't know if we can even start talking about Apple and how cool-

Marc (07:30): Oh, don't get me started.

Darren (07:32): Those were, yeah.

Marc (07:33): Don't get me started on how cold that stuff is. But how can you trust in, you know, in a huge data center of somebody else's computers, your cool Linux software might be sitting right next to somebody that's running, you know, some hot ass Windows stuff. And I just can't even imagine how that must feel.

Darren (07:54): Yeah, you have to be able to very much dial in the thermostat yourself. If you don't have granular control over the temperature of your server, and if you can't twist that controller, how do you know your servers are being taken care of in the best way? You need to be able to see, you need to be able to measure because we're all about measurability and metrics here.

Marc (08:14): Absolutely.

Darren (08:15): So you need up-to-date information on the temperature of your server room from minute to minute.

Marc (08:22): Absolutely. From minute to minute. And speaking of measuring, you know, the miles and miles of cables in your server room. And if you think about, you know, a cloud-based equivalent, like routing tables and whitelisting and things like this, it's like, you can barely even see that on the screen, the characters are so small. But in a physical server room, you can just go and you can take a cable. And if you jiggle it a little bit back and forth, you can tell where it's going and you can trace it over to the next bit and to the next bit. You know, if you unplug it, you can pull it out and you can measure the exact length. It has stamp on it, you know, what category it is. I mean, we went really quickly from Cat5 to Cat6 and, you know, we don't really talk very much about seven or eight and nine. We're missing out, you know, because of all of this cloud-based work, we're not even continuing the development onto Cat9.

Darren (09:19): Yeah, I have to agree about all of the categories, really. I mean, I don't think we really gave category five a chance. What we've been doing is we've been increasing the speed instead of increasing the efficiency of our programs and our transmission.

Marc (09:33): Absolutely.

Darren (09:34): So what all we're doing is kind of creating a network of bloatware instead of using the minimum bandwidth. And I challenge anyone who really says they need gigabit speeds. I think a hundred megabits is just fine for 99% of use cases.

Marc (09:49): Absolutely. And, you know, Darren, it's like another thing that we're missing out on here is you can tell a lot about a person about how they handle a wire.

Darren (09:58): Definitely. I mean, just looking at pictures of nicely cabled racks on the internet is a pastime for a lot of nerds. And it's just so satisfying to see this kind of cabling done and admire the professionalism that's gone into it.

Marc (10:12): Absolutely.

Darren (10:13): Whereas when you're working with something like the cloud, professionalism is often invisible. It just kind of disappears behind configurations. So no one's printing out their cloud configurations and showing them to the world. But cabling, everyone likes to see good cabling.

Marc (10:29): Yes, I completely agree. And, you know, if we continue, I've been a fan of monoliths as long as I've been alive. You know, Arthur C. Clark, you know, science fiction always predicts the future. When we look at, you know, the works of the thirties, forties and fifties, you know, that's the world that we're living in now. And, you know, we've known Arthur C. Clark's work, 2001, Space Odyssey. We've known the monoliths are the future. Why are we in such a hurry to get rid of them?

Darren (10:57): Yeah, I don't know. I mean, just look at the pyramids. Monoliths are fun. Everyone likes building something that's huge. Like when I was playing with Lego as a kid, half the fun was just building the largest possible tower you could before it collapsed. And then, when it collapsed, you rebuilt it to see if you could go any higher.

Marc (11:17): Of course.

Darren (11:18): So we have this situation where I feel like monoliths have been kind of given a bad reputation. Sure, they're huge and often unwieldy, and they don't scale well. But I just bring it back to the pyramids. Look at the examples we have of longstanding monoliths. If you want to build something that lasts and build something that represents your dedication to that software, you should just build it as large as you can.

Marc (11:45): An obelisk, a pyramid, one by three by nine, whatever your monolith is, you know, spend time on caring for it, rebuilding it, make it taller and taller as you go along, find out where the boundaries really are for the height of your monolith.

Darren (12:02): Yeah. If we take examples from the ancient world, because I think that's the most relevant place we can look at for DevOps references.

Marc (12:09): Absolutely.

Darren (12:10): Nothing that we look at from the ancient world has these small micro relics. We look at things like Stonehenge. We look at things like the pyramids. We look at things that have lasted the test of time. We're not looking at these small items, which may have been in use or may not have been. So what we have to do is we have to acknowledge that history can teach us something here and that size actually does matter.

Marc (12:37): Absolutely. And, you know, I don't consider Stonehenge to be a distributed monolith. I think it more in terms of a monolith collective.

Darren (12:47): Yeah, yeah. It's sort of like this interconnected single monolith that has, you know, kind of internal, maybe black box system where from the outside it looks like a monolith, but it's maybe passing parts around. So yeah, I would agree with that determination.

Marc (13:03): And this is a design that beyond so many others really has proven the test of time.

Darren (13:09): Definitely. Yep. Just large rocks standing in place. They are the cornerstone of the ancient world. And there's no reason they shouldn't be the cornerstone of modern software development.

Marc (13:19): Literally.

Darren (13:20): Because when we think about it, scaling, it's kind of for suckers. We can learn from the great movie Wayne's World that if you build it, they will come. So you should just build for every user you can conceive of joining. Because if you build small, what you're doing is limiting your user base. You should just basically build for everyone. If you build for all 8 billion people on the planet, then we can just assume that at some point all 8 billion people will end there.

Marc (13:48): Absolutely. Absolutely. And speaking of that, you know, let's talk about pets. I mean, how many people really have cattle anyway? But we all have pets.

Darren (13:56): Yeah. It's awesome. Everyone has a pet, no one has cattle. And everyone knows how to take care of a pet.

Marc (14:02): Absolutely.

Darren (14:03): I'm not a farmer. I've never taken care of cattle. So we use this idea, but do any of us really understand it?

Marc (14:12): No. And I think it's so important to take care of your servers and allow them then to take care of you. Name them, make sure that they're individually getting the best care, they're getting the best software updates. It might be that a software update is not for every one of your pet servers. It could be that a few of them appreciated more, and a few of them just are not ready for that level of responsibility.

Darren (14:34): Yeah, and definitely don't trust automated processes.

Marc (14:37): No.

Darren (14:38): You need to know which updates are going on to your servers. Automated processes, they treat everything like cattle. They want the process to go everywhere. And as Marc said, your server might just not be ready for that. A software update, a security update might crash your Apache server, for example, your service will be down. So obviously in that case, it makes sense just not to push that software update. It makes sense to allow the system to run without that software update, just to ensure that your downtime is minimal. So automatic updates kind of obliterate this. Make sure you do manual processes for security updates every time.

Marc (15:18): And this is a great time for a special hot tip from Marc and Darren. The more time you spend obsessing over your servers, the more you'll know about them.

Darren (15:29): If you want to spend your evenings and weekends working, just looking at your servers, checking them over, that's the perfectly fine time to do it. I mean, think about your pets. These are pets now, not cattle. You don't take the weekend off from taking care of your cat, do you?

Marc (15:45): No.

Darren (15:46): So why would you leave your server alone for the whole weekend and just expect things to be running fine Monday morning when you get back to work?

Marc (15:53): Unfathomable. But I'd like us to talk about commuting, Darren. Now, one thing that COVID taught us is we might as well just stay at home and work from there. There's no traffic, no accidents getting in the way, just a short trip in your skivvies, and you're at work.

Darren (16:10): And that's kind of interesting in a balance, because when you think about some of your most expensive assets, they're those servers that run your software. And now that we know we should treat them like pets and not cattle, don't you think it's unfair that we keep them in the office every day, shouldn't they also have the luxury to work from home?

Marc (16:31): I mean, why would you drag your software and all of that valuable data, your most valuable company assets across all the world, and let them run on somebody else's computer when you could make them a snuggly comfy den in the basement?

Darren (16:47): Yeah, you definitely want to bring your servers home with you. It's the best way of bringing your tools as close to the end users as possible. Now, obviously, this might vary depending on where you live compared to who took the server home with them, but it's kind of better than these content distribution networks that we're seeing popping up over the cloud that claim to distribute your software everywhere, but really they're not as close to you as they could be if you just took them away in a quiet corner of your basement.

Marc (17:17): Absolutely. And saving on heating costs in the winter by snuggling with your servers, you know, we care about the environment here at Eficode, and this is something that we absolutely encourage you to do. Okay, Darren, so let's recap, repatriation in DevOps.

Darren (17:36): Yep. Move everything back to manual deployment and bare metal servers, and nothing will go wrong.

Marc (17:41): All right. What about your exercising of the control that you deserve?

Darren (17:46): Yep. Don't trust third-party suppliers, just control everything yourself and do away with all the legal DPAs and that kind of stuff.

Marc (17:55): All right. How do you recommend doing deployment?

Darren (17:58): Oh, floppy disk every time. But if you have immutable changes, you can also burn a CD.

Marc (18:04): All right. So what about the air conditioning in the server room?

Darren (18:08): Maintain it as cold as your software needs and spend as much time in there as possible.

Marc (18:13): Okay. How do we deal with routing?

Darren (18:14): Manual, physical cables and cable management. That's critical.

Marc (18:19): All right. What about monoliths?

Darren (18:22): Monoliths are fun. Everyone loves monoliths. No one builds a small sandcastle.

Marc (18:26): Excellent. Should we have pets or cattle?

Darren (18:30): Pets because none of us are farmers and we don't know what to do with cattle.

Marc (18:33): All right. And finally, should we force our software and data to commute?

Darren (18:40): Definitely not. The internet superhighway isn't in the condition it used to be. So bring it home with you, bring your servers, and let them work from home as well.

 Marc (18:48): Okay. Thank you, Darren.

Darren (18:50): Thank you, Marc. This was a pleasure, as always.

Marc (18:52): It was a pleasure as always. And we would like to say a happy 1st of April to all of our listeners today. Thank you for coming to the sauna, and we'll see you next time. We'll now tell you a little bit about who we are. Hi, I'm Marc Dillon, lead consultant at Eficode in the advisory and coaching team. And I specialize in enterprise transformations.

Darren (19:16): Hey, I'm Darren Richardson, security architect at Eficode. And I work to ensure the security of our managed services offerings.

Marc (19:23): If you like what you hear, please like, rate and subscribe on your favorite podcast platform. It means the world to us.