Innovation, collaboration, and DevOps

Patrick (00:06): I've been intrigued by the whole GenAI thing. It's been a spark again that I missed for a while in the industry. Also, kind of the metaverse, and I know that got a bad rep.

Marc (00:21): This season, Andy and Marc are back with a fantastic group of guests.

Andy (00:26): I've been to depths that remain classified. And Marc keeps his head in the clouds. With our combined experience in the industry, we can go from the bare metal to the boardroom. Enjoy your time in the DevOps Sauna.

Marc (00:46): Hello, we are back in the DevOps Sauna. This is kind of a couple of things. One is that it's a little bit of a dream come true to me to get to interview our guest today. Somebody I've been a fan of for a long time. And this is also a bit of a post-game for The DEVOPS Conference Scandinavia that we had a little around a month ago. I have my usual cohort in the sauna today, Andy Allred.

Andy (01:11): Hello, hello. Happy to be here as always, especially for this one.

Marc (01:15): Especially for this one. And the man himself, the father of DevOps perhaps. I know you get tired of hearing that, Patrick, but Mr. Patrick Debois is in the sauna today. 

Patrick (01:25): Good morning, Marc, Andy, pleasure to be here, especially in the sauna with this weather. 

Marc (01:30): Yes, indeed. Patrick, I've carried around the thought of your most famous quote through much of my DevOps career anyway, can you guess the quote that I'm referring to?

Patrick (01:43): Well, the one that seems very popular is that I would say DevOps is all about breaking the silos and all the rest is engineering. 

Marc (01:50): Absolutely. I've heard you go into this a little bit in some other forums, but would you like to, for us, say, looking back at that it's a little bit longer in the teeth now this quote, but looking back, can you reflect on this a little bit? Because this is still something that we face every day talking to customers that are still in the waterfall and command and control kind of paradigms.

Patrick (02:13): Sure. It took me a while to get that quote. So there's quite some history in there. Happy to elaborate. I think, though, by accident 2009 calling the word DevOps by organizing DevOps days. We were at a setting where developers and operations were quite separate in companies. And we were having a fight mostly when new software was delivered. We call it throw it over the wall. And that kind of probably still exists today in certain companies as well. And kind of that is the first silo of Dev and Ops or to do silos, I would say. Over the years, things have evolved both in the company style of cultural organizations. We got from those two big groups working together to smaller groups, the pizza teams that I often referred to and they're now collaborating with it in a different way. They're not the big silos, but it's a fragmented way of working together. This has brought us flexibility and speed, but it still created other friction, as well. If you have many teams, now, we have a bigger problem of nobody really knows what's happening anymore. And there's a new kind of friction coming up. So this is just an example of from the old days to the new days, how it has shifted. The second thing that has happened, not just cultural order structure in the companies from Dev and Ops, is that we brought more people along the journey. Most famous one is probably the security folks DevSecOps. They were in alliance as well, the auditors sometimes referred to them as naysayers because they were protecting for the good as ops was protecting for the good, the production and the health of the company. But we got in the recent years to a better collaboration. And then I guess many people over the years have come up with acronyms. I've seen front end ops, I've seen test stops, I've seen QA ops, and everything could be included, as something with ops. So there are many groups now collaborating. Personal journey. For a while, I had a startup and that made me realize that there's even a bigger system at play that is not just technology. And there's the part that there's a friction maybe between HR and the engineering. They're not hiring the right people or engineering doesn't express well what they need. There's a problem maybe with sales and marketing and engineering. They want to have things to sell. And engineers want to make the things that more be rigorous. So there's always a friction between getting more features, making more stability. So this made me see a whole system of different groups playing out in a company having friction in some form, and the best way to deal with it is collaboration. And sometimes automation can help. And that kind of works. Now, I would say there's even a third one, which if you go in a broader picture, this until now was all within one company. But more and more, we're working together with SAS vendors, and they're outside of our control of a company. And those are other silos. We can't control them. It's by design because they have their own agenda, they have their own direction, they have their own business to run. So we started looking more for partnerships. What is a good partner that can help us on our journey, and reduce that friction? So it wasn't any more about sitting together next to each other kumbaya, holding hands and kind of getting closer. It is just how you seek that collaboration across. And that could be also your suppliers, how you deal with contracts that are more flexible, and more agile. So this led me to well, if we just keep sticking it to Dev and Ops only this is the narrow definition if we brought it up, it is all about reducing friction in some form or the other. And one of the reasons why that's important to keep hammering on this is that over the years, things have been rebranded. The sysadmin became DevOps, the DevOps became the SRE, the SRE became the platform engineer now, and they're going, but that's all good on the technical side. They have been progressing, better monitoring, better practices, but a platform engineering I think, is not done right if you don't think about having customers and helping your customers, and helping reduce the friction to deliver the value to the customers. If you're just in there for the technology, if you're just looking for performance tuning, and that's your thing, that's absolutely fine, but that is engineering. So that's where I drew the line between solving the friction, it could be procedural, organizational, technical, to overcome a problem that was created by stretching in your organization. But if it's just technical, then by all means, keep doing the technical stuff as well to help there.

Marc (07:03): There's a lot of cool stuff there. I'm glad you also gave us a new definition, I think, that we can work with, but there's a few different things I'd like to touch on. The first one is when you put the SEC in between the Dev and Ops, and you mentioned the auditors and things. Is there a difference between passing an audit and being compliant?

Patrick (07:25): Definitely yes. And we all know that, right?

Andy (07:28): We don't always admit that, but we should know it.

Patrick (07:32): Yeah, well, it's like telling ourselves fairy tales, in a way. Well, I think procedures, certifications, and whether that is in auditing or professionally, they'll help you become better at certain things, it is not the control that will help you do the better things, it just gives you an aspiration of things you can do better. So if you're just there to check off the box, yes, you can do this. But you'll probably have yourself one way or the other that you're saying like we're compliant. Shit, we have an incident, a big incident. It doesn't work like that. And it's something I've seen many times. I often hear of people doing more agile security, more agile DevSecOps is that they're bringing more proof to the table in an automated way. In a way that they can talk to the auditors with a trail that actually makes sense. And it's not just a paper and a facade. So that kind of has been a shift. Does that mean you're completely compliant? I think even the auditors in the old way of working, they knew it wasn't about that. It was all about improvement. So they will give you an evaluation. And they say, well, we know it's not perfect, but these are the things you need to work on. And that's how you have to see this as the perfect improvement. I'm probably not talking about health and other impactful security aspects. Because sometimes you just have to have it, there's no negotiation, it is part of the game, it is part of being a good citizen. But other things are often very aspirational and for us to get better.

Marc (09:11): Cool. There's another thing that you mentioned, which is something that I'm a little bit curious about right now. So one of our big trends that's going on now and in the near future is this move to the big platforms, GitHub, GitLab and all this SaaS everywhere with a monthly subscription and bandwidth limitations. And all of these types of things. And then there's this, I don't know if it's a joke, or some trolling or something, but there's also seems to be a movement towards cloud repatriation coming out of the cloud and back to the on prem, and all of this. Do you have any perspective on this? Or do you have an opinion anyway because when I look at the lack of flexibility that we get with a lot of the SaaS vendors than companies that understand this can have less friction because they go with the flow of how that SaaS vendor works. Companies that insist upon having corner cases or customizations or things like that oftentimes encounter a tremendous amount of friction, and how they work with their SaaS vendors. Would you like to put an opinion here, Patrick?

Patrick (10:20): What actually helped me when serverless first came up as this way of like, I'm going to delegate all my stuff, I don't own the servers anymore. Even the next leap of cloud as a vendor. What helped me there was a concept called Promise Theory by Mark Burgess. Everybody that we're working in make promises, but the one thing they can't do is make promises on behalf of others, you have to understand that they're going to do their best, but they can't over promise. And that's kind of the danger that some of those SAS vendors are doing anyway. And then Promise Theory also says, if your partner that makes promises to you is becoming bigger, they will be slower, and their direction often will change in a different pace. So if that direction doesn't suit you, it becomes a problem because you have no control, like I mentioned before, they're outside of your control. And the way that I think about this is that while we had continuous integration, continuous delivery with a SaaS in a SaaS world, you have to think about continuous rearchitecting, and swapping things out and in as much as you can. On the flip side, I do believe that sometimes a bigger partner does make sense because they have an ecosystem parts that are well integrated, they take care of certain pains there as well. And it also reduces costs on certain aspects. If you're not have to go multi-cloud, I would not recommend that too anybody if you don't really have to. And then the move back to the return of the on prem, there’s cases to be made where maybe some of the clouds are not the perfect use case, whether that is something for performance reasons like robotics, or video streaming, or things like that. There is a case definitely where the one size doesn't fit all. But it is expensive. But running it internally is also expensive. And will you count in innovation, engineering time, debugging time, keeping it up to date also in the cost, and then make the fair comparison? I think the jury's still out on that one.

Andy (12:35): Working as a consultant, the right answer is always it depends. And the trick is knowing it depends on what. And I think with this repatriation kind of theme, if you have a whole bunch of people in your organization who have as much gray as the three of us and have done things as long as we have done, then, hey, we need to customize our kernel extension. So we can blah, blah, blah, whatever. And hey, let's just build a new version of the operating system and still include some more stuff. And we think about that without- and it's just like, yeah, okay, do you need it tomorrow? Or is should we have it done today? No big deal, let's go ahead. But then you get people who don't have this vast experience. And it's not always as fast and as easy to do these kinds of customizations that you would need. And it becomes a huge expense to do almost any customizations on a server or whatnot. So in that case, definitely go with a cloud, just take their API say, how do I use this? How do I get it. And use them for what they're good at. So I think a lot of these discussions are missing the, what's the base experience of your team? And how easy is it to use that and capitalize that experience that you have? And where do you get the value from that?

Patrick (13:53): Yeah, ease of use and commodity. So if you can probably optimize a lot of things, but yeah, who knows? Who understands? Who can change it? That's also part of ease of use that people often think it's in the UI, but it's also definitely in engineering part as well.

Marc (14:10): I'm curious, Patrick, when you kind of look around, like at the conference space and what's going on and technical oriented social media and forums and things. People like you and Andy, I'll put you in a similar class, you have some really interesting experience and stories to tell based upon how you have built up things as a human and helped enable others, reducing friction, and understanding how different contexts can be applied for technology, and like Andy does it with submarines and technology, development and things. How do you see the young people coming up and how they are going to be able to feel like they've made a difference like you have? Or what does the next kind of hero look like in the technology space or something? Do you have any thoughts here?

Patrick (15:05): Well, it's one of the questions you asked me. And it's funny enough, asked by people like us having gray hair, how do these people will learn what we've learned over the years? And I usually give them the example of when my kids were young, they wanted to have a Minecraft server. And I said, this is interesting. I can work with my son, getting to know Linux and kind of play and set this up, but the next day at breakfast, he said to me, yeah, dad. I get it. Here's a Linux server, and I could all do those things, but I just wanted do the gaming. If I just pay $10, then I can game. And he made probably the wisest choice by saying like, yeah, he wanted to have the value. He looked for what was cost effective at that time to have that value. And that really helped him and probably I would have made the technical decision of, this is fun to play with. And in the end, I would never have played Minecraft in probably a year because it would have taken me to set up. The flip side is always when he had a problem with Minecraft, he didn't understand what was going on. So he looked around all the time on the forums, and then you have to have that knowledge. But there's a different way of looking at it. Do you need to have the knowledge? Or do you just need a good partner or something that can help you if you have a problem? I think that is a mentality shift between having to do everything ourselves. And if you're at the firm forefront, often things don't exist yet. And you're, let's build this together. I've lived through the pre-Kubernetes era, I've built all the pieces together, is that still relevant? Yes. Because I understand the architecture. Would I need it to run it? No, but for debugging, it really helps. So there's a paper out there, like the irony of automation, like the more you automate, the more you abstract, the less you understand. And I think that's also the reason why chaos engineering came back as well. We're going to do this in a controlled way. So what we're actually doing is training for when it fails. And that is the reverse flip side of us still learning, but from the perspective of on dealing with the things that go wrong, but yeah, I didn't have to know everything from a microprocessor anymore. I just did that abstraction; they will take another abstraction. And if you look at Wadley maps, things will become commodity, we don't have to know, we have people taking care of this. It is part of the fabric, electricity, consumer things. And that's how they will probably see the world. For them, things are really a given. And they're going for the next leap. Where that is going? Who knows. But I think that also sparks innovation as much as we've experienced it over our lives. 

Andy (17:51): And I guess another difference is that quite often when I look at some SAS service, I know exactly. I spent a couple of decades in operations. So I look at the SAS service, and I know how fraught this is going to be with different errors that could pop up. And I'm like, okay, I can't trust this, I have to make sure I have my checks in place, I need to make sure that I'm doing my part to ensure that everything is lining up and blah, blah, blah. But then I talked with a younger colleague who doesn't have the horror stories that I've lived through. And they're like, yeah, but there's an SLA there. So let's not worry about it. Let's just trust them to do what they do. So part of it is a level of trust and the level of okay, that's their remit to take care of it. And we just need to let them do it. And yeah, I know a bit about it. But still, I'm not the one working on it. So let them do what they do. 

Patrick (18:45): But you did exactly the same. You're saying use it, but prepare for failure. Right? And that prepare for failure is look at the scenarios. I have this discussion quite often. We made all the backups of our environment. Okay, did you backup the databases? Yes. Yeah, of course, we backed up the databases. Well, did you backup your fresh control config somewhere? Well, okay, some we do manual, some we not. So, okay, we could use infrastructure as code. Did you backup all the emails that we’re getting, all our documentation that we got? So you just keep going on those scenarios, like people don't maybe think about, and that's where this experience and this scar tissue comes in. But we've learned that if our computer crashes, our Word document is gone if we don't do save. Now we got autosave. Now in the cloud, it's all auto saved. Will somebody restore your account if everything gets deleted? It just takes new forms. And they learn that, too, but we also learn through the scar tissue. So you can't blame them in a way. 

Marc (19:49): When we were talking about AI recently and when you talked about GenAI in our conferences and some other things. In some areas of technology, we're talking about democratization, right? So AI used to mean that you have to have some PhDs that don't know anything or care about writing production code. And then you need to have a bunch of senior developers that can take that stuff and turn it into something that you can actually reliably and predictably create working models with. And now with GenAI, we've democratized the access to AI for different kinds of companies and use cases. But then on the other side, as things are slipping to the right on our Wadley maps, we are getting to the point where we have less and less control because we're just picking a service based upon a price point, and we have to take it or leave it. I have a question related to this, though, which is that Patrick, what's on the left side of your Wadley maps these days?

Patrick (20:42): Well, I've been intrigued by the whole GenAI thing. It's been a spark, again, that I missed for a while in the industry, also the metaverse and I know that got a bad rep. But I think I see it from a technology point of view, there's interesting stuff that is happening in the virtual world. I think what excites me and I call this sometimes as reality as code. Probably people have never heard about this, but it is something in my head where all those threads are coming together, we create virtual worlds, we create a virtual avatars, people, there's going to be some robotics that ties that all together. It's still somewhere separate, but I believe, how do you express this? Now with LLM maybe the texts was one of the ways of expressing things. But how do we deal with events? How do we kind of make a more orchestrated thing, whether that's vision, robotics, workflow, agents, stuff like that, so how that's going to leap into our lives. I think, this year, we saw a little glimpse on where these things are going. But then also how this affects our day-to-day work of collaboration with team members. Is the AI going to be a team member? It's just a fascinating thing. It feels like we're living the science fiction of many years ago, somehow. And then, obviously, there's more. But again, those are the areas that I'm dabbling in these days, which are really exciting, I think.

Andy (22:10): And at the recent Scandinavia DevOps Conference, after your talk about AI, we had a chat in the hallway, and you shared a GitHub link. And I took a look at that. And I've been playing around running some models on my laptop and just doing some LLM stuff with referencing my local documents with very mixed results. Sometimes it's amazing. And sometimes it's like, what the heck happened here? But it's just really interesting that if we just assume that this continues to develop at a reasonable pace, what could it look like in the future and where it's going. And it's really interesting, not just where it is, but what direction it's leading, what that could bring next.

Patrick (22:51): Yesterday, I read a paper where they compare the agents communicating together to get something done to like a kernel of a computer. They have a shared bus, they're communicating, they have a register, they have memory that needs to be collapsed. And it's just taking so many different forms. But I think you're right, looking back now for over a year how this has evolved. There's the easy part. Now a lot of people have learned on ChatGPT, you give it a prompt, and then you get like mixed results. What we learned is that one prompt is never enough. It is usually one prompt, and then a series of prompts to nudge it in better direction, whether that is your format you want to have, more specific, you give it examples. You kind of have to feed this better, or a series of prompts to get there. And then what we're seeing now, I believe, is that a lot of the boilerplate that we're now adding to make the responses better is getting kind of integrated in what I would call the GenAI fabric is what the providers are bringing. It is not just about bring the model, no, OpenAI has adjacent mode that takes care of JSON formatting and it not being correct, or retry more is building, if it doesn't give the right answer. So you will see this slowly move into a more steady state than the haphazard prompting that we've done. Will that be enough? I can't tell. I'm not a specialist on the data scientist level, but you see us progressing and the quality of models has vastly improved over this year, the multimodal things. It's hard to predict whether that is going to stagnate or it is just going to continue to grow. But from the more and more that I read, there is still new strategies, new improvements to be done. One of the reasons why I do this is that to me, it feels like another tool or a library that we can use as an engineer to develop things or get better at things and to understand the limits. It's also why you have to put in the work to learn it. Much as we said like Cloud has limits, this will have limits. To you say like, well, it's good for A, but let's say, checking whether the syntax is correct, let's just use a linter. It's the better tool, just don't try to shoehorn everything in that one tool. Learn how you can deal with this as well. I think let's keep on growing. And one of the reasons I talked at your event was to get those stories out of people trying and learning what they see what is working, what is not working.

Marc (25:26): Okay, fantastic, Patrick, thank you. We have two questions that we ask everybody that comes on the podcast. And this is a thought experiment on leadership. I was coaching someone who was dealing with a difficult team, and they were trying to figure out what were some tools that they could use? And then I turned it into a thought experiment to see what can I learn from everyone that comes on the program? So the thought experiment is this. Patrick, you are the leader, and I am a trusted team member. And the rest of the team has gathered around us and they are collectively complaining about an issue. And as a trusted team member, I look you in the eye and I say, Patrick, I can take care of that. What do you say?

Patrick (26:09): As Andy says it depends, but that is the wrong answer. I'll explain why I think it depends. If I see that that person has a good connection with the people in the room, and that there is a chance, then obviously, I would say go ahead. I will happily stand back. I’m what people say a reluctant leader. And it's not that I want to escape my responsibility. But I'm also kind of for others spending and giving the floor to that. If I feel and I come back maybe to the issue of the failsafe environment. If I don't think that is a safe thing to do with the group or the fact that there's been some issues in the past or it's heating up, I'll probably try to step in. And one of the things I would do is to likely make the discussion more objective, sometimes I would use a whiteboard for this, just so we're not pointing at people, but we're pointing at a situation in a way. I try to get more of the facts and how people react to the facts. So closer to post mortem, or have that discussion in that way. So set the stage correctly. But before usually I do that I would kind of try to calm down and get everybody to have in a listening mode. So we're not shouting in the void in that perspective. I don't know if that fits your thought experiment more.

Marc (27:33): It fits both questions, actually. But I'm going to try the second one anyway. It's always fun to listen to how people think and put things into context. The second question, Patrick, is how do we get the others to be more like you and I in that type of discussion?

Patrick (27:50): So it's an interesting question. Reflecting back on myself, I was driven by innovation, which leads me to have more sparks when people are not thinking the same as me. But then when you do want to deliver something or get something done, obviously how people are thinking similarly, but that brings us back to the reducing friction by understanding other points of view you get kind of like a nuance where people in the mix. But again, it is a matter of talking to each other, trying to understand the other position, it's also what led me to DevOps is me running around having different roles, different perspectives, doing that. So I find myself if everybody has a consensus, I'll try to be the dissident voice. If there's the opposite, I'll try to find consensus. So that's how I probably approach that.

Marc (28:55): I like this very much. Absolutely. Thank you so much, Patrick. It's been a privilege to get to meet you a couple of times and see you speak about your work and interview you today. And have you honor our questions with such insightful answers. Thank you very much for coming to the sauna today, Patrick.

Patrick (29:12): Well, thank you very much, and I hope everybody enjoys the next episodes of Sauna and keep up the good work there.

Andy (29:18): There's so much good stuff in what you said today. So I think everybody's going to enjoy it and thank you so much. 

Patrick (29:23): All right. Marc, Andy, and Patrick signing off from the DevOps Sauna. See you next time. 

Andy (29:28): Bye, bye. 

Marc (29:32): Before we go, let's give our guest an opportunity to introduce themselves and tell you a little bit about who we are.

Patrick (22:37): My name is Patrick Debois. Most people know me for getting Dev and Ops together. These days I tried to extend that to more also security and bring AI into the mix. I like to inspire people while I do the research and while I do my work. I currently work at Showpad as a VP engineering, but I'm also happy to inspire anybody along the journey, whether that's your team or your event or you are on the happy right to learn more in this industry.

Marc (30:03): My name is Marc Dillon. I'm a lead consultant in the transformation business at Eficode.

Andy (30:07): My name is Andy Allred and I’m doing platform engineering at Eficode. 

Marc (30:11): Thank you for listening. If you enjoyed what you heard, please like and subscribe, it means the world to us. Also check out our other interesting talks and tune in for our next episode. Take care of yourself and remember what really matters is everything we do with machines is to help humans.