Platform engineering done right

Nuno (00:05): The most important thing that we are now validating is that we can onboard people and they use it and they are actually having direct conversations.

Marc (00:21): This season, Andy and Marc are back with a fantastic group of guests.

Andy (00:26): I've been to depths that remain classified. And Marc keeps his head in the clouds. With our combined experience in the industry, we can go from the bare metal to the boardroom.

Marc (00:35): In DevOps Sauna season three, we'll explore platform engineering, and the people and cultures that make it happen.

Andy (00:42): Enjoy your time in the DevOps Sauna.

Marc (00:53): Welcome back to the Sauna with Andy and Marc. We have a really special guest today. I'm super happy to have Nuno Guedes from Millennium BCP on the podcast today. Hello, Nuno.

Nuno (01:05): Hello. It's great to be here.

Marc (01:06): It's super to have you here. And as usual, my cohort, Andy Allred. 

Andy (01:12): Yep. Hi. 

Marc (01:13): Hi, Andy.

Andy (01:13): Great to be here. Nice to see Nuno. And I just saw him earlier in our daily stand up. So this is a little bit different environment, which makes it fun.

Marc (01:21): Excellent. So by daily stand up, that does imply that we are working together with Eficode and Millennium BCP and Andy and Nuno specifically here. The topic, not du jour, I think the topic of the year actually remains platform engineering. It's one of our trends for 2023. We've been working on it and talking about it for years now, but it's getting really, really critical. And Nuno, I know that you have been working on a development platform, and you're taking platform engineering quite seriously in Millennium BCP. Let's start kind of like what do you have today? Like, what are the capabilities for development, CI/CD, developer experience, whatever in your development platform today?

Nuno (02:08): Sure. So we have this slide that we typically use on board meetings that says something like eight minutes from whiteboard to production, that's the sales pitch of our internal platform. But breaking it down, what we basically offer to our product teams is an experience where as soon as you describe your application in a webpage, so to speak, and say, "Oh, I'm going to connect to this and that, and I need this infrastructure requirement or whatever," and you push a button, basically. We give you all that you need to manage that application from birth to the commissioning, all of the dependencies, all of that, we like to think it just works. And that's how we get five minutes of service level it's because things show up when they need to show up, or at least we'd like to think so.

Marc (03:06): Okay. And by all the things, we could kind of iterate a little bit for our listeners, so like security scanning, CI/CD, GitOps...

Nuno (03:15): So let me run this through in my mind. First thing is that you need to have the right configuration and Git and CI/CD pipelines. So the first thing we set up for that project is all of the repos, pipelines, all of that stuff, in this case in Azure DevOps because that's what we use mostly. But that whole set of configurations that will allow the new onboarded developer to just go to a URL and do a git-pull, all of that is done for you. Then the next thing is because you specify that you want to connect to service A or B, for instance, you want to get a balance of a bank account, or a customer detail or whatever, that wiring up is also done for you. So when you do that git-pull, there's already git there using an internal development framework that has wired up those services for you. So basically, then you need to add business logic for that. And then when you commit and push that code, it is going to trigger the few pipelines, of course CI pipelines to do code validation, the security checks, net vulnerabilities, all of that cool stuff. But then we're going to build for instance, a container with your code and push it using git ops, all of the git ops wiring is also done, push that container into development cluster that's assigned to your team. Oh, and if you have an open cluster, probably there's an automation that has set up that cluster for you, namespaces, dependencies, wiring that up to the rest of the infrastructure, firewall policies, all of that stuff that you then you need to use that cluster and work, oh, it's in a cluster. Now all you need to run tests, sure, we'll run the unit test for you. And we'll even wire up that. We will run them in Kubernetes and wire up the results for you to see in Azure DevOps as well. And we'll also automatically notify the security guys that there's new for instance, roles, because your application will certainly have roles or consumers for users. And those guys will be notified that they need to approve those roles. And for instance, SREs that help that team are notified that there's something new coming up in the vault in the development environment, and they need to start looking at that. And for the SRE to start looking at that, we need to do interoperability. So then we're going to wire up automatically as well, your workloads to data log, which is what we use for interoperability and get all of that monitors, alerts and standard dashboards up and running. And, as a good enough bundle, just from your push, and all this, you get in eight minutes now, because it minutes also includes when all of this stuff gets approved. And we need to push it to all of the environments in front of them, up to six environments that we use internally. So all of those pushes, git ops operations, and all of that are included in eight minutes. So if you're really quick, and you just spend three seconds writing your business code, it’s eight minutes.

Marc (06:41):  <laughs> Man, that's really cool. And it's beautifully presented as well, the way that you told the story. So you mentioned containers. And by the way, and have you been container based for a while? Or how did you decide to go into containers?

Nuno (06:55): Well, there are multiple things that made containers an obvious choice. One has to do with agility. So we need to have a workload that runs everywhere, it's just not a matter of saying get friends everywhere, it's actually doing it on a day to day basis, we typically use two public cloud providers and a private cloud provider. And we ensure that that workload is compatible with all of these different configurations. The side effect of that is containers are an easy, let's call it easy, way to ensure that for the application itself, all of the underlying dependencies might not be so easy to run and all of those different infrastructure configurations. But well, that's the fun of it. But besides having this universal run environments for our applications, this also helped us enforce something that's very important to us, that has to do with security and compliance, because as soon as we get get a container image, we will definitely make sure that the stuff that's inside that container image is secure, and has no known vulnerabilities, the code has passed through a lot of security checks, and so on. And with all of the, I like to think attention, that we put into having Secure Kubernetes running, it’s a good match. We take care of making sure that I'd say any CNCF certified kubernetes distribution matches with our infrastructure patterns. And so, either from the infrastructure perspective, or from the workload perspective, we try to ensure that security is forefront. And so containers are a really good source of this.

Marc (08:47): Excellent, beautifully spoken once again. So the toughest part with transformation is oftentimes culture. And even kind of tougher part is it's not just the employee and the developer experience and the kind of the lower culture, but it's the management culture, and how do you get a mandate to make a change like this? In our DevOps conference, Copenhagen, there was a presentation by a couple of ladies that did a transformation for IKEA. And afterwards, I walked up to them, and I asked them, “How did you possibly pull this off?” And they said, “Management told us to do it. So we did it.” And I'm kind of curious with Millennium BCP. Like, how did you get the mandate, not only from the management to do this kind of change, you know, there's also a developer experience and you know, there's always, you know, pets and cattle, right, some people have their pet projects and tools and things. So how did you do this from a political kind of landscape?

Nuno (09:48): Well, getting the mandates is mandatory, but that's just as the beginning of the story because honestly, even if it's not something written in a slide deck, the biggest deliverable that we can provide to the organization is supporting the right conversations, meaning that people need it to move, and are starting to move from those daily chats in the halls where I'm stuck in this process so I need you to help me fill out form 33. Oh, I need to talk with someone. Oh, everything is hard. Oh, we need to move focus, mindset, energy from those conversations into, oh, why don't we do an A/B testing with this business feature and see if our customers like it? And being able to tell that story even if it's just a soft ROI on this and has to be an interesting driver for change within the organization. Telling people 'aren't you kind of fed up of spending 80% of your day chasing internal processes' is something that will probably resonate with a lot of people. If you can match that with talking with the right people to support these mandates and saying, okay, we're going to spend a relevant amount of money so that we can move mindshare and conversations to business from IT is an interesting conversation. And it's something that we've been having more and more for the last two years have supported investing in this. But that's step one, that's okay, yes, here's the blank check, you can go and try to do it, then comes the doing it. And to be honest, I think it's a bit like, everyone has opinions on stuff until you really do it, until you experience it. So our focus, and I think Andy, for instance, feel this on a daily basis, our focus is put it out there, make sure you're adding value, get the right user story to start with and get it out there. They'll show it and get those guys that are then hands-on doing it saying, oh, this is cool, this works. So that then can do the typical, okay, if it works for them, why can’t it work for A, B, C, D, X, whatever, and push it. But honestly, we really tried to move away from big white whiteboard meetings with the board room and trying to decide full one year roadmap for whatever. No, let's pick up the parts, we know where the technical risk is, we know where the business risk is, let’s manage those. If everything fails, we know the set of people that will need to do some late-night work to keep things running. So we'll run the risk. And the rest is just having fun building stuff.

Andy (12:58): I have to admit it is a quite a bit of fun in this project. And most of the planning meetings are something like 'we need to have a Kafka functionality'. 'Okay, I'm going to investigate how to do this and this and this.' 'Well, there's this operator, just put it in use, we'll start checking it out and seeing what we need to add after that.' So we are very much focused on let's get some services running, which people can consume and use and see what functionality is needed. Instead of let's do a bunch of analysis and figure out what the hypothetical best-case service would be and define it and then start figuring out how to run it.

Nuno (13:40): And if there's one thing that we internally, we're a team that's called the Cloud Center of Excellence, and we've been trying to change many things for the last three or four years. And we just have one thing that has been standard across these three years is: change your standards. The only thing we need to know is how to manage change. So yeah, we are going to change components every year in multiple stacks. And every year is probably just the average cadence. Things are going to change. Lets work from that premise and focus on building value on top of something that's always changing. That's the only primary focus. The rest is managing risk.

Marc (14:31): You're kind of like a guru. No, it's like, this is my business and you make it so elegant the way that this comes out. It's like I'm having a bit of a fan moment here. But let's turn it around. It's like are there any difficult things that are kind of stuck or, you know, especially culturally or your targets or politically or these kinds of things? Are there's some hard parts right now?

Nuno (14:54): In many organizations, we all know places where IT has been there to support the business. And their primary job for decades has been to make sure Windows is running, network is up and running, and so on. So how do you match that level of impedance, so to speak, with the scenario where, hey, it's open source, hey, it's Linux, hey, it's not Patch Tuesday, it’s zero day. And still have the guys that sign the checks happy with what IT is delivering because you're touching primal fear in an IT manager, probably if you say, “Oh, you're not going to call Microsoft for support on this.” But okay, now, it's a different phone number. Or maybe you're even going to call Microsoft, eventually, they do Linux, Kubernetes and all that stuff. And we have, for instance, here at Millennium, some good enterprise solutions running in Azure. But it's a different story. And from the point where you're starting to reach that, and in Pavlov’s pyramid, when you're reaching that basic level where people just stop listening, you need to be careful with how you pitch this and manage that on a daily basis. 

Marc (16:17): Excellent.

Marc (16:24): Hi, it's Marc again. If you'd like to get a unique view of platform engineering today, we have a great blog by Dan Glavand. I'll leave a link for you in the show notes. Now back to the show.

Marc (16:40): Let me ask a little bit about the adoption. So was this targeted originally towards greenfield? Or was it migrations from existing teams, more brownfield? Or can you talk about how the adoption has gone?

Nuno (16:55): For adoption on this platform we need it to reconcile two things. One is the platform roadmap, particularly regarding the resource types that the platform supports. Andy mentioned Kafka, for instance. Before we supported Kafka topics being managed with the platform, we tried to select or to help select workloads that didn't rely primarily on Kafka, for instance. But on the other hand, we can't just be running non important stuff. We need to embrace the challenge that is running real important stuff. So basically, what's the onboarding process for this currently, is that we basically have an open survey, where product teams can apply to use the platform. And we asked about a dozen things about that project. And then from basic stuff, like what kind of stuff do you think you will need infrastructure wise to what's your expected SLA? How do you describe your customers? How mature is your team? Do you know how to work with git, for instance, for us to get some idea of the maturity of more than one workload importance as well, but more than that, how much can we rely on that team to help us grow and use that platform, as well as how much can we expect to add to that team? And from, I'd say, monthly meeting currently, something like that, we basically end up taking some of those candidates and moving forward with them. So then we have a whole onboarding process for videos, meetings, more surveys. And just to make sure the initial experience with the platform doesn't drive them away with all of these.

Marc (18:54): All right. Is there anything kind of looking back that you would do differently?

Andy (19:01): Other than the obvious you could have brought Eficode in earlier. <laughs> I would have loved to started this a little bit earlier in the process. It's really been good.

Nuno (19:14): You know, to be honest, I think that we designed V1 for this platform almost two years ago. And when we did that, probably 80% of the technical components we selected were alpha or whiteboards or so it's easy to say today, oh, I would have chosen a different thing for this slot, for this responsibility or I would have changed the implementation cadence for this part of it but that’s to say now. If I had anything regarding that initial design where I confirmed over these almost two years is that we were ambitious, we didn't land too far from the initial design. So, cool. Could be worse. Of course, if we compare with some of the other platforms. And interesting, very interesting platforms that we see other companies bring up all over the world, we see typically a smaller scope in terms for instance, of developer lifecycle, because we even do things like ITSM tools integration, and so on, probably we could have split us up into multiple deliveries, have a different cadence, as I was saying before on this, but that's just details, you know, all in all, I think that's the most important thing that we are now validating, almost, as I was saying, almost two years after the initial design is that we can onboard people and they use it and they are actually having direct conversations. No one's complaining that the DevOps team, that cool antipattern, which is a DevOps team, hasn't implemented the pipeline yet.

Marc (21:19): Cool. I like this focus on the conversations because you know, this whole people and interactions, it's like, when I talk about where did scrum go wrong. Where did agile become a bad thing? You know, it's like, well, we forgot the very first thing, which is the people and interactions, and you keep talking about the conversations. And I think, you know, having the right conversations is a really brilliant way to structure all of these kinds of things, aiming for that. What comes next? Like, are you at a certain level where you've got like a number of the teams or a percentage of the teams? And then as you add functionality or something else, you will be able to add more? Or is it just a matter of, you know, kind of space time and moving forward? Or where are you going next, please?

Nuno (21:59): Well, I can give you a bullet list. <laughs> In my own personal bullet list, I have one, I'm looking forward to see which one when basically, when will we have the right conversations, sorry to repeat on that. But internally, because now 80% of the time within all of the teams that are behind this platform, 80% of the time, we were talking about the platform, and I like to think that six months at most, we're going to be stopped talking about the platform, because it's going to be commonplace, we're going to be talking about all of this stuff that we still want to add to it, that we want to mature with it, and so on. So the interesting part here, I'd say, is this platform is going to keep on changing. We have a roadmap for even changing some core components of it. But we're going to do that without people noticing it on one side, hopefully and besides that no one cares if everything is working okay, no one cares if Jenkins is working or Jira is up. Or if Azure DevOps is doing its thing, it just is. And so, step one, and we're getting there quickly is to reach that level of expectancy, that service level within the minds of our internal users, and then just keep on adding stuff. Now the biggest hurdle isn't IT service to all of these dev teams, because those fall into patterns typically. The biggest hurdle on organizations that have been started last year is all of the technical debts, and 'gray IT' and all that stuff that's behind the scenes. So that's I'm sure where we'll spend most of our evolution, taking care of all of those integrations, getting old stuff out of the way, replacing components just to get more efficient.

Marc (24:08): I love this continuous change focus.

Andy (24:10): I love always talking about the tech stack and the cool new toys. And this tool is really cool. And that tool is better because of this and that. But it's always been fun in Millennium, that project that we talk more about this as capabilities we need to be able to deliver so that we can have these conversations that we keep bringing up. And we talk more about how can we deliver this functionality to the developers of the teams we're onboarding so they can do their jobs? And when we ask Nuno that, what's the next steps? He's like, yeah, we're not going to talk about platform anymore. It's just going to run in the background and we'll just improve it in the background without anybody even knowing and I think that's just such a fantastic way of looking at things.

Marc (24:57): I have the same where it's like way back in my spine is let's get something up and running. And once it's running, then we understand what we have, and we can start to change it. And if we do it well, then people will only see it get better. And you know, so many times I see customers just terrified of making the wrong tool choice. So they get, you know, this analysis paralysis type of thing so easily, and you're like, “Well, we're going to change it anyway.” So let's, you know, we'll spend our time you know, replacing and doing new integration to have new capabilities, it's really cool. It's really, really cool. 

Nuno (25:30): Embrace the change.

Andy (25:32): And the more often we change something, then the easier we get, or the better we get at making change. And the more practiced we get at changing, it makes it easy to make the change more invisible and less impactful and just more of a background. This is just something which is always happening.

Nuno (25:49): That's the secret sauce, being able to change everything, and no one noticing it, unless someone is just going to say, oh, this is running even better, faster, wherever.

Marc (26:03): I think that's a really good bombshell to close with. And I think it's really interesting observation that we did all of this talking about, you know, vision and capabilities without having to go into individual tools, because as Nuno pointed out, they're going to change anyway. So Nuno, I've got two additional questions that we've been asking everybody that comes to the DevOps Sauna podcast. The first one is think back to when you were a child, what is the first thing that you can remember wanting to be when you grow up?

Nuno (26:35): I think it was some job, something related with building stuff. I have this sticker on my laptop, which that says, “I build it.” I think that throughout many years, I think that's kind of 'the thing', build stuff, put it out there, build the next one.

Marc (26:54): Excellent. And the second question is, is there a point in your life where you either realize that you are definitely on the right path, or it kind of crystallized for you or you realize instead that you needed to change the path you're on? Is there one of those kinds of points that comes to mind?

Nuno (27:12): I've had two or three points in life, where I said, this is cool, but I don't want to do this anymore. And I think all of those times, they were based on situations where I couldn't see myself on what I was building, what I was pushing out, what I was doing, I couldn't build, you know. And those occasions I really moved back into an engineering scenario. Now, I come from a management and marketing background, and go figure that and I've always been engaged in IT. I think I spent six months working in marketing saying, oh, this is cool. But I'm going back to building my kind of stuff. And all of those signs, my personal gain came from, we're building something that's adding value, we're doing stuff that is relevant for people and that can be building people, because that's the most important thing you can do, help people grow into, or help them do their job with the tools you built and give them.

Marc (28:27): Excellent. Thank you for sharing. And thank you for joining us today, Nuno, this has been really exceptional the way that you were able to describe things and the ambition level and the success and I feel the quality as well of the work that you do. Just beautifully spoken as well. Thank you so much for joining us today.

Nuno (28:44): My pleasure. 

Andy (28:45): Thanks. It's been great.

Marc (28:48): All right. Thank you once again for listening. This is the DevOps Sauna signing off. Before we go, let's give our guest an opportunity to introduce themselves and tell you a little bit about who we are.

Nuno (29:04): Hi, I'm Nuno Guedes. I am the cloud compute lead at Millennium BCP. And I'd say that's our biggest focus, and my biggest focus is building cool stuff.

Marc (29:20): My name is Marc Dillon. I'm a lead consultant in the transformation business at Eficode.

Andy (29:24): My name is Andy Allred. And I'm doing platform engineering at Eficode.

Marc (29:29): Thank you for listening. If you enjoyed what you heard, please like and subscribe. It means the world to us. Also check out our other interesting talks and tune in for our next episode. Take care of yourself and remember what really matters is everything we do with machines is to help humans.