Internal Developer Platforms: Learnings from the DEVOPS CONFERENCE 2023 panel

The DEVOPS Conference is a global event organized by Eficode that gathers software industry professionals every year.

In April 2023, it featured speakers like Kelsey Hightower, Nicole Forsgren, Helen Baal, and Dom Price, among others.

But The DEVOPS Conference isn’t just a venue that provides talks. It’s also a community for knowledge sharing. In fact, an important part of the event was the interactive track featuring fireside chats, open discussions, and live demos.

I had the privilege to host a panel on DevOps and platform engineering. How does building Internal Developer Platforms (IDPs) support DevOps? What are the advantages and the challenges in their adoption? What is their impact on culture and ways of working? These are the key questions discussed by panelists:

Bjørn Hald Sørensen (Head of Technology at Lunar)

Tuomas Leppilampi (DevOps Consultant at Eficode)

Dan Grøndahl Glavind (DevOps Consultant at Eficode)

Attendees actively participated in Q&As. We also ran a poll to understand their background—more than 50% of the audience was unfamiliar with IDPs, about 30% had at least started working on it, while the rest were considering building one.

Here is a summary of the panel discussion, broken down by the emerging main themes.

Panel discussion on DevOps and platform engineering

Are IDPs actually needed?

Mario: Why are these Internal Developer Platforms needed in the first place, or is it just like the trend that we are following now, as it used to be Kubernetes a few years ago? Any take on this?

Bjorn: I think it's mainly around the problem and evolution of how we do software development. IDPs, and platforms in general, are something that is evolving. Something that comes along with all the cognitive load that we put on developers.

How can we organize ourselves into more efficient ways of working? And I think that platform engineering is definitely one of the things that are looking to provide good value to developers right now.

Tuomas: I think it's a very big competitive advantage for you to be able to say to your developers, “You can actually concentrate on coding, you can actually concentrate on problem-solving, and then really doing the thing that you love instead of, say, dealing with Jenkins.”

Dan: I think that with DevOps, we have had this mindset of “You build it, you run it,” and what does that actually mean? Was that the whole stack from our applications down to the bare metal, right? And I think that is where the complexity comes in, and that is actually some of the promises that platform engineering tries to say, “Okay, we don't need to know all this deep level of complexity.”

Technical vs. cultural aspects

Mario: If you look around all the blog posts and material that you can find about IDPs, I have the feeling that it's more the technical aspect that is prevailing. But isn’t it actually more about addressing Developer Experience? How do you see this kind of potential, like technical and Developer Experience? Is it something that is specific to IDP, or what is the most important aspect here?

Tuomas: I think from the benefit point of view, Developer Experience is retention of good developers. You keep people in, and that's a really important part of the platform engineering trend.

From my perspective, it's really DevOps. It's really the kind of DevOps and agility that we are trying to achieve as a culture. Platform engineering and Internal Developer Platforms are just one way of attaining that kind of culture and keeping that culture afloat.

Bjorn: Pushing a lot of people to know more over the years, like having autonomous teams, means that a lot of people will burn out if they are intended to be knowing of all the things that can go wrong in complex distributed systems and that's not the purpose. That's not what is going to bring joy to our developers. That's not going to engage them in driving value for a business.

The IDP, to me at least, is not really like one thing. It's not a platform. It's not the technical thing that you take down from the shelf and make available. It's more of like a holistic view of all the things that are going to enable our developers to do the things that they need to do.

Dan: I think it’s the mental model to address some of the capabilities that we want to improve both for ourselves and other developers. It's something that's more holistic. It's something about how we can create capabilities and competencies for people to both be happy and productive.

Overhead and rule of thumb

Mario: So there is a lot of value that an IDP can provide. But if it provides everything, won’t it become difficult to maintain and manage? And a related question, is there a kind of rule of thumb? Like how many developers a company should typically have to justify building an IDP?

Tuomas: It could be just a bunch of documents, it could be a bunch of instructions on best practices. So it doesn't matter if you have five developers or two developers, and you know it will build the mental notion of building a platform that you stand on when you start to create something new, like a feature or application, or service. Once you have that, it's easier for others to come in.

Dan: That was sort of the question that I had. And so it was to what extent you as a developer have been met in terms of the capabilities in the build output, the CI/CD tooling, maybe the supply chain security, these kinds of things, and then the shift that we talk about what kind of deployment strategy.

And then what could be covered in terms of observability and monitoring, these kinds of things. It might be possible to cover all these areas at once. And if you do think so, maybe you just grow into more platform teams collaborating on a platform.

Bjorn: But what we did initially was really what you said—guides on how to do things, how to spin up a new service, how to work efficiently with software, not starting just with the basics. And we just slowly added on top of that. At some point, we outgrew that. This was something sort of like how DevOps was done, like it outgrew the teams themselves to find a good way to do it.

So we created two more core teams to sort of try and think a bit more about how we can optimize these workflows, where we can actually automate something, where we can help developers in their daily work. And that has sort of grown over the years to where we are now, where we have three platform teams providing the Lunar development platform. We are serving different user groups.

So you are building up from small because defining what value you can provide is easier if you have a problem saying like, “We want a team to solve this problem, and we understand the problem this big,” that's the value. And then we need to think about making it a platform because we can do this little tool, and we can do another little tool, but we also need to think holistically about it as time goes by to understand that this is like developers know where to go for information.

At Lunar, we have a ratio of one to ten, so we have like around ten developers to one platform engineer if you really want to put it down. Different companies have different ratios here, but that sort of depends on how aligned your infrastructure is, how aligned your development environments are, and also your history of technology.

Cognitive load

Mario: So, how much cognitive load is okay for a developer? One aspect is you have a single place where developers can do what they actually do for life. But does it address all the cognitive load? Or is it also automation, right? So, how are those two things coming together?

Tuomas: I think it's very personal, from a junior developer re-architecting and refactoring things to the so-called “UNIX guys” doing very specialized work. So it depends a lot on where your passion is. It's like, you know, what kind of developer are you?

Bjorn: I think it is personal. Definitely. The typical answer is “It depends,” but you need to talk to your developers. Just go ask them, “What are your pain points? Where do you feel like you're wasting time? Where are your engagements going?” Through the floor because you don't really enjoy working with it. That's where the cognitive load, really like the line, lies.

It’s like the 80% rule that if you're providing features that do 80% of the technology, like the organization part, then we might not be spending the time right.

If you're the only one reducing cognitive load for one team, but you could do it for everyone because there are bigger fish to fry, then you should always work on all things. And that's again figuring out how much load you can put on the team.

Dan: In cognitive load theory, there are different kinds of load. There is intrinsic cognitive load, where it's just the task at hand that inherently is complex. But the thing that we actually want to avoid is the extraneous cognitive load. Something that is just put on you and that you don’t care about at all, like dull work.

I think maybe the answer to that is if you're kind of reinventing the wheel over and over again, and you think that it could be something that someone could fix for the benefit of all, then that might be a sign to think about whether it makes sense to help more teams. But it’s not like there’s an exact number for when to invest in a team to solve common problems.

Platform engineering and DevOps

Mario: What is the connection between this platform exercise and DevOps? How many teams should there be? Or, to be more provocative, I even would say that having a platform team is an anti-DevOps pattern.

Tuomas: There's like three different areas. So one is the developers that kind of run what they build so that they have good observability, they have a chance to do rollbacks, and they have control over the operations to those areas that are relevant for them. Then there are platform teams that actually develop the platform and set the configurations and listen to the developers and listen to the third part, which I think should be managed services.

Then, we have the platform team concentrating on listening to the developers and making the best out of the configuration. And then the developers who still have enough access to the operations and the runtime and everything that they can run what they've built.

Bjorn: That’s exactly when platform engineering comes into play. You want to make sure that you don't take responsibility away from your developers—you want to empower them.

The main difference from what we had before DevOps and what we have now with platform engineering is out of mindset, compelling internal products, and understanding that we are here to serve developers.

Dan: If you take the book Team Topologies (which is a really great book if you haven't read it), it talks about team interactions. When you look at the platform team, they should provide the platform as a service or product. But the collaboration is actually the most interesting part. How do we figure out what to do now and what to do next?

The role of a developer portal

Mario: We've been talking about IDPs or platforms having a single entry point for developers to do most of what it concerns the software development lifecycle. And one of these is the developer portal. And then, when we look around in the context of IDP, we see de-facto Backstage. So is Backstage the cure, everything that you get regarding platform usage for different stakeholders? What is your take on that?

Bjorn: We were having conversations internally that we sort of needed this place where people would go and get like a single pane of glass—the entry point to all the other information. So we decided early on that we wanted Backstage in place to help us really make this central piece of information. But that's not to say that it is the core of the platform. It's more like the entry point.

So Backstage helps us create this place where we can sort of like go to Backstage and start with a search. If you don't get an answer, go to Slack. And ask the question for the team. But there are a lot of things around a platform that isn't Backstage, like CLIs tooling. All that stuff is still spread out, but you can still find information about it in Backstage.

Tuomas: I think Backstage is a great way to start the journey in terms of creating one pane of glass. But then, as you progress, you might realize that your core is maybe the Continuous Delivery platform like everybody will see how we deliver value to the customer and that that can become the core. Whereas in the beginning, you thought it would be templates you start with.

Dan: When I think of Backstage first, I immediately kind of fell in love with it because it looked like it solves a lot of the problems with software catalog and ownership, and it actually also had a kind of a systems model, which I think is super great. And, of course, the templating stuff as well, and the plugin-ins, and so on.

Bjorn: We had a clear understanding that developers are most hurt by creating new services. We had this document in Confluence—if you need these places, click these boxes, remember to set this name right, all that stuff. We were sure this was going to be the one thing that all developers love.

We asked on Slack, “What do you want us to build first?” And it turns out the software catalog had the highest number of thumbs up. It was way beyond the next one. Because the question is really what a lot of developers want to have, which is ownership. Like, what do we have? And who can I ask? That was just a simple thing that we wanted.

Measuring the benefits of the platform

Mario: If the platform is used and sustainable, then comes the next question—how do you tie the outcomes of the platform engineering to the business outcomes? Is it enough, or is it possible to have some kind of metrics? Are the DORA metrics also useful in this context? And how can you say that this platform exercise has brought value to the business or not?

Dan: If we as a platform team are measured on how we enable other teams, then those metrics, like throughput, stability, and reliability, really matter to us. And maybe we should also focus on how we can associate these with the applications that our developers put in the hands of their users.

Performance activity, communication and collaboration, and efficiency of flow. Maybe some of these on efficiency are covered by DORA. But there is more to it than just the kind of tangible quantity. Like the qualitative or perceived ones.

Bjorn: The business value that a platform can provide should hopefully be identified as the problems it needs to solve. I think it's hard to find KPIs that, on a monthly basis, will validate if it still makes sense to have a platform team.

Tuomas: DORA metrics are one option. I have not seen them so much in practice, though. And I think the fifth metric that they've considered is reliability, which is kind of related to user-facing behavior. SLIs, SLOs, and the runtime part. The experience and functionality of the runtime.

Is platform engineering for everyone?

Mario: Gartner is now saying that IDPs are necessary to scale DevOps efforts. So, how do you see platform engineering in the context of the DevOps maturity model? Is it something only for top-performing, or should it be like for all companies, or how does it stand there? What will we have, say, five years from now? Will it still be a platform? Or would it be something else? Will there still be DevOps or not?

Tuomas: I think it is absolutely crucial to scaling that we can abstract some things away because then we can really concentrate on the business outcomes. How are we actually changing the life of the end user?

I think I'm very lucky to work in an office where there are a lot of designers, and we have lunch together and coffee together. And I've picked up a lot of design thinking and service design methods that helped me a lot in uncovering the problems that a company or an organization could have.

Bjorn: I think at least there's going to be something because the problems down there won't disappear. But they might turn out to be different abstractions.

We're just going to keep looking at the next big problem to solve, not the next solution, not the next trend, and then we can look into the solution space of like, “Oh, what is out here that can help us solve this problem? Is that a platform team? Is it instead a product mindset, or is it a tool? Is it a thing?” Is it, you know, all that stuff? But we need to start identifying the next problem.

Dan: How much time do you spend on figuring out what your problems are, and how do you actually collaborate in turning these problems into tangible solutions? That’s the interesting part for me.

Wrapping up

The panel saw a very interesting discussion that brought up many different themes related to DevOps and platform engineering. It was clear that a platform is not only technology but an effort to serve developers and streamline processes.

Platform engineering should have concrete objectives, focus on Developer Experience, and foster a culture of collaboration between teams.

Every company is different; therefore, efforts in platform engineering should target pain points and seek continuous feedback from multiple stakeholders, not just developers but also product and business owners.