Talk: A 360° DEVOPS enterprise evolution: Danske Bank's true story

Spyridon Maniotis, Head of DevSecOps Center of Excellence, Group Technology & Services  at Danske bank, will walk us through three main themes, that are in the heart of the DevSecOps evolution. 

Firstly, the DevSecOps vision shaping and evolution mechanisms will be presented, providing insights on how to ensure clarity and coalition towards the desired direction. Secondly, the reconciliation approach of DevSecOps with IT Service & Portfolio Management will be presented. The third part will focus on the reconciliation of the DevSecOps operating model and vision, with that of the Cloud adoption due diligence and strategy mechanisms. Concluding, the so-far captured lessons learned will be presented, complemented by “dos & dont’s” when scaling DevSecOps, derived from our experience. 

Lande (00:03):

We have Spyridon Maniotis from Danske Bank, Spyridon who basically took us through an evolutionary journey of the DevSecOps in Danske Bank. Really interesting to hear this. So over to you Spyridon.

Spyridon (00:21):

Very well, thank you very much. Thanks to everyone for attending. My name is Spyridon and I'm heading the DevSecOps Center of Excellence at Danske Bank. I'm here today to give you a little bit of taste on how we do evolve our DevSecOps adoption looking at it from a 360 perspective. A little bit of the agenda, I'll tell you first of all who we are and give you a little bit of a sneak peek of the transformation that is currently undergoing at the firm. We'll tell you what was our starting point more than a year ago, how we decided to move ahead with the 360 model, how we make sure that we do this evolution in alignment and not relevance with these two aspects that we consider very important. A sneak peek on where we are with the multi-hybrid cloud approach, patterns and the patterns that we have discovered and we keep discovering and I will make sure... Can see the timer here if I look to my left is the timer. I'm going to make sure that I'm going to leave at least five minutes at the end for your questions.

Spyridon (01:22):

So basically going directly into it, we are a Nordic bank, we are headquartered in Copenhagen and we have been through quite a major transformation that started more than a year ago. Currently, we are being segmented into two core business areas, the one is the personal and business customers and the second one is the large corporates and institutions. We are organized following the Spotify model into 26 business Tribes and we have 12 COE's side-by-side. On our technological footprint, we have more than 4,000 people employed in technology areas while having technology centers and delivery teams in the four Nordic countries, Lithuania and India. The Nordics are our home markets obviously but also we do have global operations in 12 countries in total.

Spyridon (02:11):

Now more than a year ago we started a journey that we call A Better Bank by 2023, and this is the largest transformation that the bank has been undergoing in its history. What basically we want to achieve with this transformation is three main objectives. The first is to digitize our customer journeys both our external client to the bank, internal business units to IT, and IT to IT, uplift our enterprise agility capabilities. The adaptation of the Spotify model is one example and then a very brave and generous technology industrialization in order to be able to be future fit for purpose from a technological perspective. We work into these three pillars in alignment and we see a DevOps 360 degree evolution as an enabler, on the one hand, to materialize the ambition across these pillars but at the same hand to make sure that we move them into harmony.

Spyridon (03:05):

So a little bit on what was our starting point. So DevOps as a concept has been historically quite some years now being adopted at Danske Bank, but can only give you a little bit of sneak peek how it used to look like more than a year ago and how, to some extent, still looks like as our evolution is in progress. So we had at least 500 unofficial DevOps operating models within the same firm and you can question if those were coexisting or coming into conflict and N number of CI/CD pipelines and N means God knows, it was a little bit scary. We've done the environmental scanning of how many CI/CDs have been around. Thousands of hours of lead times on infrastructure provisioning and not only in general around technical requests, a lot of capability fragmentation and duplication. So basically, for instance, our standard CI/CD pipeline and ITSM tool integration wasn't the best it could be so the checking abilities were fragmented.

Spyridon (04:01):

And then, for instance, we had more than one observability frameworks lying around in the firm duplicating more or less the same solutions. On top of it we had concept reconciliation gaps so the agile methodology that was followed historically by the firm was not realigned with the DevOps option. Our whole portfolio planning was not aligned with management adoption and so on, and then we had a lot of awareness, so basically we caught ourselves talking about courses without really understanding in essence what they mean, or even using the same words, but meaning different things, and then basically getting lost in translation. We have at the same time been primarily focused on DDD and here we paraphrase BDD and TDD, DDD in this case stands for Deploy, Deploy, Deploy. So the most important thing it was to release velocity and to sip a new functionality and products to production.

Spyridon (04:57):

And then we didn't have stroke mechanisms about verifying both adherence to bare minimum requirements on how to build, run and deploy software, but also to the project context on where we are. Now, surprisingly enough, this has been working. So basically in our home markets, we did maintain our competitive advantages. We have a couple of products that are award-winning almost on an annual basis in European competitions. But what we came to realize is that this cannot be long-term sustainable because it has been complex, it has been costly. There was a lot of shadow IT here and there that had operational risk which was unknown and basically we had to do something about it. Now before going and telling you how we approach the situation, have a little bit of a poll in here. And we'll give you a little bit of some seconds. So if you look into this seven states of the art characteristics, if I may call it that way, can you pick which one better describes your current DevOps adoption in your organization?

Spyridon (05:59):

I think I can guess which one will come on top, but I'm really curious to see that. Right. Moving one. So basically we knew that we had some complications. Nevertheless, we knew that we had very solid fundamentals that we could build on top. Basically we came up with this one that we call our 360 Degrees DevSecOps Operating Model. And I can say with pride that is tailor made by us for us. And it starts with what are the outcomes that we want to achieve with evolving the DevOps set up. This is not exhaustive, but you see on the left hand side, but these are the main ones. So basically we want to make sure that we're compliant, and that is compliance towards our main regulators, which is the Nordic FSAs and the British FCA, but also to internal policies that are under banking secrecy and define internally what we're allowed to do and what not.

Spyridon (06:46):

Then we want to digitize our customer journeys, make everything as a service, as fast as possible from the person who would like to use our mobile banking as this person on the street to a business user that needs to be able to see progress on the release velocity of the application, to a person who wants to sell the service himself from IT to IT and get elements of it provisioned. They will want to accelerate productivity per model, looking into the people that they design, build, deploy and run software, but then everyone around them who is supportive and a function to that. And then make sure that we get aligned while we maintain autonomy on the teams that deliver software. And this actually has been one of the motives to see if there was a Spotify model to make sure that we provide enhancement to our Tribes, but at the same time make sure that this is an aligned way and it doesn't go sideways making even worse the context that we had ourselves starting point.

Spyridon (07:43):

So taking you a little bit through the model, this is not exhaustive, it's much more complex in reality, but I'm sure it can give you a very good taste. On the top pillar, we have the vision on what we want to achieve with DevOps. How can we tactically fit it into the overall adoption and what are the bank principles that we want to organize under operating in a DevSec concept and context. So our enterprise agility and DevSecOps reconciliation field guide is a good example here on what it means to go the Spotify model while injecting DevOps, organizing principles into it, then we'll do a lot around the enablement on how we can roll out this evolution. The Tice model is one example of how we can scale and measure, we have our own DevOps journey model, and then how we can amend the culture and incubate people through our coding school being one example.

Spyridon (08:36):

Now, inevitably DevOps is something that is moving quite fast, we want to stay relevant. It's one of the reasons why we're attending this conference. We want to see what happens in the industry and want to make sure that we stay on the edge and on our toes where we do a lot of stuff around environmental scanning, partnering up with companies outside ours. The core of this model now is what we call Capability Engineering and Advancement. So basically this is a set of capabilities that we see as the heart, in order to materialize the model. We have broken it down into four pillars, started with the technological ecosystem and the ambition to offer... I'm not going to call it everything because I disagree with the term everything, but as much as possible and feasible as a service and with interoperability, our multi-hybrid clouds approach is one element here, as well as our standard CI/CD pipeline.

Spyridon (09:25):

A lot of things around quality assurance engineering that we want to advance shift left. And we do a lot of work on stabilizing test environments. A quite thorough and broad SDLC engineering, and basically looking into the way that we build, deploy, run software and the correspondence processes. We are creating an observability, reliability framework actually in order to harmonize across agnostic platforms. And then quite a lot of stuff around compliance and security that we want to implement as a code and shift left. Examples here is our newest OD policy, the old one was outdated, it was not fit for the purpose, so we had to rewrite it, and be a little bit more creative and agile to call that way, and then consecutively that is requiring re-engineering identity and access management model to support that policy.

Spyridon (10:18):

Now we're looking into a lot of enablers. We've started using frameworks and patterns, which is something that we haven't really been used to due to being allowed to go different directions, a lot around policy engineering. So we want people to follow policies without realizing it, to put away. Stuff around our people's skillsets, uplift and mentality change, as well as more demarcation of ownership of capabilities. And last but not least a lot of stuff around technology which is a core enabler. So basically now, I've taken you through a high level into what we want to do, unfortunately we don't have time to go through everything, but I'm going to focus on these three areas and deep dive a little bit to how we do things.

Spyridon (10:59):

So you have seen the model and now, okay hopefully we'll have a model. Parts of it are already in place and being launched. Parts of it are coming along the way. And we have up to the end of 2022, to bring it all into life and adopt it. And you are going to ask how we do it? And basically, the main principle is that we want to do it in alignment. And we're using a model that we call TIES. This stands for Themes, Initiatives, Epics, and Stories.

Spyridon (11:21):

The Themes and Initiatives are capabilities that are to be enabled and adopted in the organization and primarily looking into that level we talk about capability enablement on the organizational level. So basically if you recall the previous model, it can be a subset of what we have as technology as a service. And then the Epics and Stories are the ones that have been broken down into the backlogs of our Tribes. And basically this is the adoption on the Tribe level. Now it is very important to mention that this applies to everyone. So it is equally applicable to all Tribes, the model, and then all the rest that are technology capability enablers are chipping in into it.

Spyridon (12:00):

What is very important to mention is that we want to balance innovation, reliability, and compliance. So if you look across the Themes, you're going to see a very fine balance between these three, while being a little bit extra cautious on the latter for obvious reasons. Now we have several things. Actually, I don't recall the number now. I think they are more than 30 in reality, but not all of them are applicable to the DevSecOps journey. On the DevSecOps journey we have primarily three of them that actually are balancing between the regulatory truck, which is mandatory things we need to do for our regulators but also banking secrecy, and then technological ones that are to be adopted at relevance.

Spyridon (12:40):

So I've drawn the three ones that are applicable to the DevSecOps evolution and give examples here in order to give you a taste on one hand of what is in it, but also how do they reconcile the three of them because they tell the same picture at the end. So we have one that we call development speed, and this is different capabilities. I want to enable them to speed up our path to production while being cautious of reliability compliance at the same time, our CI/CD pipeline completeness is an example here. Then we have the cloud-first one, which is as an example in an agnostic, continuous deployment framework for the applications that have been chosen that will be chosen to become aggressive with cloud native and aggressive cloud native in our own dictionary means public cloud included. And then we'll have a lot of controls around our SDLC and our source code repository integrity, it is one example of this one.

Spyridon (13:31):

I have this queue and the continuous cycle in the middle to tell you that this continuous realignment almost on a daily basis here and minute between these two, but a formal one every quarter. When we collectively across organizations, we also do capacity planning, benefit realization, but also an annual one in order to check whether we're on the right track. So this is a little bit on how we do it in alignment. And then yes, of course in alignment, but we are not identical. Actually, this is great because if we were all identical, it would have just been boring. So I wanted to do it in an aligned way, but we also need to do it as relevant. And basically where we see the biggest differences is across our Tribes, it is the business context. You cannot compare markets from two features with a more slow moving area which is primarily having internal consumers. The maturity variation of the DevSecOps adoption that goes horizontally across Tribes, but goes vertically into the squads in different Tribes.

Spyridon (14:31):

A lot of variation in the portfolio composition, this is just indicative, but it gives you the feeling that each squad, definitely it's Tribe, but each squad has a hybrid portfolio to take care of. I think quite a lot of differences are when it comes to the quality aspects of the applications and the most profound ones are the time to market, their resiliency requirements, availability requirements, but also the client and, the regulatory impact. So taking these circumstances for countless others, that for the economics of the session I haven't put into the slide, we'll have to come up with a model that we knew that it cannot be perfect because life is not perfect, and we don't know what is ahead of as plans are changing, but definitely will be something that can be used as a compass, on the one hand to explain to our people how they could do it at relevance, but also give a central mechanism to other areas in order to be able to capture the adoption speed and adoption success.

Spyridon (15:30):

So we came up with what we call the relevance framework, and basically this is used as a basis. Then the applications, the portfolio of applications in each Tribe, and it takes a full look in perspective. So basically it looks into how these applications are to evolve in the future. Making it simpler than it really is in reality in order to explain it, some of our mainframes we're exiting them. Some, we need to optimize them. When we talk about the distributed area, either this is going to go through a transformation, maybe be a radical one that will lead to cloud native, or they're going to stay as they are. And then the cloud native ones, they're going to build an evolution journey to become even more cloud native while staying on premises or even more cloud native whether they leave the premises depending on the nature and business case.

Spyridon (16:18):

And then we'll have a DevSecOps journey model that we have defined. This is just illustrative. It doesn't look exactly like that. It looks similar to that though, so you can get a taste. And basically what we have done is that we have run capabilities. And as you can see, these are going beyond the CI/CD pipelines. We have run capabilities that actually give you an evolution of your application and give you a compass direction of what will be your journey. The color coding is identical. So you can try and do mapping on what is applicable in each of the cases. But the most important thing here actually is that we want to enable a minimum agnostic evolution. And this minimum agnostic evolution says whatever your application landscape and whatever the journey that you've got to take, there's some very minimum adherence and economic compliance because it's not only compliance elements, but minimum adherence that you need to adapt to, and this is applicable, as I mentioned to whatever the journey of your application.

Spyridon (17:14):

But then as you get away from that and you get to the distributed and cloud native world, then comes the whole evolution at relevance. And then it is up to you following the theme work that we're doing alignment, but also looking internally into your demands together with your stakeholders, primarily the product owners on which direction you would like to evolve. And if I'm able to simulate it a little bit, let's say that the cloud, the dark cloud that you see at the bottom left, is an application that's currently in the design phase. And we want to make it the best flagship, the best lighthouse of cloud-native is at the bank. It simply needs to use some steps and follow the journey in order to get up there.

Spyridon (17:54):

And as I said, this model is not perfect because in many cases we don't know how the applications will evolve, but it can cover a lot of the mainstream examples without necessarily the need to be applied across all applications. So each Tribe that we have and the lead Tribe architect do have a view on which direction they want to go. And obviously, they start this evolution with some flagship applications that make the quality and primary aspects of them even more profound. It is important to mention that we don't mention maturity here, and that's a conscious decision we've taken. Our test around maturity assessments did not reveal that it will be effective. So, primarily what we capture is adherence and completeness, and definitely, for the minimum agnostic evolution, we enable an adherence as code practice. So basically we can see who has done the minimum agnostic adoption. And then the similar comes into the evolution at relevance where it becomes a little bit more lighter because it's not very much need to provide evidence in a generic way for regulatory purposes.

Spyridon (19:04):

It's not as simple as looking at this slide, this is illustrative, but is to give you a taste of the approach we are taking. And also to make it a little bit more visible that we didn't want to buy anything from the shelf. And we collected some good brains that we have internally, we have many of those, and came up with something that we do believe can be a fit to our context and ambitions. Now, when it comes to the multi-hybrid cloud approach, we're starting with cloud tool, this is our pride actually, it's our internal custom-made portal through which we can provision infrastructure as code, many different assets from Linux to windows machines and RabbitMQs, that's primarily based on Terraform templates in playbooks. And it's something that we have been building over the years and something that we are enhancing.

Spyridon (19:53):

We do use containers a lot. We have a quite good percentage of container adoption in our applications. Openshift is a strategic platform. We just upgraded Openshift 4, and it looks great. We still have things on open source, Docker open source applications, smaller workflows, but we have taken the strategic decision to move everything that is containerized to Openshift and this will happen in the foreseeable future. So this is our private cloud, which is on-premises, and then we're doing a lot of due diligence and preparation on the public cloud, primarily looking into what we want to achieve and what we need to do, ensuring compliance and insecurity. When we're looking into infrastructure as a service, we'll look primarily at offloading our data center.

Spyridon (20:41):

When we look into platform as a service, it's buying capabilities of the self, primarily the analytics and data sphere and not having to bring them in cloud too ourselves. And obviously, when we talk about software as a service, it's primarily collaboration tools. Now our people can have two paths to the clouds. The one is to refactor the application to cloud native and have created our own cloud native definition. So that means an application building microservices means an application using domain-driven design and be decoupled as much as possible from the mainframes, especially on the data dependency using cloud too, tp provision your infrastructure, using containers we are adopted and launching standard CI/CD pipeline GitOps frameworks, actually. So whoever is onboarded to the new pipeline, they're going to get some very nice YAML templates out of the books. Then depending on your application, the scale of it using more of other cloud native technologies for service discovery and secret management, but also prove that you can handle reliability proactively.

Spyridon (21:45):

Maybe we could do a little bit of case engineering, we're going to love hearing the story and also make sure that you can approve your compliance, adherence as code basically, and follow the guidelines in order to automatically provide evidence on that. And then the second is the lift and shift. And this is obviously if your application is already cloud native, it is in cases that the application is cloud native the journey is going to be more costly than the return of the investment. But those are the cases that we have to move something fast, and we can not waste the time. Now, in either cases, we do cloud native assessments to define how cloud native we are or not, and also have an eligibility and trade-off analysis tool that we're using in order to do the mathematics and autonomy of any potential migration.

Spyridon (22:33):

Now, what we have learned so far, and we have learned a lot, just a little bit, and we'll learn everyday. I learned something this morning, by the way. So make sure that you check the organizational temperature when you get started in continuously. For us it was for instance, very important when we got started to understand that we talk about the massive transformation, that we talk about people that are going to be very busy with too many things, people that are going to change teams reporting hierarchies and everything. So, we had to set the temple, but also this is something that we are doing as we speak in order to manage expectations and make sure that we don't try to do more than we can, they just put it that way.

Spyridon (23:10):

Then of course, make it hybrid and make it as broad as possible across your ecosystem, horizontally, vertically. Invite people, define who is a key counterpart to this ecosystem. I can tell you that in the DevSecOps here, we will have our partnership matrix, we have 14, (one four) areas on top of Tribes that we need to do things together in order to enable the vision. Avoid accessing innovation, an old Kubernetes is maybe the coolest term in IT past years, and it's going to keep being. Don't need Kubernetes all over the place and just be mindful on the one hand when you enable capabilities, but also when you're adopting. You might fall into the fallacy of trying to make it super excellent, but then have other issues on the side. And then as I mentioned, you must scale at relevance in your alignment, make sure that you all know the direction you want to go, make sure it's clear to people. Why, what, and then give them the opportunity to do it at relevance. So don't bottle everyone in the same size, because this is not going to work.

Spyridon (24:10):

Now, what not to do. Don't pretend you're another organization. I can tell you that we are not Netflix, or maybe you are, I don't know exactly who is attending, but the vast majority of you are not Netflix. So it's good to get inspiration from companies like Netflix, AWS, and Google, but make sure that you adopt things based on your context and not really try to do things that you don't need.

Spyridon (24:33):

And I can tell you that at Danske Bank we don't have global operations for 24 hours across continents. So basically the way that we do scale and can write sizing can be a little bit more naive than what Google does. Then don't boil the DevOps ocean, so, 360 degrees, yes, but step by step, here I can tell you my own romantic story. In the first presentation I've given to our management of the idea and the skeleton for the 360-degree model, it was called "overwhelming". And then I went back and looked at it myself, and it was overwhelming. So basically start with the basics, get the people aligned, get them to digest it, and then build on top of it slowly, slowly, don't go "BAM, we're going to do all these things".

Spyridon (25:12):

Don't tell your teams exactly how to do it, focus on what they need to do, and by when, but don't tell them exactly for primarily two reasons. One, you are going to kill the creativity innovation, and that will increase frustration. And secondly, you don't know their context. So basically myself being at the DevSecOps Center of Excellence, I don't know the nitty-gritty details of any application in any Squad, in any of the 26 Tribes out there. So basically don't get into wasting the time of you and your partners trying to micromanage how they're going to do things, nevertheless, do provide common partners a framework to speed up the adoption and give guidelines.

Spyridon (25:49):

Allow people to self invite themselves, I call it the Fellowship of the Ring, internally. So bringing the people together that they need to play a key role, but allow self invitations, you might be surprised at some areas you haven't really thought about engaging them because maybe didn't sound straightforward. And then they reach out to you with a couple of ideas on how they can play ball, it sounds excellent. So don't restrict yourselves into the obvious areas that don't need to come into play. And then don't realize the benefits in isolation. After hearing this thing, the one from Danske Bank in the session, how they are going to smile, 20 minutes release velocity you have achieved, and you simply don't tell me anything. So basically looking at front to back flows because speed by itself doesn't say anything, a customer satisfaction, reliability, compliance, security, team satisfaction. These are also elements that are important. So basically we try to look into the front to back flow and don't get stuck into isolated performance indicators. Maybe tell you something, but maybe that's an indication or the true story of where you really are.

Spyridon (26:54):

So this is a little bit, and I'll say this is not exhaustive, we learn much more, unfortunately, we don't have time and I want to leave you with this one. I'm not going to read it. I'm going to just give you some seconds to read it yourselves. So make out of DevOps, one, use DevOps to make what you want to make out of it. And don't try to replicate something that was implemented by the book. Then closing, I'd like to thank you all for attending. Thanks to everyone at Danske Bank, they are many, for making things happen and we have three minutes of questions.

Lande (27:24):

Excellent. Thanks, Spyridon, that was really, really good. Let me get straight to the questions. I've got two questions from the audience. One was, are you following the Spotify model by the book, or have you done some changes talking of not pretending to be Netflix.

Spyridon (27:41):

Very well, very much correct. So basically we'll have done some changes on the way that we organize within the Tribes, but also we have done changes on the way that we do the quarterly and annual business reviews as well. So I can tell you that we have a field guide on how we implement Scrum in the Tribes. This is resembling quite much the Spotify model, but if I look into the DevOps organizing principles, these are tailor-made from the ground. So we have taken what we could see being applicable. And then what we couldn't see really fitting in, we have tailor-made it.

Lande (28:20):

Cool. Okay. And then there's one more question. This one was: what is your classic security team's involvement in how cloud deployments are managed at Danske?

Spyridon (28:33):

Absolutely. So basically they're evolved primarily into three areas. One is governance controls. So basically what compliance and security controls we need to have in place. Not only when we reach the point of the CD, but working into the whole of the STLS. Secondly, they're responsible for support with policy engineering on the CI part and CD that has to go with secret management, start the code analysis, open-source scanning and validation of images in the staging area, post-deployment, all this cool stuff. And then they hold the policy around application security left, we call it. So basically there are some design principles and patterns that teams need to follow in order to implement secure applications. Now, another very important role of the hub they're the dotted line towards ITBRC, which is our IT Business Risk and Control unit, which is our interface towards the FSAs and FCAs. So there is a little bit of the channel towards the delivery areas on what is the dialogue and negotiation with the regulator as well. So they have a very important and diverse role actually.

Lande (29:38):

Excellent. And then one quick one, and then this'll be the last one. Do you build/use System Teams?

Spyridon (29:45):

Nope. Nope. I'm not going to reveal. I know where you're coming from. We don't do SAFe. I personally cannot say that loudly. I don't believe in SAFe. And our System Team is a term coming from there. I cannot reveal to you exactly what is the DevOps organizing principles are in the squads, but we don't have System Teams.

Lande (30:04):

Excellent. Okay. There's a lot more questions coming up. So I think Spyridon, only if you have a look and type the answers in, would be really helpful for our attendees.

Spyridon (30:17):

Absolutely.

Lande (30:17):

Excellent.

Spyridon (30:18):

Thank you for hosting me. It was a true pleasure.

Lande (30:21):

Oh, thank you.

Spyridon (30:21):

Thank you very much.

Spyridon Maniotis
Head of DevSecOps Center of Excellence, Group Technology & Services, Danske bank
www.linkedin.com/in/spyridon-maniotis

Watch all recordings from The DEVOPS Conference 2021 on the event page:
www.thedevopsconference.com/speakers