How software development leaders view DevOps and Agile

Risto (00:09):

In good cases, delivering or releasing to production, that can be left as a business decision. So opposing to the thinking that we deliver when we are ready, it might that be daily, weekly once a month, or once a year. Let's not go into that discussion, but having a more system, and set up, and organizational culture where the technology is not really dictating the times when to deliver. But, it's kind of the latest version, usable and working version, is always available for the business to decide that, yes, let's go to production with this version.

Marc (00:55):

Hello and welcome to DevOps Sauna podcast. I'm Marc Dillon. And I'm your host today. DevOps is no longer a new and hip concept, but rather has a proven track record in enabling business agility and performance. When DevOps has been successfully adopted, teams can react to the user feedback swiftly and release software independently.

Today I've invited two of our software leaders, Risto and Matias to discuss DevOps and how we internally view software development.

We're talking about DevOps and Agile today. So could I do like a 1, 2, 3, what is DevOps, Risto? 

Risto (01:38):

To me, DevOps is more like a kind of a culture thing. It's a state of mind. It's the way of working. And for many people, DevOps is about the tools and automation and all that what we do and use. But, when it comes to then end of things, I think that DevOps is a culture that is only manifested in the tools we use. So that's the kind of the surface that we see. 

Marc (02:07):

Okay. Matias 1, 2, 3. What is DevOps? 

Matias (02:13):

DevOps to me is both a way of working and also the tooling side of what development is and how development is made and done.

Marc (02:24):

Okay, cool. What I've noticed in my career and since joining Eficode, which is the greatest DevOps house we've been doing this for 15 years, greatest DevOps house in Europe. I've noticed that when I look at what our customers are doing, I can quite get a good sense of, are they doing it right or wrong?

It's almost binary. And I think that's one of the interesting things that we've come to see is that either people are on the right track and they're doing things in a really DevOps focused manner, or they've had some failed experience with Scrum or Agile or scaled agile framework, or they've had some tough experiences.

So how have you guys seen this kind of approach? Do you think it can be right or wrong, or do you think that there's some tendencies here? 

Matias (03:13):

I personally think that there is some sort of granularity to how it can be implemented. And it's not entirely binary. You can have sort of different levels of DevOps within a company. And again, probably due to my understanding of what DevOps is, I can sort of have a more granular approach. It isn't sort of, if you don't think of it entirely in terms of a way of thinking, then you can sort of more or less pick and choose things that are maybe more DevOps and less DevOps depending on the situation.

Risto (03:45):

Yeah. I pretty much agree with Matias. And I think this ties into my earlier statement that it's manifested in the tools that we use, how far in the DevOps journey you are. But still, if you're just using the tools that doesn't mean directly that you are in a way in a DevOps way of working.

Marc (04:06):

Yeah. I think there's a lot of this where it's like, well, we tried Agile or we tried DevOps, and this is where I say, like, it can be kind of binary. Risto, have you seen something about how people are going through the motions of DevOps or Agile, but perhaps not really having the right principles in mind or the right way of working or the right culture?

Risto (04:29):

Yes. So we've seen quite a lot of different levels and different ways of working in different projects, but I think at the end of the day, it comes to the decision that: is it working or is it not? And are you delivering value or are you not? 

Marc (04:46):

There's a term that's been trending quite a lot in the office called cargo cultism. Is this one of those things? What is it? I think you know about this, Risto? 

Risto (04:56):

Yeah. So cargo cultism, it's referring to stay things which happen to the south Pacific, where the Islanders worshiped airplanes, and act like traffic controllers in hopes for that this would bring the cargo planes back.

So in the area of World War II, when there were a lot of cargo delivered the soldiers in the south Pacific islands and some of the modern goods were distributed to the locals. So when the World War then ended, the supplies ended but the memory remained. I think the main message here is that you shouldn't just follow a framework or create a cult that someone else has brought to you, but instead figure out the best way to organize the work in a specific business and operations you're in. But still in a way that also needs that you need to master something before you start to developing it and should really understand what you're doing in the way and why you're doing something that you're doing the way you do it.

Marc (06:03):

Right. So this was a new topic for me, where the Islanders, the airplanes from the war times were dropping off supplies all of the time. And when the war ended, they were trying to recreate so that the gods would bring them more supplies. So they were even making airplanes out of bamboo and sticks and things hoping that this would bring the planes back.

And this is kind of like Agile principles. It's where people are doing scrum ceremonies, but they're missing something. And they're not really improving on a continuous basis. They're still may be doing things in this water-scrum-fall kind of mode. What do you see there, like about how when they're just going through the motions? What do they need to change in order to be able to get on the right path? 

Risto (06:54):

One of the key things in Agile development and Agile mindset is the constant learning and constant self-development. And if you're still doing in a way, a waterfallish planning and you know what is expected to be delivered six months from now, you're not really developing the way you work and the things you do in an agile way. So that's, I think Marc, what you refer to is that in a way having the ceremonies doesn't mean you're doing scrum, it just means that you're meeting once a day for a ground rubbing that, yes, this is something I'm doing, but no one is actually committing anything.

It means that you're showing twice showing every other week what you've done, but you're not actually talking 'did we learn anything'. 

Matias (07:46):

And I think there's always a possibility of you starting out in an agile manner, but then ending up in a rut and sort of just going through the motions as you sort of have gotten used to the practices. And you sort of end up not developing the practices and yourself, but just sort of get stuck in a sort of a loop, if you will.

Which then just doesn't end up anywhere. And then you sort of, you lose what the original point was. And then you start focusing more on the ceremonies as opposed to what it is that you're actually trying to achieve. 

Marc (08:21):

There was an article some time ago called Levers of change by Chris Howe-Jones. And at the bottom of the levers of change, like the least effective were things like numbers, it was metrics and targets. Like we all, like our bonus targets and things like that. But the second most effective was paradigms and the top most effective was transcending paradigms. And when I looked at that, it took me a moment to understand. 

And then I realized that, how many times have I seen a company that said, well, we tried Agile. Well, it was our own version of Agile, but it didn't really work out. And so we went back or we didn’t really continue or feel good about it. And I kind of learned that if we're taking one of these things into use, we have to really take it wholeheartedly, including the constant learning, constant development, involving customers, having feedback from the customers directly into the software development teams, having demos, having the customer present for those and getting quick feedback, having every sprint actually incrementally deliver value.

And when you start leaving out a few of those things, because, oh, well we only release once every six months. It's like, well, do you get any customer feedback? It's like, well, we've got our salespeople, but they're not really that interested. And then they start wondering why it didn't work. And then you have to really follow this by the heart before you can think that you can make any changes to it and be successful.

But once you've mastered one of these paradigms, then you can actually start to look at now, how do we mix and match the ones that are necessary to make our organization deliver value? And I think that's kind of one of the really interesting points here, which is like, yeah, you can have dailies, you can have scrum meetings, but if you're not really focusing on the content, like Matias brought up, and what you're actually doing this for, then you can kind of get lost and say, 'Oh, well, scrum didn't really work for us.'

Matias (10:18):

Yeah. I find the notion of being able to mix and match, intriguing. But it just requires that ultimately you be able to transcend the paradigms you have been working with by mastering them. And it requires just a lot of dedication and time from an organization or team to actually be able to sort of get to that level.

Risto (10:41):

Um-hm. Right. Maybe one of the topics that I think it's been in, more like on the business strategy literature earlier is kind of standing on the shoulders of giants. And I think that's one of the topics that is highly misunderstood as you really can't stand on the shoulder of a giant, if you're not big enough and wise enough to actually climb there and look further.

And I think it's one of the things that applies here also what Matias just said that you just can't go pick the berries. You need to understand the whole thing that you're doing. 

Marc (11:14):

So Matias, you work in a software development and you have a lot of UX (user experience) focus. Do you see that there is difficulty in connecting parts of teams with design and how the cycles work with pulling everybody together towards something? 

Matias (11:32):

Definitely. I feel that that's something that we see a lot of customers struggling with. And to be honest, we probably struggle with it as well when we're trying to sort of build up development teams and try to enhance the quality of flow of information between individuals in a team.

And it's sort of always, there's this risk of wanting to be agile, but then always ending up thinking in terms of waterfall anyway. Where you sort of always go from a planning and then to a design and development phase but it's not considered as a loop into the degree it should be where it's not as iterative as it should be. And it's always a huge risk. 

Marc (12:16):

Is this something that is like, cross-functional teams help with having like business analysts and design and test and developers, all on the same team? 

Matias (12:28):

Definitely when you build up a team composing of all the necessary parts of whatever product you're developing. Be it from in terms of business or design, the software development itself, it's always necessary to have the different components together and working in very tight conjunction and loops on the product. This sort of, from the way I view it, it sort of slightly changes the way, especially design should work when working on with the software development team in a tight-knit group. 

Where the design is less focused on, let's say the design artifacts themselves, but more focusing on the facilitation of design work amongst the whole team where you bring all aspects of knowledge, be it software development knowledge, or business knowledge into the same group. And trying to share information amongst the individuals and come up with ideas as a group, opposed to then simply individual ideas that happen to related to your individual area of competence. 

Risto (13:36):

If I may build on that, it's not only about design working this way. It's about investing the time of business analysts, business owners, developers, designers into kind of investing the time for them to work together and cross-pollinating their thoughts. So that actually the product becomes a whole product and not just pieces of product that are most probably misunderstood in the handshakes from one table to another. 

Matias (14:05):

It’s very well put. 

Marc (14:07):

And continuing to build on this, I've seen designers go through quite some changes as they work directly in cross-functional teams with developers. Because what ends up where it's like a 'wait until it's perfected' kind of a design model. Instead, this is 'how I can design for the current sprint and deliver a value in that sprint' instead of, and still kind of contain a long-term vision.

But the pace kind of changes such that the value out of each sprint increases, the vision can still be there, but it takes quite a bit of restraint, I think from many members of the team where there's more communication than coding or design work. And some of the greatest teams that I've seen...talked more than they seemed to be working.

And I was always like, how does this… how do they get anything done? And then all of a sudden the sprint comes and they have a new customer feature that's ready for deployment. 

Risto (15:06):

Yeah, I've actually experienced about the same. And maybe one of the things, even though this kind of difference between a designer knowing what things should look like and then adding the behaviour on top of it. And in many cases, it is the developers who actually know better how things could be done. And that then feeds back to the designer's table and actually makes the product way better than either one of these individuals or teams could actually deliver by themselves.

Marc (15:38):

And then one of the interesting things that I noticed from my assessment work in the assessment tool itself, 'four out of four' in one category was having the team’s direct access to customer feedback. So that then the teams were able to understand how the customers are using the applications or services, and then be able to take this work directly through each iteration that then we have real feedback directly from the customers. We can make designs that help the interaction. We can ensure within the same iteration that those are technically feasible and the developers to understand how to interact with the design and the customer journey in order to be able to, at each iteration, add value.

And I think that it's really interesting how some companies are tremendously good at this. And some are kind of resistant to trying. 

Risto (16:36):

It is always scary to kind of face the customer feedback and publish your work. It's always scary every time. 

Matias (16:45):

It is mandatory to be able to actually make a product that brings value to people. The only way to get that information and validation is to get it into the hands of the users no matter the costs. 

Marc (16:58): And that's one of the points of all of this agility and DevOps practices is to be able to get this kind of quick feedback and also show quick value. But you remind me a couple of the downsides are that when one starts really doing agile and getting customer feedback, all of the problems become really well exposed. 

Where in a waterfall, they’re hidden for the whole process until boom, you have a big delivery and then all the problems kind of come at once. Be the usability or a functional or non-functional problems. And I think one of the great hurdles to understand is that even at the beginning of the journey, all of the uncertainties rise to the surface. And all the things that you don't know, become things that you have to face. In addition to the scariness of going into a new process that is unfamiliar and feels uncomfortable. 

Matias (17:55):

Learning doesn't always need to be sort of enjoyable and done with a smile on the face, but rather it might come with some pains on occasion. But then ultimately the net effect remains positive hopefully. 

Risto (18:09):

Maybe one of us, we kind of started the discussion on what is DevOps. And I think Marc just kind of also gave your input to the discussion earlier. So let's just try it now when the kind of DevOps and being agile of really enables the teams and businesses and people delivering the product and service to really react and respond to the feedback. And that is one of the key cornerstones in my view, to what DevOps actually enables and brings to the table. 

Marc (18:42):

Yeah, you reminded me. And one of my customer cases, there was a bus factor, it’s an industry term, which says, how many people know a certain technology and if they were to suddenly be unavailable for work. Then you would have a big problem because all of the information is in those people's heads.

And one of the things that I saw with a customer in this area is that the more that they have cross-functional teams that are capable of delivering to deployment, the more connected they feel with the company strategy and the customer. So sometimes you have like a guy, the bus factor is one because there's one guy that's dealing with all of the production problems and the rest of the teams are not.

So that guy feels the most connected to the customer and the strategy simply because he's dealing with problems where the rest of the team can feel that, oh, well, we're kind of in this bubble and we don't really get the same feeling of connection to the customer as the guy who's sitting on the front line. Even though he's a developer dealing with the customer issues or the production issues. But do you have to release all of the time in order to be agile or to be DevOps? 

Matias (19:54):

I come from a world of startups and I feel it, YES, definitely, as often as possible. But that might not always be feasible or ultimately the best option. But if you're trying to get again, the user validation as quickly as possible, the only way to get that is to release as often as possible.

Risto (20:17):

Yeah, but that's right. I fully agree. Then again, in good cases that delivering or releasing to production that can be left as a business decision. So, in opposing to the thinking that we deliver, when we are ready, might that be daily, weekly, once a month, once a year, let's not go into that discussion, but having a more system and set up and organizational culture where the technology is not really dictating the times when to deliver.

But it's kind of the latest version usable and working version is always available for the business to decide that yes, let's go to production with this version. 

Marc (21:03):

Yeah. I think the COVID situation taught us a lot in these areas where all of a sudden companies were having new problems to solve that were turning up practically overnight, force majeure kind of situations.

So for example, thinking of all of the food and perishable goods that were delivered that suddenly people wanted to return. And you know business to business. So, if a grocery store has the right to return unsold goods, then it might be that all of a sudden all of their chains are returning lots of goods to various distributors.

And if those distributors have the online ability and the services to be able to take those things back, then it can really help with the logistical problem of going from returning a good making an order to return that or a return request. And then getting a guy with a truck there to pick the things up.

If they don't have the ability to deliver easily, then they have a whole problem to solve that they don't have the IT capabilities to be able to solve. And we've seen lots of, kind of variations on these themes in the last couple of years. So, delivering being a business decision. We deliver when we want to, or when we choose to, rather than when we're ready. I think has really a sophisticated point of view.

Risto (22:21):

Yes, I agree. And when we are looking at the way, the more traditional way of delivering software or doing software that you have releases and user acceptance testing and all that, and things are delivered when they are judged to be ready, maybe four times a year, every month. Never mind, what the timeframe is.

When we are looking, we have, oh, we all see that the kind of processes and projects in our daily work. Then we also say the opposite end of the spectrum, where there are developer or developers working on a task. And when the task is considered done, it's pushed through test automation and pull request. And if the pull request is approved, then the task is potentially shipped to production and that's kind of, it might be, in some cases, it's even pushed to production without any further business decisions.

It’s decided that when this pipeline is passed through, then it's good for production and let's go. And I consider that being rather agile. 

Matias (23:24):

Indeed, indeed. And I think it's sort of, it's interesting to think about the delivering as a business decision. When I feel that the decision to develop something has already been made and thus it should be, that should be where the decision from the business side has been made.

And then we're just now then holding our breath until we get it released, however quickly that is. And I think from a very, very sort of a small company perspective that might be probably an understandable approach. 

Risto (23:55):

You need to take into account, Matias, here also, the other business processes, which you might have, so not only the delivering of a software product, but then adding the business processes of marketing, communications, maybe even delivering a physical product based on the service that you're creating. 

Matias (24:17):

Definitely. Definitely. 

Risto (24:19):

That then brings the kind of the added needs for business decision to release a digital product out to the market. 

Marc: (24:28):

Yes, I guess. It also relates to what's the size of a release is and how much joint features or then just a whole product is related to that release. If it's these small incremental ones. And probably it doesn't matter as much, whether there is the other sort of business side taken into consideration or not, but it's a good point.

Marc (24:53):

Hi, it's Marc again. For more information on how Eficode views software development, please consider to look at our Webinar Series on Sustainable Software Development. There, you will see many things in terms of building software for customer needs, how to structure your MVPs or minimum viable products. Some great tips on using JIRA and scrum tools. And we welcome you to have a look at the sustainable software series by Eficode.

Marc (25:25):

I've heard this so many times where no, we cannot have continuous deployment because of reasons, because you have a compliance issues, or you have hardware, or you have a customer base that has certain kinds of requirements. But that doesn't mean that you can't have the ability to be ready to deliver at any moment.

So I've even seen extreme cases where I've gotten really strong pushback that no, we will not have continuous deployment. We will not have automation in this area. And then I say, okay, how do you deploy? And they say, oh, well, we have this script and whenever we're ready to deploy, then the fields are already filled out, but we change one of them if we need to.

And then we push the button and then it deploys. And I say, okay, so you have full deployment automation under one button? And they say, why, yes, we do. And it's like, okay. So that will count because now you have the sophistication that you have the security to be able to release when you need to, you have the security and as well as the quality and all of these things.

And there's a term that we use a lot in DevOps called shift left and shift left in simple terms means putting responsibilities left in the value chain, which is typically towards the development side of things. But it also has something to do with what Matias brought up about. If you have already approved something for work, then why would you delay it later for some kind of deployment approval board or something else where you've already approved the work to be done. 

When the work's approved to be done, it should get to production as soon as it's ready or the business decision is made. So you can also shift left these kinds of decision-making processes. If you put the test automation and security checking and compliance issues into your pipelines.

And the neat thing about this level of sophistication is when you practice DevOps at a high level, then it means that you can bring junior developers, competent developers who are maybe not that senior in,  and they can look around and they can feel secure to be able to make changes quickly, and they can come up to speed quickly and be productive more quickly where a complicated legacy system that oh, okay, it doesn't have that much test automation. It doesn't have that much scanning and compliance. Then they're afraid to touch anything because something might break. 

Risto (27:51):

And actually this brings us to a whole topic of developer experience as part of the DevOps movement. Maybe it kind of not only what you just said, but when you have adequate level of test automation coverage, then doing changes to a system is less stressful because you can be pretty sure that you're not breaking anything serious when you're changing something in one corner of a software product.

Marc (28:23):

Yes. I agree completely. And oftentimes, just smoke tests and unit test coverage at a high level can tremendously affect the speed that people can enter an organization and be productive, or the changes can be made affecting production and be deployed quickly. And I think that also enables a lot more focus on the value creation side of things. 

Instead of worrying about what the impacts might be of your change, you're actually able, like we're right back to the top of this podcast: you're able to have constant learning and constant experimentation to try new things and get feedback on those quickly. 

Risto (29:08):

Maybe one of the things that actually is also bringing us to the 1, 2, 3, what is DevOps question: we talked about the culture. We talked about what it is about. And one of the key cornerstones in DevOps culture to me is to praying their responsibility and mandate to decide, to as low level as possible. So, who would be the best, better person to say what we should be doing to a single feature than the person who's working on it at the moment.

Marc (29:40):

I like the four-eyes principle where we did this back in another company that I worked in, where two people together, if they agree, they can make a decision on how something's going to work and they can move forward and try it. 

Risto (29:52):

I love it. 

Matias (29:54):

Yes, that probably very much affects how a developer considers the work they do and the impact that can actually have on the work they do and the results. ‘Cause you feels that you're in control of whatever it is you're working on. And you feel that you can have a major impact towards the development and the end results. 

Risto (30:13): So maybe kind of, one of the things that I've been thinking lately is that as we're talking about developer experience and the way we’re doing work, redesigning the way we work into DevOps environment, is that as we add the level of automation. All the kind of, most of the boring stuff that people don't want to actually spend time on is automated. So that means directly that the proportion of the time spent on value-adding work can be increased. 

And the time on the tasks on a day can be reduced. So that is the same principles that we've seen in manufacturing and all the other industries leading up to this point that adds better quality. And that also probably adds also well-being at work when you can consider actually to things that are interesting and that are adding value.

Marc (31:06):

Adding automation sometimes requires increased maintenance due to the complexity of the system. Do you feel that that it's a worthy trade-off that you sometimes have an increased amount of maintenance to do with a system where it's fairly automated? 

Risto (31:22): Maybe I didn't get the question. 

Marc (31:29): Fair enough. I you have an automated system, where you're able to focus on things that actually bring more value and less focus on sort of managing the pipelines for example, but then ultimately you do have the need to manage the pipelines. Do you feel that there's any sort of trade-off between the two or is it just always more automation leads to just reduced work on a system?

Risto (31:55):

Of course, any system requires some maintenance naturally. But my pretty recent experiences is it been all of a sudden that bringing in changes to a rather large system is doable only because of wide coverage, so for automated tests. We are working on one system that any little change somewhere might break potentially something in a totally other corner of the service.

And if you didn't have the adequate amount of unit tests and integration tests and all that done automatically, no one would be courageous enough to touch anything. 

Matias (32:33):

Yeah. You sort of have a safety net. 

Marc (32:35):

There's a very interesting thing here, which is, so many of us remember the Mythical Man Month. If you have up until recently, it was the only book you ever needed to understand how software development processes work. It was written about experience in the sixties by Fred Brooks. It was published in the seventies and everybody remembers Brooks’ law which heavily paraphrased is, what one developer can do in one month, two developers can do in two months. 

Adding more developers to something that’s slow doesn't really make it happen any faster. But Brooks’, other law, the one that I've been more fascinated with in my career is, 'the complexity of a system is the square of its interfaces'. So my belief at the moment is that DevOps is the first movement that has actually started to reduce the complexity of a developer's life. 

Because instead of a developer that might be working on an application and they notice a funny behavior and an API. So they, they fix a bug or they code around the bug in the API. And their application works and everybody's happy. And then along comes another developer that notices the bug in the API and files a bug about it.

And then when it's fixed, they submit their application. And now their application is working. But the other guy who used to have a working application some time ago, now he gets a new bug filed against him because his application is broken. So, what we end up with is this complexity of dealing with systems where changes that we might not even think would affect us, can cause problems that can be very difficult to trace. 

But what DevOps and test automation and DevOps pipelines has done is it's given us the ability to say, okay, when I submit my application, here's some unit tests. When you update your API, you run my unit tests and make sure that your API didn't break my application. So the complexity of the most complicated software that's ever been created in the world now can be reduced to my hole in the pipeline that I submit my changes into with my unit tests and all the tests automation from all the other team members that are working on the software.

And now I get fast feedback to tell me, did I break something or not. Maybe even with unit tests and smoke tests along. So I think we're finally starting to crack this Brooks’ law and complexity. When we now also happen to have the most complicated software in the world. So having good Agile and DevOps practices actually allows you to try things, move faster, and be able to get good feedback on what is actually working and what's not on the system at any moment. 

Risto (35:20):

I totally agree. And I think the complexity is the square of the interfaces. If we are looking at the, like the modern-day microservices architectures and all that, the number of interfaces is growing rapidly. And there is absolutely no way one could actually manage that complexity without the automation.

Marc (35:41):

There's a lot of zeros in that number. When you start to square these numbers of interfaces. Okay. I'd like to thank Mathias and Risto for participating today. This is Marc. This has been the Eficode, DevOps Sauna, little yellow book of DevOps podcasts. Thank you for joining, you can find the links to the social media profiles in the show notes.

If you haven't already, please subscribe to our podcast, give us a rating on our platform. It means the world to us, also check out our other episodes for interesting and exciting talks with that. I thank you again for great company. I say to you, take care of yourself and remember deliver value from your software.

Risto (36:24):

My name is Risto Kinnunen. I have been working in the tech industry for about 20 years now. I focus in people in everything I do. I want to create services that fulfill the real user need. And if we are talking about processes or tool development, I want to make the developer experience and thus well-being as good as possible. Currently, I'm working as a scrum master and project manager in various projects and also as a Team Lead at Eficode. 

Matias (36:48):

So, hi, my name is Matias Merenmies, and I've been working in the IT industry for basically over a decade now in both design and software development. And I really enjoy talking about the tight-knit connection between the two. And I could probably talk about it for days on days. As a software Team Lead here at Eficode, I help serve our developers and project managers and try to coach them in their professional journey.