Navigating value streams and prioritizing feedback

Marc (00:06): It is very difficult to make promises with software, but what we can do is we can create transparency and visibility and help to be able to move value creation forward.  Welcome to DevOps Sauna, season four, the podcast where technology meets culture, and security is the bridge that connects them.  Okay.  We are back in the Sauna.  Hi Darren.

Darren (00:43): Hey Marc.

Marc (00:43): Just a little aside, we were having an interesting conversation before neither one of us used bookmarks.

Darren (00:49): Yeah.  We seem to have this bad habit of trying to build Muscle Map read books, but then I completely forgotten the name of this casting tool, so I had to be reminded.  So it's not an infallible system books, but I don't know, I think it's better than the alternative.  I work on too many different machines to migrate bookmarks in different places.

Marc (01:08): I worked in one company a long time ago, and we all had a kind of master set of Linux aliases that we all kind of referenced in our home directories.  And then I met a real guru of Unix, William Robert Hall in San Diego, and he wrote everything from scratch all the time and I asked him, why do you do that?  He said, well, I never know what machine I'm going to be on, but I know that there's gonna be VI there and I know that I can build muscle memory for all the things I need to do.  But I have a topic today and I've had some good fortune recently to have a couple of events where I've gotten to talk about one of my favorite things, which is value stream mapping.  And I think you've had a course on that, haven't you, Darren?

Darren (01:52): Actually I participated in a value stream mapping exercise and as we were discussing the topic of this, I briefly looked it up on Wikipedia, and the latter was far more informative than actually sitting through a value stream mapping in like the first time where I came in knowing nothing.  So I'm kind of hoping to take out of this some information for the next time I get pulled into something like this, so I'm a little bit more able to understand what we're actually trying to do with the process and hopefully take away some of your expertise.

Marc (02:24): Cool. This is something I've had a lot of fun with over the years.  I always think about when my genesis as a professional software developer was in the nineties, and we all had these enormous whiteboards in our cubes or on our walls, or we even had a room that was whiteboard from floor to ceiling.  It was painted with the stuff.  And we were constantly working to understand the flow of information, not only inside the software but also within the organization.  And, you know, we didn't really know that we were creating value stream maps in those days, but we certainly were, we were understanding how value flows through billing systems or wireless back channels that have all the cellular data in them.  We were looking at how do we take requirements into our team and decide which ones we're gonna work on and which ones we're not, you know, kind of pre agile stuff.  And then fast forward to some years ago when I started at Eficode, my first customer, I was doing some general consulting work in order to help them understand how to take on international markets.  And one of the things I decided to do was to take some instructions that we had in Eficode for a value stream mapping module and to put that into use.  And I brought together people from, I think two or three sets of 10, we did cross-functional developer teams with some quality and security people in there.  And we also brought in infrastructure and we made sure that the infrastructure guys also had some of the consumers of their infrastructure.  And we also tried to look at this from a product management and portfolio management perspective.

Darren (04:03): Okay.  So to me it's like, it seems it's covering these, or it's best perhaps what it's covering these vast subsections of the organization terms like I think mid where I fall down before is having it too focused on one specific goal.  Would you say that it does need a broader focus?

Marc (04:23): I think the interesting thing is always to look at the outliers and the leaves.  You know, you bring together a development team, and maybe you've got a product owner in there, and you know, if they're really cross-functional, maybe you have some people that are more, you know, test special people.  And once you understand that, it's like in so many places, we don't really need our developers to be able to develop faster.  What we need to do is remove all the impediments around them so that they are able to have conversations about how they add value within an organization or within a system rather than, you know, what are the hurdles that they need to go through and what kind of, you know, really tightly bound requirements that they get as input and what kind of pushback they get from operations areas or, you know, management if they make a mistake or what have you.  But even with a basic value stream map, oftentimes we end up with a little bit from product and portfolio management.  There might be a separate lane for how teams operate with the input and output information.  You could think of things like backlogs and refinements and planning poker or whatever estimation that they're doing, you know, kind of on one side.  And then on the right side you can think in terms of how do they handle deployments.  You know, I sat in a meeting where a person said, I'm responsible for the approval of all software that goes to production.  And then I saw that person turn white when the developer said, we only really tell them if we think there's gonna be a problem, otherwise we're delivering to production every day.  So we oftentimes do, because the first time I did these value stream maps, and in many cases there is one area on the whole thing where value is actually created and that's somewhere between checkout and commit, or someone might put a develop sticky note in there, which is for, you know, face it for most software organizations is where most of the value is created.  What we try to do in a basic value stream mapping exercise is we try to understand what are the waste areas we can think of lean waste, like overproduction or inventory transportation of things, looking for bottlenecks.  One thing like inventory that's always interesting is I might ask a side question that's not necessarily related to the value stream map, but oh, this backlog that you're refining, how many items are in it?  Yeah, okay.  So you're a team of 10 and you have 3000 backlog items.  Okay.  So that's a potential bottleneck, but let's go, let's take a step back and say what are we trying to actually achieve?  We're trying to understand how does the value flow through an organization?  Where are the bottlenecks and the handovers for that?  And where is the best bang for our buck if we invest on the process and the tooling in order to be able to reduce those bottlenecks?

Darren (07:15): Okay.  So it seems to be about making basically a priority list for which areas of the, basically, as you say, the most valuable, but also these, you've talked about waste, and you talk about the thing that caught my attention obviously is the guy who inserted himself as like a mandatory blocker and was then surprised that the development team circumvented him.  So these kind of things to me are kind of opposite.  If you're looking to kind of expand on the things that give you value or remove the things that don't, where do you put your priorities?

Marc (07:51): So for every step in the line, we look at the time that you're actually spending creating value.  So if you are a well-oiled scrum team and most of your tasks are one or two days, your develop step might be one or two days.  If you are a waterfall organization and you have a user acceptance test of two weeks before you're allowed to release, then that's a quite big waste area or wait time.  So we look in terms of what is the value add time, what is the waiting time, and are there handovers in these types of areas in order to understand what we can optimize?  And then we look at, if we have like a poll request, for example, or you know, a code review and people are on the average, you know, spending a couple of days waiting for their code to get through code review, then we find that, okay, perhaps you don't need to do quite as many reviews as you thought you did.  A lot of people are automating pull requests these days in order to reduce the amount that's necessary for people to be able to focus on big changes or important changes rather than just everything that kind of goes through.  So it comes in both sides.  So we're looking for opportunities to remove bottlenecks and handovers, and we're also looking for areas where we can kind of help things along and prioritize.  I like how you put this priority area.

Darren (09:18): It's kind of interesting, though, because I don't want to be dragging this back to security always, but you did invite a security nerd to come and sit with you and have these discussions.  So to me, it actually sounds a little bit like your threat modeling a process rather than a set of tools.  So you're kind of identifying your areas of weakness to try and minimize them.  To me, it's starting to sound like quite a similar process, at least when it comes to waste and this kind of, these delays, these bottlenecks perhaps not for prioritizing where you actually create value, but at least for the, let's say negative reductive side of it, you are looking at it kind of like a threat model against a process.

Marc (09:58): I like this a lot and one of the things that becomes really obvious, like if I talk about those first few value stream maps that I worked on, security was an afterthought.  It was in the lower right corner of the value stream map and Darren is smiling from ear to ear.  Here we have a video on the back channel.

Darren (10:16): It's more a disbelieving laugh at this point, I would say.

Marc (10:19): Yeah, or hysteria I think is also one of the terms.  So when someone with experience or a reasonable understanding of how software should be created and if you look at a value stream map and you realize that security is checking things at the end of the line and you realize, okay, could we have those people involved early and make sure that the practices and the tools are there that are necessary in order to have security as a constant presence rather than something that is just a hurdle in order to get a release out.  So that's one thing that we see a lot.

Darren (10:56): Yeah, I can imagine you do.  That's something that I unfortunately see a lot too. 

Marc (11:01): Then an extension of this, you know, testing, when we talk about shift left, and there's been a couple of interesting, there's a few of these, you know, DevOps is dead kind of shift left is for suckers people out there.  And when I hear about, you know, well shift left doesn't work because then I realize, okay, we're talking about a different shift left because shift left to me is making sure that while developers are expected to have more responsibility, for example, thinking of security early, thinking of deployment early, taking responsibility for things that we're giving them the tools and the presence and the processes in order to be able to take those, not just pile more responsibility on.  So by mapping out how people are actually working on a day-to-day basis, it oftentimes gives a really clear indication to the people that are already there.  And if there are others who are facilitating this from the outside, like you and I might do to say, hey, look, you're not involving testing until late in the process.  You're not involving security until late in the process.  You have opportunities where you could start looking at things like behavioral driven development or acceptance, test-driven development, or at least inviting the people who are responsible for quality into refinement stages of things or into portfolio management stages of things.

Darren (12:24): Okay. I mean, I do think you're right there, but the big challenge, I mean the big thing you see there is if you do that, you can obviously cut down the time towards the end, but again, my focus was to try not to drag this conversation towards security, and we're heading in that direction again.  So if we can, I'd like to sort of drag it back to the thing you discussed because you talked about the idea of waste management, and I'd like to kind of go back to that a bit if we can because I'm kind of curious to hear how you've handled it.  Because you talk about waste management and it's the idea that there are bottlenecks, there are these restrictions, there are places where things are going as smoothly as you want.  And I'm just curious how you approach those from a human perspective.  Because basically from what I see there is that those are the places where people are not doing their jobs correctly.  And I'm wondering how you've approached that.  Do you think that's an accurate statement?  And then as a consultant, how have you approached that so that you're not just the guy from the outside telling people that they're doing everything wrong?

Marc (13:30): Wow. There's a lot to kind of take back from there.  So I would not go so far as to say in most of these cases people are not doing their job correctly.  I think in terms of, you know, the system has created a set of pathologies that people have not been able to overcome sometimes by the sheer mass of those.  And you know, one of the things the, you know, your successes of the past are the things that kind of doom your future.  We've always been able to work in a certain way.  So why are we not able to continue kind of working like this?  If I think in terms of the lean wastes, I can go directly to one really interesting idea, which is global logistics company.  And they have a really great state of the art DevOps commercial organization that handles things like customer portals and websites and mobile applications and all of these things to be able to get to their services quickly and easily.  And they do 50 releases a year.  And where do those releases connect to?  Well, I said a global logistics company.  What do you think the backend of that is?  It's an ERP system now that ERP system is large and slow and difficult to change and the pure definition of legacy and they do three releases a year.  So this is the first type of lean waste over and under production.  It also includes weighting over under processing excess inventory, all of these kinds of things.  So one way to look at something like this is how can you align the amount of release capability of the backend with the front end?  And one way that we've been able to do by laying these things out and looking at all of the steps is understand that this thing in DevOps and software development called feature flags where you start work and you create a flag and you call that new work and then you do all of your new work and maybe you share that new work flag with your team and new work is turned off in production, but we can start delivering our code already and new work is turned off in production and then we can do a canary release or a blue green release and turn it on for a specific area.  We're getting a little way away from value stream mapping, but there is an idea here, which is how do you align the front end and the backend, you teach the backend ways of working that the front end has already mastered.  So this is one kind of pure example where we've been able to look at an end-to-end process by mapping it from one end to the other and realizing, okay, there's a huge bottleneck in the middle where one end is waiting sometimes, you know, for 20 releases before they can get a change through the back end.  So how can we pull these together and demonstrate that, okay, if the front end starts a new work flag, they can share that with the backend and the backend ERP system can actually start building all of their work based upon this new work type of flag.  So this is one idea.

Darren (16:36): So it sounds a little more like the idea of waste management, maybe the labeling of waste management is actually not ideal. Because to me it sounds like it's more like refining the process.  You're smirking about something now.  So I'm curious what your thoughts on that are.

Marc (16:51): Well, you keep talking about waste management and I like your earliest definition I think was your first instinct was priority, right?  So how do we help organizations clearly understand where the priorities are in order to maximize the efficiency of their value creation?  So I'm gonna jump into a couple of examples that I've kind of recently learned.  We have a series of breakfast events that we do around the world and I did one of these for value stream mapping in London at the Gerkin about a month ago.  And I created a synthetic value stream map with people from about 10 companies that were all in the room at the same time.  And what we did is we looked at where do they get all of their inputs?  They get them from customer service, they get them from sales, they get them from market research, they get them from doing customer surveys, they have internal portals for innovation.  They have all these different kinds of inputs and then how do they triage the inputs?  They have product management forums, they have portfolio management forums, they have product owners and prioritization and all of these types of things.  And then we get down into how the teams work.  And they have, some of them are doing scrum and they have sprint planning, some of them are doing Kanban and they're constantly refining their, their backlogs and priorities and this level of things.  Then we get into software development where we have architecture and we have design and we have actually doing the development and we have scanning static dynamic.  We have all of the different types of deployment.  We have infrastructure as code, another swim lane that the development uses and they may contribute to and on and on.  Development leads to testing, leads to security.  So we end up with maybe 10 different swim lanes.  So I took everybody's pathologies and best practices and I added a few of my own.  And we had one of those, you know, approval board kind of things in there.  We had user acceptance testing and a bunch of different things.  So I sent people out on a break.  So we spent about an hour on that and we ended up with maybe 60 or 70 items.  I sent people out on a break for 15 or 20 minutes and when they came back I had taken most of them and aligned them in a single stream.  And the single stream started with customer communication and then it went on through portfolio management to look at what their opportunities are.  And it went through prioritization and backlogs and refinements, design, architecture, developments, and then all of the different kind of development stream going over towards deployment on the right.  And then I started from the right, working my way backwards and saying, okay, you've got some obvious bottlenecks that are not state of the art user acceptance to testing this approval board.  We keep talking about having long running test cycles in the end where it might take several weeks in order for full on system testing for a release, a few more things that are not exactly modern, like code freeze type of things.  But then I started to learn something new when I looked at how we had aligned the value stream somehow just to the left, we had ended up with design just to the left of development.  We had ended up with design and I thought back to my first real scrum team from mid 2000s and we had a designer on the scrum team.  And what they would do is every time something new would come up, they would first do a quick design, share that with the team, talk about the technical feasibility, share it with the product owner, which was me, talk about, you know, how well does it fit, you know, even at the top of a sprint, and then the developers would start working on it.  So what I realized is, okay, we have just doubled the amount of value creation from my previous work in value stream mapping because if you have a designer working directly with the developers that can take items before they start being developed, create designs for them and then show them back to the representative of the customer, you are instantly actually creating tangible value because you can test that, you can test the customer's expectations, you can find out, you know, are you even working on the right thing, you can maybe prevent some waste, you know, unnecessary motion building something that you shouldn't be building to start with.  And I thought that was really cool.  So I took that design part and I linked it all the way back to the left, the very beginning of the value stream where the customer was.

Darren (21:31): Okay.  So more and more of these topics we talk about seem to come back to these immediate feedback loops.  Like now it sounds like you're talking about having the design process immediately able to be commented on to be refined before the work got.  So yeah, I can see that putting out a huge chunk of the development process and you, I guess saying that's not the case in most situations or it's something that happens later.  So I'm kind of curious about that from a perspective of why is this feedback not being given early?

Marc (22:07): I can think about, I did some work with a game company and it was an industrial game company, not necessarily like a modern, you know, super cell or that kind of game company, but I did something for a kind of an industrial game company and the developers would get graphical designs and kind of as input and those would evolve over time and they would be basically kind of two moving points in time.  The designs would be evolving and the developers would be evolving, but it kind of was thrown over the wall towards one another.  The designers would throw over the stuff that the developers would continue working on their enhancements to the engine, the gameplay and all of these types of things.  And then the designers would look and update designs as they were doing as part of their, you know, iterative or incremental development, but also trying to understand how those things look when the developers would work on them.  So they were kind of asynchronous.  Whereas if the designers are working more closely with developers, these things can be understood earlier cutting out a chunk of development as you described.  And I'm usually not seeing this, it's even worse in application development a lot of the time where the designs are done thrown over the wall to development, the developers start trying to make the things work, and then, you know, you've also got localization and internationalization and lots of other kind of asynchronous paths there.  So one of the things we do when we do a value stream map is we can look at these things and say, you know, would it be a benefit to you to be able to have design, you know, between the customer and the developer and be able to get feedback from both sides and be kind of a focal point there.

Darren (23:46): So yeah, it's again all about getting this feedback that's, seems, seems to be the key thing we run into that if we have developers who develop outside of this kind of area of feedback, then it's going to cause technical debt, it's going to cause development delays if they head in a direction that's not consistent with what they're supposed to be, they're supposed to be developing

Marc (24:11): The classical part of DevOps feedback to me, there's two things. One is, are you able to test in production to understand if the customer is going to behave in a way that you expect them to.  You can cut out some of this with design, you can do some of this with pilots.  You see a lot of this with like the hyperscalers in social media and, and you know, other kind of hyper scaling applications where they test things in markets in small areas with certain demographics maybe.  And then they use this to understand should they continue on this development path or not, or should they continue with this business model part or not.  So there's this area and one thing that I didn't get in yet, which is that all of this refinement work that we're doing in an estimates that we're doing in our scrum teams today or in our software teams, if we're not running Scrum we're doing com or something else still we need to refine our backlogs and we need to give constant feedback back to the roadmap in product and portfolio management in order to make sure that they understand what is the capability to deliver the things that are on their roadmap and understand that, you know, it's very difficult to make promises with software, but what we can do is we can create transparency and visibility and help to be able to move value creation forward.

Darren (25:30): Okay, I see.  So yeah, I feel like I'm starting to get more of a handle on this as a process.  Like, to me, it's a way of breaking down the processes to, well obviously tell you where you're supposed to be working, where you should be pulling your focus, and from there, I'm kind of curious, where you kind of find these areas, you've already said that you mostly think there it comes to be one area which is somewhere between the, basically in the dev area where the development work is actually happening.  But what I'd like to know from you is like good areas of where you do find this value because we've talked before about people's problems not being unique and people's problems not being as special as they think they are.  And I'm curious if that rings true for value streams.  Will people's value streams look similar?  Do you know of good areas where prioritization should happen in general?

Marc (26:32): It's a good question, and generally speaking, testing is not involved early enough.  Design is not coupled with development clearly.  Architecture is not an emergent property.  It's either too early or completely disconnected from iterative or incremental software development, which actually leads to another finding.  Actually, Darren, in that workshop, I tripled my understanding of value creation because I thought one of the definitions of DevOps is the ability to continuously deliver a value and environment of psychological safety and experimentation.  And I thought, okay, so with value stream mapping, we should be increasing the level of psychological safety because it's not about people not doing their job, but it's about where process bottlenecks have been created over time that are limiting people's ability to have the right conversation about how they create value and utilize their domain expertise.  So that's kind of an impact on psychological safety that can be positive when we start to look at how value is actually flowing and created.  But this experimentation part kind of got to me, and I realized, okay, so what I did in this exercise is I put experimentation to the left of architecture well before even sprint planning, design, or development. And I did it to make a point, which is to say that sometimes before you can refine something, before you can plan it, before you can give real feedback on it, you need to be able to try something, and production is one of the best places to try something because you have everything there.  You have your customers, you have all of your application, so maybe you can do little test features in production and understand does the customer potentially work in the way that you would like them to with a feature.  So what I did is to the left of architecture to the left of design development and sprint planning, I put experimentation, and then I drag a huge line all the way to deployment and back and said, okay, so if we can actually in a safe manner, in a very controlled small, attack surface is the wrong word, I've been looking for something for this, but in a very small limited way, be able to test ideas in production through experimentation, then I think this gives a lot of opportunity for value creation and understanding if you can actually get customers to behave and to do the things that you would like them to do.

Darren (29:08): It's quite an interesting idea.  I'm not sure it's one will be able to sell very far in the idea that production is the best place to test because I think everyone's a bit cautious of that.  But I do kind of see the benefit to it.  Like obviously exposing your experiments to the best testing audience you can is ideal.  I think people will be cautious about their business based on that.  So it's easier to sell in like open source projects, I'd feel where you can have your experimental branches and you can have your kind of less, I don't wanna say less responsibility, but kind of less obligation to sh their stakeholders and such.

Marc (29:49): And this doesn't mean just going out and hacking in production, but it does mean in a controlled manner behind feature flags, you know, being able to send out a canary and understand if you can actually do the things that you were liking to do.  And so I felt like I had tripled the amount of value creation from my previous understanding of these.  So how about a summary, Darren?  Would you like to ask me a few questions?

Darren (30:13): Sure. I think the first one we have to start is to just quickly put into words value stream mapping.

Marc (30:20): So value stream mapping is understanding, from customer inputs through deployment, how you create value in an organization with a focus on making sure that you are able to prioritize areas for improvement and eliminate as many bottlenecks and wastes as you can.

Darren (30:38): Okay.  And how would you say it leans in towards prioritization?

Marc (30:41): It leans towards prioritization because if you actually have a flow state, then there's only one bottleneck in that flow state that, if you open that up, it allows everything to flow faster, and then you need to move to the next one.  So without clearly understanding where the next bottleneck is, you may over optimize something that doesn't actually have a positive impact and completely miss where the bigger bottleneck is.

Darren (31:08): Again, how much focus do you think should be put on waste management versus value creation or something else?

Marc (31:15): I think there's a balance here, like many places.  I like your words prioritization, helping an organization to understand where the priorities are for their continuous improvement.

Darren (31:26): Okay.  And how can you use value stream mapping to enable, encourage feedback and start to break down silos?

Marc (31:33): By understanding where the handover points are and where the key areas are for having feedback or amplifying feedback to use Gene Kim's words.  Are your backlogs and refinements feeding back into portfolio management?  Are developers able to get to limit from how the applications are used in order to be able to improve their development?  Are there clear feedback loops that go back to customers to understand?  Are you building the things that the customers actually want to need and will use?  Awesome.  Thank you, Darren.

Darren (32:07): Thank you, Marc.

Marc (32:08): I look forward to the next one.  I hope I showed a little bit of expertise in something this time.

Darren (32:12): Yeah. I'm pleased we're not just talking about security all of the time.  I felt like I've been dominating the conversation in those episodes, so I think it's important people hear from you as well.

Marc (32:23): All right. But you make me feel secure, Darren.

Darren (32:25): Well, that's what I'm here for.

Marc (32:27): Okay.  Thank you, everybody for coming to the Sauna once again with Darren and I and we'll see you next time.  Bye.  We'll now tell you a little bit about who we are.  Hi, I'm Marc Dillon, lead consultant at Eficode in the advisory and coaching team, and I specialize in enterprise transformations.

Darren (32:49): Hey, I'm Darren Richardson, security architecture Eficode, and I work to ensure the security of our managed services offerings.

Marc (32:56): If you like what you hear, please like, rate, and subscribe on your favorite podcast platform.  It means the world to us.