Interview: How submarine onboarding can enlighten SW companies, with Andy Allred

Andy (00:04):

On a submarine, we have the situation once in a while that we need to move some equipment to do some maintenance and we need to deconstruct a cabinet. And when we put it back together, we think, "Hey, let's just put it in this way." And then later there's another system which we can't maintain because we just put that cabinet in the wrong place or this beam over here, or this support bar over in the wrong thing. So the maintainability of a system is always important to think about.

Marc (00:38):

Hello, and welcome to DevOps Sauna. So this is the Sea daddy DevOps Chronicles, episode one. Getting your fish. My name is Marc Dillon. I'll be your host. And I'm here with Andy Allred.

Andy (00:51):

Hello, Marc.

Marc (00:52):

Hello, Andy. I've heard you have some interesting things in your past. Why don't you tell us a little bit about yourself and how you got here?

Andy (01:01):

All right. Well, I do have a bit of an interesting past, I think. I started my career in nuclear-powered submarines. I spent 10 years in the US Navy doing electronic warfare on fast tech, nuclear-powered submarines.

Marc (01:15):

Wow.

Andy (01:16):

Yeah. After that, I moved to IT and worked in telecoms for a while. But being submariner is always the core of my heart.

Marc (01:25):

I bet you went through a lot of interesting experiences.

Andy (01:28):

Yes, indeed. And some of them I can even talk about. But submarines are really special, in that they are extremely complicated. Well, complicated doesn't sound like such a complimentary word. So let's call it complex and specialized instead. Submarines are comprised of 4,500 PSI air systems, 3000 pound hydraulic systems, a three-phase electrical generation system, water purification, oxygen generation, CO2 removal systems, satellite communication systems. Of course a mobile nuclear power plant. It's just all to support the intelligence and weapon systems that need to be available anywhere in the world.

Marc (02:08):

Oh my gosh. But are they turbocharged?

Andy (02:10):

They do have a turbo indeed. Yep.

Marc (02:13):

Wow.

Andy (02:14):

So part of our backup electrical generator is a diesel and we have a turbo in the diesel. Summaries are also unique in that there's no concept of reinforcement. You can't get extra people onboard in the case of an emergency, which we call casualties. Any type of fire, flooding, electrical, whatever, you really have to put it out and take care of it very quickly before the pressure of the sea pushes water into the people tank or all the oxygen has gone from the fire or whatever. So the crew has to be able to deal with anything and everything effectively and immediately.

Marc (02:46):

So, truly you've got a fully cross-functional team there that's capable of taking care of anything on the summary.

Andy (02:52):

Yes, of course, everybody has their specialization. But in addition to that, you have to be generally available to work on anything, if the worst case would happen. When you get to a submarine, you're really not capable of helping out. So we have a not very nice term for new people. They're called a NUB. A Non-Useful Body. This is of course not a very complimentary term, but it is somewhat inspiring and motivational. I don't want to have this label anymore. What do I have to do? Yeah. So the term comes from the idea that if there is an emergency of any type, you don't know what to do. You are not able to be helpful. You are simply in the way, so you're not very useful. So the way to be useful is to learn what to do in case of an emergency anywhere and be able to help in any situation.

Andy (03:45):

So when you have that, then you're qualified to get qualified. The first part is you have to go through all the systems on the submarine. First... Well, not first, you can go in any order you want, but you need to go through the hydraulic system, the air system, the plumbing system, the ventilation system, the electrical systems, the nuclear system, the propulsion system, pretty much everything. And you have to learn them inside and out. So you have to be able to draw the system and show this is where the water flows. This is where the hydraulic pressure is. This is where all the valves are. This is where the electrical switches are, pretty much everything.

Andy (04:19):

So, when you get checked out on a system, for example, the hydraulic system, you have to be able to point out where all the hydraulic valves are on the ship, where the hydraulic pumps are, where everything belongs to. You have to draw it. You have to find them. You have to describe what would happen in case of this being closed or that being open or whatever. Then when you are able to do that, you get checked out on the one system and then you're able to go to the next one.

Marc (04:45):

And I guess you never know where you're going to be in the submarine if there is a casualty or an incident. So then you have to know how to deal with the things in departments that might not be your own.

Andy (04:58):

Exactly. So if I happen to be walking through the engine room and there's a casualty there, then I need to be able to take care of it, even though my specialty is electronic warfare. So I still have to know how everything works back there. So you really have to learn absolutely every system on the boat. So you iterate, iterate, iterate, and go through everything until you're checked out on every one of the systems and you then are qualified on that system. Then that's kind of like a very nice, deep dive into different sections, but you still need to be able to put it all together. So once you are checked out on all the systems, to put it all together, you go to what we call a qualification board. The qualification board we'll go through and kind of ask you, follow-up questions on each one of the systems.

Andy (05:46):

They'll say where's VH one, where's TD five, where's an electrical bus, whatever. And you have to know them all. They just kind of spot-check. But then they put it all together and they give you kind of practical situations to solve. They will say if you are standing in this spot and a fire happens, what do you do? And if you're standing in that spot and there's an electrical malfunction, what do you do? Then they also make sure that you understand the entire submarine as a system and how things interact with each other. So you could get a question like, you are a molecule of air, how would you make this light bulb come on?

Marc (06:20):

Oh my gosh!

Andy (06:22):

So then you had to be a little bit creative and think, okay, I'm a molecule of air and I'm floating out in the ocean somewhere. I get sucked in through this snorkel mast. I come down into the ventilation room and you have to name the valves that you're going through. Then you get down the diesel generator room. You go through the diesel generator, get converted to mechanical energy, which then gets converted to electrical energy. Then you go through this bus, through this junction, through that switch over to here through this distribution panel, come down to this switch, and then you go to the light and make the light come on.

Marc (06:55):

Wow. That's an end-to-end use case.

Andy (06:58):

Very much end-to-end, and very intensive. And you'll get a few of those. When you finally get through everything, this whole process, the absolute, very best case you could get qualified in three to four months. Usually, it's around six to nine months. Sometimes up to a year. But six to nine months is the general time that people use.

Marc (07:19):

Wow. I mean, talk about this is literal onboarding. We talk about onboarding and companies and it usually means something like you get your badge and you know where the coffee is and get your laptop hopefully in the first week. But your onboarding actually means really understanding the whole end-to-end and holistic view of the systems. And if a sailor can do this... I'm not disparaging sailors, but if a sailor can do this in three to four months or at a normal case, six to nine months why in IT systems, do we not do more of this? Why would we not reduce our bus factors? Or actually is it increase our bus factors? Like if somebody were to not show up to work and there's a problem, then many people should be able to solve that kind of problem.

Andy (08:06):

Yeah. Exactly.

Marc (08:07):

If you could do it at the scale of a nuclear submarine. I would suspect that some of our application providers could probably also do this.

Andy (08:14):

Yeah, of course. On a submarine we do also have, this is where you can get the coffee and this is where the toilets are and here's your workstation, but that's like day one kind of stuff. The real onboarding starts after that.

Marc (08:26):

Wow. And just kind of thinking about this, Andy, it's like, if I think of just the qualification board, think if you took an average IT organization and you took five of the top experts, and as part of the onboarding of new personnel, they said, okay, you're going to find out that if there is a bug in this part of the system, you need to go find out how that you would trace that from end-to-end, or if a customer is getting this kind of problem, or if there's this kind of... The last system down problem that we had, you're going to figure out how to trace that. And you're going to come back and you get fully onboarded when you can describe to us how these end-to-end use cases actually work.

Andy (09:06):

Yeah. Yeah. So after my time in the Navy, when I moved to IT, I kind of just intuitively or instinctively, I guess, did this, that whenever I came to a new company, I tried to figure out how everything worked together. And I just assumed that everybody else did this because why wouldn't they. Isn't that just how things are done? Then I realized later that, no, this isn't how things are done. This isn't normal. And it should be though.

Marc (09:35):

I agree. And the next thing that comes to mind here, like from a DevOps perspective is the value streams. And it's like how many people in the IT organization, we do assessments at Eficode all the time. And we ask people, do you feel like the work that you do contributes to the company strategy or contributes to the customer value? And a lot of people say, "I don't really know. I don't really know." So like understanding how customer needs then flow through, we call it the fuzzy front end in IT sometimes. How requirements get made, how they get formed into tickets that developers then are prioritizing, breaking down, committing code for, and then delivering which ultimately results in customer value. That whole stream is another component of the system that I think many people don't have exposure to. And shouldn't that be part of the onboarding?

Andy (10:28):

Yeah, exactly. And if we think of like a nuclear-powered submarine, you definitely don't want me doing maintenance on the nuclear plant itself. I'm woefully unqualified for that, but I do understand how the basics work. And if there's a problem, I can say, "Oh, hey, over here, this is the part the expert needs to look. This is the team we need to call. This is the person we need to talk to." If we could do that in IT, that pretty much everybody in the entire organization can say, this is the kind of problem. And as it flows through the whole system, like the molecule of air, as you go from, here's the requirement to here's the use case, to here's the pit of code, to here's the automated testing to here's the release pipeline. And people understood that at least on a basic level, then they know who to talk to, how the value is added, how things should be working.

Marc (11:22):

Really, really powerful stuff. And I think that it kind of shows that none of these are really new problems, but they're problems that are solved differently in different disciplines, even though they could be applied across many disciplines.

Andy (11:36):

Yes, this is something which has been solved a couple of times, and there's good processes for it that people use. And we easily think that, well, that's an old-fashioned process. It doesn't apply for us because we're high tech and we do things differently. But the theory behind it is still very much applicable and finding that this is how we can use the lessons learned in the other use case and apply them to this use case is where we can really add some value.

Marc (12:04):

Really cool. So, Andy, what other kinds of valuable lessons can you share with us from your experience on the nuclear Submarine?

Andy (12:14):

Getting qualified is never easy. There's a lot of systems to learn. So we give you help. When you get to a submarine and you're a NUB you're usually assigned a sea daddy. A sea daddy is the person you can think of it as a mentor or as a coach or as an advisor, or there's many different ways to look at this. But it's the person who's going to help you learn all the systems. It's not going to be your boss. It's not going to be the person who decides your vacations or your work schedule, or even your daily tasks, but it will be the person who can help you know that, okay, this is the person to talk to for that system. And hey, you need to focus over here before you go over there and just generally give advice. And they're called a sea daddy, and then the NUB is the seapap.

Marc (13:01):

All right. So this is the person you said, it's not like your manager, but mentor or coach or advisor. So what kind of person becomes a sea daddy? Is it someone within your team? Might it be from another team? Is it like the group of people or is it random?

Andy (13:16):

It's somewhat random. And it's just based on when somebody new comes to the submarine and needs a person, needs a sea daddy, then who happens to be available? It might be somebody in your own team, generally, it's somebody from another team. And that just helps give a little bit different perspective. So, if I have a sea daddy who's from my own team, of course, I'm going to be looking at everything. We are going to be looking at everything from the perspective of our team. If I have a sea daddy from a different team by definition, I'm kind of able to look, or I'm steered to look in a different perspective and from a different team's point of view, which helps see the overall picture much more easily.

Marc (13:57):

So, the simple fact that there's an experienced person who is assigned to you to be kind of the go-to person for whatever kind of questions that you might have, it's beyond just the technical and qualification, or is it lots of different things?

Andy (14:12):

It's lots of different things. So it's mainly focused on the qualification and the technical understanding. And if you say, "Hey, I don't understand how the trim system works." Well, go talk with this guy and in seven hours that guy's going to be on watch. You can sit next to him and ask him questions while he's doing stuff and things like that. But he's also just kind of a general overall, "Hey, how do I check that my pay was correct? Who do I need to talk to?" And just a general, this is your advisor friend inside the organization who can give advice on pretty much anything on a submarine or the Navy.

Marc (14:48):

So, I think many organizations, they do try to have some kinds of this role, but simply appointing a buddy or a mentor or a human when new people join an organization to be the go-to person for whatever that person might need, they might not know, but they might be able to tell you who to talk to or direct you in a direction to get solutions to your problems. But we find in IT many times that people have a feeling that they should have all of the answers, but although you can qualify for something really complicated, like a nuclear submarine, you'll still never be a specialist in every single system technology and pipeline. And that we deal with in an IT organization on a daily basis. So having somebody who is assigned to keep you from getting too stuck for too long is a wonderful advice. Yeah.

Marc (15:41):

So you described lots and lots of different kinds of systems. So, how do you have a systematic approach towards these kinds of qualifications, or how are these things constructed? What's a system engineering look like for a nuclear submarine?

Andy (15:57):

So when you, when you get there, you get what's called your qualification card. And basically, it's a checklist of all the systems that you need to go through. And then some kind of summary reviews. So all the water-related, the plumbing, the trail, and potable water, all the water-related are in one section and all the electrical systems are next section. So you're given a checklist with pretty much everything you need to learn. But then as you go through it, you start to just figure out that there are these kinds of use cases, these kinds of areas where certain things happen. And you question that, okay, who normally works here, what normally happens here, and what systems are involved.

Andy (16:38):

And as you start walking around the submarine and interacting with different people, you get different use cases come to mind, and then you just start to think that, okay, if I'm down in the torpedo room, the torpedoman has this type of job, does this kind of thing, what kind of systems is he interacting with? And then you start thinking about it from that point of view, and it can open up other questions that you need to investigate and figure out.

Marc (17:04):

A kind of a new thought for me, which is that when you look at the system, you look at it in terms of how the roles interact with the system. So we talk about like customer journeys or interactions, but what's the developer journey or the QA journey or something within an organization, in addition to how the value flows through the system?

Andy (17:29):

Yeah. So we get the technical checklist that these are the things you need to learn, but then you start interacting with the humans. And I have come to the conclusion that the people are the system. So we don't have systems in IT or on a submarine that exist just to exist, but they serve a function which is a person, a human needs something done. And what systems are enabling that thing to be done.

Marc (17:56):

I think that in IT we look a lot at other kinds of industries. Like we've learned a lot from manufacturing, and your scrums and your conbonds and things are systems that came from different kinds of needs and then were applied towards IT. This kind of feels like another angle of those.

Andy (18:15):

Yeah. So if you think about the typical Agile scrum that the people and process over tools and whatnot, this really fits into that.

Marc (18:25):

Yeah. The people and interactions over process tools. The heart of the Agile Manifesto. It reminds me of the manufacturing or the engineering differences between German automobiles and Japanese automobiles. And one of the understandings that I've gotten is that the Germans look for the best component for every part of a vehicle. And the Japanese have a bit more holistic approach where they look at the best component for the system. And the system also includes, of course, the other components that is connected to, but it also includes the maintainability, the manufacturability can you even reach the thing to be able to update it or not? And this sounds a lot, like some problems that we have with spaghetti code and monolithic architectures and things in it today.

Andy (19:23):

Yeah, exactly. On a submarine. We have the situation once a while that we need move some equipment to do some maintenance and we need to deconstruct a cabinet. And when we put it back together, we think, "Hey, let's just put it in this way." And then later there's another system which we can't maintain because we just put the cabinet in the wrong place, or this beam over here, or this support bar over in wrong thing. So, maintainability of a system is always important to think about.

Andy (19:57):

And just like with your analogy of the Japanese automobile, if you have the absolute perfect system or absolute perfect component, but you can't maintain it, or it doesn't work well with the things around it, it's much harder to change. But when you look at it as an overall system, that how do we maintain this system as a system, instead of components individually, then maintaining the system becomes easier. Because, each individual component may not be the absolute world-class best one by every measurable standard in that narrow perspective. But it can be the best one to interact with the systems around it. And then when you're maintaining it, you're maintaining it as part of a system instead of as a component.

Marc (20:41):

Yeah, this maintenance is an interesting thing in software where technical debt or basically problems that are left for later and then continue to accumulate is something that many companies face. And then it's like, well, how much of the technical debt can you even identify and try to work through and refactor. Where, instead, constant review of systems and this constant maintenance, if the maintenance is something that's built into the system, then we don't let things get to a bad state. Instead, we try to keep them somewhat fresh. And I think there's so much holistic approach, Andy, and the way that you describe things. Like imagine companies that are doing not only regular maintenance, but whenever someone is on board worded, those people need to look end to end through the whole system and understand how all of the different parts work together. Imagine how much more business agility a company could get. The ability to deliver value quickly for their customers, if everyone that comes in understands how everything works end to end, understands how all the value flows and also participates in constantly updating those systems.

Andy (21:51):

Exactly. And we hear a lot about empowering the developers and empowering the different teams. And usually, that translates into let's let these teams make their own decisions and pick their own tools and decide their own processes and they can do what they want because they know what's best. But I think that the real meaning of empowered is if they understand how the overall system works, how value is created in the company, not just by their team, but the overall process, they see how their process fits into the overall system, the overall company, the overall organization. Then they're able to choose that, hey, this is actually maybe a newer shinier fancier tool, but this other one fits in with a process that this team, that team, that team, that team are using. So let's pick this one instead. And that adds more value to the company that makes the process easier.

Marc (22:48):

Yeah. There's so much wisdom in these words. Empowering teams, yes, it's way beyond just allowing the teams to have their own process and tools. But I love this idea that the more you know about your work, the more empowered you are.

Andy (23:05):

Yeah.

Marc (23:06):

The more transparency, the more empowered.

Andy (23:09):

Exactly. Because we don't necessarily want teams to be empowered to pick their software tool of choice, but we do want them to feel empowered to add value and do their breast, and bring their A game to the job and see that have an impact.

Marc (23:27):

And then to take this empowered a little bit further, it's like I've seen a lot of fragmentation of tools and process and ways of working in companies that used to think, well, yes, we want to empower our people. But what it means is that each team can operate on its own. Where instead it's like having this clear transparency plus the ability to solve interesting and complex problems. I think that's really empowering as well.

Marc (23:52):

It's like if the team is trusted so well that here are some raw customer requirements, can you actually, and safely quickly create some things that we can test with the customers and then decide if we want to continue this or change it or do something. Instead of this constant push to build new shiny features without taking care of the technical depth, without taking care of the rest of the systems, onboarding people directly how many times I talked to somebody and I say, "How was your onboarding?" And they said, "Well, it was into the deep end of the poll." Or, "It was trial by fire." Or something like this.

Andy (24:28):

Yeah, exactly. So if you think about the onboarding as getting people that they can get their hands on a keyboard, that's one thing, but getting them to the point that they can actually add value and do things, that's a completely different level of onboarding. And when we tie that back to our discussion about empowerment, if we want a developer to be empowered, to pick how the Git flow is going, that sounds really nice. You can choose how you're committing stuff into the Git repository. But if we instead want to empower them into, this is how you can add value into the overall chain, it's a little bit different level of empowerment from kind of more of a systems empowerment instead of a component empowerment. But the only way you can do that is if they've been onboarded and really see what the overall system is doing.

Marc (25:19):

What you just reminded me, Andy, with this systems thinking and value streams and thinks is that, if you look at a value stream, there is generally one bottleneck that is greater than the others. No two things are ever completely equal, right? So if you have a long stream of events, one of them is going to be a bit bigger bottleneck than others. If you fix something before that bottleneck, if you improve something, then you're just stacking work into the bottleneck and things don't improve. If you improve something after the bottleneck, then you're just kind of spacing out the work as it comes out, because it's already limited by this, I guess in fluid dynamics, the Venturi, that doesn't really apply. But anyway, only by really knowing and having, I think, a diverse viewpoint on these value streams, only by clearly identifying them can what the bottlenecks are and what actually needs to be improved.

Andy (26:13):

Yeah, exactly. And it's not just about the bottleneck in the process, but also tied directly to that because we said the people are the system. It's about the bottleneck of the people. So if you have one person who understands this part of the process, well, it's the typical bus factor that what's going to happen if that person isn't available. So if you're not onboarding, then your bus factors are very, very low, but if you can do a proper onboarding, get people empowered and have people who are able to really see the whole system it automatically alleviates this bus factor problem.

Marc (26:51):

Fantastic. Yeah. And you know, one of the things that comes to mind is there's such diversity in the IT landscape and new companies are coming up and we talk about disruptions and challengers and things a lot. But when a new company does come into a competitive landscape with a legacy company or company that's been doing well for a long time, it can be really difficult to understand the difference in speed that everyone says, well, it's because they're small. And, it's like, well, it's also because they can take state of the art processes. They can understand very quickly where their simple process bottlenecks are. People that are there understand the whole system end to end. The people that understand that end to end system know how the value flows through it. And everybody's focused on one thing.

Marc (27:42):

And for existing companies, they can start to take these things into use and start to say, okay, we'll take our team of experts. And they're going to understand how to qualify new employees. Our new employees come in and we're going to make them understand, end to end, how the systems work. What are all the components are and what the value is. And doing all of these potentially before you can do that first bug fix, which is the usual thing. You onboard a developer, you throw them in the deep end and say, here's a nasty bug that nobody wants to fix. Go for it. There's so much here that I think can be taken away from.

Andy (28:17):

Yeah. We usually talk about startups as these small agile, very nimble, able to react organization. And there's a lot of theories about why that is. Some people think, because it's such a small team that it's easier to communicate or whatever. I think that the real reason, these small tech startups are able to kind of move so fast is because they are so small. Everybody automatically understands the entire process and the whole value chain. They understand the system as a system. They haven't yet broken down into, you focus on this component and you focus on that component and forget about the other things. So because the small startups have this overall system view, an overall value stream kind of view. It's easier for them to know where they should and could be agile. And the big companies, it's not because they have too many people or they're too large and too established. It's mostly because they are missing this overall view of everybody having the perspective of the entire system.

Marc (29:19):

Very well put Andy, thank you. This is Marc. Thank you for listening. I have here my sea daddy, Andy Allred. This has been DevOps Sauna and the sea daddy DevOps Chronicles, episode one: Getting your fish.

Andy (29:35):

Thanks Marc. It's been a pleasure and you're not ready for your fish yet, but we'll keep working on it.

Marc (29:41):

Thank you.