Interview: Black Ops, tools, and the pipeline with Mika Aho

Heidi (00:05):

Hi, welcome to another episode of DevOps Sauna, Eficode's dev ops podcast about all things continuous delivery, DevOps and tooling. My name is Heidi Aho, I'm the host of the podcast, and I write content at Eficode.

Heidi (00:19):

Now I'll start off the podcast with a bit of a plug. Eficode had some big news last week, we were chosen as Tivi's Company of the Year for 2019. So you can go check out that news on our blog if you're interested. One of the reasons Tivi chose us as their Company of the Year for 2019 is that we have this DevOps platform called Eficode ROOT, which was built from scratch, which dovetails quite nicely into today's episode. Because I'm sitting here with Mika Aho, Eficode's ROOT resident [foreign language 00:00:00:53], and technical lead. So let's get cracking with our interview. Mika.

Mika (01:00):

Yeah.

Heidi (01:00):

This is an important dev ops topic I'd like to start off with. Given we've got the same last name, do you think we're related to each other? What are the chances? 5,000 Aho's, are we family?

Mika (01:12):

It's a small country and I think that perhaps in the fourth generation or something like that, who knows? Probably not.

Heidi (01:22):

Okay. I was hoping we could string that out into a proper 30 minute podcast, but we might need to get on with some dev ops instead. What a shame. So could you briefly tell us how long you've been at Eficode and what you do here?

Mika (01:38):

Yeah, absolutely. So I started at Eficode back in 2015. At that point I started in software development, so that's my background. I used to work in software development even before that, but at Eficode I started with software development. I think I did something like a few months of it, and then I started going into this more consulting approach with different kinds of tools and, well, Atlassian tools to be precise.

Mika (02:10):

And before long, that took up most of my time. And at that point, the ROOT team was basically forming at Eficode. It wasn't called the ROOT team yet, but it was forming. And I transferred there, and within the ROOT team I then started doing more and more consulting on different kinds of tools. And from there I got into processes, tool maintenance, and then of course, different kinds of integrations and so forth. And then as time went on and we grew and we gave to the situation where we actually started to do our own research and development. Then I saw that as a chance for me to do something more, to do something bigger, more interesting. And went on with it, and I'm currently the Head of Research and Development activities here at the Eficode.

Heidi (03:02):

Fantastic. Thanks for that. So you're perfectly poised to tell our listeners what Eficode ROOT is. So what is it?

Mika (03:11):

So yeah, Eficode ROOT is basically this service or product, actually both, that gives your organization this tailored DevOps toolkit, and you get continuous maintenance and support with it. It comes with different kinds of deployment options. So we can deploy it into your private data center. We can deploy to public cloud, or a private cloud that is set up by us. We have a lot of experience from the field, from our DevOps consultants and running these platforms for our customers for half a decade already. So we basically have the best tools for each purpose. And the information and the knowledge that we have of these tools, is what we actually offer to our customers.

Mika (04:00):

So the idea is to make sure that your tools are up to date, they're kept functional and fast, and that they're actually running 24 seven. This actually means that your developers don't have to, to waste time on maintaining tools, because that's not their core competence. That's not something that they should be doing. And basically, because we're deploying this to so many customers, the economy of scale here means that we can actually push the cost down to a level that could never be achieved if you had your internal personnel maintaining the tools.

Heidi (04:37):

Oh wow. Lovely. Thanks for that. I'm so stoked to have you on this podcast. I've been really thirsty to have someone from the ROOT team talk to us, because tooling is such a huge part of software development. You literally can't do it without the right tooling and it can be such a headache to sort it out, basically, if something does go wrong. So you started off your career in consulting and then moved into R and D. In your experience, how do organizations usually choose their software tools?

Mika (05:09):

Well, yeah, to be precise, I started coding first and then went into software.

Heidi (05:14):

Okay, sorry. My bad.

Mika (05:15):

I mean, consulting. But that being said, I have done enough of this boots on the ground type of work to have seen these different kinds of organizations struggling with different aspects of their tooling. Tooling is definitely not the only thing a lot of companies are struggling with, but it's in terms of modern software development, it is the biggest one in my opinion. And the way people choose their tools, I think there are two approaches generally speaking.

Mika (05:52):

One of them is that they're coming as given. And that means that there are legacy tools that just have been used at some point, and when you're starting new projects and so forth, the same tools are just kept in years. So basically there is something that might not even be commonly used anymore. They might be badly out of shape. They might not have been updated for a long while, these hand-me-down tools if you will.

Mika (06:24):

And then the other way is that perhaps organizations might have these centralized tools, but they are in such bad shape or that they are not configurable, that especially in larger organizations, these Black Ops tooling starts popping up. So different parts of the organization, different teams in there, they start putting up their own small tools. So a developer somewhere might put up his own Jenkins and start configuring that.

Heidi (06:59):

Like an internal black market for tools or?

Mika (07:02):

Well, not exactly, but people are just installing tools for their team's personal use basically. And what this means is that, first of all, you have someone installing and maintaining the tool who's core competence to begin with is not in there. And one of the problems there is that when this person, for example, goes on a vacation or changes teams or changes jobs, then you suddenly have this toolkit, which a lot of people are reliant on, but it's not being maintained at all. So that becomes a huge problem point.

Heidi (07:43):

So that person who goes on holiday is a constraint, and basically halts everything related to that tool.

Mika (07:49):

Yeah, pretty much so. And we actually, with a colleague of mine, we had this calculation at one point where we started going on about, for example, if you have an organization out of which 10% are consultants. And each of these consultants costs, let's say, a hundred euros an hour, which is pretty common, let's say, so now if you have a tool crashing down in the middle of the working day and no one's fixing it, how much does that cost? Does it cost you the amount of time it's down, times demand of consultants, times their hourly fee. So it costs a lot just to have it done.

Heidi (08:36):

Yeah. And depending on the consultant, he could be steeper than that.

Mika (08:40):

Yeah.

Heidi (08:40):

Yeah.

Mika (08:41):

So yeah, from this Black Ops point of view, the best way is to find something that would be in the middle. So you would have tools that are robust, that are easy to use, that are responsive and configurable, but in a way that they're actually centrally maintained. So, that the working habits with different kinds of teams are kind of uniform and the technologies that are used are kind of uniform, because there's definitely synergies in there.

Heidi (09:14):

Fascinating. You've pinpointed a number of issues that do crop up. You've got this bus factor of the guy who knows how to use the tool. You've got the fact that it might be a legacy tool that isn't fit for purpose anyway. Are there any other issues that you've seen in practice when it comes to tooling?

Mika (09:33):

Well, yes, definitely. Security upgrades is one thing that pops into mind. Sometimes you might have vulnerabilities that actually require someone to react really fast to the issue. One of these things would be the quite recent CPU architecture problems. And-

Heidi (09:58):

Can you unpack that acronym? What does that mean?

Mika (10:01):

Well, CPU is the central processing unit. So there were problems with the architecture, which basically meant that in certain environments you could access, for example, data from another virtual machine, given the correct circumstances. And this is of course something that if your company, for example, is running tools on these shared virtual machines, you don't want your company data to be exposed to outsiders.

Mika (10:28):

So you have to be really fast there to actually patch those problems. And now if you don't have a person who is responsible for the server and responsible for the tool, that doesn't happen. So yeah, definitely, missing security upgrades, missing upgrades for the tools themselves, those are huge issues.

Heidi (10:47):

Yeah. And why is this situation so prevailing, because it's suboptimal at the very best. Why aren't people sorting out their game?

Mika (10:59):

Well, one of the things is that it's been a habit for a long time for organizations, IT organizations, to handle their own internal stuff, the internal tools. But the amount of tools in the past hasn't been nearly this high. Nowadays you need a bunch of different tools to do modern software development. And this wasn't really the case, let's say, 10 years ago. So people are still stuck in the mindset that it's something that you need to do internally.

Heidi (11:32):

Is that because they want to retain that knowledge in-house. Is that seen as valuable to them or?

Mika (11:38):

I suppose that is one of the reasons, yeah. But one of the reasons has been that a lot of people still seem to think that it's something that they should be able to do. While the fact is that it's so complicated these days, that you shouldn't do it yourself if it's not actual your core competence,

Heidi (11:57):

Sp there's a bit of pride there as well. Like, "Obviously I can sort out this tool, why wouldn't I be able to do this?" Or, "Why wouldn't our organization be able to do this in-house?"

Mika (12:07):

Sort of yeah. Yeah, I suppose. And of course, partly the misconception that this should be a thing that we can handle internally.

Heidi (12:18):

So much for that. So obviously ROOT is a DevOps platform, and it's an outsourcing option for companies to outsource this whole world of tooling so that they can get on with building products, et cetera. I know we've spoken about barriers already or objections that people may have, but are there any specific barriers to people accepting what ROOT is or what a ROOT equivalent is for their company? And choosing that as their preferred tooling option?

Mika (12:59):

Well, I think one of the barriers is definitely that these tools are such a central piece of this software development, that handing them off to someone else is a risk. And to be able to accept that risk, you have to know that you can trust the partner that is going to take care of it. So the partner has to be top-notch and you have to know it. Well, we obviously are, but everyone has to know that.

Heidi (13:27):

So the nature of the risk is, "Oh, what if this partner I've chosen gets it wrong? And what if we can't develop something for days on end?" Is that the risk or are there other risks?

Mika (13:37):

Yeah, exactly. It's exactly that. So there are ways to mitigate this. For example, you could set up, or rather, we could set up a platform, and the platform could be taken into use step-by-step. So you could take, for example, one team who transfers in there, starts doing their work there. And once they've gotten accustomed to the environment, then take another team and another and another, and migrate data as you go along.

Mika (14:06):

Because one of the big risks there is that you have this major blackout where everyone jumps from system A to system B within a working day. And when something goes wrong, then you have your organization's developers do nothing for a day or two days or three days, or something like that. And that's a major waste.

Heidi (14:25):

But that is possible? It is-

Mika (14:28):

That is if you do it in a way that you transfer everything at once.

Heidi (14:33):

Have you guys done that before, where you've transferred an entire company at once? [crosstalk 00:14:38].

Mika (14:36):

There have been some huge, huge migrations, yeah.

Heidi (14:42):

Okay.

Mika (14:42):

But we've never really run into a situation where we couldn't have done it.

Heidi (14:49):

Couldn't have, okay. So it's more-

Mika (14:49):

Yeah, so we've always succeeded there.

Heidi (14:53):

Okay. So it's more a question of a company feeling comfortable with that move, because it feels like such a big move.

Mika (15:00):

Yeah, definitely. And it has a lot to do with how the partner who is doing it, is actually preparing for it. The way we do migrations is that we check out the data beforehand. We do everything that we can to make sure that the migration goes fine, which means that we tested a million times. We automate every step of the way, so that when the actual migration time comes, then all we need to do is actually just execute our automation and make sure that everything goes as planned. And if not, if something unexpected happens, then we can roll back and continue without the migration. And then do another try in a few days, for example.

Heidi (15:44):

Yeah. And I'm guessing you've done quite a number by now?

Mika (15:50):

Yeah. I've personally done a lot of migrations, and our team has probably done hundreds of migrations by now.

Heidi (15:58):

Yeah. It's not just you and the team. How many are there in the ROOT team in total?

Mika (16:02):

Well, in the research and development side there's a bit less than 10 of us. In the operations, there's something like 30, 35, 40 people. So we're somewhere around the 50 people mark in ROOT. As well as marketing and sales as well.

Heidi (16:19):

Hearing how fluently you're talking about these issues, I'm starting to think that maybe we are long lost cousins after all. Okay, so we've spoken about outsourcing or the mess that you might want to outsource when it comes to software tools. But what if a company wants to build it themselves, they want the centralized, integrated software production line and they want to do it in-house. Could you give them some tips on how to do that? Real tips, real valuable stuff, please?

Mika (16:57):

Absolutely. Yeah. It's possible to build your own tool chain. I mean our reference architecture picture, it's actually visible on eficoderoot.com. So you can go in there and check out what it looks like. The key factor here, I think is leveraging infrastructure as a code. So basically, when you are setting up your servers, when you are installing your software, you do not do it manually onto a server. You actually set it up as infrastructure automation using, for example, Ansible or Salt or something like that. And that makes it so that you can actually upgrade the instances pretty easily. And you can do things in a very, very smooth, fast fashion, as well as you then have documentation basically that keeps itself up to date. And you see the actual state of the servers just by observing the infrastructure as a code.

Mika (17:59):

But then again, when you do configure and integrate the tools, then you start running into these different kinds of struggles, making mistakes that could or should be avoided if you have-

Heidi (18:13):

Are you speaking out of experience here or?

Mika (18:15):

Yeah, absolutely out of experience, because I mean, we've made a lot of mistakes over the years.

Heidi (18:21):

Yeah. I guess, yeah.

Mika (18:21):

And we've learned from the mistakes. So we know the worst potholes you need to avoid in there. For example, if you manage user permissions on a large scale, how do you do it? How do you approach the problem? Because if you do it wrong from the get-go, it's a lot more expensive to then later on go and change that, instead of doing it right the first time.

Heidi (18:47):

Yeah. Brilliant. Absolutely brilliant. I think we could actually talk about R and D in Eficode ROOT now. Are there any cool developments in the pipeline?

Mika (19:05):

Oh, of course. Yeah. That's my favorite subject more or less.

Heidi (19:08):

Yah.

Mika (19:09):

Yeah. I mean, a billion things are happening.

Heidi (19:14):

You are currently working on one billing thing?

Mika (19:16):

Yeah, one billion things, give or take a few.

Heidi (19:18):

Okay.

Mika (19:19):

I always have a lot of plans. I mean, that is my job basically as the head of R and D. I have to figure out new ways to make our platform better, and not all of these ideas of course ever see the light of day. But there are a couple of developments that are being planned out right now, and they will sooner or later be out for you guys to enjoy. For example, the ROOT Hub. So we just released the ROOT Hub a couple of weeks ago.

Heidi (19:50):

Yeah, yeah.

Mika (19:51):

And it's basically this landing page to the ROOT platform, which allows you to navigate between the different tools and lets you see what the architecture of your platform is like. And what we plan to do is add more features upon that. So currently it's a front-end, which you can use to navigate, but we intend to add a back-end in there as well, which would allow you to, for example, do this end-to-end completely automated product creation across the whole ROOT platform.

Mika (20:24):

So for example, when you wanted a new project, you simply go in there, you select the name for your project. You click on the tools that you want to use, you press create. And what happens is, the automation goes through every tool in the platform. It creates you a group, it creates you a, let's say Jira project, a confluence space. It can create you, for example, a Jenkins folder, all the bits and pieces you might need to actually start working. So reduce the manual labor and reduce the times you have to ask for something from somebody. So make it completely self-service to set something up.

Heidi (21:06):

That's cool.

Mika (21:07):

Another very interesting thing is the track and drive part of things. So track and drive, which is our, I like to call it framework basically. It's something that we use to gather data from different tools, then cross-reference and visualize that data. So we're working on coming up with new cool metrics and visualizations for that tracki and drive framework and, well, new kinds of features.

Mika (21:37):

For example, a feature drill down, which would then let you, for example, see when you design a feature, when work starts on it, when the first code commits come to your version control system, when you're working on it, when is it being tested? How is it being tested? And then finally when it's finished, you'll be able to see how long it actually takes for this specific feature to get done from the get-go to the finish line.

Heidi (22:09):

So there's a whole plethora of data that you can just see very easily about that feature you're building, right?

Mika (22:17):

Yeah, yeah. Pretty much. And the thing I love about track and drive the most is that we've done a lot of groundwork, where we are collecting data from these different tools. And now that we have a lot of data and ways to gather it, now we can really start thinking about how we can actually leverage this data? What kinds of deductions can we do from these different bits and pieces of information? And how can we actually visualize this in a manner that the end user finds usable?

Heidi (22:52):

Yeah, and I'm guessing the end user actually has similar questions as well once they get that data, "What does this mean for my business? What does this mean for my team?" And that's why data is yeah, the gold of software development.

Mika (23:07):

Yeah, absolutely. The point here is that the whole point of the ROOT platform is to ... We usually call it the delivery track and drive. So the delivery part basically means that you have these tools and you're able to automate things, and thus deliver software faster. Then track would be to actually get data out of the pipeline.

Heidi (23:32):

Well, are you getting, or is it just giving it to you?

Mika (23:36):

Well, basically the different tools are producing the data, and our track and drive framework collects the data for those tools.

Heidi (23:43):

Okay. So it's not a lot of effort for users, you just see the data?

Mika (23:46):

Not for the end users, no.

Heidi (23:48):

No. Okay. And by end users, you mean-

Mika (23:50):

All the developers-

Heidi (23:51):

All the developers.

Mika (23:51):

... and managers and everyone who would be interested in that kind of data in the organization.

Heidi (23:57):

Okay, cool.

Mika (23:58):

So yeah, gathering the data is the tracking part. And then the actual drive part would be to learn things from that data to make that delivery process even faster, to learn from your own bottlenecks, to learn from your own mistakes. Like this, everyone's seen the DevOps Infinity Loop, which goes around the-

Heidi (24:20):

Yeah, the donut. Yeah.

Mika (24:21):

Yeah, yeah. You continuously learn from your own doing and your own mistakes. But the idea here is to actually make that live, to make that actually happen, to provide you that data so you can speed up.

Heidi (24:37):

Okay. But surely the team means to change something it's doing in order for that driving to happen. So in terms of the drive actually, you're changing behavior because here's the data, here's why you should change that behavior. Or is there something else going on here?

Mika (24:53):

Well, basically it's about the underlining of the problems. Because to be able to change the way you do something, to be able to change the way you behave, you first have to be aware of that. So we are underlying those things. And we are actually currently also moving towards this intelligent early warning systems, which would search for these patterns in your data. And then let the end users know that, "Hey, we have detected this pattern in your, for example, source code management behavior," and this usually exhibits different kinds of problems.

Heidi (25:37):

Yeah. And I'm guessing then, even if you're busy, even if your team is busy, you'll get this alert and you'll be aware of it?

Mika (25:46):

Yes, it's a collection of dashboards. So it's definitely going to be a part of that.

Heidi (25:50):

Thank you so much for joining me on the DevOps Sauna podcast. We hope to have you on here again next season to tell us more about Eficode ROOT and software tooling and all that good stuff.

Mika (26:00):

Yeah. Thanks for having me. I'll absolutely be here if you'll just invite me. I mean, I think it's very nice to be able to do different kinds of things within your working day, and having a podcast recording is definitely something very different from your day-to-day work.

Heidi (26:17):

Yeah, I agree. I agree. Is there anything else you want to share with our listeners about Eficode ROOT or where they can learn more about Eficode ROOT, et cetera?

Mika (26:28):

Yeah, I mean all of our products, the hub, track and drive, RTM, which is ROOT team management, all of those are up on our webpages, eficoderoot.com. There are blogs describing what they are and what they do. And if you're interested, please go in there and check them out.

Heidi (26:50):

Fantastic. Thank you for listening to our second episode of DevOps Sauna. If you're not following us on social media already, please do so at Eficode. Efimo, Eficode ROOT's mall mascot is a regular on our Instagram page. So watch out for that one, and see you next time.