Discussion with Beyond Identity: Software development lifecycle security

Bob (00:07):

If you're going to move to the next generation of kind of SDLS security which is coming fast which is around the integrity pipeline. Things like SLSA sigstore which starts to look at how the artifacts are broken out more like a distributed ledger. You really can't start that journey unless you anchor the commits with strong identity and commit signing.

Lauri (00:37):

Hello and welcome to DevOps Sauna. In March this year, we held our annual The DEVOPS Conference. To keep the vibes from the event going we kept inviting some exciting people to our podcast to share their backstories to the themes in the event . Today we discuss about the software development lifecycle security. There are many reasons why code in the CI/CD pipeline can be untrustworthy or what are the sources or vulnerabilities of the software delivery pipeline. One of those reasons is that the identity of the people who made the commits cannot be verified. What to do about that and where should companies start to improve their SDLC security is what we discussed today. Our guest today is Bob Burke, VP of Security and Infrastructure at Beyond Identity. He is accompanied by Niko Ikonnen, Senior Consultant at Efficode. Letøs tune in! Welcome to the DevOps sauna podcast and it's great to have you Bob and Niko today.

Bob (01:45):

Thank you. 

Niko (01:45):

Thank you. 

Lauri (01:46):

We are talking about the security of the software development lifecycle tool, so SDLC security. I should start by saying that it's really no secret to anyone that all forms of Git have really taken over the world as the form of source code management. Every time a technology gets adopted widely it also becomes a target of security exploitation. Now I imagine that even though there may still be vulnerabilities in the code of the products that are part of the toolchain, maybe they are not the main culprit for SDLC security failing. Would you agree with that or are there some other reasons for SDLC security failing? 

Bob (02:26):

Thank you for having me today. This is a great honor. Thank you Lapa and Niko. I think when you take a step back and you look at the SDLC process, you kind of have to understand how complex it is today and the environment we operate in. Right? I mean it's not uncommon for developers to get pushed to do releases. They integrate third-party software. They're doing builds on distributed build systems that may be running across multiple trust zones. They're being asked to then dump those artifacts into third-party repositories which creates yet another set of concerns. And then finally deliver to the end client. I think we only have to look to the industry to see some cautionary tales of how complex this problem has become. For example, you have SolarWinds, you have Kaseya, you have Codecov. These are all cautionary tales of really, really, great engineers and great engineering teams delivering and meeting the business needs, but somewhere along the lines something broke down in the SDLC process. You know when I look at the SDLC process today, I think because of how we integrate open source tools during the production and then through the delivery, it's across, honestly a distributed build environment across multiple build systems. Maybe it's across multiple vendors.

Bob (03:42):

Engineers have to realize today when they're working on the SDLC that they are a part of a distributed supply chain now. And the things that they do in their SDLC can have huge impacts. Again if you look at the SolarWinds, there's no surprise that they were specifically targeted. Because how large was that blast radius. One SDLC vulnerability impacted thousands of customers. And so when you start looking at it the reason it's so complicated it's got multiple stages, right. We have the build process, so basically we have the creation of the source and we have to put that source into a repository. Well, like you said I mean Git is just eating the world, right. I mean all these third-party repos artifact repositories, they're all SaaS services.

Bob (04:28):

So, just the nature of putting your source code into a repo like that raises all kinds of questions about security and vulnerabilities. In particular, how are we securing those systems, are we using strong authentication, is it integrated into SSO, understanding that strong authentication of the third-party Git repo allows you to control the configuration. But when developers do their work, they're not using SSO, they've already distributed keys to themselves, maybe across multiple trusted devices or untrusted devices. You have issues with impersonation and undefined authorship that can create all kinds of confusion. So, I think just securing the repo itself is a challenge. But that's just the beginning of the story because let's say that we can secure that code, let's say that we're smart enough to have our developers work on managed devices, and we have, and we'll talk a little bit about this later, we have strong identity anchoring and artifact provenance from the point of commit and that's when we get into commit signing. But that's just the beginning of the journey because once that's pulled out of the repo your entire build system is vulnerable, right. And it's the production and movement of those artifacts when it's in the build process. The systems you're building on, are they hardened and secure, are they immutable, are the ephemeral? When you use a build system and you're done do you throw it out and pull the new build system, build it from scratch from a secure repo. So there's no persistent threat possibilities. How do you know that the commit came from your pipeline? A lot of developers will have a master control script or a common model where they have all their keystone they push into all the environments. That's a huge anti-pattern. Are you using a pull model, are things clearly segmented between the build deploy, and run time environments. So then we can get into all of the vulnerabilities that happen as you build things and you pull in third-party software, right. So I think it's a completely new focus on the SDLC and it's kind of scary as a developer. But it's also an opportunity for the DevOps and DevSecOps teams to really shine and solve a meaningful problem.

Niko (06:32):

Few years ago when you said a security to a developer that means that there was going to be an IT department coming to you and removing all your access rights to your applications and making things really difficult. So that kind of created a situation where you had the developers living in one space and all the other people in the other space. And then you would just trust those binaries which the development sector was coming in and the security was applied when you went to the production.  But I would say the DevSecOps is the key here to make it actually secure development to make it easy and not adding any hurdles to anyone except the money source attackers.

Bob (07:27):

Yeah. I think that's absolutely great point Niko. In the end, you know everybody at your organization should be a security engineer and because things are moving so quickly and the environments are so dynamic that you really have to couple the engineers, the security operations and what traditionally was DevOps. And they essentially have to be co-located and working together and building security into the solution. What you really see and I hear shift left and now it's kind of shift anywhere because fundamentally security is shifting left into the release cycle. But now with NoOps feature teams, you have the developers kind of shifting right, right. So where they're building and responsible for infrastructure, so no, I think that's exactly right. I think it all comes down to having a really keen focus on DevSecOps.

Lauri (08:14):

 You mentioned the single sign or the other forms of reliable authentication which I take it as in many cases, when systems generally modernize that becomes just one attribute of a system. It's like okay. It's being widely adopted, it becomes, it gets SSO'ed because it's just the how the game works. But I suppose there are some other kinds of sources of vulnerabilities, specifically related to the SDLC, so. 

Bob (08:42):

 Yeah. I think the first one where it starts again is in the on the repo and it's really about having proper code repo hygiene. So you're right. The first confusing thing for a developer is the impersonation. So, how do we know that the person basically when we start distributing all these keys, right. You can SSO into a repo but all that really means is you protected the configuration of the system. Once those keys are distributed, the first vulnerability is where are those keys going and who are using them. And then when you do commits if you don't do commit signing and have verified commits that can provide complete identity traceability and providence through the through the change, it becomes very very hard to know who actually did the commit. And if you don't have some kind of binding between identity and device, it's real hard to know that it came from a trusted source. So I think that's the first vulnerability that we target is focusing on getting all our developers on and employees quite frankly on a strong authentication system, right. That eliminates all the passwords because that's where a lot of the threats are. So that's step one and that can bind the identity of the device which is implicitly multi-factor, MFA. And then we push developers to do commit signing because then that anchors what we call the integrity chain, right. You have a normal pipeline which is, yeah, moving stuff through the pipeline. But what you're seeing today is this is trend to look at more it's an integrity pipeline which is as I move the artifacts through the system, no one's modifying it. Like if you go back to some of the breaches we've had, if you penetrate one of the build systems all you have to do is trick the build worker to actually recompile code that's actually sitting locally on the build system. And then it will bundle it up and then move it to the artifact repository. And you have no idea that that happened. If you're not tracking the integrity through the entire system, you really can't say what you delivered. In fact, all you really can say is most of us know how to sign our final artifact. All you can say is this absolutely came from me but you can't say that it came only from me, could have come from someone else too, right. Yeah, but at the root of all this is really I think having a secure identity and a strong authentication mechanism because everything flows from that.

Niko (11:05):

Yes. I think it's surprisingly only few people understand the decentralized way how we it works. So many of us just use some service where you log in and add your SSC keys and push your code in or your commits in. And then it shows your face next to the commits. But many people don't understand that actually when you signed into the application with your SSC key or some SSO system, it doesn't mean anything about that data you have put in those comments. Those have nothing to do with the fact that who put them return in. So for example, for there is a developer who is installing all sort of the applications to his workstation. And then some of those applications has access to a file system which then can modify the code on the file system. Add some comments which were not from the developer and whenever the developer can push the code in the central Git location. Well, it's in there. Of course, there are other ways to limit this like if you can only push to be future prances and then someone else must also review the change before merging it into the master of development brands.

Bob (12:36):

I think that's a great point and you talk about one of the remediations. I mean once you have, you know a secure authentication you do commit signing, you also need to make sure that every commit that goes in triggers a two-person review. What a lot of times people will do engineers will do is they'll kick off merge bots that try to facilitate and accelerate the release cycle of the merging or different types of bots. But you really shouldn't allow that really at the end of the day, everything that goes into your repository needs to be signed off by at least two people. And they need to check that box they need to look at that. And also a little bit to what Niko said is you know one of the problems that I see that is, it's understandable is developers who are working all over the place and they're working at home, well, not everybody's been wrong. But maybe you know traditionally in the office and at home, they may be moving across different devices, right. And so they may have started on a trusted device. And just say hey, I'm just going to copy my key and put it on another device, my SSH key, maybe they didn't set up the same configuration. So it's not even getting committed as the same author, right. And now you have, and how do you know that other device maybe it's their partner's device, laptop, that could already be compromised. And it goes right back to what Niko said, someone could make changes to that code and you just roll it into the commit.

Lauri (13:53):

So commit signing I think we have already discussed about what problems companies typically solving when they go about comment signing. And then Niko had a great point earlier when we were preparing for the show which was the perception that commit signing slows developers down. Let's talk about those a little bit.

Bob (14:13):

Yeah, the perception really, you're right. I think the perception is, you know honestly as a developer, I developed for decades the developers always just want to get code in right they want to take their, build their stuff and just check. And they don't want anything in the way of that I think commit signing has gotten a little bit of a bad rap. And I think it's more of just how we've done things as opposed to the right way we should do it. And I think that for us, the proof is always in the eating of the pudding. So we use commit signing and every day and everything that we do here, both the developers and the DevSecOps teams. And it's important because work and 100%, IAC infrastructure is code. When you think about it's kind of interesting, we go to companies and we say you know and they start to push back a little bit on the overhead. The first thing we do is we, it takes five minutes to set up. And they're commit signing, really the challenge is, that they may actually have poor repository hygiene, right. And you have to kind of get that cleaned up because no one was actually checking in with any meaningful authorship or anything like, that it's all. The same developers moved across multiple devices and they have multiple tags, so there's that and then it can be really set up and it can be made to be seamless. And you sit down and you talk to the developers and you talk to the Dev particularly, DevSecOps engineers. And you start explaining how this is a prevention method, right. Because it's a one and done, you do it. Let's talk about the threat surfaces you've just eliminated. Let's talk about the multiple security controls by design you've eliminated, right. Because now you have a strong authentication you're doing commit signing by its very nature. And then you talk about how it's if you're going to move to the next generation of kind of SDLC security which is coming fast, which is around the integrity pipeline. Things like SALSA, SLSA, sigstore which starts to look at how the artifacts are broken out more like a distributed ledger. You really can't start that journey unless you anchor the commits with strong identity and commit signing. And then the last nail in the coffin is when you go and you say, where's your network? You go to the DevSecOps engineer and say, where's your network? They say what do you mean where's your network. Well, your network is in your repo because if your infrastructure is code, you have now moved your network into your repo. Why aren't you protecting your network and your repo like you protect your network in your corporate infrastructure? And then I think that that kind of makes the case and then when you get them using it. It's pretty seamless. It's mostly about the repo hygiene that really creates some concerns but we, you know you just have to there's some behavioral changes. They have to make it. That's basically it.

Niko (16:53):

 Yeah. I don't think that many developers actually do bad hygiene on purpose. But they didn't have any instructions, they were just need to commit. And when they first time trying to commit on something on a new device, it asks all sorts of questions, what's your email, what's your name, do you have a some keys, do you want to sign it. So I mean that's just a two three minutes of setting up it once per device and there you go.

Bob (17:22):

Yeah. One of the one of the things we like to do too is just we go in, we talk to our customers. And we start out with let's, honestly, what can you say about your repo, how many different authors do you have in the repo if you were to look at the author title, where are they committing from, how do you know who's done what, right. And I think there's the impression that we're a lot of us are more sophisticated than we actually are. But when you ask simple questions and you just say, okay, let's list the usernames or the authors in your repo. And you see bugs bunny 12 times, it starts to raise some questions, right. No wonder. 

Lauri (18:04):

It's Lauri again. We had a webinar with bank data, one of the largest financial companies in Denmark. In this webinar, they share their DevOps journey and tooling choices and focus on how DevOps practices and tools help integrate security and compliance requirements in the software development. You can find the link to the webinar in the show notes. Now let's get back to our show.

Lauri (18:33):

We had a conversation with somebody a long time ago in this respect saying that the only compiler you can trust is the compiler you combine yourself. And it sort of goes to the very root of this problem and some applications are more security intensive than some other applications. But when you look at everything we have discussed, is there a way to characterize the companies who are on the forefront of adopting everything we have discussed.

Bob (19:02):

You mean from just out like a outside perspective or they run some kind of audit. So from an outside perspective, I think again, and I'm sure Niko will have a different opinion, I think that some of the more recent early stage companies. Let's say they're under four years, who started cloud native are probably more open to these kinds of things. And they're probably using a lot of the new techniques that are available. And to be quite honest, some of the techniques three years ago were there but were cumbersome and hard to use, like you know multi-factor authentication, strong authentication, and passwordless really wasn't there. At least some of the early variations weren't all that great.

Lauri (19:46):

 I'm not challenging your answer but I'm thinking it's counter-intuitive. Because if you think the impact of an individual corporation in a country to their GDP, you could say that those companies who have existed the longest have the highest impact on the GDP. And that goes more like established banks for instance. And somebody could make a case that the biggest impact on the measures that we have discussed could be achieved from the completely polar opposite from what you have described.

Bob (20:17):

Yeah, absolutely. You can. It's very, very, simple. You can just go down the, if you wanted to, if you took a step back and you said, let me figure out which companies if they adopted this would have the largest impact in the supply chain. Because at the end of the day and that's… all you have to do is look at those companies that have the most enmeshed endpoints across that market, right. So somebody who's got, you know that's why the SolarWind attack was, so I think it was the largest distributed breach in history because it touched so many. So if I was strategically to position which companies to go after, I would just look at that mesh. And you could probably figure out the connected web and those are the ones that really need to adopt it first, right. And then the financial institutions because against the risk, they would absolutely 100 percent. Those are the hardest ones to change sometimes.

Niko (21:09):

Yes. I was just saying that I've been seeing is that even the large organizations when data start a new project or they start developing something new, they actually do out of these things. But then you have all the legacy code, legacy systems, which have been running for years, they will still be missing this even though the impact on those legacy systems would most likely be the greatest..

Bob (21:38):

No, that's exactly right. Yeah and you have to, I mean it's no surprise when you start to look at the supply chain breaches, you have the nation state actors or whoever right, the threat actors. It's clearly strategically, there's strategic reasons why they're going after the companies they're going after. Because they just have such a gigantic blast radius.

Lauri (21:58):

Yeah, the other day for my own personal curiosity, I tinkered around the government issued ID card to authenticate my own laptop. So basically, come to my home office and punch in my own government issued ID and type in the pin code. So basically, have a multi-factor authentication on my own laptop. But immediately you do that immediate question pops up but this is actually not a chain of trust, like I'm only taking care of the local authentication but there's simply no chain of trust. I cannot venerate that identity from my own desktop or laptop into SDLC or any further, so it's sort of very, very local solution. But nevertheless, it was more of a proof of content for myself that I can move away from passwords if I want in my own local domain. When you look at moving away from passwords and moving to the multi-factor authentication in general, how do you see the world going there?

Bob (22:56):

When I think about where the industry going, I think of the stakeholders or actually not just the industry because there's a consumer perspective on this too, right. So for me, when I look at passwordless, I think if there's a compliance and security perspective, there's pure enterprise perspective. I think which goes to your, you know delegated identity model. And then we have the consumer perspective, right. They just want to get to their stuff and they don't want any, it's very unlikely that most consumers trying to get to whatever they're trying to get to their bank or something, is they're going to use a key, right. Some physical or fob, it's highly unlikely. Maybe some of us will but not most. But I think from a compliance and security perspective, there's a reason why passwordless and strong authentication is so important now, and it's a big movement in the market. Because from a security compliance perspective, it just, when you nail that down, you immediately by design satisfy a number of security controls across the different families, both from access control, identity and authentication which is a big one. And then from a compliance perspective at a large corporation can start looking at, they can say I now have if you have the right deployed solution, I can control my authentication assurance level fleet-wide, right. So, it's too compelling not to pursue because at the end of the day, there's a whole bunch of things we all we're all bombarded by security controls but what we really want to do, if you want to be able to say that my authentication assurance level, right, based on this digital guidelines is good, is it bad, is it a one, is it a two? Where is all my distributed key in, are they all in trusted root of trust, or are they on disk. So the industry is moving that way and it's being pulled that way from a security perspective. From an enterprise perspective in a workforce perspective, it's basically just ease of use. Obviously, the prevention if you eliminate passwords, ease of use, eliminating phishing attacks.

Bob (23:44):

But improved user, like from the workforce, it's easy to use. But if you're doing BDC, you have a streamlined user experience and a significantly reduced risk profile. That's why you're seeing I think the proliferation of web authen and the fido alliance is a lot of people are, I mean that's quite important standard in a reduction and complexity and authentication speed. Those are important from an enterprise perspective. Because you know if you can't get your workforce to do it and use it, it doesn't help your security controls, right. So, having something very simple. And then when you get in the enterprise, you know that's where you get into whether they're using SSO or delegated identity or federated identity. And if you can come from that root of trust, when you register that person and you've now pushed the root of trust to that device, you now know that identity and that secure identity. And then you can federate that. And then from a consumer perspective, like I said I think a little bit earlier, most people, I think you know the web authen protocol is really interesting, simplifies the experience and provides a significant increase in security. I don't know about you Lapa about how many passwords you have floating around and whether they're written on a piece of paper, or you're tired of your password manager, but…

Lauri (26:09):

All too many. 

Bob (26:10):

 Yeah, it's a big event in our household whenever we have to organize our passwords.

Lauri (26:14):

Well, what I know is for how many people I have shared my Netflix password. What I don't know is for how many people, they have shared it. 

Bob (26:22):

Yeah. It's really interesting because you know, maybe down the road, we're gonna, you know you kind of start getting very close to sovereign identity and things like that down the road, which are really interesting. You know, and maybe the industry is eventually going to go there, right. I don't know. But this is definitely a step in that direction.

Lauri (26:40):

Yeah. I would imagine and Niko, please step in. But I would imagine that at some point, it would be something like bring your own identity that when employees come to the office, they basically walk up to the front desk, and say, here is my identity and this identity is issued by the source that you trust. Can you federate me? And then they basically federate you on the basis of that and then you can use your favorite identity from there on. And it's basically the company has federated it.

Bob (27:10):

Go ahead Niko. 

Niko (27:11):

Yeah. I actually liked your example of that national ID using that to authenticate your own computer but you mentioned that not into your software development pipeline. Well, that's only because no one didn't configure your software development python pipeline to trust, finish national authorities signing key for example. But instead, they configured to trust it your company's keys which you were given after you showed your passport, your supervisor, or IT sector. So I mean technically there is everything already there to move from your passport type of manual paper document to digital version. So maybe at some point when you come to the office, you just insert your national ID. And then get federated access from there. 

Bob (28:13):

You know that's a good point. I think you know in a lot of ways, particularly with the DevSecOps team, we live in a kind of a world of, in the developers of integration, innovation. Right. Like there's all these different technologies that are appearing in the market. And then the capabilities, kind of are somewhat non-linear emergent. And I think you're onto something a lot because I'm familiar with the verifiable credentials. And they have the concept of a verifier. And you know we've got the assurance, authentication assurance levels. If you go back to NIST, you've also got the identity proofing, so the identity assurance levels. All of this, so the identity proofing with the authentication and then the distributed. Like if you can get that federated piece on the back end in terms of a verifier, you literally could. I mean how far are we actually away from that model, right. I mean it's the pieces are starting to take shape and, you know I know the W3C issued verifiable credentials. And I know the US government started to adopt that for I think tracking, I think shipment of goods. My previous company we use verify for credentials for identity proofing for just a simple commercial application, a B2C. So, it's you know with the digital distributed ledgers and distributed identifier, maybe we're close, right. 

Lauri (29:32):

Yep. Yep. Coming closer and closer. 

Bob (29:36)

Yep. 

Lauri (29:37):

So, to round up, I think we are coming to the very last question. I'd like thoughts from both of you, where do I start!

Bob (29:49):

So, start with securing the SDLC.

Lauri (29:51):

Yes.

Bob (29:52):

Yeah. I mean the first place I would start is to really look at kind of how my SDLC process is architected, right. Making sure that there's proper segregation of domains. And then start to also look at what my access and my key life cycle looks like, right. Where are those keys, and who has them, and how are they protected? And so, I'd start to lock that down but then I think any company of any size can start with a strong simply start integrating ,a strong authentication mechanism and start locking down the commits. I think that's where you can start right away. I mean that that can happen almost overnight. You go and you get strong authentication. You can move very quickly on your DevSecOps to passwordless and you can start to inject a root of trust into that system. And then start working on your repo hygiene. Because that is a very contained issue, right. Because the build system's got a lot of legacy stuff in it. And it's something that you can test and validate and do a POC almost overnight.

Niko (30:53):

Yeah. I mostly agree with this. So overall just secure the whole pipeline check who has the access to what. And for example, we machine that compiles the extra proliferation should not be acceptable too easily. Then of course, I would check your Git configuration so that only you need to actually have two persons checking everything which goes into your product. So, reviews and prevent, pushing into the release parentheses just without the peer review. And maybe when these are… Okay, you can start looking at the other things of course, adding the commit signing is easy. It doesn't take too much time, of course, you don't have those for your old commits. But anyway, that's one next step on that.

Bob (31:47):

Yeah. I know. I think there is some low-hanging fruit. I mean it's just, if you look, if you try to ball the ocean, it can look like such an unwieldy problem. But there's some obvious things that you can pin down right away, and it's not okay today moving forward to basically be uncertain about what and who, what's being committed and who did it. Right. At the very minimum, you've got to nail that down, right. And then from that, it's like pinning down a sheet. You can pin that down and you can start smoothing out, you know out to other areas.

Niko (32:17):

Yeah. I have seen actually many companies in which their, for example Git repo is not part of their centralized user access management. So, they have active accounts for 50 x employees there, and especially if that's reachable from outside or any other means that's not really good thing. So, also make sure that the development processes and our tools for the company policies.

Lauri (32:49):

That's a good reminder for systems of all kinds. It's funny of enough we discussed earlier in the podcast that any change we make should be reviewed by two people. And we can say that this is the well verified checklist where to start because this checklist has now been verified by two people, Bob and Niko. So you can trust this list of where to start a little tongue-in-cheek there. It has been a wonderful interesting conversation on a very specific subject. And I thank both Niko and Bob for joining us. Thanks for the conversation.

Niko (33:26):

Thank you. 

Bob (33:27):

Well, thank you a lot for Niko. It's been great talking to you and I appreciate your time. 

Lauri (33:30):

Thank you for listening. As usual, we have enclosed the links to the social media profiles of our guests to the show notes. Please take a look. You can also find links to the literature referred in the podcast on the show notes alongside other interesting educational content. If you haven't already please subscribe to our podcast and give us a rating on your platform, it means the world to us. Also, check out our other episodes for interesting and exciting talks. Finally, before we sign off, I would like to remind you that all the talks form The DEVOPS Conference can be watched and listed online and for free. You can find the link from where else, then, in the show notes. Now let's give our guests an opportunity to introduce themselves. I say now take care of yourselves and remember: with novel DevOps services readily accessible for everyone, healthy Agile and DevOps culture,  is the source of success in delivering great digital services. 

Bob (34:21):

My name is Bob Burke. I'm VP of Infrastructure and Security at Beyond Identity. I've been working in the security space for over 20 years, started out working on session border controllers and security gateways in the .com era. Moved on very quickly to some distributed computing applications and then some banking security applications that had early secure authentication solutions for banking. Then more recently, I was at a company that did digital offers. There was CTO and CEO there. And then started Beyond Identity in 2019. My interests have always been in computing distributed applications, network security and just complex system engineering. I mean it's what gets me up in the morning.

Niko (35:08):

My name is Niko Ikonnen. I'm a Senior Consultant at Eficode. I have been working with the identities for 12 years now. And first 10 years, mostly with authentic identity and access management, and for last few years, making security more agile. And which is I think now called DevSecOps. My interest has already always been making things secure and easy. And I think there is plenty of to do with that.