npm supply chain attack lessons in security and human error

[Darren] (0:03 - 0:22) But they immediately went for greed and decided to try and extort money from people.

Welcome to the DevOps Sauna, the podcast where we deep dive into the world of DevOps, platform engineering, security, and more as we explore the future of development.

[Pinja] (0:22 - 0:44) Join us as we dive into the heart of DevOps, one story at a time. Whether you're a seasoned practitioner or only starting your DevOps journey, we're happy to welcome you into the DevOps Sauna. Welcome back to the DevOps Sauna.

I'm once again joined by my co-host, Darren. Welcome.

[Darren] (0:44 - 0:45) Hey, Pinja.

[Pinja] (0:45 - 0:46) How's it going, Darren?

[Darren] (0:47 - 0:55) Well, I'm a security specialist, so I don't know if you've been following the news over the last 18 hours, but it's been an interesting time.

[Pinja] (0:55 - 1:12) Yeah, I'm not a security nerd myself, but I'm aware of things in the general relation to the IT world and news related to that. So we love a good security breach as much as anybody else, and we're so happy that we can cover now one that just happened yesterday.

[Darren] (1:12 - 1:20) It's always a good excuse to sit down and record, but it would be nice if we could do it under better circumstances at some point.

[Pinja] (1:20 - 1:45) The thing is, something happened yesterday, today, to be clear. Today is September the 9th. September the 8th, a large-scale security incident was discovered with NPM Packages bucket manager.

And just imagine this, that you're about to install a package that's supposed to help you, but instead this is stealthily stealing your money or data. So that's what basically was happening.

[Darren] (1:45 - 2:12) Yep. So what happened is NPM, as you say, a large security incident, a supply chain attack where malicious code was inserted into around 20 packages, and combined these 20 packages have more than 2 billion weekly downloads. So that's kind of a huge number.

And the packages are just the kind of general helper tool chain. We could probably talk a bit about NPM and what it actually is.

[Pinja] (2:12 - 2:27) Yeah, let's talk about that first. So we were talking about NPM, which is a package manager. It is intended for the JavaScript programming language.

And this is actually a funny connection. It is maintained by NPM Inc., which is a subsidiary of GitHub, which is part of Microsoft.

[Darren] (2:27 - 2:33) Yeah, but you rejected my idea of starting this episode by saying Microsoft security breach for the clickbait.

[Pinja] (2:33 - 3:15) That would have been something of a clickbait. And to be honest, we could have connected the dots like that, but we did not. Microsoft is owning, indeed, GitHub, and GitHub is the mother company for NPM.

So this NPM is the default package manager for this JavaScript runtime environment, Node.js. And that's the thing. I was actually, when I went looking into this, I had to Google what NPM stands for. But we got into investigating this with Darren, because it might clearly be Node package manager, right?

But something on, was it on Wikipedia or something, said that in fact, it is officially a recursive bacronymic abbreviation for NPM is not an acronym. So we stand corrected.

[Darren] (3:15 - 3:33) I mean, I maintain that that was the stupidest thing I've heard today. It's the Node package manager. It's like the GIF discussion.

There might be an official answer, but there is also a right answer. And those two things don't always sync up. Shall we talk about the actual attack?

[Pinja] (3:34 - 3:57) Let's talk about that. So this was alerted by the Aikido Intel feed. So kudos to Aikido and your feed.

In fact, because this was, this was in fact a user error. So there was an email, phishing email that said that two-phase authentication was due. So a very basic phishing attempt.

And I think there was a LinkedIn post about it as well that we saw that says, oopsies.

[Darren] (3:57 - 5:08) Basically. Yeah. So it's an interesting thing about the ecosystem of Node.js and JavaScript in general, that this was one guy called Josh Junon, I believe he's a maintainer of these 20 packages. And yeah, he actually got a reasonable phishing email. It looked relatively genuine and the email was received from support at npmjs.help. And I think at here, we could take a pause and look at the phishing aspect of this, because I imagine there's going to be a lot of people angry about the fact that this guy made a mistake, but npmjs.help is a, it's probably, it's not an official account, but I can see it fooling people. And Aikido actually dove into it and it was registered three days ago on September 5th.

And honestly, I think that's something that email providers should be looking for. They should be notifying the age of a domain. And if it's five days old, or sorry, three days old, then perhaps they should have a note, by the way, this domain is three days old.

I think to be honest, that's a failure on the email side.

[Pinja] (5:08 - 5:32) That's true. And people on Reddit actually are really good at this. When they look into subreddits and conversations, they actually notify others very easily that, have you noticed that this is a 12 hour old account?

So the kind of the hive is working against the people misusing the trust of the community. But as I said, this was basically a human error, but based on the email address, it could happen to anybody.

[Darren] (5:32 - 5:55) Yeah. And it did happen. Like you happen to a guy who's maintaining 20 packages that are getting 2 billion downloads a week.

Like this guy is obviously a competent guy in the field and he got caught by what is a realistic human error. And I again, maintain that email should perhaps warn you if you're getting messages from three day old domains.

[Pinja] (5:55 - 6:28) I applaud the swiftness of the communication and this guy's ability to even raise this with his own name. Like, hey, it was me. I apologize.

We're really working on this right now. But if we think of why, why was this significant? Yes, it was very, the scale and the volume was extremely large.

So we're talking about, as you say, over 2 billion downloads combined with these, these approximately 20 packages. And it's the kind of volume that is now getting through with, with these attacks. So the cybersecurity focus is very much still the technical focus.

[Darren] (6:28 - 7:09) Yeah. We as cybersecurity folk like the technical because that's what we train to do. And then a random guy clicks a random email and all of a sudden things are caught off guard and we just, we don't put enough focus on this human element.

Just like I was saying, the email, I feel like noticing that it's a new account should not be a thing outside the possibility of email providers. They like this kind of trading stuff, but this was a well-crafted attack. And thanks to large language models, we're seeing a lot more well-crafted attacks because, you know, it used to be that it was things like poor language that notified people that they were being attacked.

And now that's just no longer the case.

[Pinja] (7:10 - 7:24) True. And we can talk about the impact of this attack. The blast radius is huge, obviously, because we're talking about that many packages and that many downloads a week.

But we can also argue that to be fair, that the impact was not that problematic. It was targeting crypto.

[Darren] (7:25 - 7:54) Yeah. The whole thing was designed to hijack crypto transactions and Web3 activity to redirect transactions to a third party. So we've actually been able to, with the help of some various websites, dig up the address which the transactions were directed to.

And so far they've managed to steal $493 worth. So, well, there's a couple of things.

[Pinja] (7:54 - 8:28) Yeah. Sure that the amount can increase from that, but if we, we really think of the impact, even though this blast radius was enormous. And we're talking about developers who have now grown very familiar with using open source and using these packages instead of actually writing the code themselves, which is, this is the standard nowadays, right?

So getting into JavaScript and which is because this is now embedded in JavaScript. So we were actually talking about many, many impacted people, even though the stolen amount so far is only approximately $500.

[Darren] (8:28 - 9:07) It's actually an interesting lesson on human greed as well, because they could have slipped something a lot more subtle in there, like some kind of backdoor, which would have been less subtle, but might've had a larger reach. They could have just slipped in something that allowed them to continue to update the code without the access. They could have been quiet about it, but they immediately went for greed and decided to try and extort money from people.

And the result is that this massive hack, which I'm sure affected a lot of people over the last few days, has netted them $500.

[Pinja] (9:08 - 9:09) Yeah, exactly.

[Darren] (9:09 - 9:43) Okay. So the next thing I think we need to talk about is the actual packages in question. So we have a list of them here. I'm just going to quickly run through them. We have ANSI styles, debug, backslash, chalk template, supports hyperlinks, has ANSI, simple swizzle, color string, error X, color name, is arrayish, slice ANSI, color convert, wrap ANSI, ANSI reg X, supports color, strip ANSI, and chalk. And these actually fall into basically four main categories.

[Pinja] (9:43 - 10:26) Yeah. So we've got the first four of these packages that we can talk about: the core formatting and styling packages of chalk, ANSI style, supports color, and supports hyperlinks. So for example, these are providing some methods to style in some terminal text with calories and underlining, etc.

There's code definitions. For example, chalk is using ANSI styles actually internally and supports colories, also detecting if there are terminal supports colories. And the same thing that supports hyperlinks is the terminal supporting clickable hyperlinks in the output.

Then the second category is text processing with ANSI codes. So ANSI reg X, strip ANSI, slice ANSI, and wrap ANSI, and has ANSI.

[Darren] (10:26 - 10:41) Yeah. So they're basically all for dealing with ANSI escape codes. Then we have all the small convenience things like color conversion utilities and helpers.

These are not like, they're not big things. They're just like minor convenience items that people have picked up over time. Yeah.

[Pinja] (10:41 - 11:07) And then we can talk about the last category, kind of the other. So the debug, which alone has like 357 million downloads a week. So that is a very popular debugging logging library.

And then we've got backslash. This is kind of like a very small utility, but again, alone, it's got 0.26 million downloads a week. So it's still kind of widely used, even though a lot less than let's say the debug that we just mentioned.

[Darren] (11:07 - 11:14) Yeah. That's the smallest of them, but combined, they add up to more than 2 billion. So maybe we should talk about what developers should do.

[Pinja] (11:14 - 11:28) Exactly. It is still common to, and still okay to use open source and packages. That is the standard in the industry nowadays, but we do need to figure and talk about the effective version as well.

[Darren] (11:28 - 12:22) And that's the important fact. So if you're a developer using NPM, go to just Google Aikido NPM, and this will be the first thing that pops up. One thing to notice about these systems is because they injected code, they basically released a new version, as far as I understand.

So only the specific listed version is vulnerable. Previous versions are not. And that's an important distinction in this case, because often if a vulnerability is found, it's been there for 10, 20 releases.

So they have a great package of versions listed on Aikido dev. They list every package along with the affected number. If you haven't updated that package in the last couple of weeks, you should be fine.

But if you have been doing it, then you should maybe switch over to fixed versioning and make sure you have a version that you can trust.

[Pinja] (12:23 - 12:44) Yeah. And this is why the supply chains are a problem nowadays. And it keeps on being an ongoing problem at the moment because they're massive with this one, even though the target was, yes, it was crypto and all that.

But as mentioned, the blast radius was enormous. And why do you need SBOMs is because of this.

[Darren] (12:44 - 13:25) Yep. There is no better use case for SBOM than what is happening right now, because a lot of people won't understand that they have this dependency in their code if they're not tracking their dependencies. And there are good tools you can use.

DependencyTrack, DTrack is, I believe it's open source. It's at least free for some use. So there's no real excuse to not have that on your tool chain.

And it will basically catch these things early. Obviously, the remediation tends to be either rolling back a version in this case or waiting for a patched version. Of the affected systems, I believe only Chalk has a patched version as of right now.

[Pinja] (13:26 - 14:01) To be fair, Chalk, it was one of the largest packages on this list, but still only basically won 10% of what was the amount of downloads. But if we think of what else can be done to prevent these attacks and kind of how to minimize the impact of them. So SBOM, DTrack, that's a huge thing.

But also, don't just blindly trust the packages. It is, again, mentioning open source. It is still okay to use open source, so don't be fooled by that.

But please lock in the versions that have been embedded.

[Darren] (14:01 - 14:36) We have open source review boards for this entire purpose. We should have a dependency vetting process for this purpose. So when you spot changes like this, hopefully when dependencies update, you do have a process that checks them.

And if you don't, that's something to look into. But it comes down to just checking things like this. And I think we should acknowledge how lucky we were here because this had a potential to be so much worse had human greed not intervened and saved us by targeting crypto instead of targeting something a little more nefarious.

[Pinja] (14:37 - 14:49) And once again, tipping my hat to Aikido and detecting this as fast as you did, because this was spotted immediately. So the attackers were unable to use this on a wider scale than they already did.

[Darren] (14:49 - 15:02) And I think there's one more thing we should say, which is that on the off chance that Josh Junon is listening to this podcast, it literally does happen to the best of us. And this is why we have security built up how we do.

[Pinja] (15:02 - 15:35) Yeah. And it was extremely important that Josh, on the off chance that you are indeed listening to us, but anybody just the culture of bringing up the mistakes and discussing about this. It has been studied that it is actually the mistakes of others that teach us much more and we learn much more about those than our own.

So that is also why it is important to talk about these things and to have the kind of culture that it is okay to bring that up and say, let's fix it instead. All right. But this was all the time we had for the topical topic of today.

Thank you for joining me, Darren.

[Darren] (15:35 - 15:36) Been a pleasure as always.

[Pinja] (15:37 - 15:47) And thank everybody for joining us in the DevOps Sauna. We'll see you next time.

We'll now tell you a little bit about who we are.

[Darren] (15:48 - 15:50) I'm Darren Richardson, Security Consultant at Eficode.

[Pinja] (15:50 - 15:55) I'm Pinja Kujala. I specialize in actual and portfolio management topics at Eficode.

[Darren] (15:55 - 15:57) Thanks for tuning in. We'll catch you next time.

[Pinja] (15:58 - 16:06)