Field day: Key takeaways from KubeCon 2023

Amruta (00:00): A lot of people talked about reducing carbon footprint and power consumption. There are some mixed reviews about it and see if that's a trending topic in the next Kubecon as well.

Marc (00:20): This season, Andy and Marc are back with a fantastic group of guests.

Andy (00:25): I bet the depths remain classified, and Marc keeps his head in the clouds. With our combined experience in the industry, we can go from the bare metal to the boardroom.

Marc (00:34): In DevOps Sauna Season Three, we'll explore platform engineering, and the people and cultures that make it happen.

Andy (00:41): Enjoy your time in the DevOps Sauna.

Marc (00:53): Hello, we are out of the Sauna today having a field day and Eficode had a big presence at Kubecon this year. And we've got some of our participants here in order to talk about their takeaways and the fun that we had at Kubecon. First, we have Amruta. 

Amruta (01:11): Hi. 

Marc (01:13): Hello, thank you for joining us, Dan Glavind and one of our usual suspects.

Dan (01:18): Hello. Nice to be with you.

Marc (01:21): Nice to have you back, Dan. And Zander. 

Zander (01:25): Hello. 

Marc (01:27): Hello. It's always nice to have you, Zander. And once again, my usual cohort, Mr. Andy Allred.

Andy (01:33): Yeah, you're not allowed to do this without me. 

Marc (01:40): That's right. We might have a little different audio signature because some of us are in the same room today. But we're really excited to let you know our experiences from Kubecon. Let's start with Amruta. I think you were a first comer for this conference this year.

Amruta (01:56): Yeah. I had always heard about Kubecon and I heard from the people who have attended it that it is amazing. And I wanted to attend it. And I'm fortunate that I got to attend it this time. And it was amazing. It was an overwhelming experience; I would say because it was like 10,000 nerds in the same room. But yeah, definitely a fun experience. And for me, I was there for the experience, obviously. And there were a lot of new learnings. And the whole format of such a huge conference was new to me. And I learned a lot of things and learned about a lot of new tools as well. And new technologies and some of the hot trending topics in DevOps, and CNCF in general.

Marc (02:53): Are there any specific trends or technologies that you got excited about?

Amruta (02:59): There were few collocated events on the first day and different smaller conferences, I would say. There was one called Argocon, which had everything related to Argo CD, which was quite interesting. And it seems like a lot of people are building different tools using Argo CD. That was quite interesting. Also, Cillium and eBPF were two hot topics, which everybody was talking about. But honestly, I did not join any of the tracks with Cillium or eBPF. I don't really have much to say about it, but they were definitely something that people were talking about.

Marc (03:44): Did you network much at Kubecon?

Amruta (03:46): I did manage to talk to a few people from different companies. And it was nice talking to them, knowing about them, and learning how their experience has been about the conferences. There was one guy who was also a first comer like me. And he also had similar experience as me, it was nice talking to him. We discussed about all the things that we learned from the different talks and keynotes. And it was amazing talking to some new people.

Marc (04:22): I think that's really cool. And just reminding that some of us we get really nervous when we go to the conference, but everybody wants to talk about the stuff that we're all seeing there. I think that it's something to always remember you might not know what to say to somebody, but you can say did you like the last talk? Did you see this one? Or where are you from? All of this kind of thing that people really want to hear about it. Dan, this is not your first rodeo, not your first Kubecon. What did you find there?

Dan (04:53): No, this is actually my second Kubecon. I went also last year to Valencia and I share some of the same experiences that Amruta also mentioned that the insane amount of people, I think. I heard that there were actually 2000 on the waitlist for this year as well, it's just amazing. And some of the experiences or takeaways that I have is that, of course, I actually spent the most time talking and networking with all my fantastic colleagues where some of them are in this podcast as well. But also I had the opportunity as a consultant to go and be together with the clan that I'm currently working with because they actually had a presentation and worse on stage. Denmark had a talk about the Gateway API controller that they built called Bifrost in Danish, which is pretty awesome. And something that even the maintainers of Gateway API is really, really happy to see and they were like, you just made our conference, we haven't expected to see that kind of thing already built. That was really awesome. Besides that, I think as Amruta also hinted that Cillium and eBPF is huge thing. I think I-surveillance talked about that they had 18 talks around the subject this year. That was a big thing. And then out of nowhere, at least in my head, then we had web assembly. It even added its own co-located event. I'm curious on what that will shape into in the coming years, it still feels really fresh and new, at least in my head. I'm still trying to grasp all what I've seen until now and boil it down, but that's probably the things for me, at least.

Marc (07:03): There's a lot of nodding going on while everybody's talking here, we have a little backline video. And I understand that we've been talking about platform engineering in the DevOps space and in conferences for a while now. We're moving more into the nuts and bolts and the other tools even on how to implement these things. Zander, what did you see?

Zander (07:26): A good question. This was my second Kubecon, as well. I think it was interesting to see a lot of the things from what was continued from last, and everything that was missing from last year. Things that were continuing, platform engineering was a huge topic last year. And I think it was this year as well. But it had a different spin on, where last year, everybody was like, "Platform engineering is going to be the next best thing. It's going to kill DevOps. We're all going to be platform engineers now." And now it seems like everybody created a platform team. And now we're all figuring out how exactly to do it. Me and Dan are on a team working on a platform project, so that was very topical for us, I think. I think in general, also just overall maturity of the cloud native landscape. Things like GitHubs moving from being a GitHubscon co-located event to a tooling specific Argocon co-located event. And security also being a huge topic this year. Those are my main observations. 

Marc (08:49): Was there anything specific in the security space that you'd like to bring up? It's getting more important all the time.

Zander (08:55): I think security is hard. And I think, especially big distributed cloud systems, like we're working on are extremely difficult to have good security on. I'm not a security guy. I don't think I'm an expert to ask in this field. But I think it's interesting that security is hard. And we like to have tools that help us do security better, even if we're not security experts. And I think Kubecon showed that there was a lot of work going on making new tools, making new ways of integrating security throughout all of the tooling and platforms and infrastructure that we're using, that security permeates and becomes ubiquitous within all the tools.

Andy (09:57): As an industry, we have decades of experience doing security on prem with virtual machines and physical servers and whatnot. But then how do you translate that to the cloud, and cloud native and Kubernetes, and containers, and a lot of the tools that we have existing don't really apply, or they don't give you the similar kinds of benefits. And just the tooling around security now has kind of started catching up with cloud native. And you have tools, which can give you the same level of security, but in a cloud native way, and leverage things like eBPF, and things like that that you can get even better security now in the cloud with cloud native specific tools, which are becoming mature enough with the tools, which are also becoming part of the default standard for platform engineering and whatnot, that it's still not easy to be completely secure. But getting a baseline level starts to be easier and easier because the tools are much better integrated than they ever have been, at least in my experience,

Amruta (11:09): It is often the most forgotten aspect, I think. A lot of people don't even take security into consideration. But it is only now that people have started realizing that there are tools, which can help us build a secure infrastructure.

Zander (11:26): I thought it was very telling that in one of the keynotes, they had a graph over contributions to the cloud native community. And they were divided into categories such as scheduling, networking, so and so forth. And this year security was the largest portion of new contributions.

Marc (11:47): Good to hear. And just reminding like, isn't security everybody's?

Andy (11:52): AWS always says security is job zero. Does that mean it's the first thing that happens before anything else or it's nobody's job?

Marc (12:02): Yeah, the S in IOT is for security. All right. And then Andy, I guess this is your second Kubecon. Was there anything special about it?

Andy (12:19): It was a great conference, yes. Yeah, I had a rather surprising email one time a few weeks ago, that we are giving a presentation on Cillium from Esovalent. And we would like somebody to talk about how they're using this in the wild. Would you like joining us on stage? Yes. Okay, you'll be there with Liz Rice. I'm sorry, what? It was amazing. It was really nice experience. I don't even find the words to talk about how amazing it was to be on stage at Kubecon, and just that many people. And as Amruta said, it's like 10,000 nerds in the same building. And I got to talk in front of them. It was really cool.

Marc (13:23): Yeah, and you got tremendous feedback doing it as well. And I think there's a really neat story here, which is that one of the trends that we've been talking about in this podcast is that many of us we consider making a difference in our lives with our customers and our colleagues. And then we realize that we can step onto a stage somewhere like Kubecon, and all of a sudden, be affecting thousands, or even tens of thousands of people. There aren’t many platforms for us to get out and share the things that we're interested in. And people want to hear about what you do. I think that's a really fantastic success story, and they need to get up on the stage. Tell us a little bit about your perspective on the things that you saw and what you're excited about.

Andy (14:15): Obviously, I was on stage talking about Cillium. I'm excited about Cillium. But seriously, I do think that though anything and everything around eBPF is really interesting and intriguing and going to grow quite a lot, different things like how can you observe what's happening inside of a cluster and inside of a node? Well, in order to do that, you want to know how much resources and what type of system calls and what type of connections are happening inside your cluster so you can architect your whole system in a way to optimize around that, but how can you know what's happening? And there's different tools you can look at from the outside. But with eBPF, you can look at what's happening inside the kernel. It's really unlocking a lot of observability possibilities. Also, things like when it comes to security, we don't want to let people escalate their privileges. And there's different ways to prevent that. With traditional security, you just scan and make sure nobody's trying to do something. With eBPF, you can just make it not possible, the kernel will not allow you to change this file, for example. I think it's just really unlocking a whole lot of possibilities and making a lot of things that were difficult, much more performant, and much easier. And I'm excited to see what's coming next from those. And then last year in Kubecon, Dan and Zander and I were talking a lot that we saw crossplane this and crossplane that and crossplane was everywhere. This year, we didn't really see that. But if you listen to the talks, we're doing this and we're doing this and we use crossplane to do that. It was mentioned everywhere, but it wasn't really a highlight. It's just like, yeah, we use crossplane to do our infrastructure stuff. But the interesting part is, and I'm just wondering, that is next year Cillium and eBPF going to be it's in everybody's slide, but it's just a minor note. 

Zander (16:42): I think so. I thought the same thing with crossplane. It's very interesting that it seems like everybody adopted crossplane last year. And everybody's just experiencing that it’s not the easiest thing to operate, but it solves a lot of problems in a really cool way. I also thought it was interesting that everybody was talking about linker D last year, and I didn't even see it anywhere this year.

Andy (17:06): Linker what?

Zander (17:09): I guess everybody who adopted it are happy with it and then they just kind of… I hope for them.

Marc (17:20): If you'd like to hear more about Cillium and eBPF, we'll leave a link for you in the show notes to a podcast with Liz Rice, as well as her presentation from The DEVOPS Conference 2023.

Marc (17:40): All right. And I had promised to put Zander on the spot. I have this pet project. I'm collecting use cases for Cillium and eBPF. I had promised to put Zander on the spot for some of this as well.

Zander (17:55): Yes. I tried deploying Cillium earlier today. Unfortunately, it didn't work. That's my real hands-on experience with Cillium. But I saw a cool demo at the Cillium booth where they had a cluster, and they were using what they called huddle, which is their observability component. And they will correct me if I say something that was not 100% correct.

Marc (18:21): It does it to me all the time.

Zander (18:23): Yeah. And they had this cool demo where they had a topology map that showed the network in the cluster. They have from an ingress to service to a pod through a node, I think, you could visualize the network traces, but without instrumenting all of this, which is very cool. Because I think network tracing is one of those things we all want. But it's very hard to actually implement because it requires a lot of instrumentation. And it seems like you get that out of the box with Cillium and Hubble. I think that's a very cool use case. I would like that. Is that good enough?

Marc (19:05): That’s lovely Zander, and you inspired me with something here. Let me see if I can stretch this into reality where in quantum physics, we talk about observing the thing changes, it can change its state, but the whole purpose of Cillium eBPF is the other way. It's being able to observe without changing the state. That's okay, so am I like the king of the use cases now?

Andy (19:31): Yes. I’ll give you that.

Marc (19:36): Andy is patronizing me.

Andy (19:38): No. There was a time in a previous company we were doing telco signaling software. And we had this strange bug and we couldn't figure it out. We went and enabled some tracing. And when we enabled the tracing in the source code, the bug did not happen. We disabled the tracing and the bug's there, we enable the tracing and the bug's back. We finally figured out it was due to timing, how these connections are coming in. And that handful of milliseconds it took to generate the trace inside the software was enough that you never saw the bug. But in the case of eBPF because we would not have changed the software, we would have just watched what is actually the kernel trying to call, what is being called in the kernel. And when we would have been able to see this bug without trying to change our software to enable more logging.

Marc (20:39): And just a little behind the scenes, I can see the younger ones on the call, like looking and all that yes, when you turn on debug mode, all the bugs go away. That's where we come from. Really, really cool. All right.

Dan (20:59): Yes. And actually, talking about observability, as Zander mentioned, and maybe also Andy. I think there's something interesting, I hang out at the honeycombed booth so much that eventually they handed me their observability engineering book, which I'm reading right now. And it's quite good. But the thing from me was at least that some of these things that Zander also mentioned, like tracing is something that is maybe a little hard to do to implement in teams that are not used to using that. They may be just have logs and metrics as telemetry data, right? I think from that point of view, I think that you get much more details when you get these tracing mechanisms, eventually, and I think that's also some of the messages from Charity Majors who cite their founders that test in production. And I think to do so, you need to have a really good observability setup for that. It all aligns and goes in the same direction that if we can have better observability, maybe we can get rid of these stupid staging environments that nobody uses right anyway. That's my take on how to connect those things together. At least that's what I hope to see. All right.

Marc (22:27): I'm going to try one other thing, from the top of your head, does each of you have one speaker from Kubecon that you would like to follow in the future?

Amruta (22:40): Kelsey Hightower for sure. I think Andy as well. I look forward to seeing you do some keynotes next year.

Zander (22:51): I don't know. There were so many good ones. Hard to choose one. I think one of my favorites, maybe not that much of a topic that actually relates to a lot to what I do. But that I found interesting was a fun talk. I unfortunately, don't remember what her name was. But she did a demo running Kubernetes and micro VMs for a CI on her own home lab of four Raspberry Pi's that she brought with her on stage. She was dynamically creating micro VMs, both for running Kubernetes in and as hosts so like GitHub action runners in her cluster.

Marc (23:40): Was that Claudia Beresford?

Dan (23:42): She's from Weaveworks.

Zander (23:46): It was very entertaining.

Marc (23:48): Yeah, Claudia Beresford, Weaveworks. All right. Anyone else like to say the speaker that you would most like to follow from this year?

Dan (24:02): I think I saw a talk from Katrina Verey from Shopify, which is talking about 1 million lines of Yamo. And they have struggled in trying to incorporate ways of actually making executable code within Yamo. And they figured that would probably not be the way to go about it. But essentially, the ways that they have iterated on what is the best abstractions that we can provide to our developers so that they can get their jobs done. And they had principles that it should express that they wanted their Yamo to express the intent of their users not themselves and so on. I think in the end, they came to the conclusion that KRM functions was the way to go for them. And I think it resonated a lot with me because it's the same problems that I feel like we have right now is what is the contract in terms of the API between platform team and developers using our platform. That is something that I will take with me. And that's a talk that I remember. Another talk was and it's more like in the front end, [unintelligible] did a silly demo. I think that everything he does is silly demos on, I think it's like you choose where we had the possibility through voting app to choose which tools to use to build, which container solutions that we want, which registry you want to push to how you want to deploy to helm or something else. There's a lot of tools there and it’s just funny because it was like when we voted [inaudible].

Marc (26:20): Excellent. All right, Andy.

Andy (26:23): I'm trying to remember. There were two speakers on this, sharing a talk, two co-speakers. They were from Bloomberg. And this was actually at Cilliumcon on the previous day. They were talking about how they're using Cillium, and some other tools around the cloud native space to develop their developer experience and platform engineering at Bloomberg. And the way they were talking and explaining, they had ultimate use case description. And then this is what we were trying to do. And this is how we did it. And this was the technical implementation. And I thought they did really, really well. They might be on our future episode of our podcast. But I thought they were really good presenters from the, this is what the business is trying to get to this is what the developers need in order to enable it to this is the tools that they need to do it. They had a really great presentation, and I’m definitely going to keep my eyes on them. 

Marc (27:25): Excellent. Okay. Well, thank you. I think it's been really cool. And what do we got here? Cillium, eBPF, observability, huge things going on right now. Platform engineering is coming out into more practical steps, Argo CD, really big right now, security is everybody's business. But the tools are getting better to help everybody be more secure. CNCF landscape is getting much bigger in the security space. There's a lot to watch there. Am I missing anything else you want to add?

Andy (28:05): There was a whole lot of buzz around WASM, which might turn into something and might not, but that's not something I'm going to spend much time playing with yet, but definitely going to keep an eye on that in the in the future. That might be something to learn.

Dan (28:21): We haven't even talked about that. And that's funny in itself. But developer experience or maybe something like developer portraits like backstage, there were still talks on that, I think both Spotify talked about it. And also, Luna talked about how they made their paved roads towards the developers through the backstage. I think it's worth mentioning that it's still really an exciting project to think about, I think everyone at least a lot of people fall in love with it, when they see it. And then maybe also recognize that it actually takes some effort to kind of do it's not, I don't think it's an easy one to just jump aboard. But definitely, it's still a thing to watch for. And I also see a lot of other kinds of developer portals popping up from vendors, such as Port and others, where they trying to build the same thing, service catalogs, templating, these kinds of things. They're still interesting way of seeing the interface between platform teams and their users. 

Marc (29:35): All right. Excellent. Any final words, Andy?

Andy (29:38): Let's give Amruta a chance first. Ladies first.

Amruta (29:43): Yeah, another topic that we haven't discussed yet. And which was trending was about sustainability in cloud. A lot of people talked about reducing carbon footprint and power consumption. But when I discussed it with the people I met there while networking, I realized that there are some mixed reviews about it. But it is definitely a topic that is being talked about, and people are getting aware about it. And well, honestly, I feel that we should definitely do our bit and see if that's a trending topic in the next Kubecon as well.

Andy (30:23): I think it just might be. So I think that my main takeaway feeling of the whole conference was that I went to this one with the approach of the talks are going to be recorded, I can watch them later. Some of them I want to watch in person, but what I want to do is talk with other practitioners, 10, 000 nerds in the same building, what problem you're trying to solve, why are you not using this tool, why did you choose that tool and pick people's brains. And I spent a lot of time just talking with people and doing that. And I think that was invaluable. The main things that I'm going to try keeping an eye on is what we discussed, Cillium growing, more use cases around that. Observability with eBPF, things like that. See how WASM comes along, the backstage competitors there, dare I say, how that develops, but the number one thing that I got out of this was feeling like instead of I'm trying to solve this problem, but there are thousands of people trying to solve the same problem in the same time. The more we can leverage the great ideas that are out there and say, "Yeah, this tool was perfect for me, but I didn't use it here because of that." Okay, maybe that's why I'm struggling, should I change this? And those insights are really not so easy to get when your circle is smaller, so just the sense of community and learning from people in the community was really my main thing.

Marc (32:12):  Awesome. Hey, I'd like to thank you all once again for participating. This has been a DevOps Sauna field day covering Kubecon 2023. Thank you.

Amruta (32:23):  Thank you.

Marc (32:29):  Before we go, let's give our guests an opportunity to introduce themselves and tell you a little bit about who we are.

Amruta (32:25):  Hello, I am Amruta, I work as a DevOps consultant at Eficode in Copenhagen office. I mostly work with cloud native technologies and I love Kubernetes.

Dan (32:49):  Hey, I'm Dan Glavind and I work as the DevOps consultant from the office in Aarhus in Eficode. I am very much into platform engineering and that's what I do.

Zander (32:58):  Hi, my name is Zander. I'm a DevOps consultant at Eficode in Copenhagen. Today is my birthday and I brought cookies with Kubernetes logos on them.

Dan (33:08): Happy birthday!

Amruta (33:09): Happy birthday!

Marc (33:09): Happy birthday, Zander!

Zander (33:13): Thank you, guys.

Marc (33:14):  Well, we ruined your introduction. 

Zander (33:17): That’s okay.

Marc (33:19): And I'm Marc and I am your host. And I'm a lead consultant in transformations based in Helsinki.

Andy (33:25):  I'm Andy. I'm a DevOps consultant from Eficode, Finland. And I'm currently sitting in Copenhagen, enjoying eating the Kubecon or the Kubernetes cookies which Zander brought.

Marc (33:37):  Thank you for listening. If you enjoyed what you heard, please like and subscribe, it means the world to us. Also, check out our other interesting talks and tune in for our next episode. Take care of yourself and remember what really matters is everything we do with machines is to help humans.