Two weeks before The DEVOPS Conference 2022, our CTO, Marko Klemetti, and Mike McQuaid, Staff Engineer at GitHub and the Homebrew Project Leader discuss how Homebrew came to be, how it built an approach to contributions from the community, and what companies could learn from these experiences when they decide on their approach to open source contributions.
Mike (0:09):
Encouraging your employees to feel that they're able to contribute to open sources as part of their work. And particularly, if you're using open source projects and you have to modify them internally, whatever, like trying to get those changes back into the main project may be really worthwhile for the kind of wider community rather than just keeping them kind of locked up within your organization. And also, like cutting people a little bit more slack with that, that process will take a little bit longer.
Lauri (00:39):
Hello, and welcome to DevOps Sauna podcast. The DEVOPS Conference is coming again on March 8th and 9th and you are invited to join the event. To build the excitement to the DevOps event of the year, we have invited some exciting people to join our podcast and share a bit of backstory to the themes we will be covering in the event. Before I introduce our next speakers, let me pose your question: What is common between Linux, Apache, MySQL, and PHP? You guessed it right. It’s the infamous LAMP, one of the most common software stacks for many of the web's most popular applications, but there's more. They are all open source. They are all heavily used by commercial organizations, and they all have an active community that keeps on making those products better. While using and further developing open source products is so obvious, nearly not all companies have a solid policy for their employees concerning open source contributions. Today, in DevOps Sauna podcast, we have Marko Klemetti, CTO of Eficode, and Mike McQuaid, Staff Engineer at GitHub, and the Homebrew project leader. Marko and Mike discuss how Homebrew came to be, how Homebrew built an approach to contribution from the community, and what companies could learn from these experiences when they decide on their approach to open source. Let's get going.
Marko (02:12):
Just as a context for the listeners, how did Homebrew come to be, and where they come from?
Mike (02:18):
So, a guy called Max Howell is the original creator of Homebrew. So, he was I think a MacPorts user at the time working at Last.fm and he had a bit of frustration with the approach, kind of MacPorts took to building software on Mac where on Macs back then as would now, there are various libraries that are provided by the system but MacPorts kind of built everything from scratch. So, just to be safe and didn't rely on much of the Apple installed stuff. So, Max thought this was you know, like not a great system. And you could save time if you didn't do it that way. So, he kind of came up with Homebrew, this kind of beer-themed package manager. And this was yeah, like 13 years ago, maybe 14 at least count. And so, yeah. He started building this thing, like more and more people started using it and then I think maybe about you know six months after it started, I started using it myself. And I also built a sort of similar kind of hack around MacPorts to make it work the way that I kind of wanted and I came across Homebrew and thought like, yeah, this could be sort of interesting. So, yeah. So, I kind of joined the project, sort of contributed a little bit and got asked to be a maintainer and then you know, Max and some of the other maintainers from that period have since left and I'm the last remaining old person from who was around back from those days.
Marko (03:39):
Interesting. Yeah, I remember I've jumped on the Apple wagon. So, of course with my parents, my father took home the first Nix and then number of other Apple computers but then personally I jumped on the wagon from the first Air. And I remember using MacPorts and then there was this TextMate actually which introduced the new way of editing experience and then Homebrew, which were kind of new ways of working and made me at least really happy. What's the current status of Homebrew?
Mike (04:11):
Well, so I guess over the years, we've kind of grown more and more, you know, from I don't know how many users we would have had 13 or 14 years ago because we didn't really have any analytics or way of measuring. But you know, nowadays, we've got like a fairly decent number of millions of people who kind of use Homebrew on a given week. So, we're just trying to continue to sort of keep the project running, keep the project growing, and dealing with all the kind of scale problems that result from it being … Thankfully, like being used by more people doesn't really affect our scale as much as just packaging more software and that software taking longer to build and more dependencies and all these types of things that are kind of quite tricky, and yeah. It feels I guess, like a lot of companies I'm sure, it feels like you know, we're a year away from everything collapsing into a heap at any time but you know, we managed to keep it going.
Marko (5:01):
Also, I think the addition of Taps, I’m not sure when it was actually introduced but having Taps and people being able to use their own packages and that being kind of out of the main line is something I've enjoyed a lot.
Mike (05:14):
Yeah, I think that's good. That's actually one of the last things kind of Max built before he sort of moved on from the project where the idea of Taps being like Third Party Repositories that anyone can sort of build. We've been trying because we rely more and more over time on GitHub infrastructure. So, we're using, you know, GitHub actions now and GitHub packages for our binaries and stuff like that, you know, neither of which existed even five years ago. So, yeah. I mean, one of the nice things about kind of using these things that are provided by GitHub and hosting less and less ourselves is you know, we now have pretty decent flows for effectively replicating our like CI systems and if you want to build your own packages, or test them the way you want and stuff like that. So, yeah. That's been a nice kind of evolution of the project as well.
Marko (06:01):
Yeah. Have you, by the way, followed at all the so-called Wingets which is really fresh, like first stable version came like three months ago or so?
Mike (06:11):
Yeah, I played around with it very briefly. It's funny because I used to like back when I kind of first got started with Homebrew it was probably less than a year after first using a Mac. Well, not using a Mac but first kind of using a Mac as my kind of main computer. And, yeah. At the time, I was doing kind of cross-platform development, writing a lot of code which had to run on Windows, Mac and Linux, using QT. And, yeah. Back in those days, I was kind of pretty well-schooled on development on all three platforms. Whereas, you know, Linux has always remained hard to avoid, you know. I haven't run a Linux like desktop myself for a long time. But you know, I've interacted with Linux servers for basically that entire time. And I've been using Mac as my main platform from that time. But, yeah. I'm very far away from kind of using Windows for day-to-day development. For me Windows exists for my case, primarily just for playing games. And so, yeah. I dabbled a little bit with Winget from like, maybe I can use this to install Steam and my graphics drivers or whatever, but like, yeah. For me at least, like it's not compelling enough to kind of make that an essential part of my workflow, but if I was installing hundreds of development tools, then I think it probably would be.
Marko (07:22):
Kind of I see the biggest difference between Winget and Homebrew is that Homebrew is built by an open source community and the solutions behind Winget also where the noise as it's getting enterprise. How do you see the difference in Homebrew anyways, more like a developer tool? So, if you install a package, it's the latest and if you upgrade, you upgrade to the latest package. So, it doesn't the same way work in enterprise environment. How do you see that difference?
Mike (07:55):
Yeah, no. That's a really good point. I mean, Homebrew very much grew out of individual developers using it. And you know, now, I think it's, I would say, probably still the majority of our users. I mean, I don't have any real evidence of this beyond speculation and bug reports. But I think probably the majority of our users are people who write some sort of code in some sort of way, you know, like whether that's R or bash scripts or web design or whatever. Like, it's pretty broad at this point. But I think that's, you know, a big difference compared to as you say, kind of more maybe enterprise package management systems where the primary kind of users or not even the people who are consuming the software. They are the people who are like, you know, SaaS admins or network administrators or whatever, who are kind of trying to, like update a fleet of software. And as a result of partly where we come from, and partly where we still are, like, I think Homebrew has, I mean, I think it would be fair to say kind of an uneasy relationship with the enterprise in some ways and that there's a lot of things that a lot of kind of big enterprises want to do when they're using Homebrew. And we kind of say, like, well, what you're asking for would require a hilarious amount of work on our part. And we don't want to do that. So, we're going to build the tools so that you can go and do this yourself if you want to, but we're not going to do it for you because we don't want to and it's not fun, and we're volunteers. So, whereas yeah. If we were like a commercial company, or if we were a department inside of Apple or Microsoft or whomever, then I think, you know, we would have a very different relationship with that audience.
Marko (09:25):
But then again, representing a slice of the community, I can say that that's also the appeal of Homebrew definitely. Like being able to just having the latest package or are there new updates? Yes. Let's just upgrade. And there you go. So, for development purposes, definitely exactly what's needed, and going around the office, everybody knows what Homebrew is.
Mike (09:48):
Yeah, I think as well, there's been you know, I think there's been quite a lot of improvement in the last I would say like four or five years. And I think big cases like the Log4j stuff recently has highlighted this but I think more now than previously, enterprises realized, like, “Okay. Well, we have all these dependencies, and we want to keep things up to date.” Whereas, you know, like, I think the peak time for arguments, you know, maybe five, eight years ago or whatever, were like this sort of default situation for a lot of companies, including GitHub, back then which itself has got dramatically better at this stuff, particularly pre-acquisition of, sorry, post-acquisition of dependable but you know, the attitude was you get a say, a Ruby on Rails project, you get all your gems to versions that work. And then unless there's a known security vulnerability, you just sit on those versions forever, basically.
Marko (10:38):
I know what you mean. I so well know what you mean.
Mike (10:40):
Yeah. And the Homebrew in some ways is designed specifically to not really let you do that because, like, I personally feel like a responsibility, but I would rather a bunch of people get frustrated because something gets upgraded and broken every so often than knowing that we have built something that makes it very, very easy for people to set on known exploitable software and have their machines potentially, you know, compromised, because we kind of wanted to give them a nice easy way to do that. So, for me, like the kind of the Homebrew I guess is what you would describe as a rolling release package manager. So, we don't have unlike Ubuntu or Debian, this idea of like, every so often, everything has a nice stable base. And then we just do security updates. You know, if you've installed like MySQL, say, if a new major version of MySQL drops, you're going to get upgraded to that unless you tell him not to do that. So, yeah. We've always tried to have that attitude of like, you know, if you want to sit on an old version, you could do that. But you need to be quite intentional and explicit about doing that.
Marko (11:44):
Yeah. For me, actually, the only package that causes constant trouble is ImageMagick. Goes back to the legacy stuff, that's one of the only Homebrew packages that causes because of the other dependencies, maybe also open SSL, but that's where you're going to have those problems. So, talking about that open source and developing open source and of course, me as the CTO at Eficon would like to understand also what's the difference between individual and company responsibility for open source today. How would you see open source development, do you do it as an individual or as a representative of some community like the company?
Mike (12:26):
Yeah, I think that's an interesting question. I think it can be quite hard to tell because, you know, I do some of my work on Homebrew. And I guess particularly, you know, if you were to look from the outside, and you know, maybe haven’t read anything I've said or whatever, and you're like, okay, Mike is Project Leader of Homebrew. Homebrew is used by a lot of people. Homebrew’s a project on GitHub. GitHub uses Homebrew, Homebrew uses GitHub packages and GitHub actions, say like, presumably, like Mike’s day job is primarily work on Homebrew. And that's not actually the case. And we've been talking about this actually, within GitHub actually, like where you know, the idea of kind of whether someone's kind of an individual or kind of corporate open source maintainer or contributor or whatever it is. There's a fair spectrum where on one end, you have maybe, say, like the Linux kernel project where you might have someone who maintains a given driver and is employed by the company that makes that hardware, for example. And if they were to kind of quit, like that company, there would be no expectation that they would continue to work on, you know, improving the hardware for the company that they've left. And similarly, you know, if they work nine to five Monday to Friday on their job, that's when they're going to be answering emails related to kernel development on this hardware. And on the other end of the spectrum, you have like the far extreme other end of the spectrum you have someone who maybe works for a company that is very hostile to open source software, and that person, if you submit an issue about report or a pull request or whatever, they are only going to respond after 5pm, Monday to Friday, or at the weekend. And there's literally no overlap between that person's job and their open source development. But I think increasingly, more and more people are somewhere in the middle. So, in my case, I'm an individual. Nowhere GitHub has ever told me to kind of do things on Homebrew or work on Homebrew or well, people have occasionally tried and I politely declined because that's not what my job is. Yeah, but at the same time I do do some of my Homebrew work during my spare time and some of it during time when I work at GitHub and if I have a performance review at GitHub or whatever and I need to write things that I've done last six months, I may mention things that I’ve done on Homebrew that I think are relevant to GitHub because there is an overlap between there because you know, a lot of our engineers use Homebrew and like things like our use of GitHub packages before other projects did and stuff like that is kind of you know, good for GitHub as a company. So, yeah. There's that interesting kind of balance there. And I actually think like that, personally, I think that's maybe the best place to be and I would like to see more and more companies and open source projects work in that way where you have people who are doing things that are beneficial to their employer and they're encouraged to do those things during their work time, even if that is not always you know, I'm not getting… There's no project manager at GitHub who assigns me issues or things to go and fix on Homebrew and then I come back to them and say, “What should I do next?” And equally, the Engineering Leadership at GitHub does not tell me, you know, “Go make Homebrew do this. Go stop Homebrew doing this,” whatever. Like, so there is that detachment, but at the same time, I think it's a healthier balance. And I think both parties benefit from that because Homebrew gets some of my time and GitHub gets you know…
Marko (15:46):
What you produce essentially.
Mike (15:48):
Yeah, they benefit from Homebrew being better and they get to have open source maintainers in the company who can provide feedback because open source maintainers are very important to GitHub as a company.
Marko (16:00):
Whereas, at Eficode we have a few projects that we've been involved with for example, Helmsman, which has to do with Helm charts, and Kubernetes. And then we've been part of Robot Framework, which is an acceptance testing tool written based on Python. And it's become part of the DNA of open source to when we build a library for either to then release it as an open source, but I noticed that that's something that happens automatically. But then we are mainly a consultancy company. And I see being a consultancy company, and working with different open source tools. It doesn't automatically happen so that you would actually contribute back to that open source tool, but you rather try to make the environment work according to how the open source tool works today. Of course, there are exceptions, but that's my experience. So, it's kind of easier to build something small and release it as an open source than to actually join a bigger project. So, you can pick Homebrew or another example if you want, but how do you see people… We already see something like 10k people who have contributed to Homebrew. Are there people who have just decided to jump on board because of some trigger, or is it more of these like random contributions?
Mike (17:27):
Yeah, I think it's a bit of both. I mean, I've always thought like, I guess I've been involved for a little while with the open source community. I think I first got involved with kind of the main project I was first involved with actually writing non-trivial amounts of code for hours KD back like 15 or 16 years ago, whatever it would have been. And, yeah. It's something that people used to talk a lot about back then which they don't as much nowadays, but I would like to see a resurgence is the idea of open source being about scratching your own itch. So, like, it's about solving problems that you yourself have. And I guess the situation you described there, Marko where, if you have an open source tool, and you're wanting to use it, and you use that tool, and you don't contribute back because the tool kind of, you know, you instead, you maybe might modify the way you use the tool rather than modifying the tool. I see that as a great outcome. Like that's perfect, I'm sure the maintainers of that project will be happy to know that you didn't have to, like hack it to pieces to make it work the way you wanted it to. And I think that the perfect, almost like, I don't think firstly, that every user or even, you know, large numbers of users of a given open source project should contribute back code, almost for the sake of it. Because I think, you know, unless they feel like they have a need to do that. It's not terribly beneficial, but I think most of the people who get involved with Homebrew, myself included, all the other maintainers included, and I would imagine the vast majority of the contributors, not all of them, they bump into a problem with Homebrew that they specifically want and sometimes not even a problem. Sometimes just a thing that needs done, a task that it will make their lives better or easier if that gets done.
Mike (19:07):
So as a result, they do it and then we review it and then we merge it. And if they get very good at doing that, then they become a maintainer. I actually, yeah. I think that's the best way of doing it. I think at Homebrew, we have a really large number of contributors because again, we've taken a slightly different approach to package manager than a lot of other projects do in that we kind of like crowdsource updates, basically. So, you know, not something super popular, like W-get but say there's a lesser-used package, maybe you know, gets 100 or 1000 downloads a year or whatever. We're not going to notice, you know, at this point thousands of these packages, we're not going to notice necessarily when that gets updated. So, we're trying to build more tooling so we can alter to take some of this stuff, but a lot of this thing through the majority from registry, that gets updated when someone who uses it is like, “Oh, I wanted the latest version of this, and it's not in Homebrew.” So, I'll submit a pull request, update this thing. And what we've tried to do is instead of automating that way, we say, “Well, we're going to try and make it as easy as possible to do that.” So, there's like a Homebrew command you can run which literally you just type in the name of the package, the name of the new version, and then it will like basically test that, make sure that the new version actually exists or whatever, and then it’ll open a pull request for you. So, you can do that without really ever having used GitHub before. And you know, you’d need to have an account but it's basically trying to lower that barrier of entry so people can get involved much more easily. And I guess we, you know, some of those people will do that once and never come back. Some of those people will do that. And they'll only ever use that tool. And then some of those people will use that tool and then they'll kind of grow beyond that tool and they'll start making contributions. And maybe become a maintainer or whatever it may be. But, yeah. I think that the idea of kind of trying to make it really easy for people definitely helps, combined with the idea of basically being a bit lazy and being like, “Well, we're going to let our community handle some of this rather than the expectation that we're going to handle it all for them.
Marko (21:09):
Yeah. Sounds very reasonable. Also, probably something that's kind of a modern approach to open source, making it easy for the contributors, and also like exporting the parts that you simply couldn't maintain to the community in such a way. You've written a lot of the ways how people should contribute. And of course, we see the guidelines and automation and everything in place in most modern open source projects already. What would be like main tips for a new contributor?
Mike (21:46):
Is that for a new contributor who wants to get involved?
Marko (21:50):
In an open source, yeah, you can pick either Homebrew or just open source altogether.
Mike (21:54):
Yeah, okay. Yeah. So, I would say if you're, you know, say you're a random person who can code already, but you want to get involved in open source, a good starting point is that I guess scratching your itch thing as I said earlier, where, if you think about, like, what tools you're using on a day-to-day basis, like, which of those are open source nowadays, probably many if not all of them. And being like, “Well, what's the smallest change I could make that might be useful to me?” Like what's a little script or… And some of that may, you know, I think that's the other thing is with, say, if you're using VS code, the smallest change might be making your own plugin rather than modifying another one. And I think that's, you know, a valid contribution to the ecosystem even though technically that would be probably filed under creating your own open source project. But yeah, but say you found another plug-in you're already using for VS code or whatever. And you kind of modified it and got it working locally, then it's kind of you know, just the process of submitting a PR, like making sure the tests pass, stuff like that. Like I think it's hopefully like route. It varies from project to project exactly how projects want that done. I guess, in terms of like higher-level principles beyond just the scratching your own itch, I would say quite important one when you're starting out, it may well be if you're a contributor to open source project that you could be anyone from someone who learned to code last week from someone who has been building applications for 40 years. But if you're going to another open source project, and you're submitting your code, even if you know better than they do, you need to rein that in, because ultimately, you're going into their place and saying, “Hey, please will you accept my code?” And then I guess, the thing people maybe don't think about always is you're saying, “Please will you accept my code,” and then when you merge it, “Please, if there's any bugs with my code, I want you to fix them on my behalf in future basically.” So, it's giving someone your codes but then they have to look after it. So, I've heard that kind of in both angles. There's been some good articles going on recently at the idea of, you know, the discussion and kind of, open source of like free as in speech versus freedom as in, like, you know, like a free beer or whatever. But then the other idea of like, free as in a puppy. So, if someone gives you a puppy like that can be a nice gift, but then it also brings, if you accept that gift, you now have a responsibility to do stuff and like look after that puppy and feed that puppy and pet that puppy and whatever. And I think both for open source project creators, but also for contributors, you need to think about what you're doing as a puppy. So, if I submit a pull request to a project, and it's 1000 lines of code, I'm giving that project my puppy and saying, you know, “If this puppy meets your requirements, please merge it in” but then it is your puppy now and then if it breaks the project, like, I'm not going to be getting issues sent to me being like, you know, your pull requests broke by workflows. That's probably going to be directed to the project owners. So, if project owners or maintainers seem in pull requests like that, maybe being a little bit pedantic or asking lots of questions or whatever, and you're thinking, “Oh, why didn't they just merge this? I know it works.” It's like, well, they need to know it works as well because they need to be confident with looking after the puppy that is your change.
Marko (25:08):
This is really interesting. Because for me, thinking about open source, it's always like, I come from the Finnish background and looking at our Finnish consultant and the whole thinking in like Finland often is very humble, which means that I'm not good enough to contribute to open source. And as you already said, you said the keyword like even if you've coded only a week, you could already make a contribution. And many contributions would be just upgrading numbers in readme files or just Wiki pages, or similar. I would see that as valuable work for the community as the code itself.
Mike (25:48):
Yeah, definitely.
Marko (25:49):
But the puppy concept is pretty cool. So, I'm going to ask another question. How do you see the puppy being given out? Can the person themselves decide if they want to also take care of that puppy? Or is it always just like, given that the initial thing is that you give the puppy out and…
Mike (26:09):
It depends on the project. And I guess, yeah. The puppy metaphor originally was, I forget who created this, a complete genius. I could probably find it and we could put it in the show notes or something. But the original idea was the idea that like an open source maintainer, like when I say you can use my project, I'm saying, you know, the gift I'm giving you is like a puppy because you can use my project, but from the open source licenses, there's no guarantee that that works. There's no guarantee it will meet your use cases, as I mentioned earlier with the big enterprises who use Homebrew. Like, there's no guarantee that we want to support your use case. So, that I'm giving you something, but with the expectation that you may need to do work to make it work for you. And you can't come back to me and be like, “Hey, I don't want to do that work,” because you've agreed to that. But I guess in reverse, like the kind of contributors contributing their puppy to the project idea, like I think, yeah. There are some projects where, you know, if you have a single contribution, they will add you as like a maintainer, and then you can immediately start maintaining your own code that you kind of have merged into the project and stuff like that. I think it depends from project to project, whether that's the way they do things. So, if it's like an MBM, like, say, an NPM module, often those are so small that like, it doesn't make sense to have like 40 maintainers for something which has like 50 lines. But then on the other side, if there’s something which has like a very, I think Node.js was famous for being very liberal with adding lots of new maintainers if people contributed. But part of that I think was on the back of like they had a you know, a release process and the QA process to make sure that kind of you know, any maintainer could randomly start changing the code, but like we were going to make sure before we did a new release, like a 1.0, 2.0 or whatever, that everything worked the way it was meant to. And if not, then we could fix things and work around things or whatever. Whereas in Homebrew with it being a rolling release, we kind of by necessity have to be a little bit kind of stricter on that. Because if you have the ability effectively to push out new software versions to millions of people, like I want to see more than like a single good PR before I give you that much trust, basically. But yeah, but on other projects, like if I was making a GitHub repo that was just a collection of lists or whatever, then yeah, sure. Like this very limited damage you can do to people by just being a maintainer there. And so yeah, like I would be more liberal and willing to kind of have people come and join more readily.
Marko (28:38):
I would reckon also that that comes out of the community work and the experience, how it eventually lands into being… So, that's also interesting in the open source community for me. Like in any community work, usually you end up into a pretty good solution as a community, as opposed to just one person doing or just having a business target for certain thing. So, it's like crowdsourcing in a way.
Mike (29:05):
Yep, exactly.
Lauri (29:10):
It's Lauri again. At Eficode, we want to build the future of software development. We are part of European research projects on test automation called Testomat and serverless technologies called RADON. People at Eficode also contribute to a variety of open source projects and are responsible for several open source plugins. These are associated with a Robot Framework Jenkins, Kubernetes, and Helm. You can find a link to our open source projects in the show notes. Now, let's get back to our show.
Marko (29:44):
So, I know that a few years back, GitHub promoted the open source Friday as the day to work on open source. What other ways of contributing to open source would you see for companies such as us, for example?
Mike (30:01):
Yeah, so I think the open source kind of Friday concept was one that we've been doing at GitHub internally for a while, and then we sort of rolled it externally with this idea of, you know, you try and contribute to open source a little bit every Friday, at least. And that's, you know, like, not everyone in the company kind of follows that and some people do more than that. Some people don't do that or whatever. But it can be a nice, almost like framework, I guess. Almost like a kind of new year's resolution you might be like every Friday I'm going to go for a run like instead every Friday I'm going to well not instead, you don't have to pick between running and open source but doing both at once, I think is quite hard. And the laptop tends to fall if you're running too much. Sorry. But yeah, no, I think for companies, there's definitely lots of different ways of doing things as well. So, like as a base level, I think encouraging your employees to feel that they're able to contribute to open source as part of their work. And particularly, if you're using open source projects, and you have to modify them internally or whatever, trying to get those changes back into the main project may be really worthwhile for the kind of wider community rather than just keeping them kind of locked up within your organization. And also, like cutting people a little bit more slack, that process will take a little bit longer. Like it's a better long-term solution for I mean, I personally see almost like internal forks or open source projects as inherently tagged at essentially. Whereas if you can get those changes back into the main projects, then that's going to be better for everyone involved. But then if you can't contribute code as well, then I think sponsorship of open source projects is worthwhile as well. So, either through something like GitHub sponsors, or again, if you have like personal connections as well, there's the potential to do things like sponsor meetups for that open source project or conferences or whatever it may be. Or then there's, again, depending on whether open source projects are run by individuals or groups of people this may or may not be possible but with some projects as well, like if you are kind of having a lot of difficulties with a project, like you know, rather than going on the issue tracker and being really demanding and stomping your feet or whatever, then perhaps reaching out to the maintainer and saying like, “Look, we're a company. We maybe have tight timescales or whatever, would you be open to our consulting arrangement, potentially for where you can help us with an open source project but kind of for money.” I think that can be but sometimes not like directly kind of advertised and not every open source maintainer would want to do that. But I think that's like can be a good way to sort of help open source projects and help kind of companies benefit from open source and individuals benefit from that. But yeah, I think my go to for this stuff would be even doubt, like, so if the project has a GitHub sponsors page, then yeah. That's generally a good way to kind of get started and that generally sets good expectations of like what the maintainer of that project would be willing to do or would expect from or given return for any sponsorships.
Marko (32:53):
Yeah. And that's actually also non-obvious. Two non-obvious things I noticed there. The first one is the sponsoring and indeed the GitHub sponsors, and also having people like doing contract work based on for example, an open source library is really good. You oftentimes see in Reddit or similar just is funny puns of maintainers of projects that there are like job applications with a certain amount of years of that particular library, and then the maintainer don't even have that amount of experience with it. And it's kind of like the exact opposite.
Marko (33:28):
So, I'm not sure how much is talked about open source and DevOps and automation. You've already mentioned about Homebrew various automation methods and continuous delivery and automated releases and everything and I see that from your blog, if someone's interested can also read heaps more of the way you've done, like continuous delivery. What I've been thinking also is in open source community, it's not a topic that's spoken a lot. So, because the core of open source is usually a product, something that you create for a purpose, and then you have contributors to fix it. But surprisingly, most of the open source projects do have proper testing and proper release methods and good practices and pull requests and all that in place. Where do they come from and how does the community know to build this so-called DevOpsy environment around also open source products?
Mike (34:25):
I think that's interesting, because I think what you see there is kind of the sort of mutual influence the open source community has with the kind of commercial community where there's definitely almost… Well so, I've never had a job doing DevOps stuff, probably and certainly not any real scale before GitHub, you know. I've probably done a little bit of what I would have called system administration, but not much in the way of kind of organizing like, you know, large-scale CI or CD or whatever. And I think Homebrew’s definitely taken things that I've learned at GitHub, and kind of made that like ways that we use things, you know, feature flagging code and stuff like that as being kind of a useful deployment method. But then, you know, I'm able to take some of that back into GitHub and things that I learned from Homebrew. But I think with Homebrew, there's always been a little bit of a kind of bland, maybe, I guess, DevOps, in a word from the fact that we haven't had almost like dedicated servers and system administration folks who were like managing those and then the people who are actually making the application. There's always been a little bit of everyone does everything and the like most of the kind of CI/CD stuff, like when I joined Homebrew, we didn't have any CI. Like every change was manually tested on a machine of a maintainer or just, you know, idle scanned and looks good, ship it sort of thing. And as the project's grown and kind of scale and the number of contributors or whatever, like we dealt and I guess particularly I, like I'm a sucker for like automation, like more tooling, more automation, like doing less stuff manually, because I'm exceptionally lazy, basically. And I'm lazy and I'm not good at doing things in a high-quality way unless like, like automated or documented, like how to do so.
Mike (36:12):
So yeah, it started off for me with like, as testing user contributions as being like a little command I could run and I just copy-paste a pull request URL, and then it will churn away on my machine and do essentially like local testing because this is before we had any hardware. Then we did like, this would be in like 2012 or something we did like a Kickstarter because that was how you got money on the internet back then. I got some hardware but then again, this was very manual because you know, I had a friend who run an ISP and I like physically took the Macs and set them up in the ISP and then you know, we set up Jenkins to kind of run on these machines. And then over time, like, you know, we were able to kind of move more of this stuff to the cloud eventually.
Mike (37:54):
And then by the time GitHub actions comes along, we're able to move effectively, the Jenkins piece into GitHub actions and then we just have our self-hosted runners now which are running in Mac stadium, like on our kind of virtualized machines, most of them. So yeah, so like, we're definitely kind of a lot more like DevOpsy than we used to be in that regard. But yeah, it's funny because it's just felt like a fairly slow but like natural evolution of almost like building up like, what do we need to do, what are the tasks that we're doing right now that could be automated, and partly as we want to reduce those times, but also as the project grows and the scale of the project grows, and we aren't able to scale our people as much as we scale users or contributors or whatever, then being like, yeah. We have to automate more of this stuff, so yeah. It's funny because the expression continuous delivery is something I probably only heard like four or five years ago or something like that. But it's something I've always personally internalized from like through I guess more or less my whole career. Like I worked previously at a start-up called Mendeley and I only worked there for a year. I was the first employee but we had, you know, effectively on any build, you could like push a commit, and then it would spit out like an installer for Mac, Linux, Windows, which you could like download and run on your machine or whatever. So I got fed up with like my bosses being like, “Hey, we want to run a test build of the latest thing” and I'm like, “I'm fed up with like, you know, manually making your test build every time.” There you go, right. The server now, you just go to this folder, and you can always get the latest test build and you don't need to bother me anymore.
Mike (38:28):
And yeah, I'd taken the same approach into stuff like Homebrew where, like the actual process for accepting a user change is very much kind of automated. So, if I have a pull request on Homebrew and it's green, where I can basically just approve that pull request and then binaries will get uploaded, it will get merged into the main branch, it will get deployed to users. And it's only when the pull request is read that it's maybe a little bit more time and effort is required to figure out like what the problem is, why is this happening? Maybe explain it to the user, and maybe debug things, you know, maybe some of our infrastructure is broken, whatever it may be, like making almost like the happy path really, really easy and really nicely automated across our infrastructure has been a really big win.
Marko (39:11):
It's interesting, I get two different concepts from here. I've always thought… So, my first, I think my first open source contribution is actually to Cruise control.
Mike (39:22):
Oh, yeah, yeah, I remember that.
Marko (39:23):
Jeffrey Fredrick was working on it. And then there was something like back then there was like Java build environment, and there were some libraries that were incompatible. So, that was my first one. So, continuous delivery already from there has been in my blood and how I see the DevOps work, it's more like a lazy but efficient, in a way. And it was funny that you mentioned it yourself because I can identify myself there. But then again, also as DevOps and all these continuous delivery and automation has matured, I can also see the other side of the things which is we need this package and we want to try it out. We want to ship it to our customers is that you created the like installation package creator for the exact purpose of being able to deliver. So, quite literally explaining the DevOps and it's yeah.
Marko (40:19):
I remember when Travis CI was introduced the first time, they were like GitHub started showing these labels of build passing. And then there was this code quality passing and similar. And now of course, we have GitHub actions that's already doing that for us. And also, if we look at release notes and everything, everything has been more or less automated, and it's contributing to libraries. For example, GitHub, or say Google manages. There's also automation for the license agreement of actually approving that when I'm giving you my puppy, it's okay to give this puppy and you own it now. So, I guess one question comes to mind when looking at what's happened in the past in the open source development and of course, your experience from prior is, now that there's more automation, I noticed that you said that the automation works as a more gentle commenter on things that should be changed on the pull request so people easily fall in love with their changes and if they don't quite match with the contribution to the actual core code. If you comment it as a person, it's more easily taken as a negative. Whereas, when you're doing it with automation, it's kind of you're like, “Okay, this is the automation. The automation says that I should take it…” like, it's like the automation versus empathy thing. So, how would you see automation changing that now and in the future?
Mike (41:52):
Yeah, I think that's a really good point. And I completely agree, like I think, like there is a different dynamic from getting the same responses from a bot and from a human. And I think, partly as well, and it's like, I guess working in GitHub, the company and also on GitHub, the platform, like I live in Scotland, a lot of people who work on Homebrew live in America, and it doesn't work very well from us to have a continually back and forth because it may be, I make a comment and then 24 hours later they well, not 24, 12 hours later, when I'm asleep, they respond to my comment and then 12 hours later, I make another comment, you know. So, they can quite easily take a week that way, whereas if they can push a change and then get some like that feedback straight away, and then they can have that back and forth with the bot rather than with me. You know that is, in my mind kind of inherently empathetic, where they might not like what the bot is saying, but at least they're able to get that feedback quickly. And hopefully the bot can communicate how they could maybe even run that on their machine to get that even more quickly. But I think as time goes on, I think we're going to see more and more both open source projects and kind of commercial projects kind of lean into this sort of automation because I think something that I've always felt is and I try and nudge other Homebrew maintainers and I guess people inside of GitHub to do is you know, if you're making the same comments on pull requests again and again, you need to ask yourself pretty quickly like how could this be done by a bot rather than by me. Because in the software industry in most of the world, like CPU time is way cheaper than human time. You know, in the software industry, more or less, the most expensive resource is the time of the people that work within it.
Mike (43:33):
And particularly, I guess as you get more senior and hopefully well-compensated, then your time becomes even more valuable. And you know, you simply can't afford to go and spend hours and hours and hours nitpicking things that could have been easily done by a bot software. And yeah, I think that's like a direction of travel that we're going in, but I think there's a balance there where it's making sure you don't try and automate the parts that should not be automated. So, like I remember when people were leaning more and more into the stuff a few years ago I saw some open source projects would automate a lot of bots would say, “Oh, thank you for your first pull request or whatever.” And to me at least like that doesn't mean as much. Like I don't mind a bot saying my code is bad. But when a bot says my code is good, then it's like well, you know, you're just a robot like you're not, you know, there's not almost like emotion or empathy or human connection in that part. Whereas when like a human says, “Hey, like, you know, I'm really happy to have you do this. Thanks very much. Like I'm grateful for your work or whatever,” then that human connection’s a really nice thing.
Mike (44:36):
It's funny because people have, I've met various people who've contributed to Homebrew over the years at conferences and things and people come up to me and say, “Oh, you know, my first contribution you sent this really nice message.” So, thank you for that. And it's funny because you know, people have different views, but I have like a safe reply on GitHub, where I say, you know, like, “Thank you for your contribution,” like “You rock” and then it kind of inserts the person's name in there. And pretty much everyone gets the same safe reply. But, you know, I still mean it. Like it's just that's a way for me to communicate what I feel very quickly. And I still personally think that that safe reply coming from me as an individual means even though you know, it's a couple of key presses, it means more than it does, even if it was longer and more articulate, coming from a bot when people just know that that's an automated response.
Marko (45:24):
Then again, I always look at things from scaling point of view, and I'm quite conscious with my GitHub profile, as in how many comments do I do because it also it reflects really well my mental well-being and if I'm somewhere at the level of 2k comments per year, I know that I've been pretty healthy that year. And if I don't do comments, it means that I have had something that has like, taken the jewels to do actual work, or actual coding work and looking at, for example, your profile and the fact that you really do review a lot and a lot of pull requests, being able to be just scalable in the way that you give feedback and give props to the people who actually earn it is for me a natural way, the same way as an automation being lazy but effective.
Mike (46:19):
Yeah, I like to think so. Because, yeah. I mean, that's what I want to do, right, is I want to make someone feel as good as I can in the shortest span of time. And I mean, I guess in some ways, it's like any other human relationships, right? Where like, if I got my partner some flowers or chocolates or whatever, like it would mean more if I like, maybe grew those flowers myself in a forest and then give it to them or I baked those chocolates by hand. But at the same time, like, you know, that maybe doesn't scale and if I said, you know, “I baked these chocolates by hand, but I'm afraid I didn't have time to like, pick up the children or buy our food or do this because I was too busy making those chocolates,” then they might not taste as good as some nice ones bought from a shop. Still, you know, my heart went into it, but I've also done the chores as well, you know, like, so yeah. I think it feels like I don't know, like, as I get older and more experienced in the industry and in open source and in life, it feels like everything's a tradeoff really where like, you know, and what I guess what you're trying to do for me with like open source is like figure out those places where I can kind of dedicate my time most effectively so that, you know, if I double the time, I'm going to get double or more the impact. And be very wary of the cases where I spent twice as much time and I get 10% more impact or whatever it may be. Because you know, I guess as a counterpoint for an example, you know, people have pointed out to me in the past that well I went through a period of kind of maybe being more involved on Homebrew but my responses were more terse, and some people were upset by them or like thought I was a you know, an a**%$%@ or whatever it may be. And that was never intended, but it was just like when I'm trying to go and reply to 100 comments in 30 minutes, that I'm just going to be like, type, type, type, answer next tap, type, type, type, answer next tap, you know. And yeah, it's probably been a little bit better for me to slow down a little bit, write longer, nicer messages, maybe get a little bit less done, you know, than have people think rightfully or wrongfully that I'm an a**%$%@.
Marko (48:25):
On the DevOps side, like last question on there would be do you see a balance change in the actual logic? So, as you said, you build Homebrew without any automation in the beginning and now of course, we have all these GitHub actions in place and similar. Does the balance change in creating the automation or the harness for the product and then the product itself somehow? Or do they work hand in hand in such a way that when you have the harness, you're also faster in developing the actual product?
Mike (48:57):
Yeah, that's a really good question, because I think there's, yeah. There is often this kind of balance between say, you know, say I've got a project using GitHub actions, like I could probably, you know, I could either have maybe one step which runs a Ruby file, which then runs 20 steps, or I could have 20 steps, which individually run 20 different Ruby files, or, you know, use GitHub script to embed JavaScript in the Yamo or whatever it may be. Like, yeah, I think it's always worth trading off those two things and thinking like, where does it make the most sense for that code to live? And also like, where is most easily testable and stuff like that? So, for me within, like GitHub actions, if there's code that will literally not work, unless it's running GitHub actions, and there's no way of testing on my local machine, and it won't work on my Mac and all this other stuff. Then to me, it's like, well, there's not really any harm in that just being embedded in the Yamo or, you know, some script that is in a repo that has essentially no testing other than GitHub actions running it for you. But the minute you're almost like you're trying to emulate something which I would want to reproduce on my local machine or whatever, then that's when I personally try and push everything to the application logic. So, like a lot of Homebrew CI operations, they run like a brew command, which then you know, takes environment variables from GitHub actions and does various things with it, or our testing framework, whatever it may be. And the motivation for that is so that we can more closely match the environment on our CI/CD DevOps platforms and on our local machines and try and, you know, have that abstraction live in the application code, rather than that abstraction be something which the user has to kind of figure out themselves if they're trying to reproduce those sorts of issues or errors or, you know, differences between platforms.
Marko (50:51):
Yeah. Interesting. So, last question for you, Mike. What's next for Homebrew, or Mike?
Mike (50:58):
For Homebrew, next is like we've got you know; we've always got various different people working on different things on the go. So, on the DevOps side, we're trying to move towards having we've got another Scottish maintainer right now called Bo, working on making our CI workers more ephemeral. So, we have I mentioned before this kind of like VMs that are running a lot of our build jobs, but they're not destroyed and recreated on every job when you have like an X code update or whatever. Like there's a certain amount of manual clicking around that’s required to get stuff done. So, we're basically trying to automate more and more of that as much as we can. So that we can have a CI environment and sort of more easily repaired, etc. And, you know, again, requires less human time to do stuff. So yeah, so that should be exciting. And then on the kind of Homebrew user side, we're trying to move towards so like one of the slowest parts of Homebrew at the moment is the auto update process. So, doing like a GitHub update of our kind of biggest repository because it's been around for a really long time and had a lot of contributors is really slow. And so, we're trying to make it so that you can like basically generate an API for that so that you don't unless you're a contributor need to have that Homebrew core or Homebrew cask repositories on your local machine. So, that again, that's been kind of worked on at the moment, and when that kind of rolls out properly, then that'll hopefully result in some pretty big speeds up for things like you know, updating and auto-updating and stuff like that. And yeah, what's next for me? I don't know. I'm just going to continue working along GitHub, work on Homebrew like I've got two small children. So yeah, they stopped me from being too over-ambitious with what else I'm going to achieve outside of work nowadays, so.
Lauri (52:41):
Thank you for listening. If you want to continue the conversation with Marko and Mike, you can find their social media profiles in the show notes. You can also find the links to the articles that were referred in the discussion from the show notes alongside some interesting related content. If you haven't already, please subscribe to our podcast and give us a rating on your platform. It means the world to us. Also, check out our other episodes for interesting and exciting talks.
Lauri (53:18):
I would also like to invite you personally to The DEVOPS Conference happening online on March 8th and 9th. The participation is free of charge. You can find the link to the registration page from where else than in the show notes. Finally, before we sign off, let's give Marko and Mike an opportunity to introduce themselves. I say now, take care of yourselves and remember this, much like with puppy in the open source, if you know what you're getting yourself into upfront, it can be a rewarding experience.
Marko (53:40):
Hello, everybody listening. My name is Marko. I'm the CTO of Eficode. Eficode, as you know, is a company employing roughly 500 people in eight different countries. What we do is DevOps consultation and different solutions in DevOps, our own tooling-as-a-service solution. And then lots with work with our partners, namely GitHub, Atlassian and similar. I just took a look we have at Eficode. We have Wait4 which was built for Docker Alpine images, to wait for other containers. It has something like 1400 stars currently and that would be my biggest open source contribution. Of course, lately I've been working on the number of DevOps tools like the Azure, GitHub actions and TerraForm, Google provisioning tools and such. Personally, I also enjoy working on open source.
Mike (54:33):
I'm Mike McQuaid. I'm a software engineer working at GitHub. My title there is staff engineer on the community's team so our community's part of the company focuses on building communities on GitHub, mostly the open source community and the sub- communities within that. And then outside of my day job I'm also the project leader for the Homebrew Mac OS and more recently Linux package manager, which I've been working on for I think about 13 years.
