Dan Lorenc, CEO and co-founder of Chainguard, joined Dennis Fisher on the Decipher podcast last week to discuss the rise of software supply chain security threats, the challenges of asset inventory and management, and the value of Sigstore for code signing. This is an edited and condensed transcript of their conversation.
Dennis Fisher: Where did the idea for what Chainguard is doing come from?
Dan Lorenc: I think the overall idea in the space of supply chain security kind of came up gradually. I was at Google for about nine years like you said I started there back in 2012 I think it was and worked on a bunch of different things throughout Google cloud platform. Kind of from backend infrastructure and then later out toward kind of open source developer tools in the container and Kubernetes space and Google in the 2012, 2013 timeframe that was right when big nation state attacks started happening to most of the big tech companies. I’ve heard similar things from Microsoft and Amazon. Pretty much all of the big tech companies started noticing. This was happening around then a lot of it’s been talked about since at the time nobody was talking about it at all. It was top secret but it was kind of like a crazy revelation that you know in your job you might encounter nation States trying to attack you and you know they might even go as far as having folks go get jobs at the company and try to compromise it from inside and that kind of caught everybody in the industry by surprise back then. It’s a little crazy to think about now when it happens all the time and it’s pretty obvious, but back then it was new like folks weren’t used to operating systems that way.
And so we spent a couple of years after that dealing with the fallout from realizing that you can’t actually just blindly trust all employees to have access to all sensitive data when your company that size is dealing with that much sensitive information and systems had to be architected completely differently and baking in that culture of multi-party review and two-factor review to every single thing that happens not just access to production but code review, compilers kind of all of that stuff, and then when Kubernetes and containers and public cloud and Docker and everything started catching on a few years later and I started working on that, it was like stepping backwards like almost a decade and it was like wait a minute all that stuff we just built is gone now. Everybody’s building stuff on Jenkins machines in closets and under their desks and nobody’s tracking what goes into software and how it’s getting built. And so that kind of made me pretty paranoid for some of the stuff I was building in open source and shipping and kind of led me down this rabbit hole. It was pretty boring for a while like yeah, nobody really cared about this at all and honestly just felt you’re bothering everybody. Until SolarWinds happened honestly at the end of 2020, then it was like a night and day switch went off and everybody was like hey why haven’t we been doing this for forever. It’s so obvious in hindsight and so that’s sort of how I got into this field, and the field kind of grew up.
Dennis Fisher: There was a lot of outward-facing changes that Google and other companies made, encrypting the links between their data centers and all that kind of stuff but it’s cool to hear about the internal stuff too where you’re looking inside and you’re looking around and being like well why do we trust this system. Why do we trust this person?
Dan Lorenc: It’s a big shift in the way you build systems and you know there’s no perfect answer here. The best you can really do is have multiple people look at something in those situations because at the end of the day you are trusting people. Trusting people blindly is also terrifying especially when you’re working on open source landscapes where you’re taking code from anybody on the internet basically and if anybody has spent time on the internet, you realize that not everyone on the internet is a nice person and deserves your trust, and yeah it leads to kind of these inverted security setups in a lot of companies that we see too, just based on policies being applied. If you want to get a new vendor approved at a company you have to go through a crazy vendor approval process and security audits and budget approvals and all this stuff and it can take months. But if you just find an open source project on GitHub you can pull that in without asking anybody in most cases.
Dennis Fisher: You mentioned when the SolarWinds attack happened, which was the end of 2020, it seemed that was kind of a watershed moment for a lot of people in the security industry and also in the broader software industry I think too. They started looking at the dependencies and how many people had SolarWinds in their environment and how would they know if their version was compromised. So did you kind of look around and say, I told you. I was trying to tell you guys.
Dan Lorenc: Yeah, sort of. You know, a lot of it was like well you know you take this as an opportunity to do a tabletop exercise at your company. if this happened to us, like how hard would this be for us to detect and remediate and fix and do we actually have any controls in place that would have prevented this? A lot of organizations around the world are probably doing that around that same time and you know I’ve seen spreadsheets from CISOs of massive companies. They’ve showed me right after SolarWinds, that attack happened. You know we did an audit and you know we found 400 different Jenkins servers that were in use today across our company and it took us six months to do this and there’s probably 100 more that have been spun up since then. And we really need to get a handle on this and it kind of raised that level of awareness to the executive level which is great, is kind of the only way you actually address something like this across the industry.
Dennis Fisher: Also I think there were a bunch of organizations that discovered that they had SolarWinds after that. I remember hearing stories from people that were like you know we found out four months later that we did have solar winds in our environment. We didn’t even know.
Dan Lorenc: Yeah. Accurate asset inventory, accurate asset management, shadow infrastructure, kind of all of those things are a prerequisite to even being able to get started on supply chain security and a lot of folks are still struggling there.
“If anybody has spent time on the internet, you realize that not everyone on the internet is a nice person and deserves your trust.”
