DJ Schleen: 

I’m not the most active user of any social networking platform, but when I do engage it’s normally on LinkedIn –  and the first thing I usually see is a great article, video, or post from Chris Hughes. 

He’s a content machine – an active podcaster, and I can tell you that when his upcoming book “Software Transparency,” is released, I’ll be the first to pick it up and read it.

I had the pleasure of meeting Chris in person recently, and he’s a remarkable person whose presence immediately establishes him as the smartest person in any room. 

He was just about to give a talk about Software Transparency and the Software supply chain. I was blown away by the amount of knowledge he shared, and the clarity in which he delivered it.

In today’s episode, I’m extremely excited because Chris and I dig into a diverse range of topics, and we explore the crucial concept of transparency at the crossroads where government, vendors, and consumers all meet.

Welcome back, to daBom.

Hey everyone. Welcome back to daBOM. I’m here with Chris Hughes from Aquia. I’ve been taking a look at some of your background and the things that you’ve done in the community, especially around supply chain. It’s pretty impressive. Tell me how you got into supply chain and cybersecurity.

First off, thank you for your service. You’re a veteran. is that where this all began? 

Chris Hughes: 

I always was interested in computers and gaming and stuff growing up, and then joined the Air Force and got put into cybersecurity. I didn’t realize at the time, how fruitful and a good career it would be. That’s what got me started down this path. I’ve been in the career field, a little bit over 15 years. I was, active duty in the US Air Force and then a federal employee with the Navy. And then also, FedRAMP, if you’ve heard of FedRAMP, I was on the FedRAMP team for some time, reviewing cloud service offerings, coming to the government through FedRAMP.

 I’ve just been doing cloud cybersecurity and DevSecOps for the last several years basically in the federal space. I noticed software supply chain security was a hot topic and was picking up. I found it fascinating. Just the last several years I’ve really went down the rabbit hole on that reading and writing everything I can get my hands on, and just learning everything I could about it.

DJ Schleen: 

So even before the military, were you involved in technology? Were you doing security work or interested? Were you a hacker back in the day? 

Chris Hughes: 

Growing up, I would build computers and play some games and maybe, find ways to manipulate the games to my advantage. But, I wasn’t, formally in cybersecurity in that capacity before the military. No. 

DJ Schleen: 

Tell me about what you’re doing in supply chain now. I know that you’re doing some work around SBOMs and of course when you think about supply chain security, this is exactly what SBOMs are about.

 What are you doing today with, the supply chain and supply chain security?

Chris Hughes: 

I’ve been reading and writing about the topic a lot lately. I have a blog where I write a lot called Resilient Cyber. And then I also actually am in the process of finalizing publishing a book with Wiley, called Software Transparency, which focuses on software supply chain security, including SBOM and many other topics.

My company, we’re working on software supply chain security. One context is around SBOM at a federal agency where we’re helping them, formalize how they go about procuring and obtaining software including SBOMs from vendors as part of cybersecurity executive order, and, office of management and budget requirements and things like that, that are emerging. Helping them, aggregate SBOMs at scale, enrich ’em, store ’em, report on findings and so on. 

We’re also doing a lot of work around SaaS security. and SBOMs from SaaS vendors too. We think about software supply chain security. obviously there’s COTS and trying to developed software that’s using open source software components and things like that. 

DJ Schleen: 

You mentioned the executive order, 14028, we have some guidance coming out with the cybersecurity strategy for the nation. You’re on the front line of solving this problem with the government, and with these agencies. What’s the biggest challenge there? Is this just unchartered territory that you’re trying to navigate and come up with solutions for? 

Chris Hughes: 

One of my friends said that the market failure that is cybersecurity is not going to fix itself on a voluntary basis. So we’re advocating for secure by design, secure by default. We’ve been saying these kind of things for a long time.

 But when it comes to, manufacturers or software and technology suppliers, they’re not really incentivized from a market perspective to do these kind of activities voluntarily or even provide transparency like SBOMs, for example, on a voluntary basis.

The biggest challenge that we’re dealing with right now when it comes to cybersecurity and software supply, chain security and things of that nature is, shifting from the paradigm we have now, which is voluntary, we look at the, guidance from CISA, secure by deign, secure by default, it uses the word should 51 times in a very short document. 

Anyone who’s been in cybersecurity for some time know there’s a lot of things that we should do, and we’ve been advocating for development teams and businesses to do these things for a long time.

 Until we move from a paradigm of what you should do or what we recommend you do to what’s required to do, whether it’s regulatory perspective or changing the incentives within the market somehow, I don’t think we’ll see much change. 

DJ Schleen: 

You said a lot of shoulds in this documentation. There’s no musts. Do you see this moving on from that and going into this, more of the responsibility coming on the vendor?

Chris Hughes: 

We’re going to see a shift in that direction. But, it’s a give and take. It’s a constant battle. for example, there was language in the NDA A, the National Defense Authorization Act at one point that used the phrase of SBOM and it being required for software vendors working with the DoD. But it also was a bit misguided in the sense that it called for vulnerability free software, which we all know is a fantasy. We’re not in the business of eliminating all vulnerabilities. We’re in the business of managing risk. 

I’ll also say that as you point out, there’s cybersecurity executive order, and then there’s also the OMB memo that, starts to call for organizations to self attest to things like NIST Secure Software Development Framework and potentially provide SBOMs.

 Right now they’re not required. but I think this is where we’re seeing the federal government try to use their massive purchasing power their procurement arm to get some of this language into things like the FAR, the Federal Acquisition Regulation, for example, and drive some of these requirements on software vendors in hopes that if the government leaves the way, maybe industry will follow.

It could happen that way, but it also could create a situation where if we make it too cumbersome or too burdensome, we can actually limit the pool of innovative, technology providers that the government will get access to, which can hinder things like national security or, citizen services, critical infrastructure, things like that. So it’s a delicate balance 

We’ve seen industry push back quite a bit. Department of Commerce and others, nonprofit entities that represent industry have pushed back on SBOM, saying that it’s still a maturing concept. The tooling, space is still maturing. We have competing standards.

There’s a lot of validity to that, but there’s also some aspect of people being afraid of transparency and pulling the curtain back to see how many vulnerabilities are being shipped to purchasers and consumers. 

DJ Schleen: 

How have you seen vendors deal with these mandates? Do they staff up? Do they divert budget? this is a lot of regulation and things that are changing that vendors might have to catch up on. 

Chris Hughes: 

That’s a really great question. Honestly, it it varies depending on the vendor. For example, there’s organizations like Microsoft and Google and Amazon and others who have incredibly robust and capable security teams and massive budgets.

 They’re in a better position to respond to these changes where when you look at the, the broader defense industrial base or the broader industry of small mid-size companies and startups, they’re going to struggle a bit more when it comes to complying with these requirements due to, the lack of expertise, lack of resources, competing priorities of getting to the market, getting market share, things like that, return on investment for investors. 

It’s going to be definitely more challenging for smaller organizations to meet these requirements than it is for some of the bigger, more capable, technology success examples in the industry. If it becomes too cumbersome or bureaucratic, it can actually hinder adoption of innovative technology for the government. I said I worked on the FedRAMP team at one point. There’s tens of thousands of cloud service offerings in the marketplace. Now there’s less, than 300 authorized in the FedRAMP marketplace. That’s a very small subset of the cloud service offerings that the federal government should be using if they’re using only FedRAMP authorized services. 

We could see something happen like that on the software procurement side. If the requirements become too cumbersome or bureaucratic or costly, they can have an impact on the market and accessibility to innovative offerings.

DJ Schleen: 

What would you recommend for those small organizations, from an information perspective. There’s so many things that they just can’t consume. If it’s the bread and butter, the lifeline of your business to sell to the federal government, you’ve really got to take action. How do these small organizations do that? 

Chris Hughes: 

I think you’re spot on. It’s going to be dictated by how much of their revenue is tied to this sector, this industry, this customer. And that’ll drive how much of a prioritization they place on it. Same thing, as you point out, some may take the wait and see approach and see, what gets finalized into federal acquisition language or requirements.

 Also we’ll see an industry pop up to serve these entities, to help them understand the requirements, advise them on SDF and requirements related to that and OMB and how to produce SBOMs, how to provide SBOMs.  There’s a lot of, innovative open source software tooling out there to help facilitate some of these artifacts and creating these artifacts, for vendors to provide.

 It’s going to be a combination of all those things, that’ll play out over time. 

DJ Schleen: 

Let’s talk a little bit about the life cycle of an SBOM. 

Chris Hughes: 

This is a really fascinating topic that doesn’t get discussed enough. We’re starting to see more dialogue from industry and entities like CISA and so on is the, there’s been a lot of guidance from NTIA and others in industry in terms of creating an SBOM.

That’s great… but what do you do with it? Where does it go from there? How do you make it actionable? How do you store it? How do you enrich it? How long should you retain it? As you’re pointing out, creating an SBOM is a great first step, but now there’s an entire lifecycle associated with that.

 We’re starting to see, organizations both in the federal and the commercial space, try to flush out how long these things are retained, how frequently they wanna receive them. It should it be every single new release of software or should be on a certain cadence that’s not too cumbersome for vendors.

The organizations that I’m working with are striving to have SBOMs created for every release of software. And then ultimately take those, enrich ’em and store ’em so that they have, information on hand on the latest software they’re using and consuming, but also have some, I guess you could say incident response or forensics information in terms of maybe a vulnerable component was used in a previous version of software that was in the environment for some time.

Those kind of situations are playing out now across the ecosystem. 

DJ Schleen: 

Once a federal agency gets an SBOM, what do they do with it? Do they just file it in an S3 bucket and like a pack of baseball cards and, just, leave it there… or do they actually have some sort of workflow that happens after they receive it?

Chris Hughes: 

I’ll say for the organizations I’m working with, there’s definitely a workflow in terms of enriching it, to get, more, vulnerability data, for example, from multiple different sources, and then also move beyond, like CVEs for example, or lagging indicators of risk. At that point, there’s a vulnerability, there’s some risk. Starting to try to get some leading indicators of risk too, or key risk indicators, things such as pedigree and provenance, number of maintainers, national origin of the contributors, other projects that they may have contributed to that may be also malicious. Start to try to enrich it with key, risk indicators or leading indicators of risk.

Then also there’s something to be said about the lifecycle of the SBOM, isn’t going to be on par with the lifecycle of vulnerability data. You may have an SBOM where the components haven’t changed, but as we know, new vulnerabilities are emerging all the time. You need to revisit the SBOM even if the software hasn’t changed to see has there been new vulnerabilities that have emerged that are now applicable, that didn’t exist or weren’t applicable at the time. 

You’re definitely seeing kind of a separate activity when it comes to vulnerabilities and enriching SBOMs around vulnerability data, beyond just the SBOM itself in terms of the lifecycle may not be the same.

DJ Schleen: 

What kind of feedback loops are you seeing in the industry of people reaching back out to the vendors and saying, Hey, now there’s a new vulnerability,

What have you seen about that feedback loop, or is there one? 

Chris Hughes: 

Honestly, it’s still evolving and maturing. I think it’s being figured out as it goes. It’s very challenging to do in large, complex environments where you’re consuming software from many, many different entities and sources because you have that dialogue back and forth and, in terms of, hey, there’s a new vulnerability, what are you guys doing about it, kind of thing. But just because there’s a vulnerability doesn’t mean it’s actually exploitable or presents risk to the consumer.

60 to 80% of modern code bases are made up of open source software components. And we often try to apply the manufacturing analogy to software, unlike in manufacturing, we’re using things that are not from “suppliers”. We have no expectation of them providing any kind of SLA or response time or response whatsoever even in any case, unless they chose to. 

 We can’t force them to adhere to anything or respond to anything because they’re doing this voluntarily free in their own time. They don’t owe us anything. That creates a kind of, another complicating factor in the sense that open source software contributors and maintainers, which are being used heavily in modern applications, they’re under no requirement to respond or address things unless they chose to do so. 

DJ Schleen: 

With those open source projects, do you see SBOMs going down further into the supply chain that we’re creating software, we’re asking for software bill of materials from these open source projects. 

Chris Hughes: 

It depends on the individual or the community of the project, whether they’re willing to go along and provide these kind of artifacts and meet these kind of requirements.

Or they’re going to say, Hey, look, I’m doing this on my own free will. I don’t owe you anything and I don’t have to align with any kind of requirement that you’re. pushing towards me. 

But we are seeing efforts, not only from the open source software contributors and maintainers, but from platform providers like GitHub.

They’ve released some, native capability to produce SBOM, from repos and for projects and so on. So I think there will be a kind of relationship between the maintainers contributors and then the platforms that they’re developing and producing and distributing software through to meet this from multiple angles.

DJ Schleen: 

So as a CISO for a software company, what keeps you up at night around SBOMs and supply chain? 

Chris Hughes: 

What really I found fascinating and scary is a couple different things. It’s honestly mostly related to the open source software ecosystem, not because it’s any less secure than proprietary software, but because malicious actors have woken up to the reality that, Hey, I can target a single entity, a single target, or I can target a widely used component or piece of software from a proprietary vendor, open source software project and so on, and have a massive downstream cascading impact across the entire ecosystem and impact thousands of organizations and millions of individuals.

 Not only that, a lot of these projects that are being used, the overwhelming majority of ’em have less than 10, and often not much more than a single maintainer or contributor.

We just don’t have good visibility of the ecosystem, of what we’re using, who’s making it, how it’s being made. and it’s concerning, for lack of a better way to put it. 

DJ Schleen: 

Let’s circle back to the minimum elements of a software bill of materials as described with the NTIA response to the executive order. Is that enough? As everyone should be aware, SBOMs are more than just open source components. They’re files, they’re hashes for files. There’s licensing information in there. There could be VEX vulnerability information and attestations in those, billing materials as well included as part of it. 

Do you think that we’re just at the beginning, the cusp, or are these minimum elements just a guideline to get started?

Chris Hughes: 

We’re definitely at the infancy of this endeavor of trying to get this information. And as you point out, there’s a lot more information in there than just vulnerabilities. There’s things like licensing that could be informative for procurement or acquisition. There’s a lot more information in terms of hashing and integrity. 

Looking at the NTIA minimum elements, not only is it not enough, cause you talked about it deals with direct dependencies, not transitive dependencies necessarily. studies from, groups showed that six out of seven vulnerabilities come from transitive dependencies.

 If we’re trying to get to vulnerabilities and drive down risk, a lot of that’s going to be coming via the transitive to dependencies that we just don’t have good visibility or inventory of or how to respond to. 

Another aspect of that is there’s another group that did some research, around the quality or state of SBOMs in the ecosystem at the moment, aligned with NTIA’s, minimum elements. What they found was essentially almost none, basically fully complied with the minimum elements as defined by NTIA. And we’ve seen the same on our end so far. We have a long way to go to meet that minimum element aspect, let alone build on that, to more robust, complete SBOMs, in my opinion.

DJ Schleen: 

On the licensing front, how are organizations reacting to that? Now that we know what’s in our software and we have this visibility, are we taking licensing and, risk around GPL licenses more seriously? 

Chris Hughes: 

I think it depends on the entity involved.

if you’re internally developing software and using it for your agency or your enterprise with no intent necessarily to sell it or something like that, it may not get prioritized as much if you’re looking to sell software or you’re looking at things like mergers and acquisition, and now you have this visibility of the licensing in any potential conflicts or concerns associated with that. They’re perking up a little bit more in terms of licensing conflicts and concerns because it’s more at play for them and they potentially run into more legal ramifications.

DJ Schleen: 

That makes sense. What do you see the next 10 years? if you could just have a crystal ball and imagine where all this software bill of materials conversation is going, where do you see that taking us over the next few years? 

Chris Hughes: 

I think we’ll see, SBOM continue to evolve. and there’s going to be a lot of different, stakeholders with different opinions involved. Some will be big advocates for this, level of transparency. Historically there has been Information asymmetry between the software supplier and the consumer. SBOM helps to address that asymmetry of information. 

 It also isn’t perfect. A lot of people have pushed back, on the tooling or competing standards or the completeness of the SBOMs… but security has never been perfect. It’s always an endeavor to improve on, where we’ve been and where we’re headed. People need to understand that. 

 In terms of how it will unfold and play out is going to depend on the extent that the government is willing to go, to start to formally require these things, have consequences for failing to use their purchasing power to drive some of these systemic, changes across the ecosystem.

so it’s going to be a combination of all those things in terms of how successful it is and how long it takes us to get there. But I will say, having this transparency is something that we didn’t have historically. despite software asset inventory being a critical control for, ever, we just didn’t have good visibility of the software components and libraries that we’re using in enterprise environments.

 That’s why you’ve seen a cyber safety review board, showed that one agency spent 33,000 hours just tracking down where Log4J was in their environment. Having this visibility is a massive value added activity, but it’s not perfect and we have a long way to go still. It’s going to take some time. 

DJ Schleen: 

So basically, correct me if I’m wrong, you’re thinking that this is going to decrease response time when it comes to issues. Is one of the things that we could anticipate? Where do you think security is going to fit into that? 

Chris Hughes: 

It’s definitely going to aid in areas like incident response or responding to compromised components as malicious actors. Can you continue to target, software components or libraries that are widely used, across the world. You start to maybe improve the hygiene of your software and be more cognizant of the components you put in your software.

DJ Schleen: 

So one last question for you, Chris, before we wrap this episode up, where can people find more information from you about supply chain management software, bill materials? 

You have a book. You also have that podcast. Can you give us some information on where to find these things? 

Chris Hughes: 

Definitely. As I mentioned, I’m super active on LinkedIn, so you’ll find me on there very active in terms of sharing this information.

I do have a book coming out from Wiley named Software Transparency, Supply Chain Security in an Era of a Software Driven Society. Other than that, you can also find me on substack. I publish typically an article a week or so on areas like cybersecurity, cloud security, software supply chain security, open source, so it’s resilientcyber.substack.com.

And then the podcast is also named, Resilient Cyber. 

DJ Schleen: 

Chris, this was a great conversation. I am looking forward to reading your book. For all of the listeners, I recommend you read it as well because any information on SBOMs and where this whole movement is going is going to be valuable as we start figuring out how to implement these processes and techniques in our own enterprises. 

This episode of daBOM was created by me, DJ Schleen, with help from sound engineer Pokie Huang and Executive Producer Mark Miller. This show is recorded in Golden, Colorado, and is part of Sourced Network Productions. We use Captivate.fm as our distribution platform and Descript for spoken text editing.

You can subscribe to daBom on your favorite podcast platform. We’re going to be releasing a new episode every Tuesday at 9:00 AM. I’ll see you next week as we continue to diffuse daBOM.