DJ Schleen: 

Every one of us has a few of those people in our lives that change the trajectory of our careers, and for me, Dan Walsh is one of them. 

It was just a few weeks after the world shut down during the pandemic when I was introduced to Dan by a mutual friend of ours – Aaron Rinehart – after Aaron heard I was looking for my next big adventure. He introduced us via text message and when I got a chance to meet with Dan We talked for over two hours, and I think we cracked a few brews along the way. It was a conversation that was filled with ideas, possibilities, and dreams. 

Although I never met Dan in person, it didn’t stop me from going to work with him in one of the biggest healthcare groups in the world. 

We still hadn’t met in person when I followed him to another company in the healthcare industry. We were just talking heads on a screen to each other at that time. But it was a new world, and none of it hindered our innovative spirit and friendship. 

As the pandemic restrictions started to wind down, I arranged a trip to Chicago to meet my team, and as I landed, I hoped that I’d get to the hotel on time for a quick drink before the bar closed.

I’d arranged to meet up with Dan. In person. 

It was almost two years after we first talked on Zoom and here my plane was delayed, and it was really late. But I did get to the hotel… just in time. 

I’ll never forget walking into the lobby bar at the W, in downtown Chicago and seeing Dan with 4 full pints of beer in front of him.

“It was last call” he said, 

“you’re taller than I thought you were”, I responded. 

Welcome back, to daBOM.

Hey everyone. Welcome back to daBOM. I’m here with Dan Walsh. We’ve been acquaintances for quite a long time. 

I’ve been part of two of your teams that you’ve grown from nothing to something and actually nothing to a lot, and built up different parts of the organization.

it’s been a pleasure to work with you because, I always love people who let you run until you’re tackled and then figuring out these new technologies and where we go. That’s a testament of the teams that you build to the quality of the software that I know that you guys are creating on a day-to-day basis.

 Tell me a little bit about your background.

Dan Walsh: 

I’ve been a CISO now for a number of years. Currently the CISO at VillageMD, one of the larger provider groups in the country. It’ll be three years in September. Prior to that, CISO at Rally Health, where we met, which was acquired by UnitedHealth Group, I believe, in 2017.

Spent seven and a half years in a variety of development and security roles, leadership roles, at UnitedHealth Group, and then started my career at Vanguard, the financial services company. Advised several startups, work with different VCs on security products. Also been an adjunct faculty in the past and contributed to certifications like ISOC and a few other organizations.

So really kind of eat, breathe and sleep security and that’s who I am in a nutshell. 

DJ Schleen: 

How did you get into security? What brought you down the CISO path? 

Dan Walsh: 

I did software development, earlier in my career. And as part of that, security, as you and I have always said like security is an aspect of quality. You want efficient code, you want code that’s going to run well, it’s clean, it’s tidy. it also has to be secure. 

A lot of developers are focused on feature delivery. Not so much the security of it, because security hadn’t quite peaked yet. I always viewed it as like, No, security is part of it. I’m not going to have a vulnerability. It’s irritating to me. 

When I began doing that, when we would have annual pen tests on the applications that I was either, contributing to, or that I was responsible for, cause I was leading teams, they always had almost zero or a couple lows or a few information, or that one known medium because, we had to accept that because of the way the software was written.

 That got the attention of the security team and they said, Hey, how are you doing this? First of all, can we talk to you about this? Cause we’re looking at all these other internally developed applications and there’s like critical vulns out there for over a year and what’s, what are you doing? What’s going on? 

From there I was able to get into security and I’ve been in security ever since. It’s been a blast. It’s also why SBOM really resonates with me because, there’s a lot of applications for both, the developer as well as the CISO. It’s an area that is going to continue to have focus. 

Being a CISO is trying to look ahead and extend the horizon of the organizations that I support, to make sure that we are deploying the best and the brightest. We’re with the best and brightest people, and we’re also working on the best and most intelligent approaches for the business so we can defend the business in the most efficient way. 

DJ Schleen: 

I hadn’t met you until our second job together. I remember being introduced to you by one of our mutual friends, Aaron Rinehart back pre covid, then coming to work for you at one of the largest healthcare companies in the world. We started looking at this thing called supply chain. And we’re talking about this new technology, or not even technology, new formats of getting information from our software vendors called SBOMs.

 Seems like we’ve been talking about SBOMs for years. Where did this even come from? 

Dan Walsh: 

It came from the fact that we were working for a company that was a product company and we had hundreds of devs and we had two people I think dedicated to DevSecOps and application security.

It was really a matter of scalability, which is how do we get in a machine readable format, something that’s consistent across all the repos that support the product that our devs are developing, and how do we put that on the level playing field? If we get hit with log4j or WannaCry or HeartBleed and there’s particular components that we need to very quickly be able to diagnose. Or how do we put the risk of those repos or that software, whether it’s first or third party on the same playing field, and be able to understand what the age of it is, what the components of it are, what the families of components of it are, what the vulnerabilities are. That’s really where it started, was trying to just get a level playing field in a way that we could quickly grab that snapshot if you will, of that software.

DJ Schleen: 

We met each other at Rally and I had never met you in person. It was during the covid times. We never met each other. We worked together for almost a year, upwards of a year. You left to go to VillageMD and I followed you. 

There’s always that one person in your life who’s one of these great leaders and mentors, and for me, you are one of those. Following you over to Village was definitely one of the, the best things I could do at the time. When we did that, we were continuing this work on software bill materials, open source software, trying to really pin down the inventory of what we have. 

 Tell me from a CISO perspective, what is important about SBOMs for you?

Dan Walsh: 

There’s a number of things. We wouldn’t bring a vendor into our environment unless we did a vendor security review from a third party point of view. So why would we bring in open source software into our products if we’re not doing the same thing? SBOM is one way that we can achieve that.

Similarly, if we are bringing a vendor in, typically we rely on vendor questionnaires, whether it be from a GRC platform or whether we send out the old fashioned Excel spreadsheet. But really, those answers can be positioned, they could not be answered truthfully, and hopefully that’s not the case, but if you ask for an SBOM from a vendor and you can scan that and understand what the components are, what the vulnerabilities are in the current build, it gives you another level of understanding of what that risk is of using that software. 

The other value for SBOM, DJ, is that it’s an extension of your asset inventory. Organizations struggle with asset inventory in general, but in general we like to understand what kind of laptops we have in our fleet for our employees. We like to understand what is our network architecture and design? What does our infrastructure look like? We like to have an understanding of all these things so we can protect them. 

In a similar fashion why don’t we want to understand the inventory that our software that we are both building, so first party software and we are using third party software, is comprised of. So when we have issues like log4j or HeartBleed, we can understand very quickly, because time is of essence, where that risk lies and what our exposure is. 

DJ Schleen: 

It’s interesting because when we started asking our vendors back in the day for software bill materials, a lot of them looked at us and said what are you talking about? This was SBOM before SBOM was cool. That was a challenge. It was like the early days of the internet, right? Where people didn’t understand what we were talking about, and they would give us all these crazy formats.

I know what the challenges were at that point, but tell me a little bit about when we got those, what were your thoughts of what did we do with these things now? 

Dan Walsh: 

There were some limited tools that you could use to kind of review them and scan them. Depending on the, type of component, whether it was like a Java component or a rust component, there’s different types of tools. Some are better at others than scanning them. 

The first part was, can you just give us an SBOM? That’s the first test. Can you give us an SBOM in SPDX or CycloneDX? Because if you can, that demonstrates level of transparency and a level of sophistication. I know what my code is that’s supporting the product that you’re purchasing from me customer, and here’s my SBOM. We have that level of sophistication. It’s indicative that there’s probably a level of sophistication in other parts of the security program 

Once we get them, can we scan these for vulnerabilities? What does that look like? And then when we present that to the vendor , are they surprised by it or are they sending it along? Are they able to use something that’s open source to scan that, and then see what the risk profile is because of the vulnerabilities.

DJ Schleen: 

There’s this freshness issue, how are you dealing with updating SBOMs as vendor software changes, new version comes out.

Dan Walsh: 

Right now, we will take the position of, If a vendor is going to store, transmit, process, or have access to our most sensitive information, most sensitive data, then that’s something we want to update on an annual basis. We’re going to want a fresh one, fresh SBOM as part of that.

In the same way that we’re going to want to fresh vendor review just, Hey, what’s changing your environment? How has your code drifted? How has your security posture changed? 

 Where we will want to go in the future, and I don’t know how far in the future this is going to be because I think the tech industry has a vested interest in making sure that this doesn’t happen because of the cost of technology will rise because of the cost that maintenance will rise, will be a dynamic connection where you can see as the code is updated on the supplier side, on the customer side, the SBOM being updated, regularly. Whether it’s every two weeks or every time there’s a major release. All the legal documentation that’s going to go into the master services agreement and the legal documents between the supplier and the customer are also going to have to change. 

Because right now, if you look at a typical security addendum, there might be a right to audit. There’s probably something around if you lose our data, here’s what’s going to happen to you legally. We might be able to get an annual pen test done . We still haven’t come to the place as an industry where we’re saying we’re going to continuously aprise you of how your risk from using our product is evolving.

 Vendors and suppliers and customers and clients are going to have to get comfortable with living with vulnerabilities and living with that risk and understanding what your risk appetite is. If it goes above it, what actions , should the vendor take, in what timeframe, what rights do the customer client have, in response to that.

 It’s going to create a whole new dynamic around some of these legal obligations that we have between vendor and customer. 

DJ Schleen: 

That’s a good point. A lot of this negotiation has to happen in the contracting phase when we’re you’re procuring software. Have you seen any resistance from vendors to actually provide those?

Dan Walsh: 

If you’re like a medical device manufacturer, you’re obligated under the Patch Act to provide an SBOM. And there’s certain parts of, certain industries that are required to provide that.

I think when it comes to general, like SaaS products that companies, organizations use to run their business, right now the current feeling is look at best cases, we’ll give you an SBOM, let’s not put that in the legal agreement. That’s a bit overkill and we’ll just have like a gentleman’s agreement that we will provide that.

 In the long term that’s not going to be good enough because we’re going to see, as other zero-days come out that cause damage and harm, the government, taking a harder look and a more of a stronger position on that. But what will be interesting to me, DJ, will to see what the large tech companies do. Imagine how much Microsoft’s profitability would be impacted, or Google’s or Amazon’s profitability if they have to then provide SBOMs and work within this dynamic of basically broadcasting the risk up to their customers and then that additional tax or overhead of maintenance in that and hitting it within SLAs or within a certain timeframe, because all this is so very dynamic.

DJ Schleen: 

Do you think there’s a worry there about secret sauce and providing information for attackers that could potentially look at a software bill of materials and say, oh, there’s a potential downstream attack that I can, inject in? Do you think there’s privacy issue around the vendor supplying SBOMs?

Dan Walsh: 

That’s given as an excuse, but I guess what I would say in response to that is the Verizon report came out, the IBM report came out earlier this year . It’s still the basics. Misconfigurations, phishing, human error. None of that are supplying a list of components. It’s not going to be found in the report. Transparency in terms of how the software is built is not in there. 

Patching is, don’t let a vulnerability go too long, make sure you patch. Human error is. Misconfiguration is. I don’t really buy that. What a legitimate excuse or reason will be, it will drive the cost of software because you’re going to have to have somebody do that thing, which is to create, build, maintain the SBOMs. And then have that, build those APIs, build those platforms out with customers and with, clients, to be able to provide that.

 That is not something that software suppliers are equipped for at the moment. 

DJ Schleen: 

Do you think that GRC teams in general or security teams are equipped for receiving and managing these things? Do you anticipate in the future having somebody responsible, or a small team or a specific IC role that deals with SBOMs and those vendor relationships in the future. 

Dan Walsh: 

The GRC team will have to evolve from a team where it’s mainly audit focused, compliance focused, and security awareness or human risk management focused into one that is comfortable talking about a critical vulnerability in a certain repo or in a certain product with a, either a developer or software supplier. And understanding what the risks are of that, and how we mitigate those risks. Think about how we score vulnerabilities from the CVE all the way down to the CVS and  how we rescore it because we’re given additional context into the environment. That is also the exercise that should be happening now in vulnerability management programs across security programs. and it will have to continue to happen as software transparency, SBOM, the software supply chain evolves in this way. 

DJ Schleen: 

Having an SBOM from a vendor, having a collection of SBOMs internally from the software you create and you generate. What happens in the log4j situation? 

Dan Walsh: 

There’s major companies that add exposure. They’ve had it on their website, so we are exposed. We’re investigating, and we’ll let you know as soon as we know anything, and we waited four or five days to hear an update. Now, if that was actively being exploited, one day is too long.

What was going on there? I don’t know cause I didn’t work there, but I know that in other organizations, when HeartBleed came out and when other WannaCry, another type of, zero day or one day vulnerabilities came out that were being exploited in the wild, it was, let’s email the developer teams, let’s get a shared spreadsheet or shared Google sheet going, and let’s try to manually identify where it’s at.

Then from a software consumer point of view, if you had an external software supplier you’re just in the waiting game. You can ping them, email them, you can call their see ciso, and you’re at the, mercy of them to get back to you.

Now if you’re spending a lot of money with it, they’ll probably jump on a call and basically tell you like we’re looking into it. We’ll let you know as soon as possible. But having the SBOM both for first and third party software enables you to very quickly locate it and then very quickly get a plan together.

Because the longest time in all this DJ is locating whether or not you’re exposed. And then once you are, because you can see specifically in what repo or what part of the software it’s in, how close is it to internet facing. What are the other services that are talking to it? That’s when you can then go into okay, what are the compensating controls that we can put into place? What are the mitigations? How do we remediate this? 

But the longest part is identifying where it’s at and then how you’re exposed. Understanding what the SBOM is, allows you to, with speed and accuracy , figure that out. 

 Because the goal is not zero vulnerabilities. The goal is to manage risk. 

DJ Schleen: 

We’re never going to get to the point where there’s zero vulnerabilities on anything. Every day there’s a new one that comes up. the challenge is are we affected? Where is it? And then, what are the compensating controls? 

Dan Walsh: 

Are we using something like exploitability prediction scoring system, right? Something that says, yes, it’s vulnerable, and then here are all the other pieces of metadata that tells that it’s actually risky. And to what level of risk it is, that helps security teams not deplete their resources on something that’s not risky. There could be something that’s vulnerable, but it may not be risky, And we want to focus on the things that are risky. 

DJ Schleen: 

We had Lisa Bradley on the show a couple weeks ago, and she had this great way of tracking if a specific vulnerability was exploitable by using their ticketing system and having the developers actually fill that information out. And then that could come back and be centralized as, yeah, we know we have this component, but we don’t use that called method, so hence we’re not vulnerable.

That’s going to be the panacea of where we get to with vendors creating these SBOMs and giving them to us. But right now there’s not a heck of a lot of concrete guidance on what to do with SBOMs. 

When we talk about inventory, we need to talk about storage at the same time, and searchability. What are you thinking about that space right now? How are you addressing some of those issues? 

Dan Walsh: 

The S3 bucket continues to be the primary way of, addressing that. There are certain GRC platforms that are beginning to do integrations with, different type of SBOM tools. That’s going to continue to go on. 

Ultimately the curation and searching of SBOMs is going to be an area that expands both through SBOM startups, as you and I are both familiar with, as well as some of the major application security players like Snyk, Sonatype. 

That’s a natural extension. They already have most of that information anyways, and it’s not going to take a ton of time or effort for them to pivot to that. 

But ultimately it has to come down to the private and public sector demanding it.  If you think about it, there’s not a huge incentive for companies to provide SBOMs. In fact, I was talking with one medical device manufacturer. Their position is yeah, we have to produce SBOMs, but we want to produce the skimpiest thinnest SBOM possible because we don’t want it to be scrutinized. We’re afraid that a customer or maybe a regulator is going to look at that, see vulnerability and then we’re going to get into an argument over what the risk of that is. which is a very real concern. That’s where we’re going to have to go back to my earlier point of building out like the, what is the legal recourse? What should the operating model be for us to have software transparency?

 Those are some of the outstanding items that we have to run to ground in order for SBOM to really take off. 

DJ Schleen: 

Being in a regulated industry, like heavily regulated, when we come to healthcare and you’re dealing with these manufacturers and the SBOM mandate that they’re under right now, it makes total sense that there’d be some apprehension.

 How do you find these new regulations, cybersecurity strategy, the minimum requirements for SBOM, how does that affect what you’re doing right now and the software that you create. Do you need to provide SBOMs to other companies yourself? 

Dan Walsh: 

Not currently. We have patient apps, and they’re actually built pretty well, quite well. Those are mobile apps that are specific for our patient population.  In a typical B2C situation, customers are not demanding SBOMs. They don’t even know what they are.

if we ever decided to white label it and sell it to somebody else, that would be a valid question for them to ask. That space will continue to evolve. The requirements for SBOMs will continue to ratchet up. And then you’re going to see it spill into financial services. You’re already seeing it spill into public sector. and then once financial services and healthcare takeoff, you’re going to see it in high tech and go from there. 

DJ Schleen: 

Do you see this becoming just part of everything we do on a day-to-day basis, or do you see this in specific industries and consuming and generating SBOMs based on “Do I need to” rather than, “should I have to”. 

Dan Walsh: 

My hope Is that the industry takes a risk-based approach. So do we need an SBOM for a video game that we’re selling, on an I iOS, play store? I don’t know. Probably not. Maybe, I don’t know. I hope that they’re used in a risk based way for ultimately human life and safety. Financial safety, data safety, human risk management, that SBOMs are used to support those areas. 

We’ll see like. With anything else, it’ll evolve. When the internet first came about , that has certainly changed now. We’re seeing it right now with artificial intelligence ChatGPT that’s going to evolve several times.

 SBOM will also have its similar evolution, as we, work towards better software, supply chains. transparency. 

DJ Schleen: 

Do you think there’s a correlation between software updates and automated software patching and SBOM. The more frequent we update our software or our components, if we’re always late as why would we have an SBOM?

Dan Walsh: 

That’s a good question. You can still have zero-days. The other thing is, it’s important to understand, what your assets are. The latest isn’t always the most resilient. There are also dependency issues, compatibility issues. There’s going to be business reasons as to why something isn’t the latest or greatest or why the latest and greatest might actually not work really well.

It goes back to, do you understand what you’re using? And is that what you need it to be for the business that you’re trying to operate? 

DJ Schleen: 

We’ve talked about in the past, you and I, DevSecOps pushing responsibility and accountability for security left towards the developers. How do you see SBOMs and even in general more supply chain management, helping developers build quality software sooner? 

Dan Walsh: 

It’s been my experience both from being a developer and from being a CISO, that developers, when you see an SBOM from a repo that you created, it shows you all the warts. It’s just garbage in, garbage out. It’s just an SBOM generator, so whatever there, it’s going to spit into that SPDX or CycloneDX format. 

What I’ve seen across different companies is that when we generate SBOMs, all the garbage comes out. Whenever a light’s shown on your inventory, think about like your sock drawer, right?

You go into your sock drawer. Are you just throwing socks in there? So when the dryer eats one, is there an orphan sock in there? SBOMs shine a similar light as the sock drawer. In this case, your repo or your software suppliers. It’s going to help you understand and say, you know what? Maybe I want to organize my white socks and my colored socks, my red socks, my blue socks. Oh, turns out I have 17 socks in my sock drawer that don’t have a match. Maybe I should get rid of those, or maybe I’ll hang onto them in hopes that the other one returns home one day. 

It helps developers have better hygiene. and it is less is more because now you’re not wasting time scanning repos that have garbage in them. Now your noise, your signal to noise ratio is getting better. That SBOM in the same way that it provides transparency from supplier to customer or vendor to client, does the same thing for developers. It helps them focus on the essential and really what they need to work on, what’s in front of them, and it gets rid of all garbage.

DJ Schleen: 

This episode of daBOM was created by me, DJ Schleen, with help from sound engineer Pokie Huang and Executive Producer Mark Miller. This show is recorded in Golden, Colorado, and is part of Sourced Network Productions. We use Captivate.fm as our distribution platform and Descript for spoken text editing.

You can subscribe to daBom on your favorite podcast platform. We’re going to be releasing a new episode every Tuesday at 9:00 AM. I’ll see you next week as we continue to diffuse daBOM.