DJ Schleen: 

There’s no better way to get to know someone than staying awake for 24 hours straight while moderating sessions of the world’s biggest virtual DevOps conference – All Day DevOps. It’s One of the many times I’ve gotten to spend with Hasan Yasar over the years. 

We were hunkered down in an office in Tyson’s Corner, just outside of Washington, DC, broadcasting throughout the day to an audience spanning the world, introducing some of the world’s most talented minds before they shared their stories.

Hassan and I met back in 2017 when we were both speaking at DevOps Connect at RSA, and I was floored at the wealth of knowledge he had about DevSecOps. He’s done the research, knows the practice, and has the mind of an architect. 

Hassan isn’t only a speaker in the community, though, he’s also an organizer of events such as DevSecOps Days Istanbul, DevSecOps Days Tokyo, and one very memorable panel I was on at an event hosted by the Software Engineering Institute at Carnegie Mellon University. Hassan placed me on a panel beside Brigadier General Greg Tohill in front of an audience of military personnel to discuss DevSecOps. 

I will never forget fielding a question with General Tohill from a member of the Air Force. They asked “how do you fail fast with a ballistic missile?” 

” You better have some good simulators.”

When Hassan and I caught up again at the RSA conference this year, our conversation turned to the topic of Software Bill of Materials and how they fit into the SDLC. 

… and then Hassan started talking about how we could shift them extremely far left… 

Welcome back, to daBOM.

Hey everyone. Welcome back to daBOM. I’m here with Hasan Yasar. You’re from Turkey Tell me about Turkey and let’s hear a bit about Hasan. 

Hasan Yasar: 

I moved to the US 1996, My background is Double-E, Electrical Engineer. But I really spent all my life since 1989 in the software engineering world.

I moved to the US to support the US Air Force simulators on F-16s and other things. I had been working that space. Actually I was working in Turkey in that F-16 simulator as well. 

I moved to the US and after that a lot of startup business, work in organizations. While I was working a lot of industrial arena and working for many other smalls companies for the DoD, I moved to the startup business and building up a scanning solutions and digitizations and running a IT shop. 

Then I end up at the CMU (Carnegie Mellon University). They were looking for more industry experience. People get into SEI or the CMU and improve the state of practice. 

The main mission of SEI as part of Carnegie Mellon University, we would like to increase the state of practice with respect to software engineering. Our main focus, even though coming from a DoD perspective, we would like to increase or improve or find a way to improve the state of practice with engineering.

I joined the 2010. Probably was the perfect timing for me to join SEI because I brought the engineering experience that I did for years and years. When I started I said, “I’m going to change something here in the CMU-SCI, make it more modern way of building”, which is exactly 2010, that’s where the DevOps started. DevOps started in 2009. I said, “I’m going to bring that concept, make it more broad thinking as software engineering and using modern techniques, modern components and modern implementations.” Specific for Department of Defense, but more about the highly regulated environments, the concept.

That’s how I joined SCI. Now I am leading 35 plus people who are focusing on DevOps, DevSecOps, Agile practices across the Department of Defense helping many units, many programs, to improve the state of practice with software engineering. While I was working at SEI, the concept was very new, a DevOps concept or DevSecOps concept, and CMU always forward thinking, CMU said, can we do some teaching on DevOps?

 In 2013, I was the first instructor, first professor in all around the world, brought DevOps courses into the University. Since that, I have been teaching DevOps course in three times in a year; summer, fall, and spring semester, talking about the DevOps throughout the year. I have 80 plus students for every session. It just keep growing. 

When I talk about the DevOps, I’m talking about engineering practices of DevOps.. It’s more about what it is, what needs to be done, what engineering mindset looks like, what are the requirements for DevOps and what are the key elements of implementations, including hardware and software?

DJ Schleen: 

It’s interesting how when we met DevSecOps wasn’t a thing. You created one of the first DevSecOps reference architectures.

 Where I’m working now,  I’m seeing things written by you and written by myself eight years ago, defining what DevSecOps was. But you did one of those first diagrams. Are you putting SBOM in there anywhere right now?

Hasan Yasar: 

Actually, SBOM is part of it.  If you look at all this graphic right now in the community that we have been using, it looks like it’s a two dimensional of the presentation of those DevOps or DevOps practices. But in reality it’s a three dimensional of thinking DevOps concept. 

When I say three dimensional means, we always have kind of representation of the CI/CD practices, which is automations for infinite loop for CI/CD. But there are two other dimensions we had to think about. One is the collection of the data monitoring pieces is very important. But there’s a human elements in picture, which is one vector will be a people, another vector will be data collection, another vector will be in an automated part of the machines. 

So then we’ll cut that angle. Really, we have to visualize that component make it a three dimensional of DevSecOps; human and data collection pieces, which we see very important right now. Data is becoming a very crucial driver for improving the state of practice in engineering. 

We did not spell clearly in the SBOM as part of our drawing and writeup. We always mentioned the concept of Infrastructure as Code. I clearly remember almost 10 years ago of a specific talk about the version or the libraries that we are writing, IEC script. We have to think about the secret aspect of those IECs Think about and write those libraries and version and keep track every changes. 

So indirectly we were referencing the SBOM concept as part of the DevOps concept, as part of Infrastructure as Code, deployment scripts and make sure that we are following the version as part of our writeups.

Now we are talking about SBOM is basically creating the similar things. We should know our dependencies. We should know infrastructure. We have to know where we are getting it from. We’d have to keep track those information as monitored through the pipeline. 

 We did not clearly say here is the format that we will like to use. It was part of infrastructure. It was part of the build and part of the deployment process at the moment. 

DJ Schleen: 

Now that you have that diagram in place and we have that three dimensional view, are you extracting data into an SBOM format for observation, for monitoring. How does that plug into that DevSecOps architecture that played out? 

Hasan Yasar: 

Perfect. Thank you. That’s a great, diving to the DevSecOps and SBOM because we have been seeing so many SBOM local implementation ideas. What I have been seeing so far, DJ, the SBOM seems more about creations either getting from a vendor as a file, either CycloneDX or SPDX. These are typical format as we know. Getting those files and storing in repositories, it is good. 

But it should be dynamic because when we look at the concept of software that we are building, it just keep increasing or keep changing. There’s so much dynamic things moving around. There’s so many moving parts. We basically keep changing the files frequently, continuously with it. Because we are building applications. 

You may say how much we changing infrastructures. That is true. We are not changing infrastructure a lot, but dependencies keep changing because sometimes we see the surface, we don’t know what is happening behind the scene. 

What literally means an SBOM in DevSecOps?

There are multiple gates or multiple phases throughout the DevSecOps pipeline.

So what does pipeline means? We have planning, which is the first phase of the software lifecycle. We have a design architect phases. We have development phases. We have tests and built and all these phases. Actually, every phase has some interaction with the SBOM. Every phase either generating the SBOM or verifying validating SBOM. There’s multiple things. What we should do? Getting a file from the vendor is good, which is part of our planning, which is where we start, which is a design SBOM. If you look at the CISA website, there are about six type of SBOM mentioned. There’s a design SBOM. There’s a source SBOM. There’s a build SBOM. Analyze SBOM. Deploy in a runtime SBOM.

DJ Schleen: 

That’s why we call this daBOM because it could be called *BOM. 

This is the first time I think our listeners might have heard about six different SBOMs. You guys are following this trend and you’re doing a lot of research. Do those actually give value to people along that SDLC?

Hasan Yasar: 

It’s very important because we always focusing only either build or a runtime SBOM. They’re limited.

Why it is limited as build and runtime, because runtime you’re executing the application. You need to know the dependencies. Yes. We have to. That’s what everybody’s talking. That’s what we are generating. That’s what people are asking. Tell me your runtime. The runtime requirements exist every time when we talk about application, yes, we should know. 

But what are the other SBOM types like the design as an example. If I’m talking about the design SBOM, that means as soon as when I start to think about application or a system including hardware and software, by the way. In the design phase, there is an opportunity at the moment that we are identifying our applications. It’s very crucial. Why? If anybody’s changing the design, if anybody’s adding any new libraries, what they do, they are working this design or planning phases.

Maybe we’re acquiring a third party libraries. Maybe we are recording a solution. That’s exactly happening during the design phase. That’s what the design SBOM means. We are not creating a new type, but we are creating a SBOM file in the design phase. 

Let me go into a specific example. Let’s say I’m going to build up an AngularJS. An architect said we are building up a pops up services. The pops up services will use the AngularJS and NodeJS. This is solution, which is architectural design. If I know that solution at the moment, I can create a container version of my shell as my VM, and automatically I can create my design SBOM at the moment, as soon as I know that pops up component that I’m going to build up.

I can look at the licensing pieces as an organization. Do I have a licensing right to use those libraries? So now once we create those design SBOM, we’re going to carry out through the lifecycle when we go into the source, we will see that we’re going to compare, do we have the right design SBOM that we generated? Are we following it? Is there anything that you keep changing? Is there anything keep updating? We can carry out the file building and next one, and next one on top of it. 

DJ Schleen: 

That’s really cool because if you have architects specify the components that they expect is in the software that is getting created you can track architectural drift on components at that point.

Hasan Yasar: 

Yes, yes, we can do that. Think about the containerization. We use container a lot, right? If I know my higher level requirements with architectural choice, going to use AngularJS as an example, I can easily look at my base container images, which we are shifting left. 

This is the exactly shift left our SBOM to early phases, in the planning phases, in the design phase for a design drift. And also we can see the licensing. How do we know that we are following the right licensing? 

DJ Schleen: 

This is an educational opportunity for the developers too. Especially around the licensing.

Yeah I had never thought about bringing an SBOM that far left, but it makes total sense because you’re setting the stage for everything that happens to that product from then on.

Hasan Yasar: 

Exactly. So when we put it in the DevSecOps pipeline, here’s the beauty. 

If there is anything changing, if somebody’s changing or architectural change, or one of developers may be looking for different libraries, we have a traceability of our architectural choice from the beginning from a licensing perspective, libraries, and developers maybe do something else. So we see that changes. Either developer will notify the right people, right team members, why I’m changing. We’re making more living documents of SBOM. 

That means we are starting from our planning design phases. When we get into the source and build, we are carrying out that information, adding additional required or updates as we go.

 We can generate SBOM in every phases. Once we have the architectural choice, once we know if the developer start to write the code, we can share the base images, we can share the base VM for testing. That base image or base VM will have the libraries already in it. We can generate those SBOMs as developers start to write it.

 If I know my choice at the beginning, I can build up artifactories. When I go to the build phase and artifactory will be part of my CI pipeline, Continuous Integration, I can pull those libraries from my artifactory. 

Who’s going to create artifactory, DJ? Somebody has to tell the artifactory who is managing those artifactory libraries. They have to say, here’s my library I’m going to use. Who’s going to create those libraries? How they going to learn libraries? They’re going to learn the acquisition phase or the planning or design phase. That’s what the design SBOM means, that we can generate it as early as possible. 

Another thing that’s important, I’m trying to connect an acquisition team member and the developers. Sometimes there’s no connection. Why there’s no connection? Because we are acquiring a software. If you really get the approval process early in the acquisition, early in acquiring the software, we can get SBOMs from the vendors, we can look at the dependencies, we can look at the policies, we can look at the licensing.

Now we can carry out our infrastructure because we know exactly what is happening. When we acquire, we know all these components are tipping in. Which we are shifting as far left, day number one, by enforcing the policies, which goes back to the three dimensional thinking. 

 I can enforce the policy at the beginning when I acquire the libraries. When I purchase it. 

Then I get the SBOM, then I can put in environment and I can make actionable of SBOM as I go to the pipeline. Writing the code. Building the code. And then compiling whatever the artifact looks like I’m creating maybe JAR file, maybe exe maybe DLL, which is the build artifact. Combine all the dependencies and create a new package, then the next one is runtime. Instead of just focusing only at runtime, it is limited to my opinion because there’s a lot of things we missed there. 

DJ Schleen: 

As developers start working with things, you’re going to get more data, and then when they start releasing, you’re going to get more data. What are you guys seeing from a data management problem with SBOMs?

Hasan Yasar: 

Great. Great. That’s a great question. First of all, let’s ask ourselves what’s the purpose of creating the SBOM? The main purpose of creating or knowing SBOM, be ready for any incident. Be ready for any vulnerabilities, which we learn our lesson for Log4j. 

Statistically 75% of organization, they were struggling to find that, do we have a Log4j or not? That’s what is struggle. We are trying to be ready for any new vulnerabilities or anything that we are talking about, which is the main goal, our purpose. 

When we look at those data collection, yes, we are calling a lot of data in terms of libraries. But the key point is how can I do traceability from the beginning of those libraries that I’m going to use?

There are many tools available to show that traceability. It’s not a big problem, be honest with you, to monitor. The big problem is creating the data at the moment that we are using it .The monitoring or connecting dots, it’s much easier because we have all that existing technology. The key problem, we don’t create those files as we are building the code. We are creating ad hoc as maybe the part of the build or maybe the part of the container, which is more on about the right side.

 What we have to do, what we saw already, which we created a pipeline exemplar that we have been for many of the other organization right now. Think about SBOM day number one, and then try to build up the changes as you go. Again, collect the data, which data is about the libraries, package URL, version. We have a traceability. 

DJ Schleen: 

It’s interesting that you say that you generate when you need to, almost like on an ad hoc basis, because that’s one of the things I noticed developing Bomber. I would tag releases and then I would generate an SBOM for the release, which is great. But just because we have the ability to generate SBOMs doesn’t mean we need to do it all the time. 

Hasan Yasar: 

Another reason that we need to know our application version, because if something happens on our application running in the production environment, if it does some incident management or if it any type of response as an engineer, as a software engineer, first thing I ask, what’s the version?

I’m asking the version of my application, but indeed there is a version of my dependencies. How do you know that you have the right dependency version running an infrastructure match with application version. That says we need to create those SBOM as we are doing some activities in our pipeline. As we are building my code as part of the CI, I have to create SBOM right away. 

If I’m designing, I have to create SBOM at the beginning, which is more documentation. I should create an SBOM as dependencies on applications goes through the process because that’s what we are bringing application together with the dependencies and artifacts together. 

DJ Schleen: 

What the industry needs is use cases or guidance documents of like how people are doing things. Because I’ve never heard a lot of these comments, like having the ability to have a collection of documents that says, Hey this is where you need to create it and this is the requirements for this.

Hasan Yasar:  

I did the one presentation in the Vancouver Linux Foundations Summit Conference. I specifically tell the people, if I’m looking for a supplier name as an example who’s going to create a supplier name for me? There is an ability to create those data fields, and if you remember, there is a minimum elements of SBOM that’s recommended from CISA.

The minimum elements are the supplier name, component name, version, and package url, and authors. About seven data fields required. But nobody’s really talking about how can I generate those files or the data elements? Where can I pull it?

If you ask architects, it’s too much work for me to do that. If you ask the developer, they can say too much work to do that. If you ask the tools what the tool is going to do, the tool will generate those SBOM probably most likely called checked in or like using a GitLab or other type of tools, there is a plugins available generate those as soon as we get in the code. 

It’s too late for me, honestly, because we are missing the design, we are missing the licensing pieces.

Why? Because we are not able to tell the developers or architect how can we generate those files as part of our living, our daily job. We need some sort of industry guidance. How can we generate those files as part of our code practice? 

We know the coding. We know how to write the code. We know how to design. But my opinion, we don’t know how to generate actionable SBOM as we are writing the code. That portion seems missing to me. 

DJ Schleen: 

Hasan, tell me a little bit about what SEI is doing with DoD or the Department of Defense and SBOMs. What are you doing there? 

Hasan Yasar: 

The SBOM became a very crucial component of the ATO process. ATO is the Authority to Operate. Every application, every system has to go through the approval process. The approval process is looking for the safety controls from beginning of application until the deployment of application.

SBOM is mandated as part of the approval process. Should be created, should be monitored, and should be inserted into the approval process. So that’s kind of relation with SBOM and the DoD. 

What we are doing is at SEI, we are inserting SBOM practices, try to build up a reference architecture. What we do right now, creating an exemplary pipeline using the various tools for verification of the SBOM, creation of the SBOM, looking for the vulnerabilities on each libraries, which I’m using the Bomber, thank you, as the creator. It’s all at the part of it to show the people how much we can use those files, generated automatically, look at those file content, look at the vulnerabilities.

Basically, at SEI, we are creating the best practices of actionable SBOM throughout the DevSecOps infrastructures. 

We are also creating as the concept and process and ideas how much we should do when we are practicing agile practices on planning or postmortems or sprint planning. Creating some techniques, creating some ideas, creating some solutions so people can use and people taking advantage of early indicator, early application of SBOM. 

I have been working as a team about the threat modeling. There’s a direct connection between SBOM and threat modeling. If I know my threats, platform specific, how do I know my threat? SBOM will tell me my design components. I can look at the potential threat of libraries, so I’m trying to connect the threat modeling and SBOM as well as the platform pieces.

DJ Schleen: 

With the research that you’re doing and the work you’re doing with DoD, do you plan on publishing anything to the wider community? 

Hasan Yasar: 

Our website is publicly available. We always publish and we always write and get technical papers or a conference speaking or a writing research paper. As soon as we have a good data set about SBOM, SBOM limitation, we will be publishing what our findings looks like. What’s the reference architecture? We’re going to publish those. 

As of now, we have a DevSecOps space, an agile space for SEI website or people can search my name. A lot of publication has been done in that space as well with DevSecOps and software supply chain too. 

DJ Schleen: 

Hasan, one more question for you before we wrap up. Where do you see SBOMs going? What’s your research telling you and your experience working with them? 

Hasan Yasar: 

What we see right now, we just dealing about the tip of the iceberg to be honest with you, DJ. Because we are talking very high level. But when you look at the reality of the complex systems, we have so much interdependencies of the libraries and component. We don’t know how can we deal as such a mess of connected or hierarchical libraries dependencies. It is very difficult to deal about it.

Think about the very complex systems like a car as an example. How do we know that all the SBOM that we know, all the files, all the dependencies that we can control. It’s getting very challenging. 

Another challenge I have been seeing, they don’t want to share SBOM. Why? How do you know that we’re going to protect those files appropriately in infrastructures? Because technically, if you share how you build up your food as an example, you’re sharing every ingredients. But you’re sharing your secret at the same time. 

There is licensing, there is a security components. Protect those libraries and protect those files. I see people hesitant to share those files or in depth details. They don’t want to do that because it’s a security problem right there. If I’m sharing my libraries, if I share how I build it, it’s going to be a little bit challenging to protect those components. .

DJ Schleen: 

So tying this back to DevSecOps, you and I have always talked about tools, technique and talent or people, right? Tools are coming. We’re still working on technique. And we have to educate people and let them know how to do things. 

Hasan Yasar: 

Yes. Especially our new generation of developer. I have to say it because I have been teaching at the CMU for years. Most of my students, if I say write the project, they just Google it, open Stack Overflow, “building an application”. They don’t really, I’m not saying intentional, but they don’t really care. They just writing the component as quick as possible. 

It’s about education. It’s about learning. It’s about rating the transparency of our code.

Copying code from Stack Overflow or somewhere else, or getting code from a GitHub without knowing the details, it’s a problem, it’s a learning curve.  They have to learn. They have to learn, they have to spin it, they have to understand what the complication means, what the issues are, what the problem they may face is. 

I always advise my students that if you are copying anything from somewhere else, take a note for yourself. Take a note, what the note means. It’s a source of the SBOM, we are looking for the URL and package url. We are looking for the vendors, right? It’s a habit. It’s a cultural change. It’s about the people. They have to follow those policies and guidelines. We have to educate those people. 

That’s the tough part, to be honest, because the demand is so huge and, sadly, developer are the pressured on the time. They will like to get things done as quick as possible. They will likely remove the application as fast as possible. They need enough time to think about it, to write the notes, to think about SBOM, to build up those requirements. 

It’s just not happening because there’s so much pressure from the business aspect of it. Writing the code seems lesser and lesser. It looks like we’re going to write less code. We just create the configuration of the code.