Jay, welcome to the Noise 2 Signal podcast Welcome, how are you doing, brother?
Good, good. Thanks for having me. Great to be here.
Jay, you are one of the original creators of the EPSS scoring algorithm, which is the foundation for many vulnerability management programs in the world now. And you're also a co founder at Empirical Security, which powers the EPSS it powers the EPSS work.
Mm-hmm.
the way I wanted to start the interview is I wanted to get to the origins of EPSS. How did it get started? Because you are not part of the original Kenna team or the empirical team as it has today. You had your own firm, you were doing something on the research side, you were associated with the DBIR project, you were working on the D B IR project in the early days. You created Scientia, or at least you were founder of Scientia and doing all this work, research work. And the next thing I know is Jay Jacobs is the co-author of EPSS. So
Let's start the interview there. What's the origin story of EPSS? How did you get started there?
Yeah, boy, you mentioned so many things there. It's hard to know exactly which one to to pull on first. yeah, so Scientia was born out of the DBIR. So the the
The guy who started the DBIR, Wade Baker, and I had such a great time working on the DBIR in the early days that we started Scientia to sort of keep that going, but without Verizon. and so Scientia basically works with vendors, works with the security data that they have, and tries to make sense of it and tries to produce reports for them internally or for public facing. Most of them were public facing, had like a marketing sp spin to them, but
One of the earliest customers there was Kenna. And so they came to us with their massive quantity of vulnerability data. you know, not only the usual like here's the CVE and the record and stuff, but all the scan data for hundreds and hundreds of companies. And so we could see what's actually going on, which made it really interesting. And at the same time, another early customer was Fortinet working on their threat landscape report.
And so we had Fortinet with all these detections of exploited vulnerabilities and Kenna with all these sightings of vulnerabilities. And we thought, hey, what if we bring this together? And so working with the two companies, we said, Hey, can we use your data for this? And can we can we, you know, do this extra thing on the side just to see if it's possible? And that's basically how it was born, trying to see, you know, we had these vulnerabilities, we had all this information about them, and then
what people were actually going after and all the alerts and all the, you know, things that people were seeing on the internet. and so that's how EPS FOS was born to say, can we look at what we have and predict what's gonna be coming for the exploitation activity?
You know, the funny thing is, one of the people I'm interviewing as part of the season is Alex Spinto, and Alex Pinto is one of the authors at D V I R.
Mm-hmm.
And he was telling me the process of DBIR and in terms of how they take the data and they come up with the report. And I had a hunch, I had hunch. This sounds like EPSS. This is essentially what Ed told me that you know they stumbled on this data between Fortinet and all these other things. And I never knew there was a direct connection between DBIR and EPSS. So it is now clear that you know, you obviously being one of the early authors of DBIR with Wade,
Yeah.
Now all kind of makes sense. the the question I have in mind is, and this is where Kenna Kenna really sh you know really shined really well, was that
Yeah.
It was able to say, well, you have all these hundreds of thousands of vulnerabilities, but there are only two or five percent, two to five percent of the vulnerabilities that really matter. And no one really knew, no one really knew why two to five percent. Like what gives you that authority to say it is just two to five percent, why not thirty percent? Why not ten percent? And that was the basis of your research, right? So you were seeing something in the data, which basically said, those are the only things that we are seeing. So what was happening here?
Right.
Yeah.
Yeah.
Yeah, and it's really hard to say like that upper limit, like where where does what is the maximum number, you know, maximum proportion of things being exploited. because essentially like we can say at least this amount. This is what we're seeing. These are the exploits, the activity that we're seeing. And it's depending on how you slice it and which time period you're looking at, it's two to five percent. We've seen it up to seven to eight percent in different slices.
and it could be higher because this is essentially the base of what we're seeing. yeah, and so I mean, like that's a big
Discovery in my mind. There's a couple of big discoveries, but like that's one of the big ones saying, like, whoa, wait a second. Like we've got, you know, thousand millions and millions of open vulnerabilities. A lot of companies literally, like one Windows desktop could have 150 vulnerabilities on it at any given point. It is not. And when I went into it first, I was thinking like, I was thinking of like a you know, the food supply and like the pack growing to the food supply, and the food supply is sort of
Yeah.
Yeah.
you know, maintaining that balance. But no, this is not what happens in vulnerabilities. Like the food supply of open vulnerabilities is way more plentiful than the attackers eating off of that food supply. You know, and so there's always there's always more vulnerabilities to exploit. There's something else limiting that.
And were you plugged into what you know when you did this work? Were you plugged into the vulnerability management ecos ecosystem? Were you like plugged into C VSS? Because I read recently one of the things that you said was, what is C VSS even measuring? You know, that was you had this, you had this post that, you know, even after all these years, you've been at this for maybe 15 years or 10 years. and you know, for somebody like you to come back and say, dude, what is this C VSS? Right? So
Yeah.
Yeah. Yeah.
when you compare your number to two to five percent with EPSS, what was the equivalent in the C BSS terms and how did that change?
Yeah, the the C VSS is a big topic and we could probably spend a lot of time there. But yeah, I mean like going back to what you said about what is it even measuring, and I think that's a really interesting topic to talk about. But you know, it's been around for this long and people say severity and you say, What's severity? And they're like, Well, it's you know, you get different answers depending on who you ask and stuff, and that's there's something really interesting going on there. But when you talk about C V S, like that the way that's built is be by
you know polling a bunch of people in the C VSS SIG group and giving them vectors and saying which which one of these is worse. They're comparing two of them. And so C VSS is really measuring a a practitioner's perception of how bad a vector string looks. Right. And that's it. Like we're measuring opinion judgment in a very complex situation based on a you know eight nine different vectors. and so like comparing that to EPSS's apples and oranges and
there's we can use some corollaries because people understand C VSS in practice. You know, they they've built up some kind of intuition about what it is. So we can transfer that and talk about how some of those lessons or intuition about C VSS can transfer to EPSS a little bit. But that's about it. I mean like there's a limit there.
Yeah.
But I I will I will say this. The pre C VSS world was worse than the post C VS S world. So yeah, I mean, regardless of how much I mean the C VSS deserves all the criticism in the world, but the pre C VSS world it also needs credit because the pre C VSS world was bad because I I remember I was doing I was writing Nessus plugins, this is pre C VSS and you could have
Yes.
Yeah.
Yeah, but it also gets credit, right?
high medium cr critical as a researcher. I think this is a critical and this is a critical. And but no one knew what is critical. And like Jacobs could say, Well, this is a low, well so so the pre C V S world was definitely worse than the post C V S S world.
Yeah.
Yeah. Yeah.
Yeah.
Yeah.
And then but that's a really interesting observation too, because like that is how science evolves, right? You get someone who says something and people are like, Hey, that's kind of helpful, and we use it for a while, and then we learn from that and say, Hey, what's next? Well, how are we gonna improve on that? And so I think that's where we are with stuff like C VSS and even how we talk about vulnerabilities in general. I think we're at a cusp. I mean, hopefully we'll talk about AI soon, but you know, at we're at a cusp where where we need to now reevaluate some of these things and go after and improve on what we've been doing.
'Cause it did improve where we were, you know, twenty years ago when it came out, but yeah.
Yeah.
you know, you so you you had this hunch that only you had this hunch that two to five percent are getting exploited in the wild, and then can you partnered with Canna? And was there like aha moment because you this vulnerability data was separate, the 40 minute data was separate. And was there initial success that you were seeing at Kenna where people were like, Wow, this is awesome, I can use this, I this is better than C VSS, and then go from there. Was there I I I believe there are like a volume one, volume two you were doing these quarterly or maybe yearly report.
Yeah.
Right. Were you seeing the initial traction with these reports coming out and people saying this is awesome, this is great?
It's yeah, it's really hard when doing research like that because i I've done so many there's probably, I don't know, maybe a dozen things that I published that I was like, this is gonna change the world. You know, like this is awesome. Like as soon as people see this, their eyes are gonna be open. It's gonna be wonderful, right?
And i it rarely happens like that. And but the some of the stuff that we did with Ken, especially early on, you know, EPSS when we were publishing it, it was like, boy, this is gonna change the world. And like it it it was a very slow burn. It was
In in in your defense, it was a proprietary thing. So it is very hard to adopt. It was very hard, you know, it was just it is essentially Kenna talking its own book. Hey, look at look at our score, our score is better, everything is crap, right? So it's very difficult to get credibility for something that is proprietary and a black box kind of a thing. And that you know, that's actually a good segue to get to the next topic is you eventually decided to just gift gift it away.
Right. Yes.
Yeah.
Yeah.
Yeah, to give the score away.
And or maybe like move away from the Kenascore to the to a public score, which became the EPSS. Right? I there's one question I want to ask. Was there any reason that you kept the C E PSS to it was as a play on C V S S? Was there intentional? Or it was like you know, well it sounds good.
Right.
And
It's there's there's a whole theory in naming things like that and when when you have a new concept, you don't want to push people too far from where they are.
Right. And so like you wanna get something sort of adjacent to where they are. And so when they hear something like EPSS and they're like, hey, that's sort of like C V S S, is it? You know? And so like it's it's sort of seeding them thinking, it's close to that. There's some it rhymes kind of somehow. I'm gonna associate it, you know. So yeah, there was there was definite thinking of doing a slight play on C V S S with that, but but not trying to, you know, total.
Mm.
How did that come about? How did the how did the whole process of giving away the scoring algorithm to the first organization come about?
It so first is just a partner. We didn't actually give it to first, but yeah. We I knew that EPSS was gonna succeed through data partners, right? Through like Fortinet donating data to EPSS so that we could do this score. And the only way that would work is if it wasn't proprietary or commercial.
Mm.
Right. And so we wanted to make it as open as we can. and there's some limitations how open we've made it, but we wanted to get that score out there and really benefit everybody. So then we could go to, you know, like we went to Gray Noise and Alien Vault and several other companies saying, Hey, would you participate? And and they were like, Yeah, it's a great effort. Like, you know, you've got that great paper and yeah, let's let's support that, you know. So that's really the the impetus for getting that out there and giving it away and making it public like that.
But well I I'm sure this Kenna saw this as its own score and it must be difficult for them to let go of this core. Dude, we have built our proprietary business around this core. and you're talking you're telling me you're gonna give this away? how how was that? How was that? How did that play out?
Yeah.
Yeah. There there were some delicate times. but it it was largely you know, like the the folks at Kenna were I mean, still are very bright people and they they could see the the path, you know, they could see that this probably would not grow as a commercial entity, and supported the opening of it and you know, and they they obviously still got recognition and definitely benefited as a as a company from it. So
And then why the first organization? Why was that the choice? Or was there or is it just happenstance or something more to that?
It's
It was it was a like a a close thing that people knew about. so as initially I was working with Sasha Romanowski and he's been a huge, huge influence and and driver of this. And I I don't think EPSS would be where it is now without Sasha Romanowski, but he really wanted to get a group
built around EPSS. And so what we wanted was like a calendar and a meeting space and a mailing list and you know, things like that. And he had been working with First in the past. And so.
And this is pre this is pre EPSS. This is before the EPSS thing was officially launched, or this is something post EPSS that this
We had done at that point we had done the paper when we did a presentation in 2019 at Black Hat, Michael Reutman and I, another co-founder at Empirical. and so we had four authors in that original paper. and so and we had that paper out, but that's all we had. and the first version was thinking like C VSS. I think it had 16 variables and you had go collect it yourself. And we gave you you published a formula for, you know, doing an Excel or something like that. but Sasha was like, we gotta keep
momentum going, we got to get a mailing list and you know make this public. And you know, he was also an advocate for public. And so that's how we got into FIRST because he was familiar with it. And they had, you know, offered mailing lists and SIGs. And, you know, so we started the EPSS SIG at first. And FIRST actually really helped. And they developed the API. They're still running the API. so it's a lot of it is through FIRST. A lot of the growth that we've done is is partnering with FIRST.
And is Sha Sasha was involved with the first organization before or was he part of the first organization?
Yeah, and he's he was one of the original folks on C VSS and works at Rand Corporation. Just tremendous researcher, doing a lot of great work out there.
And then what goes into, you know, you mentioned when we were discussing C C VSS, this was these were maybe ten or fifteen parameters that the the practitioners used as a sense of severity of the vulnerability. But when you were thinking of EPSS, what went into your thinking for creating the EPSS core?
There's so if you think of an attacker, right? And this is what I was trying to do. Think of an attacker, how do they choose what to go after? Because, you know, we know if there's only two to five percent being exploited, there's a massive things just staring them in the face that could be exploited. How do they choose what to exploit? And so you think of things like, hey, how hard is this? You know, buffer overflow versus SQL injection have very different complexity surface attacks, you know. and so trying to
Trying to look at these attributes, like what makes them more likely to be exploited? you know, like why would someone choose something over another? And and you get some very clear signals, like, first, if you see something in Metasploit, that's a pretty good signal. because and I think we're seeing like 80% of the the modules in Metasploit we see for active exploitation activity pretty continually. and so it's a pretty good signal.
Yeah.
but then you know, like exploit db is less of a signal, even though it's the same exploit code oftentimes, or you know, like a modification or an additional exploit code, even though it's exploit db and you have the exploit code out there, it's it's a lower prevalence of activity. and so we're looking for these signals, like what if you know, if I'm a 16-year-old kid or something and I want to go after something, what am I going to pick? How am I going to decide what what to scan for, what to find, things like that. And so we're looking at things actually like pen testing tools.
What pen tests do often mirror a lot of the attacker styles. And so we're going after what what tools do they use? And then if you look at those tools, they're not scanning for 360,000 CVEs. They're scanning for a few thousand often. You know, and so what are those few thousand? What are those attributes? And so those all become signals into the EPSS model.
And then in do you have a hunch why the Exploit D B didn't take off but Metasploit took off? but why
Yeah, I mean Metasploit you hit a button and exploit DB you have to grab it and compile it and, you know, make it run. that's the major difference there. But now a lot of the exploit data's on GitHub too, so
Absolutely.
And do you then factor in different weights depending on the source? So if it's Metasploit then you know it gets a different weight. And if it is like exploit D B or packet storm gets a lower weight. if it should open shadow server, then you know that gets a different weight kind of a thing.
Yeah.
And yeah, we don't assign weights. I mean, this is part of the in inside of EPSS. It's all driven by the data. So this is machine learning and the the machine learns these weights through iteration and things like that. And then we also get interaction effects, which makes it even more complex.
You know, so like you might have Metasploit, which would be sort of a default way, but then hey, this is actually in a Microsoft product. We need to adjust the influence of Metasploit in there, right? So you get these interaction of or it's not met Microsoft, it's this weird thing out in the middle of nowhere that might actually lower the fact that it's in Metasploit, might have a lower influence then. So yeah.
Yeah. So let's let's get under the hood with EPSS. Right? I mean one of I I don't know how you how I don't know how you cross this bridge because historically people were used to just one score. You get a score
high, medium, low, and be done with it. But EPS has changed the game in the sense that you could have it as one today and maybe point three tomorrow, or maybe a point three today and becomes one tomorrow. Right? So it's very dynamic in nature, as you said, right? You know, the weights are dynamic. You don't you don't set the stage you don't set the weights. It it's machine learning happening behind the scenes. I'm curious how often are these scores run? You know, how many sources are you bringing in? And
Right.
What's the variability that based on what you've seen?
So it there's a lot of factors that go into that. And I mean, the first thing about deciding to make this dynamic was because that's how the world is. You know, if you're if you have a vulnerability in your environment and you say, hey, it's a medium or something, but then tomorrow we see, you know, 14 headlines about it and we see that Metasploit is a module and exploit DBA, and there's 14 repos in GitHub with exploits, you know, that's gonna change your reaction to it. And it should.
Right. You should change to an evolving landscape, whether or not EPSS does, right? But you should. and so that's what EPSS is trying to do is to mirror that reaction to what actually changes in the landscape. and so yeah, there's a lot of those that shift like that.
I'm curious if there is a cross-site scripting vulnerability that is blowing up on Twitter and blowing and there is like an exploit everywhere, does it would EPSS ever rank it as a one? Given Or maybe like yeah, like point nine one, you know, on the top of the line. Or is does it be a criticality of impact?
like a full hundred percent, we are a hundred percent sure.
And yeah, I mean like yeah, if yeah, if there's if there is like really credible evidence that it looks like all the other things we've seen exploited, it will rate extremely high, probably not a hundred percent. I the model I don't think allows a hundred percent, but ninety-nine point nine percent, yeah. Yeah.
So it will so if even if it is you know from a practitioner's point of view, a low severity vulnerability, but if it is getting exploited in the wild, EPSS will consider active exploitation in the wild. Go for it.
It it considers it only during training. So the the way that we set it up, we we didn't want to put like, hey, this is being exploited now, you know, so probably will be tomorrow. We didn't want to put that in there because for multitude of reasons. but without that in there, we are just sort of looking at the attributes and the reaction to it, not necessarily day-to-day activity. so and
Hm. And in the in the scores they they they do get updated daily though, right? and or is it multiple times a day?
Yes. Yeah, they're daily. we do it multiple times a day, but we only publish once a day so we have data coming in all all the time. Yeah.
And all of that data goes.
And and the data gets published to the first website. Every every
Yeah, yeah, we actually publish it. Empirical has their own CSV download thing. we have a GitHub repo where we're putting it. And if you want the history, you want to look at all of the scores since 2021 when we started publishing. They're all on a GitHub repo. And then first has the API. If you want to integrate something and just look up individual CVEs real quick, you can do it through the API.
And do you ever get pushback from this stuff where hey dude, yeah, this was a 0.3 yesterday, now it's a 0.9, what's going on? Yeah, or like you know, Jay Jacobs, this EPSS thing doesn't work for me, kind of a thing, and like rage baiting, and you know, rage baiting, that is one part of it. And the second is the second question I have for you is how do you know it is accurate? You know, if let's say it goes from point three to point seven, then how do you know, well, how do you you know make sure that the model is not
Lying, not lying, like you know, like falling off.
It's yeah, no no no being being tricked, being fooled. Yeah, absolutely. this is essentially like the entire premise of data science, you know. So like I mean, this is why data science is what it is. Yeah, sorry, but the but I mean like this is this is why, you know, s there's a science behind data, you know. when we make a model.
Now now you're now you're now you're sounding like Anthony Fauci.
insecurity to like when i when i i went to statistics and machine learning because i came up in security and i would see these risk models and the way the risk model was developed is that someone would identify some things put some weights to them generate a score and then see if anybody complained right and that that's how the models worked and they still a lot of them work that way right and but when you get into statistics that is
That is not how the world works, right? You you build a model and you have to test it. You go back and you say, hey, I'm making these predictions. How did it turn out? Right. And because now we've had EPSS for I think we've been publishing scores for over five years, every single day is a statement, a testable hypothesis. We could, and it's trying to predict in the next 30 days, what do we estimate the probability of exploitation activity? And it's very specific actually, exploitation activity that we are trying to gather.
Mm.
not in somebody's environment, you know, not just randomly in the wild, but out of the data we're getting. And so what we can do is we can publish our score, wait 30 days, get all the data that we have around expectation activity and say, hey, how did we do? and so we were putting a lot of plots. We just we're just released a new website for EPSS and we've got plots in there talking about how we calibrate the probability, how we know, you know, like generally
It's not perfect. to your point, like, hey, this is a this is really low and I see exploitation activity or something. that it's not perfect. And you know, we want there's a plot in there talking about the the coverage versus the efficiency, and you want that line to be in the upper right, and it's sort of going through the middle, sort of pushing to the upper right. So it's gonna, it's gonna have some things high, some things low. But the the challenge is you wanna be better than any alternative.
So if we go back to something like C V S, you would just wanna outperform C V S.
I
I have a very f I have a and I should have asked this earlier. I have a very fundamental question. What is EPSS predicting?
Yeah, it's a probability of exploitation activity being observed in the next thirty days in all of the sources we're collecting for exploit activity.
And it is it is thirty days only, like not sixty days, not in year, like likelihood of exploitation in the future. This is like thirty days. That's the metric you measure your sample.
Those are yeah. Thirty days. Yeah. That's how we measure our performance. So it's it's very correlated to sixty days and one day in the future, right? And there's a a little formula you can do to like change that probability into one day or sixty days, things like that. but yeah, it's a 30 day and that 30 days largely arbitrary, you know, sort of patch Tuesday-ish 30 day outlook once a month.
and so that's I mean, that's how we do the model. And by having that very strict bounds, now we have very strict guidelines on which to measure the performance and see how we did.
Let's just double click on the thing that you said. It is much correlated to the one day and the sixty day. What do you mean by that? Is this like a fundamental concept of like data science and statistics that where you see a thirty day if you see i the thirty day and you know the variance between the thirty day and the sixty day is very marginal to the point that doesn't matter. So you might as well just go stick with the thirty day.
no, I mean there's a relationship. So like, you know, the the probability of you know, getting a disease in the next year is correlated to the probability of giving the same disease in the next five years. I mean, they're gonna scale depending on your time that you're looking at. So like the probability of exporting something in thirty days, if it's high in thirty days, it's gonna be high in sixty days and high in a one-day probability. So it's just, you know, it's scaling that depending on the time window.
It's so it is statistically significant. Is that is that what you're saying? That when you say it is you know yeah, is it the thirty day window, then it's going to be likely to see you'll likely see activity the sixty days and the year and so on? Got it. Awesome. So you talked about the new website, but you didn't mention there is also a new version of
The Yeah.
Yeah. Yeah. Yeah.
EPSS. I mean that my assumption is the new version drove the new website. So the the version came first before the website, right? and one of the things that I remember from the announcement of EPSS version 5 is that it is 23% much more accurate or much more accurate. And as somebody who's not into statistics, this may not be very clear. And my understanding of it is my and correct me if I'm wrong.
Right.
Yes.
Yeah.
Yeah.
So my understanding is that 23% is a measure of how ac it is twenty-three percent more accurate in predicting exploitability in the next thirty days compared to version four. Is that accurate or because there is this all this science? When I go and talk to you, you'll me, no, no, no, no. There is like area under the curve and then the this curve goes down and this curve goes up. Like there are many things that go in. So what's the real deal behind the the metric?
It's Yeah.
Yeah.
So the the twenty-three percent, so anytime you do data driven communication, the the the marketing people
Yeah.
They have two things. They want a what's called a hero graphic, right? Give me the one graphic that's gonna be like when you put it in LinkedIn, like you're gonna see that graphic. And then the other one is a hero stat, hero statistic, right? What is that number that's gonna grab attention? And so with something like EPSS, you know, we're we're creating these curves and we're doing this plot and like how well is it performing across all of the scores and you're trying to look at three hundred yeah.
Yeah.
No, before we go there, can you explain the curve? Because I think that is important. Can you explain the curve goes up and then it goes down, really? Or maybe it goes up it stays straight and then goes up. Or one of those things, right? So can you explain the curve? And you know, when the post edit, I'll overlay the curve when we when the video goes out. So go ahead.
Sir.
Yeah.
Yeah, so the it's if you
Yep. Yeah. And so if you the what we're plotting is called a precision and recall curve. and so there's a great Wikipedia page on what precision and recall is. And basically what and we rename it to coverage and efficiency because that's a lot more intuitive for security people. But coverage is, you know, if you take everything that was exploited in the next 30 days, how what good coverage do you have of that? Like how much of the things exploited did you actually remediate depending on how much you did?
And then efficiency is sort of that flip side out of everything that you did in your environment that you tried to remediate or prioritize, how much of that was exploited? And so, you know, if you're spending a hundred dollars on remediating, maybe two dollars of that went to things that were actually exploited in the next 30 days, and the other ninety-eight dollars could have been delayed slightly or something, right? And so that's efficiency. And so coverage is mainly what people care about. They want, you know, they want to.
patch things that are going to be exploited. So coverage is the main one. But we and so because EPSS is a continuous number depending on the CVE, right? Between zero and one, what you end up with a is a curve. So like if you did, hey, 0.99 and above, you're going to have really high coverage because those are really likely to be exploited.
you're gonna have low efficiency or really high efficiency, I guess. You're gonna have high efficiency but low coverage. Sorry. so you're gonna be very efficient because most of those are exploited, but you're not gonna get most of them that are exploited because you're only doing a little bit. Right. And so as you slide down that, you're gonna be changing your trade-off between coverage and efficiency. Right.
I have a fundamental question. Maybe you said it, but maybe I didn't understand it. Is there a way? Is there a is there a way to measure? Let's say in the next thirty days, two hundred CVs got exploited, but but EPSS only predicted one fifty of them in the world, regardless of the remediation, regardless of all those things, but just genuine exploitation activity that we've seen in the world, and there were two hundred, let's say two hundred C V showed up, but
EPSS only predicted one fifty. Is there a is there a curve or a measure that you can say how EPSS was accurate compared to what you've seen or no?
There so EPSS never says this will be exploited or not. There is there is
The the I mean what I mean is like the Yeah, yeah, yeah true. Yeah, true. It isn't it is not binary. It is it is over the curve. Would it fall under the curve?
But but what happens, you create a threshold, right? So like as a practitioner, you say, like, hey, I wanna I wanna remediate things over this score and I I wanna delay things under that score. So then you do have a binary, right? So if you set that at like a point nine, right, most people think or point five, let's say fifty percent above probability, which seems somewhat high, you know, but it's not. It's like
less than I don't know one percent of the data or something. It's a really small portion because things are not exploited. So most of these are scoring really, really low. So because you know two to five percent are actually exploited.
That means that two to five percent are gonna i if we had perfect, two to five percent would be at a hundred percent, and ninety-five to ninety-eight percent would be zero, right? So you would expect over ninety percent of the data to be close to zero, and that's what we see in the score. so but if you say over fifty percent and you got two hundred that are exploited, you could say what proportion were over fifty percent or over ten percent or whatever. You set that threshold.
Right. And that's how you create that curve is that you're setting these thresholds and sliding it through and saying at this point, if you say over ten percent and above, you're gonna have this coverage and this efficiency.
And what's the deal with the new website? Why the new website? To drive the
Well we n we needed to update it. and there's two two big reasons. Yeah. Yeah. Yeah, we
That is true, right? That is that is evident, but like wha what apart from what what apart from it? Are there new charts, new information that is published on the website?
Yeah, and largely I think this evolution of the website is sort of a a a good point to sort of look back and all of the feedback that we've gotten about EPSS and try to capture everything that people
either complain about or ask about or don't understand or get wrong. and so and the other thing we wanted to do is make it a lot more LLM friendly. so if you just wanted to grab Claude or GPT and say, go tell me about EPSS, it can read that website and summarize it pretty quickly and hopefully represent it as accurately as possible because there are
when you get into machine learning, especially probability, things get really weird and people have very bad intuition about probability. And so it's it's a challenge to talk about it sometimes. So hopefully LLMs will help sort all this out for folks.
And one thing we haven't talked about is we've talked about EPSS a lot, but we haven't touched upon the role of Empirical behind EPSS. So Empirical is the organization that is funding the compute and all the infrastructure that is required for that is required for EPSS. So you obviously publish the scores for EPSS, but then there are I guess proprietary versions of EPSS that are packaged under Empirical. Maybe speak a bit more about that too.
Yeah.
Yeah, and so EPSS, you know, could because we made it public and we made it, you know, sort of community through the the the EPSS SIG and stuff like that, it was it and it somewhat still is a volunteer-driven program, which makes it very problematic to get resources sometimes. And so at Scientia, I, you know, managed to get resources there. I convinced my partner that, hey, we need to do this, you know, support that.
And the other thing is there's a lot of stuff sort of untapped for EPSS, a lot of insight, a lot of really, really juicy, good research bits that I know would help companies. And so there's two. So we started Empirical basically to tap into that extra stuff to really get into all the stuff that companies would really find valuable. but also to to give EPSS a home. And now because we've got empirical.
Mm.
We've got this commercial backing where we can go out and get more data. We're actually buying data, which we couldn't do before, right? EPSS was totally reliant on donations. But now we're able to buy data. And so we're getting data subscriptions, we're taking in commercial data, we're taking all these other things, trying to improve the number of signals, the range, the variety. And so that's really one of the benefits. And then also that also helps us feed the commercial side where we can do a lot more interesting things with the research and have it actually funded, which is really great.
You know, given all you know, you you've been at this EPSS for five, six years. Are there any success stories that come to your mind that you are like, Wow, I didn't expect this to get this far? Because one thing that recently came up on my radar is the Mythos thing came out and then EPSS was, you know, they use the EPSS core or in their blog and mention, right? Are there other stories like that where you were like, you know, you're drinking coffee like you're doing today and then
Yeah.
Something jumps out and you know, EPS is what like help somebody do something better.
Yeah. Yes, there's lots of them. and every single one surprises me. Like it's really amazing.
Yeah, that's all good. So any examples come to your mind?
Yeah, I mean like there was I remember being on a conference in the Netherlands and someone came up and was just enamored that I had created EPSS and was so happy to meet me and they were talking about how much it helped in their work and all this stuff and just stuff like that, where it's like out of the blue, you know, I get stuff like that. But then on the EPSS website, Patrick Garrity has helped curate this list of companies who are using EPSS and that is well over a hundred companies now.
and like I was talking to a really, really large company who I don't want to name yet, but they they were gonna put EPSS into their advisory feeds and they wanted to talk to me about how how they could trust it and stuff like that. And and on that list was that company that was talking to me, just a different division, is already using it their products, you know. and so things like that were like it's already out there, it's being used heavily. and I think
It's also been really slow. Like our first paper was seven years ago at Black Hat. So coming up, you know, next week is Black Hat. So seven years ago at Black Hat, right before COVID, we did our first presentation on it. and I think it's still I would consider it still young and growing. and so I think the the better stuff is ahead of us. and like you said, we just came out with a fifth version. Our first version, even the second and third version to some extent, don't look anything like where we are now.
Hmm.
And so as we get to the tenth and fifteenth version, you know, I expect to to have some really, really amazing stuff as we're evolving and and following the trends.
How did it feel when the Mythos rage came out and EPSS was like front and center on that? Like w were you like, yeah, but it's like, yeah, then like you know, because then I saw like a marketing push for empirical. Like you know it was like a big push, dude. We are a mythos, right? So how did that feel?
It Yeah
Yeah.
Yeah.
It it's good, it's really good, but also like you know, it was yet another thing. And I'm like, all right, now now we're serious, we really gotta focus on this model, right? Like, I I never wanna like rest and be like, we're good, like we're a meth, you know, anthropic mention us and mythos is quoting us, whatever. Like you're not done at that point. That's when you start, right? That's when you you you gotta dig in at that point. You really gotta get focus and stuff like that. So yeah, it's motivational.
We are legit now.
is motivational. so, you know, on the topic of meetos, let's talk about the future of let's talk about the future in this AI native offensive world that is coming our way. it's much easier to write exploits, create exploits. Previously the writing of exploits was dependent on the human talent. I mean I know this personally. I used to write like exploits for Nessis a long time ago and it took a lot of time and skill.
get to that point. Now you don't. You can prompt your way, you can prompt your way to an exploit. So which means which means by definition that it's going to get easier to write exploits. So more and more of these things will show up in your data fields where exploitation will get easier. vulnerabilities more disc discovery of vulnerabilities will go exponential from here. At least that's my sense is where you point a model and you point you point
Yeah. And it is. Yeah. We're already seeing it, yeah.
You point a model to a code base, ten thousand vulnerabilities will show up. what's the what's the future in this AI native offensive or AI driven exploit world that is coming our way? Exploitation world that is coming our way.
Yeah.
Yeah, it's I I'm excited. you know, and a lot of people will say, Hey, with all this change, doesn't that
Are excited for the offense that is coming our way or are you excited about EPSS? Very important distinction.
I know I'm excited about EPSS and the defender aspect. There's definitely gonna be, and we're already feeling the pain of this.
So this change is being thrust upon defenders, right? Whether we like it or not. This and this is going to give the attacker an advantage initially. you know, we're already seeing it. And so now what we're seeing is this bottleneck for P-serts, you the product security incident response teams. they're the flood of reports of vulnerabilities that they have to deal with. I mean, Microsoft had their largest patch Tuesday, Oracle just had
the largest day of published CVEs ever in the history of the CVE program, over a thousand CVEs that Oracle just published in a day for their quarter I think it's quarterly release, but and but how how we how we tackle this in the future is basically with science in my mind, you know, like
If if the landscape is changing, let's watch it. Let's figure out how it's changing and figure out how we're going to react. So if you say, like, hey, we're gonna see a whole bunch more of exploit code being written, it's super easy to write. We should see a whole bunch more exploits. Well then hey, let's go figure out how to write detections automatically. Let's increase the amount of things we can detect for exploitation activity, because then we can know.
Hey, what what is the new AI bots really good at writing exploits for? Where do they struggle? Do they have problems with timing-based attacks? Do they have struggle with very complex things? Or do we see the opposite that, hey, we've seen some evidence that they're really good at chaining vulnerabilities and finding, you know, sort of these three or four vulnerabilities that will come together and chain for an exploit, which is going be much harder to defend against. But the way, the way to go after that is with data, is with data analysis and exactly what EPS.
S is doing. I think the the scale and the rate is going to have to increase a whole bunch, but this is how we're going to tackle it. We're going to tackle it by watching what's going on and being able to react and and put the resources where they need to go.
The the th the thing that is not clear to me is how how and this I was talking to Patrick Garretti. He just came out with his AI exploitation report earlier this week. and he was saying to the effect that he's just not seeing the evidence of AI exploitation in the wild. And my question to you or somebody like you would be, how would you know this is AI driven exploitation? Because you know, it the script is a script. You don't know if a human wrote it or an AI wrote it, right? So how would you know if it's an AI written AI driven exploitation or not?
Right.
I I don't think it matters, honestly. Like if you're a defender and like you're seeing exploit activity, you don't really care, like how did that come about? You know, no. You you go after it anyway, right? So
True, so who can order it?
No, no, no. But in terms of classifying, in terms of you know, like the one narrative is we are seeing increase in AI driven exploitation. How would that narrative get set? Because the evidence would not show up and hey, I am an AI bot, I'm trying to exploit kind of a thing. You see what I mean?
Yeah, yeah. I I I mean like I think we could do it with by seeing an increase that we'd never seen before. You know, I mean like you could say, hey, we were two to five percent, now we're at fifteen to eighteen percent, you know. and and we might see something like that where we're seeing just this massive increase in general volume. but knowing this one versus that one is AI versus not, we're gonna have to do something other than detection, right, to figure that out.
Is irrelevant. Yeah. And and and to some extent, as you mentioned, it is irrelevant because b it the only thing that matters is is it exploitable by AI or not exploitable by AI? And maybe that becomes a factor into into the algorithm. last or maybe a second last question for the day. Gun to your head, Jay. you have to pick a threshold.
Yeah.
Yeah.
I know. Yeah, there is a security practitioner CISO is frustrated with all these vulnerabilities and he's getting even more frustrated with these AI vulnerabilities. Comes with a gun to your head and says, Dude, tell me a score I need to prioritize on. What do you say? What what number do you give him?
Yeah. I so this is something I've always resisted this. So gun to my head, I gotta answer it, but I've always resisted this because it it depends on who you are, how much money you got, how crazy your environment. You know, there's so many things that go into that threshold. And you probably wanna have other factors than just EPSS in there if you have any kind of mature program. But
If you're not a mature program and you don't have these resources and you're just standing there saying, tell me what to fix, right? I would look at something like like I said, C VSS people have an intuitive sense about. And if you look at C VSS nine and above critical, right? The critical, these are critical, C VSS nine and above, that gets, depending on how you slice it, between 10 and 13% of vulnerabilities are marked critical through C VSS.
So if you take the the let's just say 10% of the EPSS scores, that's about a 3% and above. So like 3% probability and above is about 10% of the data. And so if you know, around there, about 3%, which is crazy to people, because if you think like what's critical, no one would say 3%. You'd be like, maybe 90% and above should be critical. But no, like if when you're looking at the volume and looking at these probability.
it's gonna be about three percent and above. Now when we publish the score, we publish a percentile. So you could just go to the percentile and say point nine and above, which is the top ten percent. Right. So you can do that percentile and just look at the top ten percent. But that would be gone to my head, you know, that would be critical in my mind.
What w what is the equivalent of two to five percent in EPSS world? You know, the two two to five percent that you see in the what would that threshold be?
Yeah.
Well that that actually dictates more of the base rate.
Of exploitation. And so what that says is on average, the probability is going to be come out to about two to five percent. and so that's why when I say three percent and above is critical, that basically means you know, if we're saying two to five percent, the top 10% are rated over three percent. So that's what that actually does in EPSS. There is no threshold that says you're gonna get this two to two and a half percent exploited.
You know, so we we do see things that are rated really low, they just have a low probability, right? So it's very unlikely they're gonna be exploited, but they're still exploited, right?
So so so in a sense, I mean my my my my assumption was point nine and above is the most critical, but what you're saying is point three and above is the is the one that you have the point three and above is the point
Yeah. Point point three. So like it's point three three percent. Yeah. Point three
So so point is where you really want to be if you want to prioritize correctly EPSS.
It's there is no prioritize correctly. I gotta make that very clear. Like, depending on your resources, if you if you wanna make sure you're not gonna be exploited, you gotta do all of them. Right? Like, that's it. Like even at three percent, we're gonna miss some things. And actually, there's the in the new website, we've got some charts that show where that stuff is. Like you're gonna you're gonna miss some things under three percent as well. There's things exploited that are rated lower than three percent, but
Yeah.
Different.
you're gonna put in more effort to get down into those.
Jay, last question for the day. What gives you hope? What gives you what keeps you excited?
There I think that there's so many good questions to ask. and so many good areas to research yet. a as I've been working on this, I mean I've been working with Vone data specifically for over ten years now, and the
The amount of questions in my head that I want answered has grown exponentially since I started. there's so much to work on. And I I think with AI and all this stuff, even though it's like crazy, there's a lot of people having a hard time with the explosion of things, and some people are not even affected, like you mentioned, Pat Patrick. We're not seeing huge explosion everywhere. We're seeing a few pockets here and there. But I think I think it's all good. I think that we're, like I was saying, we're we're on a cusp.
where we can it's it's so important now and the money is there and we can say we've got a drive to do this research now. We we want to take a step back and say, whoa, we gotta we gotta approach this differently because what we're doing isn't gonna get us there. Right. So I'm I'm excited about that. I think there's there's so much work to be done and I I love it.
That's that's a good point to end this interview. Jay, so grateful for you to come to come on the Noise to Signal podcast. I hope you come back when you do the version six and the version sevens and maybe the version eight website too.
Yeah.
Yeah. I would love to. Thank you for having me.
Thank you.