Alright, folks, welcome down the security rabbit hole. Welcome to the BSides edition of Down the Security Rabbit Hole Podcast. You know me, I'm Rafal Los. And this week I am scheduled to record live here at BSides Atlanta, the Georgia Tech Hotel and Convention Center. Conference center? Convention center? Both. Over the next few episodes, I've got five really cool speakers for you guys. they're the sharpest ones, I think, from the conference, some of those cool topics. Mehul Revankar is gonna join us.
He's the guy in front of me right now on the future of cyber defense in the age of AI. Phil Brass is gonna join us on LLMs and how they're changing the pen test game. Good old Wes Winer on unlocking open weight models. That's an awesome one. And as a treat, Mr. Xavier Ash is gonna join with some stories from the trenches over his time. And then as a bonus, Matt Coons on Agentic Detection Engineering. Boy, we've got a big, big, big conversation to be had. I'm gonna start with.
my new friend here, Mehul Revankar. Hello and welcome to the podcast.
Thank you for having me. This is gonna be fun.
Well, you are the co-founder and chief product officer at Quantro Security, where you're building agentic AI for enterprise cyber risk management. That's a mouthful. my take is I think I've read that you spent more than what 20 years building security products. You stopped at Tenable, SaltStack, Qualys, and where you've led product engineering research teams. I read somewhere that earlier in the year you talked at RSA conference about AI agents for vulnerability management.
Well, you've had some vulnerability management episodes on this podcast recently. So those were some of our most popular. I see that you're also a fellow podcaster like myself with the Noise 2 Signal podcast. and those are more long form conversations. I'm looking forward to to digging into that. But here we go. Let's talk about let's talk about agentic AI. give give me give me the three second intro of like where we're headed in this show.
Where are we headed? I think the way I think about this is the entire infrastructure of cyber defense that ha that we are used to that is set up is designed for a human-centric attacker. Okay. And we are moving towards an agentic attacker. And in in in that era the tools and the metrics that we have used in the past are no longer
Fair.
relevant because the AI agents don't necessarily think in terms of how much time it's going to take. It's all compute. And if you have unlimited compute, they can break and exploit any vulnerabilities that they can. So in my humble opinion, things that really matter in the age of AI are things that are vulnerable, reachable and exploitable.
Okay, so g give me some examples 'cause I I think I I think we've talked a lot about as an industry for sure, about really like AI is gonna change the game, everything's gonna be different, like how is it gonna be different?
So the way I think about this is there is there is a lot of false sense of security when it comes to vulnerabilities, right? I mean there are for for example, if you think if your patch if your systems are patched, there is a perception that they're good. Yeah. That you don't have to worry about it. But then you have to think about in the terms of the models and the model releases, every new model finds new types of vulnerabilities in application software. And if that is the case, you might have a completely patched
MySQL server. New model releases, zero days could be found in that MySQL database and could be exploited within a day or two.
Yeah, well we've seen some of that already.
And just because just because you have a patched asset exposed to the s system doesn't mean you're safe. So that is how I think it is fundamentally changing is the speed of response. How quickly can you respond and what are you relying on to patch these systems?
you've got a you've got a statement that the only limitation on AI is compute. Do you think that do you do you think that that goes for everybody? Is that an attacker limitation? Where where is the you know like we all have we all have I think everybody listening, if you don't you probably should have access to some sort of, you know, Claude
open AI, some kind of AI agent, some kind of AI interface, and it's it's important to play and understand these, right? But compute time isn't free necessarily, especially if you're using a frontier model. alternatives are using your obviously your own environment, right? Hosting it locally, but that compute still not free. So what does that mean if the only limitation is compute?
So I'll give you example. So we recently did a study as part of a research team, which we published earlier this year, and that was the basis of my talk, where we exploited around 3,000 vulnerabilities. The average cost of exploiting a vulnerability was less than $3. Less than $3 in tokens and around 10 to 15 minutes in terms of time to create an exploit and successfully exploit that.
So you're talking about identifying so the time between identifying the exploit, creating the the con validating it, creating the conditions, creating the exploit code and validating that it works? Yes. Fifteen minutes.
The end-to-end. So going from and this is I'm not talking about zero days at this point, I'm talking about disclosed vulnerability. Disclosed vulnerability. So for example, there is an undisclosed vulnerability in WordPress. You don't know what that vulnerability is, but you probably have access to the patch files, right? So going from going from knowing what the vulnerability is to creating an exploit, creating an exploit to testing on vulnerable systems, to testing on patch systems, making sure it works end to end is like 11 minutes on average. So that is that is what we are looking into. So
Okay, disclosed bar, okay.
The the thing we ran into in our research, we were limited in terms of how much compute we wanted to spend. Right. So so our the you know I talk about this in our pipeline. There is in the in in our pipeline there are there are two key stages. One is the model escalation loop. The model escalation is we start with the cheapest model and we go up all the stack up to the frontier model. And there is the exploit mutation loop where we where we mutate the exploit to try try different combinations. We are an early stage startup.
So we had limitations on how much money we could find. Even with our limited resources, we were able to exploit seventy-two percent of those vulnerabilities of the 3,000 vulnerabilities we tried to exploit. And so and so my my and the the vulnerabilities that we were not able to exploit, it was mostly because we didn't want to pay hundreds of thousands of dollars in compute to break those vulnerabilities. If we had, we would likely have.
a ninety-two or a ninety-five percent success rate, breaking these vulnerabilities. So if you're a state actor, if you really want to go after certain vulnerabilities, you can actually now and go exploit these vulnerabilities.
So
And I that's why I think compute is the only limitation. If we had given our exploit harness unlimited budget and unlimited compute, our numbers instead of 72% would likely be closer to like 95%. So
Right, fair enough. What does that do for what does that do for like the whole notion that we can simply patch ourselves out of problems? 'Cause there's a there's a lot of there's been a lot of conversation recently, especially on on my show here, if you scroll back just I mean literally ten episodes, where it's become pretty clear that we're not
a as a community, we're just not very good at understanding what's exploitable, what's not exploitable, what's real, what to patch, how much of it to patch. And it turns out now this is all pre AI, right? So this is I don't think we're really factoring in a AI the the way you're ex describing it here, but we're saying attackers really only go after a s a couple limited number of vulnerabilities. And if you get those, you're fined. Now y what you're suggesting is like that maybe potentially throws that out the window. What do you think?
That is the gist of my my argument that all these metrics, all the tools that we have built in the past are designed for a very human centric attacker. And humans are lazy. Humans humans give up. I if they give up, they try, they give up. Agents don't.
Well they sleep. humans sleep, right?
Sleep, but they're also lazy. Like you know, if if something is very hard, if you've tried to break, you know, if you try to reverse engineer something, it's it's a thankless game. Yeah, at some point you will give up. But the agents don't. The agents work on a reward function. And the reward function is successful exploitation. And the limitation is the compute budget that you give to the system. So in in that world, in that world, do you really care?
whether something had a so something is has a EPSS score of less than point two five. Do you really care if this is not exploited in the wild? Because the agents only have to compromise one system. They are not you know, they if they have been given a target.
to break into I don't know where we are, maybe Georgia Tech, right? Here's the ten here's a ten thousand dollar budget, go figure out a break to break in. I just read a article somewhere where the the open open source pen testing platforms are able to break into retailers for twenty five dollars a pop.
Yeah.
You know, for twenty five dollars they are able to get in a and then go from there. So I so my my th my thinking is these metrics, these tools are essentially not relevant and I think you should really focus on you should really focus on is this exploitable by AI or not exploitable by AI? If it is exploitable by AI, how hard is it? Is how much compute it takes? Does it require a frontier model to break through?
Couldn't couldn't it be argued that like the answer to that, you know, is it exploitable by AI or not is just always yes.
That's that's true. G it's always a yes with unlimited budget.
Right, okay. So okay, so you've got a you've got a modifier there of some kind of like financial cap.
Financial cap is if if you're really determined, if you're really determined, you can figure out a way. Because you know, you've seen these situations with Hugging Face where the models broke out, you know, or the agents broke out and they started to compromise systems. That was the function of the reward function. It was not that the agents thought of one day, let us break into Hugging Face. That was I I feel s I feel like that's the false narrative. That these agents have sentience, they can think for themselves. No.
You gave them an incentive to gra go break into systems. Once they broke into the systems, your argument was, look, these ma these agents are bad. They did exactly what I told them. Regulate us. But stop these other open source models.
They did exactly what I told them.
Yeah, right, right. Yeah. Well, we've got a we've got a conversation on open weight and open source a little bit later. That I think that's gonna be an interesting one too. so okay, so i if if dollars essentially are now the limiter, I would argue that it's very similar to what we had previously, right? 'Cause we always said for the last twenty years, before AI showed up and made things complicated or less complicated depending on how you look at it. So everything is everything's exploitable given enough time and money. Yeah.
I think the given enough money part sticks, the time part falls away. Right? 'Cause now it's not not you don't have to say like given enough time and money 'cause time is now it used to be like weeks, months, whatever. Now we're like five or seven minutes as you're describing.
So the example I will give you is ninety percent of the vulnerabilities we exploited as part of the research study were under thirty thirty minutes. Ninety percent of the vulnerabilities that we exploit. And these are not like easy vulnerabilities, these are like remote code execution vulnerabilities, major commercial software, open source software with POC, without POC.
CISA KEV, not part of CISA KEV, all kinds of competitions because we wanted to make sure that the corpus of vulnerabilities that we give the model to exploit is a good representative of a typical enterprise organization and it doesn't feel like, you know, we have we're exploiting all these cheap vulnerabilities.
Yeah, right. So how does that how does that what does that loop look like? how do you describe that loop then? Because the notion of, you know, a bunch of hackers sitting around a dark room wearing hoodies, banging away at computer screens in the dark, right? Like I feel like that's changing.
You don't even have to like babysit these models. You know, the the the one fun fact I'll share with you 77% of the vulnerabilities that we exploited, they were exploited autonomously without a human in the loop. The second stat I will share with you, 23% of the vulnerabilities that we exploited were exploited in the first freaking try. So
That's not good.
So you st you gave it a C V E, go exploit it with the cheapest model with the first try, boom, exploit it. Right? And so doesn't so this is where this is where the entire argument of we are going to we are going to model the the we have going to have a threat model, we're gonna have all these weights to figure out what is what things you really need to prioritize. And I'm like, why does it matter at this point?
Does it bode well?
At this point, you know, you just go. You just point the perfect exploit harness and go.
So wha so that you can look at this in one of two ways 'cause that either sounds like, you know what, screw it. None of this none of what we're gonna do matters. Throw your hands up and go, well, you know, YOLO, right? It's over, like we've lost, machines have taken over. What's the what's the opposite of that? So how do we defend cause it I can't imagine a world where maybe 'cause I'm just I've I've been doing this for so long, you know, co the old way, but I I don't imagine a world where we let
some AI system a automatically start patching our software or or or creating bugs or mitigations. What is the what is the fix look like for you? So
one thing that I've and you know we we talk to enterprises all the time. One thing that I've seen work surprisingly well is that the compensating controls they really work well. If you have a good web application firewall, if you have good firewall policies, you can actually limit exploitation to a big degree. So in our cases what we typically see is we try to exploit from the outside, most of these requests get blocked.
But as soon as the whitelist our agents, as soon as the whitelist our scanners are agents, then we see a lot of vulnerabilities that we can fly. We cannot patch everything, especially like you know, if you're going to have hundreds of thousands of vulnerabilities, we cannot patch everything. You have to have really good cyber hygiene.
That includes includes having all the right kinds of controls in place so that you limit exploitation in the first place. Ideally you don't want to have ideally you don't want to have a class of vulnerabilities in the first place. Like for example, if you're not using a s a certain application which has a bunch of vulnerabilities, just get rid of it. Rather than rather than keeping it around for for you know, this these systems to come and exploit it later. So removing the attack surface itself is a b i is itself is a big deal.
Are we I'm trying to figure out how to say this 'cause I'm I'm I'm I'm sitting here thinking to myself, this is this feels pretty apocalyptic it to some, right? If we're most companies how about this, most companies still carry a ton of legacy software. When I say legacy, I mean some of it comes back to the seventies, some of it, you know, was just from last month.
But it's before we started throwing AI at the software development process, which theoretically and somewhat provably is limiting the number of vulnerabilities we're introducing anyway. What happens to all that old like the old the stuff that's been out there before? In the before times, the whole like be in the before times, before AI saved us and then also tried to kill us.
The way the way I think about the way I think about this is I think we'll see I think we'll see an explosion of vulnerabilities for next few years, which are found by these very models that we talk about. And these these new systems, a a I guess AI powered systems, will define all these vulnerabilities. So these code bases get b secure by design over a period of time. So
The easy, low-hanging, stupid human code written vo vulnerabilities will be found. Yeah. They'll be fixed. it's not going to be an instant process. It's going to be like a three, four-year cycle. And then it plateaus out. So that is one scenario I feel like is likely, where all the old human written software is fixed by AI. In the process of it, we find a big bunch of vulnerabilities, like 10,000 vulnerabilities a day or whatever number you want to come up with. And then there's a plateau un unless
There is a new class of vulnerabilities that are found by AI systems. Then we then I don't know how the world th then I don't know what level what what world we will live in. But I feel like that if we follow this trajectory, we should reach a state of secure by default in an in a optimistic world. Okay. In a pessimistic world, basically new classes of vulnerabilities are found. And you know, you essentially have have to have
AI agents defending against AI offense.
Right. But that so you've got a I I I think we should look at this short, medium, and long term. So short term, you're talking AI built mitigations in real time, right? To try to com identify and combat what's happening, right? The the dare I say AI detection and response.
Yeah, I sh it's short I think short term is just buy time. Short term is buy time through compensating controls or whatever it is. Medium term, fix all the human written code to have, you know, AI vetted to the extent possible. Now, I p I f I first thought this AI written code is probably better in terms of security vulnerabilities, but the flaw is the AI models are trained on human written code, which is filled with bugs. So what the the code they write is also filled with bugs because they've been trained on human written
That's the But that's the problem, right? Like that is that is historically the issue. is now that we've introduced a way to limit bugs, you look at all the software it's out before every time we we create a milestone call it a milestone in software development.
There's always the before times. And the before times don't end right there. Right? There's this adoption curve. So now as we'll race to ad adopt AI in a meaningful way, because it's not like you're just gonna jam it into everybody's software development lifecycle tomorrow, and we're gonna vibe code our way into zero vulnerabilities. Like that's not a thing. Yeah. I'm not delusional. I don't think that's gonna happen that fast. So in r with that in mind, we have
fifty, sixty years of software that's still sitting out there. So you you gotta have this real time it's like E D R but you know, like
You know, it just occurred to me, you know, one one way to deal with this, one way to deal with this is to decommission the old software. I say that I say that because the the cost of software was a function of the engineering talent you could get. So software was expensive because humans were writing the software. So replacing the software was expensive. Okay. But in this world where you can buy
the greatest model for two hundred dollars a month of good enough token usage, it might be good to just decommission the crappy old software and bring the same workflows and the same solutions in an in an AI native world.
Yeah.
I wanna be I wanna be optimistic about that, but but there's this whole forget if we take the cost of rebuilding software down a couple of zeros, right? Because software that maybe used to be a million dollars can now be sold maybe less expensively. Yeah. but there's this whole thing of like implementation and validation and making sure all the things that were here's catch.
all the the add-ons and the things that pe people have built over top of it over the years that were dependent on the bugs. Right? 'Cause cause you know that exists. Like we all know that exists. How do we that that's where I struggle is how do I meaningfully raise the bar for everybody that didn't start today?
I don't have a good answer for it, but my sense is if if you if you if you needed ten people to service the software in the pre-AI world, you probably need two. So you you still you still don't need as many people. And and then and then in the and the two people can actually service and code. So they're not just servicing or just coding, because you're the if you if you get enough technically adept people.
I mean that's fair, that's completely fair, yeah.
you can actually code and service the client. You know, I I'll give you example. I've built I've spent my entire career in product management. And product management is all about roadmaps, deliveries, all those things. And my roadmaps would run for like three months, six months. You know, feature requests would come in, they will come out six months later. One of my one of our customers, one of our customers that we had, our time to feature delivery
was an hour and forty two minutes. From the request coming in from a request coming in to sh shipping into the product was an hour and forty two minutes.
Ha ha that's okay. I I I I mean listen up that gives me hope. See, you said that scanners have to be rebuilt. Talk talk talk about how.
I I I think most of the software is very rules based. So they are they are coded with a bunch of rules, like you know, if you see this port do this, if you see this do this, if you see this banner do this. But I don't think they can think on the fly. They are not designed to think they are not designed to so they can if there are five if conditions they'll try the five if conditions and then it will fail. But in the in the world that I've envisioned, the scanners can think on the fly as they are scanning the service. You know they
That's well yeah, they weren't designed to do that.
You might think I say reason, is that the same thing?
I think reason yeah.
Like you have an open port. What is this open? Is this a web service? What kind of web service is it? Let me probe it. Let me try try all these different combinations. That is a very AI native agentic scanner that is reasoning on the fly. I mean obviously you can encode all the rules in the in the system, but then they can actually
a reason as they're performing the scan. Okay. Go deeper, leave leave this year, go to something else. You don't have to encode all this logic. You just have to tell them try to find as many open ports on this house again.
And then and then try to get in. Do we do we if we turn that on, in in in exactly that matter, say we have a scanner that that can do that exactly like that tomorrow, I for some reason I hear one of my old old CIOs in the back of my mind saying, Don't you dare you'll break everything.
I mean if you if you don't do it the attackers will do it anyway, so you know, w d do you want them the do you want the good guys to do it or the bad guys to do it?
Yeah, that that's well I would love for that argument to work and make sense. It makes sense it I don't know how often it's gonna work.
But I think what has happened one thing w I've s one thing I've seen is there is a mind shift in is post Mythos. I think I have to thank the folks who who created Mythos and the people surrounding Mythos created all this hype and fear, uncertainty and doubt, where now there is l lot of attention around this topic. So there is less pushback around solving these issues. Okay. Th than in the past. So I think you there is there is a higher likelihood of getting through now than in the past for these kinds of things.
Hm. Okay. Alright, so three minutes, parting thoughts. what do you wanna what do you wanna leave people with?
I think AI native offense is getting cheap and fast. That doesn't mean AI native defense has to cost more. I think we have to figure out a way to arm the defenders to defend and the only way you can do it in my humble opinion is have to use AI to defend against AI. So you need yeah you know you need AI native defense for AI native offense.
Okay. That's that's a I think that's a good place to to leave us. I th this has been a conversation to me about sort of a new reality we live in. Yeah. we've we've touched similar we've we've touched in this before, but just you know from different perspectives. I think the the the thing I'm I'm kinda keyed in on this last section we talked about that could the new versions of vuln scanning. it's less probe open port.
Wa you know look for this banner, try this, move on. I I I I am I am very hopeful that something like that can be built and and and deployed cama without destroying our infrastructures as we go.
well the funny thing is Quantro builds that. That's w that's why I wanted to start my own company around because I feel like the entire aspect of the cyber defense has to change and that's why we started Quantro.
Well it's been great having you on the podcast. Thank you for joining us. great talk today. you have internet presence. How do people find you on the on the social
I'm on LinkedIn, Mehul Revankar, I'm on X, Mehul Revankar, and then obviously Quantro Security is the company.
Awesome. Well thank you so much folks. Thanks for catching this first episode as we do this live from BSides Atlanta twenty twenty six. stay tuned for the next one. I've got Phil Brass coming up. And until next time I'll catch you another time, another place on another down to security rabbit hole podcast. Ciao y'all. Bye bye. Thank you.