Dan Jones, thank you for coming on the Noise to Signal podcast. Welcome.
Hey, good to be here.
Dan, you are you're a red team engineer at Cloudfare. and the work that you did at CloudFare around vulnerability discovery hardness caught my attention a few months ago. And it turns out you're also a friend of the pod, Jackie Dutton. So I thought this would be a great opportunity to get you on the pod and talk about some of the amazing research that you and your team are doing at Cloudfare around vulnerability discovery hardness. And I believe
Cloudfair was one of the first organizations and enterprises to have access to these cutting edge models like Mythos way early in the way early in the stage. I believe this was February or March. So I want to get started. I want to start the interview there. I want to start what was the origin story for for this work that you did around vulnerability discovery harness, vulnerability validation harness.
at CloudShare. So let's start there. What was the origin story? How did this project get started? How did you get involved in this project?
Yeah, of course. So the starting was really it was in towards the end of last year, in maybe October or November. we sort of realized that soon there's gonna be something that happens. We're suddenly gonna have hundreds of thousands of bugs coming in potentially of aut on hacker one with automated detections coming in because LMs were getting good. They were getting really good and it was only a matter of time before this was gonna be a problem that we would have to deal with. So
We then started thinking about building vulnerability discovery hands then, and this was six months before Mythals even existed. we started building up a platform to do this, to test this, to find different ways of doing stuff. there was a few of us working on it in originally to come and find the most the best ways of doing it. We weren't necessarily talking to each other because we were thinking, cool, let's do this. No one's done this before. This is kind of new, no one's talked about this, there is no current best practice out for this stuff. so we were thinking, okay, cool, where do we start?
So we sort of started doing things in a different way, like all just trying different things to see what worked. We went down the sort of route of like more skill-based stuff, which is kind of where we started. we went down the routes of like really, really in-depth harnesses that are trying to map logical flows through the entire code bases, and we experimented with a bunch of different stuff. All of this stuff kind of worked. What we realized quite quickly was that you using a skill and stuff like that, it's pretty good. But back then skills were nowhere near as good. Now they're probably better and you'll have more luck with it with workflows and built in things to harnesses.
The question I have for you is, you know, when you're starting this project six months ago, nine months ago, and I personally experienced this in my interaction with these models, but they're not good at helping you with vulnerability discovery. They would just say no. They would just say, Hey, I can't help you with this request. You start with something, that is too dangerous. But then you come in and you say, Hey, my grandmother is dying. Her last wish was she won you know, to know about this vulnerability and she, sorry for you know, sorry about your grandmother, let me help you out. Right. So
But like mm-hmm.
Were you having those kinds of challenges when you were doing this six, nine months ago where the models would just flat out reject? Sorry, Dan, can't help you.
Absolutely. Yeah, absolutely. This was especially working on a red team sort of framework. Like a lot of the stuff I build, I build like sort of offensive tools that I can use to, you know, f and perform cybersecurity operations against Cloudflow, because that's what I do for my job. And that's how we make sure Cloudflow is secure. So even with regards to that, before vulnerability answers, you have all sorts of rejections of stuff. and of models that are saying no, please try, please don't ask me this, I can't answer this. and then you get the classifiers on top, which is when you have essentially another model looking at your input and your output, and they say no as well.
But essentially, yeah, these are these are big problems. But when you're dealing with vulnerabilities, if you give the model enough context to know that you're doing a vulnerability scan, you're doing all this stuff, they kind of let you go and work with it. I think that most models in general they don't say no if your request is reasonable. If you then leak that, you know, you're trying to build exploits or something like that, that's when they generally reject it because they will generally push back against weaponization of stuff. but apart from that, like if you give them enough context, then you're like I'm not gonna sit honest with them, but you tell them a coherent story.
to make sense with hunting vulnerabilities. That's when you're you're gonna have quite a lot of luck of getting them to do what you want.
Yeah.
And this is w this is with Mythos as well. Were you also because my sense is when you got access to Mythos, it it removed some of the guardrails for some of the frontier organizations that are out there? Well, I believe Cloudfair was the initial was the initial enterprise as part of the Glass Wing Glass Wing project. So when you were working with Mythos, did you still have these guardrails or were the guardrails completely removed?
Absolutely. so guardrails on models come in, as I said before, so kind of two forms. You have like the classifier type guardrails, which is where something else is watching over the level of it and say and then cutting off cutting it off or downgrading you and that sort of stuff. And you also get native model rejections. Both of the so native model rejections are something that's been trained into the model. These are things that the model has learnt to do and learnt to say, which is it's essentially the model itself having some sort of personality and deciding it's not going to help you with something.
Yeah.
and then the classifier rejections are when there's a hard stop and a break in the conversation and you can't ask that specific question. these are two kind of separate things. when you're in Project Lastwing and these and these things with these other cybersecurity companies, they will loosen the classifiers. They will because they know you're gonna be doing uncertain stuff. They know that you're going to be looking at very specific things and that you're a trusted organization. and for our part in that, we're very, very careful with everything we do with these models. We don't wanna have any sort of
issue with these models going out and doing something they're not meant to do because that would be a really, really terrible day. so we you know, and so they've trusted us that we're gonna do that properly. and as a result we have loosened classifiers and this stuff.
And then did you run into, and this is my personal experience. I want to get your experience as well. When we ported some of these models to our code bases, it found thousands of vulnerabilities, but then were like a lot of false positives. There were a bunch of false positives. And and then we had to then had a a validation system as you did to make sure that you know we are not chasing these random ghost bugs, right? Were you running into those issues as well, where then the findings were large, but then there were many of them were just essentially false positives.
Yeah, absolutely. Like a lot of this stuff is like the findings you find according to the model, the model would probably argue they're not a false positive. Because if you're standing in the right place with the right axis of the data and can change the right things, you can probably do a bad thing. so in order to combat that, we've got multiple levels of validation. I like to picture it as a really, really wide funnel. You try and find loads of really wide things, loads and loads of primitives, all sorts of stuff like that. And then you narrow it down. because you don't want to miss stuff. You don't wanna you want to be able to combine stuff in the right way, you want to get the models more context, you wanna
You'll you wanna let the early hunting models find as many things as possible. You don't want to restrict them too much. but then later on you have to in general we run multiple validation steps. and these validation steps are aligning stuff like aligning our findings with the actual security model of the target application. and also comparing it to what production looks like, what our production config is. Like you try and line up the shape of the vulnerability with the shape of production. And if they match, it's something that might
that might might be applicable. This is also it's not a science, right? It's kind of like it's something that we're still trying to figure out. I don't think anyone's got the exact answer of how to know whether a bug is a real bug. Like especially we're talking about lots of bugs. That's really hard.
Yeah. And you know and I'm curious, like, you know, for the sc for something like with the scale of Cloudfair, you probably have like ten or different, ten or fifteen different programming languages that are used for different projects. Did you see any did you see any like world abilities or more more when s with certain languages like with Python versus C or or Rust? Like, you know, like for example, my sense my sense would be the memory unsafe.
languages it would find more vulnerabilities. did you see that? Did you experience that as well?
I think that's sort of yes and no, because there's a lot of hardening stuff that goes into even memory safe languages. Like we do have some things in C, we're trying to minimize that and get rid of it like all the time. But even so they're pretty well hardened. we put a lot of effort into making sure that the VdeVot entire class of vulnerabilities, even those are written in C, are not necessarily a problem. There's a there's a lot of things you can do. You can use compiler flags, you can you know not allow the use of certain types of constructs. it's harder in C. I think we ha we actually have more C than C, which this stuff is easier in. But yeah. and
With a lot of this stuff, you c you can put a lot of mitigations in place. You could like and it can done a bug from yeah, it's bad, but it's just a crash. Like even stuff ASLR actually reduces the impact of a lot of this stuff. and we we have a lot of mitigations in place to mean that even though in these memory and unstable languages, bugs do exist, but there's a lot less of them you'd expect because our code bases with these stuff is incredibly hardened, because that's kind of the point. but I think the different classes of bugs show up more. So for example, you're not really gonna get mem memory vulnerabilities in Rust, but you do get them in C.
So I would say that in terms of like logical flaws or mixed up authorization, like IDOR type stuff, they are they're pretty much equal across every language. I think it's more the the thing that you're using the language for kind of describes what sort of bugs we're gonna get. obviously some languages are predisposed to memory bugs, like C is, and some languages are not, like Python or Rust. but there's all sorts of weird inter other things that you find as well. And the most interesting bugs are not normally really hard to trigger memory corruption vulnerabilities.
Hmm.
the the simple things that someone's got wrong when you've got two systems that are talking to each other and they're not properly lined up. because that's where you've got assumptions that don't line up with each other and you get these across all languages. And you probably get it more across two different languages because you can't necessarily have a perfectly shared interface between them.
Yeah. One of the fascinating things that I found in your blog is, you know, you started the end outcome or at least the starting point of your outcome was writing the scale. Like you started with this four fifty line security skill. And I
That is one part. And the second part that you mentioned the that is that you know you you cannot just depend on one model to do all of those things. So so maybe we can dive deeper into the skill that you wrote. And I think that is in that is that is open source and it's up open to public, post it to GitHub. maybe we start there and then talk about the fine-tuning that you have to do with the 450 line skill.
Yeah. so the skill itself, the way that works, it's essentially it was built on like a mental model of how a human finds vulnerability. The first thing you do is you understand what the land looks like, you understand what's there, you understand what's built. and then you go, okay, I'm gonna look for injection vulnerabilities, I'm gonna look for memory corruption vulnerabilities, I'm gonna look for all this sort of stuff. it's not quite the way that a real human vulnerability bug hunter sort of does things, but it's kind of semi-related. You firstly understand what's there, and then you would try and apply different learnings to that and how it works. So it's built off that.
So it's essentially it's just th it's a three-stage skill. you find the bugs, you find where the bugs might lie, you try and find the bugs, and then you validate the bugs, and then you come up with a report. it's it works quite well. There was quite a lot of fine-tuning involved initially, but also like really good models, like even modern models, for anything from like, you know, Opus 4.5 or GPT 5.4, all of this stuff, when you give it, you know, you tell it what to look for, you don't need to fine-tune stuff that much. Like fine-tuning gives you fractions of a percentage points of difference. and
No.
Because of the way that you're changing the context and the way that the actual models themselves, the way they think, and all this sort of stuff. Like if you're putting loads of stuff in that's unrelated, you might have issues. But if your skill is very tight, well defined, it's explicit what it wants, it's quite easy to make a good skill work. It doesn't have to be perfect. Like there's quite a few basic things that if you apply them to a skill, it'll be better. Like you if you're trying to write a skill with AR models, they will all and it does something wrong, and then you say, No, why is this gold on that? They will always put in rules that say, No, don't do that, no, don't do that, no, don't do that. But to make a good skill, you want to tell it what to do.
You don't want to tell it what not to do. Because if you tell it what not to do, you're putting the idea of what not to do in its context. And then when it's 150,000 tokens or 200,000 tokens or 400,000 tokens later, it's already got that idea somewhere in the back of its context. And it may decide, wait, I can do this. And even so, like if you've got a list of rules that you're following as a human, you've got to go, wait no, I can't do this, wait, I can't do wait, I can't do this, it's almost going to use up more thoughts in your brain. So I generally find that skills perform better and more reliably when you don't say don't do a thing, but you do say go and do this thing.
And you give it all the tools available to go and do the things you tell it to do.
I and once you have the skill written, do you then give the same skill to different models to test out the results of the outcome? Or do you have a skill per pipeline? Like you have skill and a model and it runs in in in the pipeline, you go to a second stage, use it, use a different skill, different model. How how do you how do you think through that? Or like are you trying to maximize coverage with multiple models with the same skill? Get up get a dump of all the findings and then prioritize from there?
It's it's a mix.
It's it's it's an absolute mix. So when we initially did the skill, that was we kind of did it with all Claude Opus type stuff, just 'cause that's the one I was most familiar with using with doing a lot of this work. and then when we did a lot of skill that's just on that, there wasn't really much diversification of models. But then we as we've built the harness and we've developed the harness, we've got loads of different models running loads of different things. So currently, I spin up four different models to hunt for bugs, I think. there's a bunch of models that look around where the bugs might lie, and there's like recon stages, there's like analyzing stages. It
Yeah. Yeah.
the it's got quite a lot more complicated than the actual blog post says now because it's been it's been constant improvements. and a lot of this stuff it's about using the strengths of different models and it's also about cost efficiency. Because you could use mythos for everything, but mythos, as I'm sure you know, is a very expensive model. and that's not necessarily the best way to do stuff. So it's working out what capabilities a model has to have in order to do a job and also breaking down the tasks into smaller things. and putting all the context in the right place to make it much more efficient and much cheaper. and the but then Yeah, sorry.
But then sorry, go ahead, finish the talk.
I was gonna say and and on that stuff, I o with this stuff I have done a lot of testing, a lot of fine tuning of switching models in, switching models out, working out which models do the best task better, and then analyzing the traces to work out why a model does a certain thing, why it has a certain behavior. Because one of the things that I think some people should do is if they're building any large scale agentic system at all, you should be keeping your traces. You should be looking at your transcripts, you should be working out, okay, but why did the model do that? Why did the model do this? And that's one of the best ways that you can self-improve on stuff because a lot of the learnings that we got with mythos, we were able to then
go, okay, Mythos does this and this and this and this. Let's pull those out and give those in bullet points to our hunting agents to make them more reliable, more effective and cheaper. We still use Mythos for a lot of our bug hunting, but it means that you can also use other models in there as well to get a similar effect for a lower price.
You need my
And you mention you mentioned that Mythos is expensive. Obviously you cannot use Mythos for everything. Do you then like use a strategy where you're using open source models for the wide discovery and then the most promising candidates get fed to Mythos for I guess verification? Like you just bring in Mythos for like the complex corner cases, but like the generic things you use the open source models.
I think so I th I think you I think you've kind of gone a little bit backwards. The way that I think about bugs is you're trying to find all of them and then you're validating them. But then the complicated bit, like because a really, really complicated bug is not normally one bug. It's normally multiple bugs stacked together. one bug on its own, one primitive, whatever. These are also things that we emit. We find things that are nearly bugs but not quite bugs. and then we have another stage, which is something that we do use mythos for, which is for combining these into an actual workable bug. so I think
Yeah. Yeah.
Yeah, yeah.
I don't think you because you can't necessarily look at a bit of code and go, this is simple, this is complicated, therefore I should use this or this. You can you can get heuristics into it, you can give things turn limits. So let's say you're using Kimmy, K two, K three, whatever. You can give it a turn limit. You can say, Okay, you get fifty turns to look at this, and if it's still not found anything, you can then upgrade, is one way to do stuff. It's not normally what I do. No normally when that happens I just note it down and log it, because then it feeds into continuous improvement of the whole harness. Because if you're doing that, you're then burning turns on a model when you could be doing a mug spence model later.
I see.
There's so many different trade offs you can do with all this stuff to optimize a model to work in the most effective way for your environment.
I guess my takeaway from what your your comment is cost is not a factor for CloudFare. You guys can just burn those tokens. You can just burn
Co no, no c cost is very much a factor. It's very, very much a factor. Like we're not using we are not using methods for ethic at all. like only on some things like so there's there's so many adjustable tuning knobs we can change, we can switch in our harness in order to essentially bump the cost up or bump the cost down. and it's all about maximizing the effectiveness with the lowest possible cost for that effectiveness. And then we sort of draw a line, how effective do you want this to be? It's not what percentage of bugs do you want find, it's what bugs was someone else not find.
Okay, cool, we don't necessarily care about those. Because there is always bugs in every bit of code. It doesn't matter how much code you've got or how well reviewed it is, you're always going to find bugs at any bit of code. So eliminating as many of them possible for as cheaply as possible is one of the priorities. And the other priority on much, much more important stuff is cool. How do we make sure there is not a single bug here? That's an impossible task, as I'm sure you're aware, but we can still strive for it.
Are you are you, you know, you've been at this almost for a year or more. Are you convinced that for a for a harness to work at your scale, you have to be model agnostic? You cannot be relying on a single model, you have to use a combination of different models. And more importantly, when you think about these models, are you always thinking about I don't know why I'm hearing my echo, hold on. I can't do anything about it. But are you when you think about
Mm-hmm.
There is an echo I'm hearing. I don't know why.
I don't think it's coming from me.
Yeah, unless it's my speak says. Yeah, that's interesting. It's picking up my headphones. It's picking up y my microphone is picking up you through my headphones.
Yeah.
Hello.
Interesting. Can you hello? Yeah, I can hear myself.
I don't know if I can change. Hello. Yeah, I can't hear myself. Did it was it doing that the whole time? No, just came up. That's really weird. I don't know, I'm afraid. I think some noise can noise cancellation is turned off or something automatically. Maybe it's I think one of the speaker directs.
No, it just came on.
Maybe
Maybe can you go to the speaker directly? Or how about now? No.
Mehul (17:42.492) Hello. Better? Yeah, better. Hello.
Ready? Yeah.
No, it's definitely on the speaker.
Hello? Yeah, that is much better. Yeah, that cancelled it. Okay. So this is good.
Okay. I mean I the problem is I can't really hear you if I do that.
How about now? This is good. I I don't hear my account.
Yeah, okay. How's that?
Okay, perfect. where was I? So yeah, the model yeah, so you know so you you are convinced that it the the the harness at your scale has to be model ex agnostic. The question I have is when you pick these models, are you always going to like the you know the top ten and picking the models from there or is there like a different cra criteria to pick these models? How do you evaluate which models are good enough to come into your harness?
Yeah, so for for us that's kind of easy because we've got all the evidence and all the traces and all the data from running our harness and finding tens of thousands of bugs. So I can literally drop in any one model into any of these once and ours and see how it performs. So from there I'm able to benchmark s the each individual model against what it can do for cybersecurity stuff and for the non cybersecurity parts of it. Because from my point of view as well, it's not about f finding the bugs is actually the easy bit, it's actually fixing them in an effective way that's the hard bit. and actually getting those fixes out and applied.
Mm.
which uses a very different model and very good very different set of skills. And you don't always need the best model for everything. and in terms of different providers, we use as many different providers as we can. we use open source models, we use the anthropic models, we use the open AI models, we use all of them because that's just the most effective way to get the job done. It's the most cut cost effective way, it's the most just the most generally effective way. And it also means that if any one provider has any issues or we lose access to certain models, we've got other ones we can flip over to. like to start with, we were
the harness was built pretty much for the anthropic ones with mythos, but that mythos was the primary one we used, but we very much quickly moved away from that. because that's just not a good way to build something that you want to be using long term.
I want to I want to put you on a tough spot. Is there is there a favorite model that you have, given all the work that you've done? is there s is there is there a
it changes like every time I use one. Every time a new model comes out, it's my favorite. I find new annoying things about them. And I think people in most workflows that do a lot of models you will just use a combination of them at all times. some like different orchestrator models, different models for reviewing, different models for writing code. They've all got really different strengths and weaknesses.
it. let's get under let's get under the hood with the with the pipeline itself. So the the the pipeline or the harness as I understand it it is a two stage process that is the VVS the vulnerability verification system or validation system and or the the disc the vulnerability discovery harness that's the stage one it is the whole the purpose of that stage is to just find as many things as possible and they use a set of model
And then you have a second stage, which is the vulnerability verification system. So all the things that have been found, all the things that have been found are then validated or verified by this system. So that you start with maybe 100,000 and then you get to maybe a thousand. I don't know. Let's dive deeper into it. So let's get into the discovery hardness. How does that work? what are some of the fundamental principles behind it?
Yeah. So the discovery harness is the bit that does all the finding of the vulnerabilities, right? That's the bit where we're doing the threat modeling, we're doing the scanning, we're doing all of this stuff. and from there, we then, you know, we we find the bugs. We then in that we also analyze them as well. We try and validate each individual bug. We do the entire process in there, including the validation we do in the actual discovery harness to start with. the vulnerability validation system is less of like it is very much agentic. We do have agentic workflows in there that do things.
like again another set of validation. but like that's more of a separation of concerns. Because if I'm building the harness, I could end up making changes to stuff that then looks really, really good on my local set of statistics when I'm building stuff and running traces and looking at stuff, it's like, wow, this is really good, it's producing loads of really, really high high quality bugs. But then if I start sending sending them through to another system, it's an entirely separate system, it's got an entirely different format, it works in a very different way. It means that A, we can validate that the quality of everything that goes in and ever reaches a team is always high.
and B, if I was to mess something up, we would notice because they're very as I said, they're very separate, they're split apart. It's like building separate software systems, like separation of concerns. One's job is to find the vulnerabilities. I aim for the vulnerabilities to be incredibly high quality coming out of the harness and fully validated, but sometimes stuff is missed. And the second thing is deduplication, because the vulnerability hardness itself, we use these frontier models, these side models, these mythos models, but they don't have access to any other systems at all. They can't access our geo, they can't access our wiki, they can't access our backlogs of bugs, they can't access any of this stuff.
So the actual one of the other things a validation system does, but yep, absolutely. our yeah, we're when we're using these advanced models, they don't have access to the outside world. That's quite dangerous, right? We this is this is something that from the start we've been very, very clear on. We don't want to ever have an instant where a model that we are controlling is then reaching out to the outside world and doing things we don't want it to do. So we just lock it away because it's
And that is intentional? Is that intentional? Is that intentional to not give
Or reaching out to reaching out to hugging face and compromising their systems, right?
yeah, we w wouldn't want that to happen. and that's mm hmm.
There is one question. There is one question I want to ask you. And this is something that I experienced is the models are just good at lying. In the sense they just lie flat out. in in in our in in in my example, we were trying to exploit a vulnerability. The model was failing to exploit the vulnerability, but the model knew my success criteria was a green checkbox at the end of the pipeline that I was able to successfully exploit. So it in our case, it just said it it faked up a fake server, web server.
and faked and then and and faked up and ran the exploit, had the response that I wanted, had the green check mark, me I was exploit able to exploit this vulnerability. I'm curious, like I'm curious, like when you're doing this vulnerability discovery, you know, stage, have you seen s have you seen situations where the the models are like injecting code and then creating false
vulnerabilities and then validating, hey, I was able to exp exploit this vulnerability. Has that happened for you as well?
Yep. Absolutely. That very much happens. That's one of the reasons that our validation stage is really, really vigorous. the validation stage I currently use, I think it's three different models I use. I have a Opa I think I'm using Opus five, Opus four point eight, and GPT five point five, all to validate that the actual bug itself is a real bug according to the threat model the threat model that we've defined.
And they will do a lot of, you know, they will do a lot of processing of the bug. They will read the evidence, they will look at the code. They will reconstruct the chains that the hunters have said they've found. because they do this, they lie, they come up with things that are not real. So I don't give them the code. I don't give them the actual code they've written that says that exploits the bug. They have no they're not operating in the same sandbox. They have a description, and they have line number references and all that sort of stuff, and that's it. And from there, they if they can't make the real bug happen again, we know it's not true, we know it's not a real bug.
And and d the do these models also have ability to make changes to the code or?
Yeah, well, so absolutely. the models are encouraged to cut up the code into separate sections. They spit up subsystems, they will build things, they will take parts of it and fit and use it as like a library and interact with certain literally functions of the code to see if a bug actually occurs, and then they can go from there and then you know apply it over a lar a larger bit of code. because if you've got a really big system, especially this is something that we run into at Cloudflow a lot, we're a really complicated system. our infrastructure is massive. We have
So many different connected interconnected systems. And actually emulating all of these in one place and running a real testing environment against all of this stuff is actually quite hard. so we we don't do the entire we don't spin up any of anything into this entirety of Cloudflare. we spin up like small separate systems of stuff. Like we get the models to build up like mini Cloudflares. We have full full flows of things that do a thing. We spin up containers in our testing container, we have systems that integrate together and talk to each other.
and the models will just all build this and they're you know driven to do all this stuff. like so like one of the things that I do is I build like I have like a mini architecture agent which gets all this really up early in the hunting pipeline because then all the hunters can interact with separate versions of stuff. But also they can customize it, they can spin it up. It's almost like a tree of branching environments where they can do stuff and build stuff in order to just experiment with the code.
And one you know, one thing that you haven't touched upon, I don't know if it is intentional or not, or maybe you don't you don't run into these situations, is my sense is given your code base, you should be running into context exhaustion windows. Like you already have a million token context window. Your code bases are definitely much bigger than that.
Then how do you like, you know, what kind of challenges do you run into? Do you run into these? I mean, at least we run into this context exhaustion windows where, you know, the the context keeps on grow growing and growing at some point, the real work that you want to do, there is no context left, right? And so then it starts to hollow hallucinate. Have you run into those challenges at all?
Absolutely, yeah. yes and no. So when we were doing our initial stuff with AI models, we thought, okay, context windows are problem. And we also found out pretty quickly that models perform much better towards the start of their context windows. Because that's what you know, they just the focus is much, much tighter, they can do everything much more precisely. so with this stuff, we realized that that's the real secret to a harness. It's managing the context. So for example, if I'm hunting for a bug, you build your threat model.
Yeah. Yes.
Which you know takes things out of the context window. we then analyse it to work out where different parts of the threat model apply to different parts of code. And then we have a hunter task that does that and its entire job is to look at this subsystem from this angle to find these bugs. and then this is omitted to a separate system. and then the validation is entirely separate again. And all the validation gets is the bug and the actual answer is is this real? And these are much, much smaller, simpler tasks, which means that you don't really run into the val these issues at all.
And how and how long how long do these how l how long do these systems run? Like you know, do you run into s systems where you have crashes, you lose the context, you lose the memory, and then you have to start over again? You know, do you have those kinds of situations as well or is that not
Absolutely. Yeah. this this happens all the time because when you're dealing with these systems, they're sometimes a bit fragile, you know. Maybe one of your libraries that your hands is built on updates and something slightly changes. But or you know, you run into a weird edge case where your MCP tools are timing out or whatever, because you've got something that's taking over fifteen minutes to respond. This is all things that can happen, this does happen, which is why we also every single transcript is recorded, every single turn is recorded, every emission is recorded, the state of everything when it starts is recorded and also what it does exactly in order. Because then we can just restart from wherever we are and wherever we've got to go.
And this is something that you improvised as you went along. So you're running into these crashes, you realized you're shutting over, wasting all burning all these tokens, r you know, spinning spinning up all the i lim API limits and whatnot, and then
Yep. It's
I mean at the start it wasn't necessarily about burning the tokens. It was about just getting stuff done efficiently because some of this stuff just takes a long time to run. Like and it's especially when you get into multi threading and burning things at much like a larger scale and all the parallelism and stuff. having a way to resume from anything, like you know, if I've SSH to something and burning it in an SH session, you know, if my SSH connection drops, I lose it. That's really annoying. Like it you know, token efficiency is really, really important too. But it was also more for me as a developer, I'm really annoyed when my sh test session dies. I'm not using T mux 'cause I forgot for this one and everything's gone.
Like I run it in Docker anyway now, but to start with, when I was developing and testing, that was one of the drivers for actually doing stuff in a database. And since then it's proven like so invaluable to store every state of everything. Because it means that I can literally replay things with the exact same model. If something breaks, I can resume it, I can switch one thing and see if it acts differently. and I can test things on like a cell by cell basis to see how it all interacts. And this is all based on the same developer frustration to start with of no, my SH connection died.
I'm curious, and you you don't have to answer this question, but I'm curious, are you on the Max account or are you on the API account? Like, you know, when you're going through all these processes, are you just
So yeah, at at Cloudflare we have API stuff.
so you have to use the API account. wow, these those are those are expensive runs. Dan, let's talk about let's talk about the success of this initiative from your point of view. So when you started, how many how many what was the scope of discovery of vulnerabilities from your point of view? Just, you know, including the false positives. And then going through this multi-stage process, how many real bugs were you able to find in your
in your in a variety of code bases. And then more importantly, how many of them got fixed in the in the in the time trip. So let's talk about the success for the entire initiative. Where did you start? When you started, where were you and now where are you now?
so where we started, like so originally when we started building vulnerability councils, that was the security organization doing a security organization thing. I was like, okay, I'm on the red team. I want to find a way that I can hack Cloudflare more effectively, which was how my foray into it started. and that solution was obviously gonna be LMs. And then moving on to through mythos, through building the harnesses, through scaling up the program, the success it was generally just how many things can we scan at the time that we have. because when we got access to the model, we didn't know how long we'd have the model for.
We don't know any of stuff. So we're like, okay, cool, let's let's get going as quickly as we as l as quickly as we can. We've got all these learnings we've got before. What do we need to do to make it work, make it happen, make it happen effectively? so we started there and we we started going. We hit the ground running quite quickly. I think we were able to get scans running at massive scale quite fast, which is really, really awesome. and then in terms of like actual scale of bugs we found, it's I think it's in our blog post we had in terms of harness emissions, which is not even harness emissions, that's like things that the harness found. I think there's like twenty thousand, twenty-one thousand.
Mm.
And then through the deduplication, validation, et cetera, I think that bore down to about 7,000 bugs. in terms of these that are fixed, I think we've fixed. I think we're probably over half. Like a lot of the ones that are left are lower lower severity, or for whatever reason, they're not necessarily applicable to the threat model that we're acting with. So one of the things I've been working recently is trying to figure out what the actual threat model of a repository out of a product is.
Yeah.
Because you can have bugs that look real from depending on where you're standing. And this is, I think, the biggest problem when you're doing LLM based vulnerability scans. You've got to know exactly where you're standing to work out exactly what's going on. and if you're not doing this, you're not gonna know whether a bug is real or not. because you know, the LLM does not have the knowledge of all the developers and all the teams. I did not have the knowledge of all the developers and all the teams. So what I'm doing now is I'm reaching out to them all and going, okay, what bugs are real or not? This is what I'm gonna go, hey, look, this is a bug, this is not a bug. And they got and then they come back to me and we talk to each other and we have an agreement. We go, okay, cool.
Does this work? Does this not work? Are these the things you want to see? Are these the things you don't want to see? And then we can link all this back through the entire harness, through the hunting for bugs, through the validation stages, even through the way that we fix bugs and the approach that we take to actually fixing the bugs itself. It's all very much done in tandem with teams, which is it makes the process take a bit longer, it's a bit slower. We've got all the data now, we've got a lot of the bugs fixed, but it's also a a process going forward because we want to minimize the amount of bugs that exist at Cloudflare.
But is there any level of human validation in this process or is it completely automated? The discovery, the duplication, all of that?
Yeah, so I validate a percentage of the bugs. I go to them and I understand exactly what they are and how they work and all this sort of stuff. Just to I like I I mean like twenty thousand or whatever, I can't do that many of them. I probably do a f a couple of thousand is probably what I've looked at and actually looked at. So which most of the critical high severities I've definitely looked at. ones that we count exploitable now, which is another sort of metric that we use that based on the config and what could happen, is this a bug that someone could exploit now in production? They've all been fixed.
These are the ones that I've all looked at and validated myself. and other people on my team have also looked at a lot of the same one to get the same get opinions that, you know, mesh on all these to try and get some sort of consensus.
So when you when you say when you say when you say validated, you have spun up the container, spun up the vulnerable service, exploited, run the run the exploit, verified you were able to maybe do do a command injection or do some validate the entire exploit works. Is that what you're calling a validation or just you know manual review of the just looking at the code and say,
Ma manual review. because the nature of a lot of these bugs, especially when you're talking about like cross-tenant isolation or memory corruption or this sort of thing, you can't test these live against production. That's just not safe. the risk is too high. And we have test environments, so where possible we test them in the testing environments. But no testing environment when you're talking about some vulnerabilities is ever good enough. Like you're always gonna have some different config options or some like even the level of use, some it's possible that you can have a mem memory corruption vulnerability that only exists.
Yeah, yeah.
If a process has been running for a certain amount of time and had a certain amount of activity, you're not going to see these in a testing environment. Which is one of the issues that we've had when validating bugs at Cloudflare scale. Like we do use our testing environment, we try it as much as we can, but it's not always possible. And also it's not always necessarily possible to tell that your exbox has succeeded from where you're standing as someone that's validating stuff. Because if you're causing a process crash on a remote server, you've then got to track down the exact correct server that you've hit and the exact crash that you've caused.
I mean, ideally we don't actually cause any crashes on any cloud flat production servers, but even in testing it's a similar thing.
Yeah. And what's like what's the end state then? Like is the end state to keep these harnesses running, keep finding f keep finding, discovering new vulnerabilities, validating them, and then autonomously remediating? Is that your end state that you're looking for? I guess my question is what is the what is the limiting factor? Because you are essentially moving at the speed of compute.
Whatever compute you have, you're you're not blocked by human talent or I mean, maybe some in to some extent extent your validation is is a is a limiting factor, like how fast can Dan validate these findings? But if you were to think of an end state, where do you see this entire hardness going in terms of securing the enterprise?
So I think one of the things that's really, really interesting is working out how to do vulnerability scanning at scale going forwards, which is something that we've been looking at a lot recently. So for example, you you clearly can't do a full vulnerability scan on every repository every time someone commits. That's not going to happen. That's really expensive. That's really slow. So the way that it's probab it might look going forwards is still in planning. This could all change. I don't know. I don't think we have the answer to it. I don't think anyone has the answer to it. we're looking at things like just understanding the shape of the change and then spinning up specific hunting tasks in our harness based on that exact shape of change.
to see whether any bugs do appear there. and the other thing we're looking at is actually like c take taken from kind of the threat model approach. if you've got a pull request that dramatically changes the threat model, you've probably introduced a bug. Unless of course you intended to add a new fen no new functionality, new endpoint or new some new new some sort of new attack service. If you added something like that, you know, you might have accidentally changed something. And this is not necessarily the stuff that static analysis or even model LM analysis are going to pick up now. You want to be able to con that's so that's something that we're looking at now is comparing
what the threat model of a repository looks like now and then what it looks like after the next change has been merged. And if they change, that could be a problem. Maybe we need to look deeper. because that's almost that adding like a a more knowledge based semantic layer on top of your general security code reviews.
And the remediations that you do, are they are they recommended by the model, or is this something that you personally provide guidance on the remediation? So you said there were seven thousand five hundred validated findings, three thousand five hundred have been fixed.
are those fixes also created by the the hardness in terms of what you should do? And and you mentioned that all you know all these models are cut off from all the other tools. Like you know, the you I I don't think these models can create pull requests or create tickets or whatnot. So who's then who is doing that part of the is that part of the also automated where the validation goes back Dan has approved it. Once you approved it, does it create a ticket? Or do you have to manually create it?
So that's that's where our vulnerability validation system comes in. So one of the things we realized quite quickly is a solution like Jira, which is what we use internally, can't keep up with this stuff. when you've got to have systems that are talking to agentix systems and doing stuff or even like do duplication of things at scale, you don't want to create a ticket for every single vulnerability that might not be a vulnerability. So you've got to store them somewhere. so then this developed into us building our own the vulnerability validation system. and this system essentially is somewhere that all the volunteers sit, where they live, teams can look at them.
it contains a fix for every single vulnerability. the fixes that we started doing originally were not you know the best quality that we realized quite quickly there's a quality issue with some of the fixes that we were giving out. So we've tried to go back and change those and update those to find a better way to get all these fixes to teams because that's the hard bit is you know you've got the bug, now you've got to fix the bug. and if you're giving a team like lots and lots and lots of individual bugs, this doesn't help because they've still got to deploy them in a safe way. So we've got to then work out mitigations, we've got to work out you know the priority of getting all these things out, and that's probably the limiting factor.
And do you envision a world where this becomes a self serving system where you are not like babysitting this entire or you're not see babysitting the entire organization and developers are like self serving, they know hey, this is my code base, this is these are the things dance hardness found. Here is my recommendation, recommended remediation, review it and then deploy it so that you know you are you are essentially removing yourself as a bottleneck from the entire process.
Yep, that's the that's the end goal. That's where I wanna be. I wanna go back to to doing red teaming work. I don't want to be doing vulnerability harnesses and finding all this stuff. so I've been trying to automate away this entire process. it's slow to do because automating stuff in a way that works for other people is hard. because there's no point, you know, coming up with solutions that don't work for other teams and other people. That's just never gonna go. That's never gonna happen. People are gonna if these if the bugs are low quality, people are gonna be annoyed by them. So we really wanna put a lot a lot of effort into making sure that the bugs are higher quality. well
Hi calling.
And the fixes are high quality and all this sort stuff, because that's the only way that we can actually get things working and that it's actually sustainable going forwards. it's not been perfect. We've made mistakes, we've given bad bugs to teams, this has happened. But we want to go and try and fix that going forwards. It's all about getting feedback, looking at you know, so for example, on these tickets that we've made, teams have left comments like, this this isn't applicable because X, Y, or Z. So it's going back and looking at all of these things and working out, okay, why did we think it was valid when it wasn't valid? Which then involves things like going back and actually looking at the traces, like our our s our validation traces, our judgment traces.
The actual hunting traces. Why did we think this was a bug when it's not a bug? Is it really a bug? Has the team mis misunderstood it? So it's all feedback loops on feedback loops on feedback loops. It's not it's never gonna be just one and done.
And does the and does the feedback look go go back to the hardness where you know when the team said this was not a good bug and then it goes to the next time both and right now. Got it. and what's the
Right right now it does not, but it will do. currently that's me running separate tasks and then running things and copying files around and whatever.
If one way one way to lose all the credibility is bring bring up a bunch of false positives to the team and tell them to fix it, right? So my question to you is what's the confident confidence level from the teams in the harness in terms of when the bugs are reported by the harness? They know this is coming from the harness. if you were to make a guess, what is the confidence level? Is it like at seventy, eighty percent right now where they are in terms of like believing as soon as the bug reports come out? Or
We're a pretty good place. I think we're probably between 70 to 80%. It's not perfect. I would love it to be better. But it's also a process. Like it depends on what the product is, how the product interacts, where it sits at Cloudflare, and also the different threat model of each team. Some things we've had much, much higher success with. There's certain products in certain areas where we found bugs and they've been ridiculously reliable. They've been really, really good, and they've been very well received by teams. because these are hard to find bugs and these are hard to fix bugs. And we give them like you know a heads up and we've got involved in the process early, which is really, really great. Some other teams we've had
less good communications with is because as these things scale, we started with two or three teams. It's more of like a white level approach, trying to work out exactly what what works, what doesn't work, and then scaling up for everyone else. and some of the times when we've scaled up everyone else, there's been a repository or a product that our harness hasn't properly understood. There's been some assumption that we've made or an LLM has made, which meant that the bogs at the the bogs, the bugs in certain places are not quite as high quality as we would want them to be, which is a problem for us, right? And we don't know about this because the you know
When you're scanning at scale, you can't look at every single individual individual bug, you can't validate every single assumption. We've tried to do that as much as we can, but it's a really, really hard problem to solve.
Then you have been
At the the bleeding edge of this kind of research. There are very few companies and enterprises that I've seen that do this kind of research. I wanted to get your thoughts on, you know, if if an enterprise is looking to do this kind of work, like where should how should they get how should they get started? What should they do? how would you recommend they get started with this process? Because my senses as more and more vulnerabilities are discovered with these AI models, with now even like the Kimi K3, we were just trying it out yesterday.
That model is pretty good. That is model is pretty good. So open source models are getting closer to the frontier models. So my sense is many organizations, many enterprises, especially the Fortune hundred, 1000, let's say, they would need to have these kinds of processes, tools in place. How would you recommend they get this get started? you obviously have open source or published your security skill on GitHub. What else can they do to get started?
I think the first thing that teams have got to work out is, okay, let's assume already that we've got a vulnerability harness that works and catches loads of bugs and the bugs are really reliable. But how do we actually fix them? Because that's the really, really hard bit. H can we even fix out bugs at scale? Can we roll out patches? Are we behind a CDN like Cloudflare that can provide someone like a WAF product? Which means that if we know that bugs exist in our products, we can stop them with a config change rather than having to roll out a new product. because this stuff is really, really hard. And that's I think
one of the hardest things for people to work out because finding bugs is really, really easy. You can open any coding agent with any model and say find bugs in my code, and it'll find bugs in your code. Some of them will be real. Some of them will be rubbish. Some of them will be security related. so until you've got a way of handling bugs at scale, or you've got an idea about what the code quality of your code is, just in general, I think you're gonna have a really, really hard time. So I think in order of things to do, work out if your code quality is pretty good. If if you're not using LMs at scale now and you're a you know you're a business that you know with a lot of surface area.
it's probably something you should look at doing, even with open source models, with chief models, just to get an idea of what your security looks like. And also then, if you realize that, no, yeah, there's things that gotta change, you've got to work out ways that you can manage these bugs, that you can roll out these fixes, that you can even store the bugs. Because if you find a bunch of things, let's say you find a thousand bugs and then they're sitting on a database somewhere, you've then got a database somewhere which tells someone how to hack your company. That's not something you want to have lying around in any way that's not access controlled, not logged, and also not having people actively working on fixes. So
Really, the first step is just work out is this something you want to do? You probably should do it, but if you have no way of doing it, it's not worth finding the bugs unless you can fix them.
My recommendation would be hire somebody like Dan. Start with start with somebody hi hire somebody like Dan and then I guess start with pointing some of the frontier models to your code bases. I mean I I don't I don't think everyone needs to build I don't think everyone needs to build a hardness, but b maybe you start there and you point point your code bases to some of the l some of the
frontier models or even the frontier open source models, get started and then at some point in time you'll have to build. I think I've I mean the more I l the more I listen to you and then the the re the research I read, I feel like this is something that many organizations will have to build in-house. In some sub just getting a sense of because most of these one thing that I've seen is most most of these static analysis tool analysis tools, dynamic
Dash tools, they're not finding the bugs, kind of bugs that the open source the the frontier models are able to find. I mean, you you probably had all these tools before the hardness was built, and your your these models still found all these bugs. They were able to chain these vulnerabilities and bugs and exploit them and so on and so forth. So the you have to be, if you're essentially if you're a sp especially if you're a f you know Fortune 500 or Fortune 1000 company, you have to invest in this kind of technology and building these harnesses to get ahead of
ahead of the defense before the offensive tool t tools and teams, you know, you know, come come in and and break into your
Absolutely.
Absolutely. Yeah. I think the main difference between like SAST and DAST and using LN driven vulnerability stuff all your SAS and your DAS. They will essentially find they will find your small primitives. They will identify your simple buffer overflows that if you do you'll get a buffer overflow here. That's something you definitely should be facing and definitely should be doing. But it's not going to find the flaws of your business logic. It's not going to find the places where two systems interact and there's one assumption, three systems downstream, that affects one assumption, four systems upstream.
You're not going to find that with static analysis. The only way you can do that realistically is with LLM-based scanning and building a harness that can work out how all of these systems interact. And then doing something that constrains the context in a certain way that you can literally map out every single path. If you don't do that, you're going to probably have a really bad time trying to find these really complex vulnerabilities. And these are the ones that might bite you because the actual input could be really, really simple. You could just change something really subtle in one input, in one packet, in one reb request, in one whatever.
And it can have really serious downstream consequences just because nothing's ever expecting it to get that far. And this is the stuff that you're going to really struggle to find with sustain dust. Or you may even write it off as a false positive, because if it's three systems downstream or ten systems downstream, you're probably going to assume that something upstream fixes it. But does it? Does it actually validate everything?
Dan, last question for the last question for the day. What's the next big project you're working on? When when is the blog update coming for all the work that you've done since the last blog you wrote?
so the what what we're trying to do now is just do it what do it at bigger scale, do it cheaper, get more fixes into more Talf Lab products more effectively, and then productize the entire thing. Because what we've got what we've got right now is something I can run scans, I can run scans one by one. I think the scans are pretty good. Some teams may agree, some teams may disagree. So firstly, I want to get all the teams to agree the scans are really, really good. And that's working out what changes we can make to our harnesses to make the output better and more reliable for these teams. And then also building stuff so it all works in CI. building stuff so it works with our appsec pipeline.
There's a I'm doing a lot of work right now with our AppSec teams in order to automate a lot of this stuff to work out how we can fix bugs at scale. And this isn't just AI generated bugs. This is bugs that come in from our HackerWagon bug batch program. This is bugs that we're finding internally with internal sources. Because with AI, all bugs are coming in much, much faster. And if you're not scaling up the entire system, you're gonna have a really, really bad time. So that's what we're doing right now. We're trying to scale up everything. It's not just the bugs that I've d built with this program, it's all bugs from any source. How do we fix them? How do we make sure they're not a problem in the future?
And then how do we actually validate they've been fixed? Because validating a bug has been fixed exhaustively is also a really hard problem. And how do you validate each fix hasn't introduced another bug? That's also a really hard problem. So coming up with sensible solutions to all of this stuff, it's just a really, really open question. Because one code change can change something else. so validating that you're not breaking other things or that your assumptions don't go downhill or whatever else is is the current hard problem.
I
I I should have asked you this question earlier in the interview, but it kind it just it just came to my mind. I mean it's very you know, it's very easy to find or at least relatively easy to find vulnerabilities in in software that you a have access to the code base, either it is open source or your first party source code. But then how do you deal with the enterprise applications that are deployed inside your organization and finding vulnerabilities in them? Do you have a path to make sure your enterprise software deployments are also secure, or is this like a phase two or a phase three?
So at Cloudflare, we're really lucky. Most things at Cloudflare we build on Cloudflare. We have we've run them ourselves or they're open source versions or whatever, which means we can scan these and we can patch them ourselves and we can submit patched up scream, which is something we do. There are some closed source things that we do and we do and we have scanned. and these are things that we're working on because these require relationships with your vendors to get them to agree that yes, you're allowed to scan this, you know, whether it's a binary, whether it's a service. because some people are not necessarily happy with you finding bugs in their system that are using AI. That could be against all sorts of terms of service. And this is
This is a complicated thing to manage. I'm managing this in an effective way that means that you can make sure that A, you can keep using this product that you rely on, B, you can fix any bugs in this product you rely on. And C, you're keeping your vendor happy because you know you're gonna have some contract which actually talks about what you can and can't do. All of this stuff has to be balanced against each other. It's something that we're working on, getting through. And we have found bugs in some of our upstream dependencies and some of our closed source things, but it's all a process. But these LM models, like they're not a smoking gun. They're not something that you can point at any black box.
And it'll work out entirely everything about it and be able to find the zero day and everything. That's not what they are. They're pretty good at finding stuff when they can figure out how it works. They can probe a lot. They can do things a lot quicker than human can. But they can't just, you know, you can't just pop open any system like a can. That's not what happens. That's not how hacking works. In the future, maybe we'll find some crazy new paradigm of breaking systems. Maybe that'll be AI enabled. Maybe it won't be. But that's not where we are right now.
Dan, that's that's a good point to end this interview. Thank you for coming on Noise to Signal Podcast. This was a fascinating interview. I really learnt a lot, learning you know, through your processes. So thank you for coming on.
Thank you very much. It's been really good to be here.