Vulnerability Research Is Not Cooked w/ Thomas Dullien

Thomas, it's such an honor to have you on the Noise to Signal podcast, or as the world knows you as Halverflake. You know, the thing that I find most interesting about you is that you are a trained mathematician, you're a trained computer scientist, but in this odd way, you find yourself in the world of vulnerability research, reverse engineering, cybersecurity, and so on and so forth. So I want to

To first of all, you know, thank you for coming on the on the podcast. I don't know if you know this, this is the first episode of season two, so super honored to have you on this episode. I wanted to start the interview in terms of your origins into this space. you're popularly known as Halwerflake. I'm curious where how did Thomas become Halwerflake?

Yeah, so I mean when I started doing computer stuff, this was the very early days when there were still BBS and like computers you would dial into from your machine that you could then leave messages on for others and once a day that BBS might or might not connect to the internet to send newsproof contents and so forth.

it was there that I chose my first nickname. and then a little bit later on I decided I want to start doing copy protection removal. And in order to learn copy protection removal cracking, I had to join RRC. Now at this point in time the internet has arrived. and you need to pick a nickname. And I had been on vacation about a year earlier and

And I had long hair, was a bit rowdy, I've always been not the slimmest of people. And the other kids had called me Halvar because there's a cartoon character, like a small Viking boy called Vicky from a village called Flake, and his dad is a rather fat, rowdy Viking by the name of Halvar. And I decided that Halvar is a good Nick as Annie.

and used that as a nickname. And then it stuck. I used that nickname while I was in the cracking scene doing covered protection removal.

But then what happened to the flake? How did the flake come along? So the halber made

I mean it was i i w it was the like the village that Vicky comes from is called Flake F L A K E Flake. and oftentimes when you entered your name somewhere you had to have a last name, so I used Flake for it. this all happened while I was in the US with a high school exchange in Minnesota of all places in a host family that had Norwegian ancestry, where the host grandfather was still mainly speaking Norwegian. So

My English had acquired a Norwegian accent on top of it. which made the nickname, which sounds vaguely Norwegian, made the nickname very plausible, right? Blonde person with a Norwegian accent, slightly Norwegian sounding name.

Interesting.

I'm curious, do you have Viking and you know ancestry? Like are you like a descendant of the Vikings in some sort or is it does it is it not relevant?

Thomas Dullien (03:25.563) I mean, it's hard to tell, right? there there was a lot of I mean i if you realize what what the Vikings did, there was a lot of looting and pillaging at the time. my like both my father's and my mother's family come from the Baltic Sea coast, which is now approximately Lithuania, or a Russian enclave in Lithuania called Kaliningrad, which was

ethnic German at the time, but clearly it's close to the coast and it's close to where all the Vikings used to raid. So who knows.

Yeah.

It's possible. It is plausible that there is some Vik Viking ancestry in Harbor or Harbor Fey. I'm curious, like, you know, you you you you I assume you were a teenager at this point. You know, you're getting just now you're just getting on the cracking scene. What was the motivation for you to get into, you know, the kind of work like copy production or breaking copy production, doing that kind of work? look very y you're probably very young, and not have you know, not have a good sense of the world, what's going on in the world and whatnot.

Yeah.

I'm curious what was driving you at that point?

So it's important to know I've got a brother that is six years older than me. and he was always a bit of the nerd in the family. When he was eight or nine, he bought himself like a a home computer assembly kit, like a TI one, I think. I don't I don't recall the exact model, but one of these computers with like sixteen kilobytes of RAM and a data set, like a cassette tape that you can save data on. And my dad

who had not had a very successful career up till shortly after I was born, who was a trained lawyer of all things, saw a newspaper ad from a company that decided that hiring people that don't know how to program, but they know something about business and accounting, and teaching them how to program is easier than to teach a programmer about business and accounting. So he called that number and in his forties, he started programming.

So there was a computer in the house. My brother was very smitten with computers. and I was six years younger, and the niche of being good at computers was already occupied by my brother. So I just played games. like I would try to write down the listings from the happy computer magazine. Like there was a German print magazine that would

You would buy and would have basic listings for computer games. So you could just type them off and then hopefully run them. provided you had made no mistake in typing them off. So yeah, I I only played games and then you run into copy protections naturally. And my brother like a might might have been 14-15, so he knew how to remove them. And I might have been eight or nine. and I asked him, Hey, bigger brother, how how do I remove?

Copy protection. And he said, it's really easy. You just find an inverted jump. And I didn't know what a jump was or anything like it. so for the next couple years, the only thing I knew about copy protections was I need to learn assembly and I need to find an inverted jump. So I mostly focused on creative pursuits like drawing, 3D animation, whatever you could do at the time.

And I ran into copy protection issues all the time because all the shareware software had limits and you could never do much. and then I went to a high school exchange in the US, and this was rural Minnesota, like an hour in south of Fargo, in a in a tiny village. And the big event on the weekends was you would drive up to Fargo and you would go to a Barnes and Nobles bookstore. And in that Barnes Nobles bookstore, they had a book that is called Masterclass Assembly Language.

by a company called RoxPress at the time, and it was written by a bunch of Russians that actually did reverse engineering as well. It was a very good book. Like it walked you through step by step through setting up a DPMI extender, moving the CPU into protected mode, essentially writing a small mini-operating system. And it had an explicit chapter on reverse engineering, even. So I went through that book and I think that book was the best investment I've ever ever made.

Mm.

and a couple of weeks later I managed to crack my first copy protection, which was a terrifying butcher job inverting jumps all over the place until it worked, but it worked.

And what software what software was that?

I was a a a Quake 3D model editor because I had made Quake animations like a hopping rabbit and a few other things. And because I was really into animation at the time. my my desired career was in character animation, either 2D or or 3D. So yeah, I cracked that software and then through cracking that software I found the cracking groups online.

there was a an IRC channel at the time called Cracking for Newbies, which was very inviting and in some sense became like there's surprisingly many people in security nowadays that went through cracking for newbies at some point. I mean Rolf Rolls works for hex-rays these days, as I think the chief scientist. and I met Rolf for the first time in Cracking for Newbies. So it was a

an RC channel where people congregated to learn cracking. And then once they were any good at cracking, there were the WAS release groups. So you would join a either a cracking group or a WAS group and start cracking for them. And there was a sort of competition amongst them about who could crack the most and the quickest. there was a a at some point a group called Core which automated a lot of that stuff.

and they had like a a few Chinese crackers, one of them was called Xyrex, which just had a phenomenal output in terms of hundreds of key generators a month. Key generators were the the upper tier of what you could do reputation-wise, because you wouldn't modify the program, you would figure out how the licensing key scheme works, break it and invert it, and then we just generate valid license keys. yeah.

So so that makes a lot of sense that you you know, you're a teenager, you're playing games, the games are probably restricted, you use copy protection to unlock these games so you can continue to play the games, do the animation. The thing that is not clear to me is you are known as a legendary shell coder. Like you're known f your shellcodes are legendary. The thing it's not clear to me is how did that version of Thomas

become this legendary hacker and reverse engineer and creator of these like you know super small shellcodes. Like what was that journey like? How did that happen?

So it's important to remember that a lot of the names that you see today in security come from the cracking scene and pretty much all crossed over at about the same time. Stefan Esser, cracking scene alumni, Solar Designer, cracking scene alumni. Anders Voch works like a distinguished or engineer, like a highest level of engineering at Intel, former cracking scene. The cracking scene people were very comfortable writing assembly.

Mm.

The hacking scene people were usually not so comfortable writing assembly. So what happened is that hacking scene people reached out to cracking scene people to help them with shellcode writing. And then the the cracking scene people were like, this is easy. Wow, let's do it. like even funny enough, the Grug had his origins in the same same field, like in the same general youth culture. So

there was literally a moment in time where several different hacking groups had an influx of the same group of teenagers essentially. Like the there were was a hacking group called TEZO, there was a hacking group called ADM, there were a bunch of different hacking groups, and what happened is essentially a large group of the cracking scene split off from the cracking scene and started doing hacking.

And if you're very used to writing a lot of stuff in assembly and reading assembly all day, then shellcoding is actually very easy. So that's how even as a mediocre assembly person from the cracking scene, you would get a reputation as being a wizard at assembly in the hacking scene just because the bar was at a very different level.

Solo, yeah. The bar bar of solo in the hacking scene, an average cracking scene guy would become like the elite.

Kinda kinda like that. Yeah, like the at least the the low level assembly stuff was was pretty easy.

And Thomas, one of your greatest contributions, or one of the greatest contributions that you've done, is your creation of Bin Def. that was when it came out, it was a revolutionary, it was a revolutionary product in the sense that you could easily figure out where where the changes were made by the vendors in terms of the patches that were they were pushing. My sense, my remember the thing I the way I remember this is.

Well Microsoft would push out these patches and not not explain what they fixed. And it was very difficult. It was very difficult to figure out what was what was patched. And you had and if you and if you looked at the assembly code or if you looked it up in IDA Pro, it was very cumbersome, if I remember it correctly. And then bin diff comes along and it completely revolutionized the game in terms of finding what changes were made and so on. So I'm curious

Yeah. Yeah.

What was your thinking behind writing Bindiff? Was it like a side project that that morph into something big?

Yeah, I mean I had just begun studying mathematics. or was perhaps a year. So I graduated from high school in two thousand, had to do my mentor in military replacement service, so I started studying in late two thousand one. and I think the the relevant bug disclosure that triggered me to work on Bindiff was at some point in two thousand two. I don't remember the specifics too well.

But in essence, a Polish hacking group called LSD had found a whole bunch of issues in Microsoft's Dom implementation, which allowed essentially remote compromise to any Windows machine on the internet at the time because that stuff was open to the world. and Microsoft wouldn't provide any details about what they changed. that was very frustrating for me because I wanted to know the bug, clearly, right? so another complicating factor was that Microsoft's

release binaries were shipped with profiling guided optimization. So they had the hot code paths all at the beginning of the code, the of the executable. And then all the error parts had been moved to the end of the executable to improve caching and memory footprint and so forth. But when they issued patches, they didn't run them through the same profiling guided optimizer. So the binary layout would be dramatically different because now all the functions in the patched version would be in one place. Whereas in the original release version there would be

What?

split into the hot path and the non-hot path. Which made I mean byte-based comparison was infeasible by the rearrangement, but also by different compiler optimization settings. So we needed some way of doing the comparison that was resilient against compiler optimization changes and this profiling guided optimization.

And what do you do when you're you're studying mathematics and they try to beat into your head to always look at the structure of everything? You're like, well, there's a graph structure here. Let's see what we can do with the graph structure. And yeah, that's where the idea came from. And then a prototype was written. the idea using the graphs was in the air at the time. There was another researcher at BindView that had a similar idea and posted a similar

prototype at some point. And turns out that Bindiv solved a real problem for many people. And then people wanted to buy the software and I didn't mind selling it, but they minded buying it from a student. like they wanted source code escrow and and all sorts of things. I was like, hey, if I had a legal entity, nobody would ask me these questions. So I

during my undergrad essentially went and set up a company in Germany so I could sell the the tool. And like I I was when when I went to like left left home and went to college, I had decided that I like for a variety of personal reasons that I would prefer not to take any money from my parents just to to earn my money by myself. famil f families are complicated at the best of times.

so selling Bindif, like I would finance my studies by doing trainings, like essentially reverse engineering trainings. and selling Bindiff allowed me to do fewer of these trainings and spend more time actually at home doing some studies and yeah.

I'm curious who who are your customers? Like who was interested in this in those days? Like who who cared about this much that they would pay money for it? Because this was like very early. This is like you know early early very early stages of the internet, right? It's not like it's not like, you know, what it is today.

Yeah, well it's this is t two thousand three, two thousand four time frame. So we're we're already well into the internet. and

No, but I mean I the internet was more around the commerce, not like the hacking and you know, that's what I'm trying to get at. Like who was so interested in this reverse the the hacking scene and you know, who were the types of people you ran into to who wanted to buy in dev?

Well

The there there were were lots like it it it fell into different buckets. There were lots of military and intelligence people in the English speaking world that were quite interested. there were a few intelligence services and military folks in the Scandinavian realm that were interested. there were

And the motivation and the motivation was to do the kind of work that you were doing is to figure out where the patches were, c figure out the exploits for them was like, Hey, you know, was that the motivation for them or something else?

I'm in.

I mean the the tool itself turned out to be useful in a variety of situations. you had the the clear offensive use where you could just try to patch gap, like take the patch, turn it around into an exploit quickly, use it before it gets patched. So there were some customers that were doing that. there were other customers, like there's a company called there used to be a company called Determina that would do

Mm-hmm.

Mm. Yeah.

Essentially intrusion prevention by understanding the patches and then writing detections for the attack and the patches. So some of our customers were on the defensive side trying to write detection for the bugs. Some of the customers were interested in porting information from existing disassemblies to a new version of the same software. So if you've already disassembled one thing and you want to not lose all your work to that point when the next version comes out, Bindiff was very helpful with that.

Some people wanted to compare malware variants. so when you have a complicated piece of malware, you want to compare them. Some people were car tuners, like we we had multiple car chip tuning customers, which surprised me quite a bit. in essence we had a very, very random assortment of very interesting customers that very often couldn't talk to us.

Mehul (19:29.323) I'm I'm curious, you know, given the kind of work that you were doing and the software was enabling, did you see any pushback from the government agencies, hey, you should not sell it to these people or you should not sell it to those people because they could use this you know, like the same narrative that we see it with today with Fable and all these, you know, latest technologies that are out there. So do you did you sense that when you were doing when you're trying to sell been diff at all?

Yeah, I mean Bindif, I mean, first of all, Microsoft was very much not amused, right? And I I learned later that there were discussions inside Microsoft at the time how to best crush us. and like not official discussion, but people talked about these things, right? in the end, we're quite thankful that they didn't. but yeah, I mean clearly we

Really?

We try to introduce a sort of know your customer program. So the secretary slash admin would have to try to vet every customer. which is kind of difficult sometimes because sometimes you had sketchy customers like some random person in Virginia that sets up a fake custom like a fake fake thing. Or you had had you had front yeah, or you you have

Yeah. A front front for you know for yeah.

You had people in like at some point somebody called the office and was yeah, we we really want to buy your software, but we need to pay in cash. Can we drop by the office and pay in cash? And it's it's these situations where you have to just try to force everybody into good behavior and be reasonable about who you sell to. at some point, I mean, most of our our income was coming from the English-speaking world, and I was at a conference, and then somebody who was working for the US government at the time.

was telling me well you know we can't tell you whom to sell to or whom to not sell to but just be aware your selling decisions impact our buying decisions and I was like well you know that's 60% of our 70% of our revenue. can you tell me who I can't sell to? And then they're like, no. All right. can you provide an oracle that if I'm doubtful on whether I can sell I can ask somebody that can tell me yes or no. And then I'm like no. So

Wow.

And at that point you you just try to do your best, knowing that you will eventually fail. Right. one one example is we at some point we got an order for from the the People's Liberation Army, like the Chinese military, for a large license of Bindif. And then we decide, well, we can't sell to you for reasons.

and then a couple weeks later we get an order from a Canadian company called Hongbin Immin Export. same order. And we are like, yeah, no, can't do that either. and then a couple weeks later we get an order from a Chinese origin architecture student in Sydney, like an individual person trying to order like an enterprise license. And we're all again, no. But I mean you've got to be realistic. If somebody wants your software,

They'll get it.

They they like a a nation state can put in a lot of effort. And as a small company, you will not be able to prevent a determined nation state from getting the software at least.

I'm curious, are were you bootstrapped until this point? Like you in

Yeah, yeah, yeah. No, the the the entire company was bootstrapped. there were other funny situations, like around the Stuxnet time, we got somebody calling us and be like, we need reverse engineering trainings and price doesn't matter. like charge whatever you want, but it has to be either in Turkey or in Malaysia. And we're like, that's a strange request. And then we figured out that Turkey and Malaysia were one of the like where were the two countries where Iranians could travel without a visa.

It does.

Interesting.

And at that point we again had to cancel. And we the the the company also did reverse engineering trainings in Frankfurt on site, which was often funny because people like all our like all our diverse set of strange customers would congregate there and eye each other suspiciously. and

And why Frankfurt? Is like Frankfurt has this special visa situation?

we no, no, no, we were just based in in Bochum and Frankfurt was the international airport that was easiest to reach for everybody. But the the part is that at some point post-Saxnet we got essentially no sign-ups outside of Iranian people. And at that point we're like, listen, we have to cancel these trainings, we can't can't do it. And

And were these trainings done by you or do you had like a team of people like these are are the were these trainings like they were they specifically we want Thomas on the on the training team kind of a request? Or was that like anybody from your company could train?

not anybody could train. I did most of the trainings, but we also had a few trainings where I was not present. So it depended on the situation, it depended on the subject, it depended on what had to be done. Most of the time I was pretty instrumental in running the trainings.

And then you ended up selling the company to Google. and curious, what happened there? Like, so you know, you're doing well, you're doing your trainings, you're like the talk of the town in all the elite Intel circles, like everyone wants a piece of bin def and wants to buy your software. what happens next? Like, how did how did the Google acquisition happen?

Yeah, yeah.

So the important thing to remember is that selling reverse engineering tools back then was a terrible business. Like the the Yeah, I mean I I I can't tell. I haven't tried again, because I I think I learned from my mistakes. But I mean we we when I tried at one point to do a business plan to invite investment or to get investment, I realized that the business plan is essentially we're gonna solve a bunch of really hard computer science problems.

It is still a I think it is still a terrible business, right?

All right.

And we're going to sell it to like 120 people for $800 a pop. and that's just a terrible business model. That's like I'm going to do handcrafted pianos for one-armed ballerinas. It's like a just not a viable business plan. And Zynamics largely survived by virtue of the fact that German salaries were reasonably low, even for competent people. So we could pay relatively low, like by today's standards, low salaries. we all liked what we're doing.

Yeah.

But also it required an enormous effort, an enormous investment of energy from everybody because the underlying business was actually not a good business. So over time you exhaust yourself running that business. I ended up having all sorts of health problems and so forth. And it became clear to me that something's gotta give. And Arrow, my business partner at the time and me, we then started discussing: okay, this is not sustainable the way it's running. We need to find

A good path out. And we so one of the things we had is we had a protocol of the X Class, which was very good at malware classification, and we tried to sell it to antivirus companies at the time, not realizing that the antivirus companies at the time were no longer capable of innovating. And then at some point after like our product was good, but we should have not tried to sell it to the antivirus companies. We should have tried to set up a separate antivirus company that competes.

Because in some sense, if you look at what CrowdStrike did, is CrowdStrike, instead of trying to sell something to the A V companies, just decide we're gonna eat those dinosaurs and successfully did so. So yeah, ran low on energy. We were thinking about who we could reasonably sell to without screwing everything up. And then we got very lucky that Google got hacked by the Chinese government just at the right time.

Because about a year before they were not very interested in what we're doing. And then Aurora happens, and all of a sudden, they're very interested in what we're doing. and me and Aero at the time discussed, well, exiting to, I mean, this is early Google as well. This is Google at 10,000 employees, which is not comparable to the behemoth it is today. And it was it was a young and exciting and idealistic company with lots of compute.

Mm.

So we're like, okay, Google is a good choice. Like we'd rather sell to Google for price X than to Semantic for like two X, because we'll hate our lives for the next couple of years. yeah, and then it was a very painful, very drawn out negotiation process with Google because we were not a typical

Yeah. Correct.

Why w why was it why was it painful? What happened there?

So we were not the typical acquisition target. Like Google was very well set up to buy a typical VC VC funded Silicon Valley company. They really were not well set up to buy a small engineering-centric company in Germany in a foreign jurisdiction with everything that that entailed. So the negotiations were difficult. They're always difficult in MA. They're doubly difficult if it's your first negotiation, like your first rodeo.

Yes.

Mm.

because in some sense as as a founder you're stepping in a ring into a ring with a professional boxer, right? Because the other negotiator does nothing but screw people over all day. That's their job. and you're just very like I was very inexperienced. And yeah, and then there were complications where Google like Google had the process to do something, as they do.

And that process is largely the same for buying Motorola and buying Xenomics. Like it does scales up arbitrarily, scales down very poorly. And one of the special parts of the German legal system is that if you sell a company, you have to go in front of a notary and the entire acquisition contract has to be read out to both parties in full to make sure they both understand what they're getting into.

But Google's legal department insisted that they have to take the American, like the contra acquisition contract they use for in the US for under American law and so forth, which is hundreds of pages, and insisted on making that work under German law. And like Google's German law firm, like the law firm they had hired, was begging us to tell Google to please come to their senses and not force them to translate hundreds of pages of contract into a different legal system. And we're like, we we we can't tell them what to do.

So anyhow, the the day of the sale I spent eight and a half hours having a notary read hundreds of pages of a US contract to me.

So the

So the painful the painful process was the legalities, not like the pricing discussion, the valuation. That was not that was not that or was that also adding to the mix into it?

all of all of that is yeah. I mean all of that is painful. It's particularly painful the first time you do it. It gets easier as time goes by, but that was very painful. I'm I'm not gonna lie.

How was the experience post acquisition? I remember you were part of the Project Zero from

Yeah. no, w w we were not part of Project Zero. We Project Zero didn't exist when we got acquired. that was a much later development. we joined a trio of of teams that was trying to protect Google from government attacks. and I have to admit like I have to say that as painful as the negotiations were, the actual like I was very, very happy the day I signed. I felt it is

A great outcome for the team, great outcome for the technology. I think we were like overall it was a very happy. Like as as far as acquisitions go, it was one of the happiest acquisitions I know of. like most of like everybody came along to join Google pretty much with like with the exception of the the admin.

Hmm.

And pretty much like most of them stayed at Google for a very long time thereafter. There's still a couple of Xamics people at Google now.

And did you move to to the did you move to the Google headquarters or did you continue to operate from Frankfurt?

We we all moved we were not based in Frankfurt, we were based further north, but we all moved to Zurich, which is in Switzerland. So but that was also helpful that you're you're moving into a city where the language that's spoken is at least pretty similar. Swiss German isn't quite German, but it's related enough that picking it up is possible.

But then did you have a role in the whenever the Project Zero got incubated? Would were you part of that team? Did you contribute to that team or was there not was there nothing to do with it?

I wouldn't say nothing to do with it. I got I was watching it from from a couple yards afar and I was in favor of it being created. but I wasn't part of it. I joined like I I spent four and a half years integrating the Xynamics technology into Google. Then I went on sabbatical and when I returned, I returned into a role at Project Zero. But Project Zero had been formed by then and established and was an existing team.

Okay. Thomas, one of the things you mentioned earlier in the interview is one of the motivations for you was this compute. The compute that you would get access to if you get to to go get to Google. And I want to bring this discussion to the current day because compute is the most scarcest resource of today. If you have the compute and you have somebody like Thomas, like a very elite researcher, s you know, reverse engineer, and so on, what happens

yeah.

what happens to this field of yours with elite talent and infinite compute? Like what's the you know, what's the future of reverse engineering? What's the future of cyber offense with this level of compute? Because now the compute is like almost a million X than what you probably saw at those times, right? Like now given all the GPUs and all the technology that is out there. I'm curious, you know, so w w

What is the future of AI native offense with this level of control?

Well yeah. I mean I I I have to caveate everything I'm saying now. because my predictions about what would happen on what timeline have been wrong a couple of times in recent years. So

What was the first prediction that you made that was wrong?

So when I saw LLMs work for the first time, I did not anticipate that these could be turned into problem solving engines. Like I I did understand, hey, this is really useful, for example, to get a textual explanation for a piece of code.

Yep.

Because I I mean I had observed the sudden improvement in machine translation with the introduction of the transformer. And as long as I could imagine the issue at hand as a translation task, I was like it was within my modeling that this would be feasible. So for example, a simple C function. Can you explain this function in human language for me? I was like, okay, perhaps that'll work. I I can see that working. Or

I write a human language description of a function in some detail and then it translates it translates translates it into code. Yeah, I I could see that. I did not foresee the level of agentic autonomy we have now. I did not foresee the level of problem solving and reasoning ability we have now. my my brother and me had a had a conversation about this recently because we're

We seem to be very different in the way that we think about things because I'm very much a shape rotator. Like I think very geometrically about almost everything. And very often it's very hard for me to move the geometric intuition that I've got into words. So I've got an image in my mind. And then translating that back into writing was also my biggest difficulty studying mathematics. I would often have a an

a geometric explanation for why something has to be the way it is in my head and then translating that into a proof was very, very hard for me. There's a real translation step. I often do not think in a in an inner monologue stream of consciousness. My brother on the other hand does.

I I are you saying are you saying that you don't think in language, you think in abstract visual context? Is that how you see the world? Like you see

Not not not always, but most of my like almost all of my deep thinking is experienced by me as nonverbal. My brother, on the other hand, seems to think at least he said so in the discussion, seems to think in an inner monologue of sorts. So to to some extent I did not because I I think because I don't think in an inner monologue, it was hard for me to see how language modeling could lead to problem solving, because that's not how

I think

I perceive my own brain to be working. So I was very surprised when when that started working. Right. I'm I was pretty wrong about how far you can push language modeling. and now I look, yeah.

Mm.

By the I by the way, I want to make one comment. For for all the perceived deficiencies that you say in terms of your writing capa capabilities, I think you're also an amazing writer. I just read your recent piece on the advice to founders. I think that was one of the epic write ups I've ever read, because it is like very comprehensive. So please sorry. I had to say that because you write really well too. Regardless of what you think of yourself as a writer, I think you're a great writer too.

No. thank you.

Thank you. Well, I I like I like language and I like writing. It's just I I did not foresee that language modeling would get us to where we are. Like I I think and because I was so wrong about this, I'm now very careful about making predictions in the sense that I might be right, I might be wrong, but I make a very wide confidence interval around all my statements, right? So I think we were talking about what's what's the future of reverse engineering now that we are we have infinite compute.

I mean reverse engineering, creating exploits, breaking into systems, because it this required somebody like yours. This is something that in my humble opinion required a really good la long amount l long time of training and skill and expertise to get to a really elite level. But now that not may not be necessarily I mean, obviously you can still get better at whatever you do, but even like somebody who's not as skilled as you can get farther along. Like, you know, going back to your initial discussion of the

Yeah.

the cracking group and the hacking group, right? You know, it is now possible for somebody from the hacking group group to get closer to the cracking group, but just by using some of these tools that are out there.

Yeah, yeah, definitely. I mean a lot of things that I mean computer security work has always included a lot of perspiration and l l only a small dash of inspiration, right? And a lot of reverse engineering was always an endurance task. I used to to joke, like there the there's a saying that in some sense the Tour de France, the the bicycle race is a pain Olympics, like whoever endures the most pain wins.

and there's something to be said that a lot of reverse engineering and vulnerability development is like it teaches you that if you just apply yourself for long enough and endure enough pain, you can eventually resolve the problem. And there is an argument to be made that modern LLMs kind of commoditize the ability to endure pain because they'll endure the pain for you. You can just tell.

Tell an LM okay, crunch on this for three days and and do this, right?

And there is the there is this tense of infinite pain. It can take infinite pain. Like you can keep prodding it, keep prodding in and keep working. Yeah, yeah, yeah. But like so if money is not a you know, a const constraint, you can keep you can go down to this infinite level of pain where you can keep prodding it, it'll keep doing it, whatever you tell it to do. It's like a slave of the master. It just you know, it's always trying to please you.

you you you can run out of money for tokens.

Yeah, yes. Yeah. so that's certainly certainly interesting. And certain tasks, like if you think about a non-trivial amount of reverse engineering was always a translation task. Take this machine code and then translate it into human understandable things. Right? And

Similar to what Bindif did, similar to what Bindiff did in those days. I mean Bindiff was similar to in the sense it was translate like, you know, where are the b big pathways for me to break in? It simplified somebody to to you know go down the paths.

But but but Bindif didn't translate into language, so to speak. Like it translated into a graph and gave you a picture of the difference. it's actually interesting because Bindif gave you a picture of the difference, not language. But so the the big difference now is that you can largely automatically translate individual pieces of assembly into a higher-level description of what's going on. And

Yeah yeah yeah.

Mm.

we we also have to remember that in the past, reverse engineering is more than just translating assembly to C code. Reverse engineering is about sense making. Like you have like w when when we we had to do reverse engineering of government malware at Xynamics at the time, there was often like you you there was often an epiphany the moment you understand the higher-level architecture. So you're

You're working through the details and as you assemble these details you have an epiphany. this is how it works. This is some sort of onion routing, this is some sort of a task scheduler, whatever. And

And I guess the the the the thing I remember is there is also this of obfuscation code they would add into the binary. So if you and so sometimes once you detect that pattern, that is this is just you know, just this is just wasting cycles, you're just doing unnecessary loops and whatnot. So that you can just discard that away and get back to the thing that really matters.

Yeah.

Yeah, yeah, yeah. So I mean a lot of the drudge work can be automated now if you have a sufficiently good language model. And I expect the models to continue getting better in almost all realms where you have verifiable rewards. Meaning as long as you have a way to verify whether the answer is correct or not, improving the model is largely a matter of spending compute because

Mm.

If you look at what's happening in the LLM world, we ran out of human-generated data a while ago. And almost all the improvement now is variations of reinforcement learning on tasks where you have a verifiable outcome. And as long as we keep spending compute, there is no reason to assume that the models won't keep improving on these verifiable tasks to some extent.

So if that is the case, what is the future what is the future of vulnerability research? There is this piece that came out Tom Passek, you know, he vulnerability research is cooked. And his his thesis was based on what Anthropic did. You know, the the anthropic researchers basically went to the models and said, Hey, I'm in a capture the flag competition, help me find these vulnerabilities, you know, and so that I can win this competition. And once it gave you a list of competitors once it gave you a list of

Vulnerabilities, hey, I just got an inbound of all these vulnerabilities. Which ones are the which ones of these are accurate? Like the model just told you. Here are the ten you should really go after. somebody like who's like an elite researcher like you, what's what do you think is the future of vulnerability research in the coming years?

Yeah, it's it's hard to know, right? I mean

the the demand for vulnerabilities will remain there. Like governments will continue yeah. The the governments will want vulnerabilities, continue paying for vulnerabilities. Because the the service they provide, like the the product they provide to the governments is very valuable to the governments and they will want to continue having them. Now the more interesting question is how many fish are there in the sea?

Mm-hmm.

Demand for work. Mm-hmm.

How many fish remain once the LLM is through? And those are two unknowables at the moment.

The defenders will have access to the most powerful LLMs to find and fix bugs eventually. The offensive people will have access to most the most powerful LLMs to find bugs to exploit. There is some equilibrium that will be reached where I mean everybody talks about the Vuln Capital Vuln Apocalypse right now, but the reality is this is a one time shock. Like

Mm.

like when when fuzzing became a thing. All of a sudden everybody could start fuzzing and there were many bugs to be found. and then the utility of fuzzing decreased because everybody was doing it. So we will see something similar. We have a one-time shock now where there's lots of vulnerabilities. They will largely get fixed. There's a certain rate at which new vulnerabilities are introduced by either manual coding or vibe coding. We don't know.

c we don't quite know where this will leave us. Like will vulnerabilities will there be sufficient vulnerabilities left?

Do you do you think there is there is going to be a new class of vulnerabilities that will come up through this experimentation that will happen over the next year or two? Because my historically, my sense is most of the exploitation was through a single vulnerability. You would you would try to figure out a remote code exploit, break into the system and then go from there. But my sense now is that you can now chain a lot of these vulnerabilities, which was d which was which was done before, but now it is much more easier to chain vulnerabilities. So the chaining of these vulnerabilities would become much more

Well, I mean

exponential in the i in the in the coming years. Do you see so so my question is one is the chaining aspect, second is the second is like new class of vulnerabilities that we have we are not even aware of.

Yeah, so I I think that chaining has been a reality what six, seven, eight years now? Like any

Not at scale, right? I mean w it was not at the level I mean humans would have to like figure out okay, I can use this and this and this and then I'll

yeah, so humans humans would have to do the chaining. And some of that chaining can be automated now, which is interesting, right? Because it it might help you build better chains, clearly. the the thing I find somewhat fascinating about LLMs though is you run the same LLM on the same code base three times, you get different sets of bugs. Right?

You mean with the same prompt or with a different prompt?

Same prompt. It's just randomized, like there's randomized like LLMs are inherently randomized, right? so it's it's really hard to tell where this will land. What is has gotten dramatically cheaper is rewriting things in Rust. I mean, the fact that Anthropic rewrote bun in Rust by spending like $170,000 in tokens.

The the ability to rewrite legacy C code bases in memory safe languages has definitely changed. so there there might be a lot of rewriting that'll happen. but it's there are like I'm I'm doing a lot of vibe coding.

Mm.

And models have strange failure modes. They're not human intelligences. They are strange creatures. they're not like they're big matrix multiplications, but the way they function in the end is very different from how you would think a human would function. And they have very surprising weaknesses. Because we have like something like Fable will crush many math Olympiad problems.

It'll fail to read a clock reliably. The models are very, very good at reading code sequentially now. I don't find them very good at doing reasoning about multi-threaded issues. And if they're bad at reasoning about multi-threaded issues, they'll also be bad at writing multi-threaded code. So it's really unclear to me.

where the equilibrium will land in terms of how many bugs there are, how difficult they are to find. we well we we shall see, right? It's a it's a brave new world in some sense.

Thomas, let's switch to your latest, your latest work, which is Optimize. and curious what is going there. You know, you've done all these different things. You've been, you know, from like you know, playing games to hacking games to copy protection to like bin diff to all the work that you have done in World of Dritt Research and now doing you're doing Optimize. There are two questions I have. One is

so optimize was done four years ago. So I I did I did optimize from twenty nineteen to twenty twenty one.

So so I want to get your thoughts on Optimize and also and as I said in the interview, you're a really great writer. One of your recent pieces, one of I I th I think it should just be hanged in law. Like that is like an epic piece of writing. It's very comprehensive. I don't know why you out wrote it. I want to get your thoughts on both Optimize and

Like advice for young entrepreneurs and young researchers on how to think about this world that is coming our way in the next four to five years. Because I my assumption is we talk I talked to you in two years. The world that we live in will be completely different than what we are experiencing today. So I'm curious what your thoughts are on that.

Yeah. Yeah, I mean optimize I I got tired of security in in around twenty seventeen, eighteenish and I wanted to do something that was technically as interesting but had fewer complications because security in the end is a zero sum game. You're always defending something or attacking something against somebody. It's always about human conflict. I got really tired about like because of that.

I wanted to do something which has like a positive effect on the world. And at the time, GPUs weren't really a thing, and II wasn't really a thing. So CPU code optimization was actually a very low-hanging fruit to have a positive impact where you could do technical work and then save power and earn money and save energy. yeah. So

my co-founder Sean and me started a company that then did fleet-wide profiling where if you have a thousand machines, you can install the software and it measures where are all the CPU cycles going. that was capability that existed inside Google for some programming languages, but we were the first to use eBPF in the Linux kernel to make it widely available to everybody. that was very well received. and we then in a

Complicated set of events get acquired by Elastic. And Elastic ultimately open sourced the relevant code, which is now part of OpenTelemetry Profiling. So if you're the open telemetry profiling agent that you can just install on your machines, that'll tell you 20 times per second what's the stack trace that's currently executing. Like through through a whole bunch of programming languages, like from the Linux kernel through user space C into Python. That's the formal optimized code, and I'm kind of proud of

of that 'cause it is in wide use and saves CPU cycles everywhere.

I'm curious, you know, you said you you lost interest in security, bec because of all the work that was happening in twenty seventeen. H has your interest rejuvenated after the things that you've seen in AI?

So

E somewhat. certainly like say security was very stagnant in 2018. Like it hadn't changed a lot in in 18, 19 years.

Computing as a whole is changing very drastically. And when Optimize got started, CPU waste was the primary motivator. If you look at where energy consumption and compute is going, it's all going into matrix multiplications in GPUs now, right? So very clearly I have a great interest in efficient inference now or efficient training. Efficiency in in the internals of AI, because that's where all the compute is going.

Yeah. Yeah.

And I have a great interest in how AI is restructuring all of software engineering, including security. So in some sense, security is exciting again, because something has moved for the first time in more than a decade. So I I personally find the the pace and change that we currently see from the LLM side to be really quite exciting.

Yeah.

I don't think I've been as interested in the changes in tech as I am now in a while. So

What you know, one thing that would be cool for you to do is to build a chip of your own. Like build your own inference chip. That would be the most epic contribution to mankind. Last question for you, Thomas. what what advice do you have for founders and researchers for the next five years?

Yeah, good question. I I don't know. I mean, any advice is always overfit to the extent that's based on the past and the future is a very different country.

I think one of the things to take away, like people look at LLMs and are like, no, we're obsolete as software engineers or something like this. I think that's the wrong lens to look at this. The right lens to look at this is what is now possible for me, in the sense that how what's the maximum ambitious project that I can tackle using these helpers? Because they are amazing, amazingly powerful machines in some sense. Like if you think of

The advent of the steam engine, you were very limited what you could do alone prior to the steam engine. And once you have a steam engine like a source of energy, there's lots of things you can all of a sudden do that weren't possible beforehand. And I think LLMs are for sure a similarly important development as the internet. They might possibly be as important as the steam engine or

Or the mechanization of industry. So the thing to think about is always what can I achieve with the tools that are now available to me? Paul Graham has something about what's possible today that wasn't possible three years ago is usually a good way of ideating business ideas. And that's not wrong, right? Because

A lot of things are possible now that weren't possible three years ago, and those will be interesting. I think another piece of advice is

Personalized tutoring in any subject you're interested in was historically not available. And I've always been like there's an argument to be made that I really my prime motivator for anything I've done in life is curiosity and the joy of learning. I like very few things give me joy, like having understood something that I didn't understand previously. And for me, having

a sort of personalized tutor that isn't always right. That sometimes tries to tell me bullshit.

Well, that is only if you know if it is bullshit. If you're not trained in that area, you might accept it as like really good. Like but if you know the domain really well, then you can say, this is bullshit.

Well, I mean you

Yeah, yeah, but I mean it it's it's it's good exercise. Like talking to like talking through a complicated topic with an LLM and catching the LLM and making errors usually is an indicator that you're starting to understand the topic well. So my advice is pick things you want to learn and make the best use of the personalized tutoring that the LLMs can give you to learn the topic as best as you can. because the fact that we still have to cor correct

Yeah.

LLMs quite often, even the frontier LLMs on on domain specific topics. Like I I think we're a long way away from complete obsolescence of the human input. I think a human with a tool will outperform just the tool for the foreseeable future. So try to be that human with the tool.

Awesome. Well that that's a good way to end this interview. Thomas, super grateful for you to coming on this pod. this was an amazing interview. looking forward to what you do next and hopefully you build that next generation influence chip as well.

I probably won't, but I'll do something else.