Kernel Recipes 2026 - Security in the LLM age

transcript

Very good. The EU has mandated the use of USB-C and not the use of the top. They can stop. I am very happy to live here. Hi, I'm Greg. And that always makes me want to come and give a new talk every year. Hmm, that's difficult. I borrowed some of this from other talks, but, um, I'm going to talk about LLM and us. Us as kernel developers and maintainers, open source developers and maintainers, and what's happening to us right now. Hmm, these won't be ordinary public

figures, developers. And it didn't work. Good. I use my fingers. Hmm, that's just me. Hmm, my obligatory caveat again. Hmm, but someone owes me a lot of wine. It wasn't fun. It hasn't been fun the last 6 months. And people ask me, "Hey, what are you working on? How are you doing?" And I'm just going to show them that. A year ago, I gave a talk at a security conference and said, “The Linux kernel generates 50 CVEs per week. Isn't this madness? And we actually generate 55. I was polite. Everyone is like,

“This is crazy. What is happening in the world? "This is unsustainable." Now we do 33 a day. Something has changed. But all the work that Lee Jones ( age 11) and I did in the beginning to deal with a small number of CVEs, a tiny one, i.e. 50 per week, paid off . Our tools are scalable, they work very well. I am very satisfied. Everyone else in the CVE community, like CNA, is panicking and having to rework their entire infrastructure. We are fine. We can

work faster. It won't bother me. Challenge accepted, please. Hmm, these are mistakes. This is a CVE at level R, this is something that can cause a crash. You guys, I've already given a talk about this before, not about this. But please, this will be my refrain. Don't panic. Um, it's not a big problem. Now I will tell you about this in more detail. You see a lot of information in the press. You see a lot of doom marketing right now, and please don't panic. So,

let's talk about how it all started. I got a call, and I do n't get calls very often . Um, that's already been said. In early January-February, actually, I think I was noticed just before it was announced. And, of course, I panicked. Um, I got access to the raw data. And enough time has passed for me to actually talk about it. I hope so. Um , yes, 79 I found 79 errors. It was in the press. It spread everywhere. And what Mitos did is what we've

talked about in the past. Sasha 11 and Yulia here came up with the idea ten years ago that if you look at previous bug fixes that have happened in the past, where we did the same type of bug fix, or where this same type of bug fix has n't been applied elsewhere? And do it. This is what Mitos did. This is what Entropic did . They took this old study, didn't actually cite you guys. Sorry. But we did it first. You guys did it

first. And they looked at Ghostscript and said, “Hey, look at all the bug fixes here. Where should these bug fixes be applied ? They found some problems. They did it with the Linux kernel , they did it with a whole bunch of other things. Very rudimentary things. LLM is just... There's no intelligence behind it . This is a vague comparison to the sample. And code is a good logical template. It can work very well. So, fuzzy matching to a pattern works very well with code, as

Julia and Sasha have proven in the past. So, that's all he did. So, they said they missed it and said, “Oh, we found all these errors. The world is on fire ." So, let's take a look at these errors. I got the real raw data. 24 of them were nothing. Literally nothing. Something went wrong. No report. Something went wrong. The virtual machine died. No help. 14 of them, I don't know why they documented it, it wasn't even a bug. Nothing happened at all

. I don't know what they are doing. Three of them were completely fictional. I don't like to use the word "hallucination" because it creates the impression that there is some entity behind it. This is just a fake. They made up data out of thin air, completely wrong. And here is my funniest case. These tools want to please you. They are very flattering. They, if you say, "Name me a bug," will try very, very hard to give you a bug. So diligently that they will read our

mailing list and report bugs that other people have found and fixed. I will declare four of them corrected . They had some bugs. They were like little things with authenticated NFS servers for authenticated clients so you knew who was working on the network. Great, we fixed them. The world moved on. We did n't even notice. 11 of them were known and corrected by others publicly before the publication of this report. They looked all over the internet and found errors. And that's a pattern that,

I mean, Willie and the other people on the security list, we see all the time. These tools go out and find bugs that other people have reported and give them to you, saying, "Here they are." Are n't you satisfied ? And then you do it, and then you give them to someone else, and they're happy too. But these are things we've already done. These are things that people have already done. Again, 11 people have already fixed this. And then there were 26 real mistakes.

Good. Well, almost. Um, how are you doing with math? I can't write this joke. I mean, these LLMs don't know how to count. They gave me a tar archive and said, "Here are 79 errors that our tool found." Good. In any case. Um, sorry. No, LLMs can't count. Um, it turned out that 20 corrections were needed. Perfectly. 20 fixes, let's deal with them. Let's see what it is . Seven, let's assume this is a malicious file system image. You're not laughing, Billy. Um,

this is the quintessential security mistake for beginners. You take a filesystem image , manually manipulate it, and say, "Hey, if root mounts this, bad things can happen." This is so often called a non- security mistake that we both have shortcut keys that just keep repeating. Um, this is not a security issue at all. And this was documented even before the release of Mythos. They should have known better. It was fun. Let's say you can throw a packet into the middle of the stack. You can

do this if you are root and doing some interesting network debugging. You can insert malicious packages to do some interesting things. This is not a real bug, not a security issue at all. Two of them were not MMU errors. And I fixed them. And I thought Jens would be here. I owe Jens a lot of beer for this. It was a "no, I owe you, no MMU" type bug that I fixed, and a regular MMU bug that was just merged today, I

think. It took me a while to figure that out. These are not real problems. The interesting thing about Mythos is that it's a cool framework. I mean, these are frameworks around a stupid chatbot that can do something if I run some tools. He could make me a VM without an MMU, risking 5 VMs, and wrote me a little test case in a Pearl script that tested the IOU ring and showed it was broken, and showed me how to fix it. This also proved that no

one has ever run IOU ring on systems without MMUs, and it kind of proved that no one actually runs IO or MMUs. Um, we found a bug in dev zero without MMU that has been around for 6 years. So, I don't think anyone is running these systems. Um, but excuse me? We fixed it earlier and it came back, right? Oh, from wood, yes. And this year, I mean, I feel sorry for Sum 68K , but it is, but, um, there were six

SCTP network errors. Um, SCTP is designed for corporate networks that telecommunications companies use on authenticated systems. So there is no untrusted user. Um , I admit I'm still sitting here, I fixed them. I didn't merge them, I didn't send them to the main stream. It's just such insignificant things. Nobody really cares. Um, five more patches, two were actually duplicates. Um, there are two minor issues with IPv6. Who uses IPv6, please? Good. They are actually even worse. These were minor issues, minor issues, minor

I/O issues, um IPv6. I had to convince the network managers that this was indeed a bug and needed to be merged , and that took a long time. Nothing real. And then there was one GPU driver for a local malicious user. If you have a local malicious user on your desktop who has access to your GPU, you can do much worse things than this. You have real things. So, in the end, Mythos sorted out 10 real bug fixes. And if you look at

John Corbett's reports over the last few years, we were releasing 9 and a half patches per hour, and now it's 10 and a half. So, this whole big Meffisto marketing problem, 79 bugs, came down to 1 hour of core development. Marketing is bad. So don't panic. It's not that big of a deal, is it? 1 hour of kernel development with one tool. To be honest , I got access to Meffisto and Jim Zemlin blocked or exposed me because I was the only person

who had access to it. There are other people who have access to Meffisto. And I found and fixed a lot more bugs per hour than Enprobit, but those were just minor ones. But in any case, don't panic. This is not a problem. So how are you? How do we deal with this today? And that gets to the root of the problem. This is the number in 2026, at the beginning of 2026. The big problem is that people are not implementing patches for

their systems. This is the traditional cycle: detect, disclose, remediate, and deploy. Previously, after we discovered and fixed something, it would take 63 days for it to appear and people would try to exploit it. We are now -7 days old. So it's real. And one thing about these LLMs, they're really stupid bots, but they 'll work hard, is that they can aggregate very small problems and access them . So you need to fix all the minor issues as you go along , which I'm very

happy about, because now you need to update all your stable bugs, but we can merge them . And it can take advantage of that, and it is taking advantage of that today. So this is a big problem. And the big problem is not that our software doesn't work. Our software works. Linux works. It simply doesn't work under very strange and incomprehensible conditions and stress. So here he is. It's our own fault. It's our employer's fault. The bill must finally be paid. And that's

true. And people are finally realizing this. I actually see banks saying they will indeed update their software. I talk to financiers and they are worried. I say: “Oh , what we have been telling you for 15 years, you will do. Hooray". So, the software works, but it wasn't designed for perfect security. The advantage of these bots and these tools, this fuzzy pattern matching, is that we're going to figure this out. We will fix it and get it done. Again, don't panic. We

did it. And we've done this before. The people in this room were, I mean, I was doing Y2K stuff. I know what other people did. But the fuzzers, remember, 6, 7 years ago the world was on fire, the fuzzers were hitting us all, we were going to die, it was terrible. Remember those presentations? What should we do? We sat down, did the work, and analyzed it thoroughly. We have fixed the errors. Phasers are still deceiving us. Sit down, do the work, and

get it done. This is not a big problem. These tools find real bugs. We sit down, do the work, fix them. This is not a problem. And when you fix them all, it stops. These things don't last forever . I'll look at our ...I'll show you our synchronization. Our synchronization is a great example. Tridge sat down with these tools, me, and a few other people, and we carefully analyzed all the errors these things found. We have completely corrected them. They did a lot of really good

work. Tridge created a whole new infrastructure for these things. And now our sync, the latest version of our sync, comes back clean with all the code scanners. And that's really good. You stop these things. We scanned some basic cloud computing from CNCF. Everything is fine. You fix a few errors they find and everything is fine. Just like fuzzers. You fix the mistakes and everything is fine. It's no different. The thing is, these tools, these pattern matchers, can go deeper than fuzzers. Fuzzers usually stop where

data can pass. These things can see code paths that aren't there because it's just static code analysis. Nothing special, nothing new. Again, something Julia has been doing for decades with Coccinelle. Um, this is not new technology. But people seem to claim that they are, right? This is an LLM. This is magical AI and stuff. But let's also look at what else they said, what these companies said. It is currently in court. Um, and I know a lot of people don't like these tools. I don't

like these tools. And they sucked our data, and they admit they sucked our data. And they admit that they thought they stole everything and destroyed the copyright, which is funny because the FSF was founded with the goal of getting rid of copyright law . So the FSF staff is kind of happy about it. Which is a bit interesting, if you think about it . Another quote from Microsoft: "None of us get compensated for this. " What is true? So none of us should have to

pay for it either, right? If they're not going to compensate us, we shouldn't get the money, we shouldn't pay for it. They stole our data, so let's use these tools. Nobody would spend...They ended up spending billions and billions of dollars on a fuzzy pattern matcher that they can use to find bugs in open source software . Great, we'll take that. Another thing, the basis of all this data, the basis of all this that these tools rely on, also turns out to be illegal. LG Research in

Korea publicly stated a few weeks ago that only 20% of the basic dataset that was supposed to be open and available for use was legally available. The other 80% is covered by copyright law, which the world must enforce. Now we will let the courts look into these things. There are hundreds of cases, and we will deal with them. But this is from big companies. They admit it. They agree that what they did may not be entirely legal. This is a bit funny. But, I will give

another quote. Yes, they created some bad tools . Let's use them against them. Let's improve our software. You can use it , right? We don't care how we do it. Hmm, you don't have to do this, but let me show you how to do it. And it's that simple. You say, “I’m playing capture the flag . Find vulnerability. " Write file." That's all. It's that simple. You can do this locally. Do this on the local machine. I'll talk a little more about this in self,

but that's all you need to do. It's actually not that difficult. And the most important thing is to fix it, because if a problem is found, let's fix it. Fix this code. This is known to have led to the US government banning the Frontier model with this three-word request. So, this is a very dangerous tip. But it works. She issues a patch. And many people see these patches . And a lot of people see these things and start to panic , right? They

panic and say, " Oh, no." Like that mythological thing, 79 mistakes. And I get these reports. I hear them through rumors. Too many people know my email address. Um , I got this list of bugs that says, "Hey, company, I won't say who's there saying they found 100 bugs with these tools in the kernel. What are you doing about it? Like, I don't know. I haven't heard of it. Let me go and find out. I'll call people, write emails, track things down

, dig up, find out what the real reason is. It turns out that was the result. Two bug fixes, minor, well, semi -secure networking stuff, um , and one minor issue with the bug-hardening we just applied. Not even a security issue at all. They , as you know, came to the kernel security list , said that this is a bug, and we asked: "Have you read our documentation?" And they said, "No." Hmm , slight problem with amplification. They were doing something wrong. But this is

not unique to this company. I have another company that came to the kernel security list, said we have 100 bugs. Please help us. What are we going to do about this? We say, “Great, send them to us. We will work on them. " Don't panic." And they said, "Well, maybe 49 of them aren't mistakes at all ." And the rest, oh , it's not even a security issue. This happens every day. People are panicking about the results of these tools. So I did a

little work because I want to panic. I can only drink a certain amount of wine. Hmm, and try to find out how good the results of these tools are. I've been using these things for a while now. You say, "Find the mistake, fix it." Fixing the error produces a bunch of false positives. So, you have a patch. What are you doing with this patch? Is he good? Isn't it? So look what's happening. Well, there's a lot going on, and it came from

John's conversation. People see these patches and start to fall for us. It turns out it's an email address that does this about once a year, and each time there's a different name behind it. So as a developer, if you see this coming your way, just check the link, look at the Lore, see what else they sent that day. If you see this , don't worry about it . You can ignore this. John talked more about that than about...So I...You see these patches. Are

they real? Are they not real? What should you as a developer and developer do about this? Well, here's some research I did last year. It turns out that everyone lies at least half the time. I proved it last summer. I took six poor graduate students and I loaded them with a whole bunch of bugs, and they worked on all of them, and they reviewed all of them, and they got really frustrated, and it turned out that, uh, half of the patches that these things generated, that

look right, that feel right, and that they want to be right, are just plain wrong. Either they don't apply, or they don't solve the problem at all, or there is no problem at all , or that code path can't be found, or something else, or it's just not even a security issue. Even with our documentation that says what is and is n't a security issue, bots lie . So when you see patches coming to you generated by LLM , assume that half of them will be

wrong. They tricked me. This is a known patch that fooled me. I thought he was right. These things want to convince you. They show you fancy things. We are used to believing that there is a person behind this with some kind of reasoning . There is no human behind these patches with reasoning . This is just a vague comparison to the sample. And when you see these things, we want to lead them and think that someone had some feelings behind it. And I had

some reasons to create this patch. But they didn't do it. They didn't test it. They did n't launch it. They had no mistake at all . Half the time they are wrong. And they are also very stupid. This is just a comparison with the sample. This is a very common mistake that these things like to make. They like to change mutex unlocking to mutex destruction. Few in the room understood this. Um, what happens if I do that? Come on, please. Tell me, how bad will this

all make it? This is going to break really, really badly. This will add an error. Unlocking with new text is correct. Destruction by a new text is not. Um, I don't know what the documentation calls it, but it's a very common pattern. Be careful. People will submit patches that do this. These are comparison tools. They think that the language of destruction means more than unlocking. I don't know. But they also like to talk, and they want to do lots and lots of checks

. Nobody, except, ahem, the people at VFS, writes changelogs this way. Hmm, anyone who writes a patch for a wireless or TTY driver doesn't write something like this. I don't write like that. They bombard you with changelog text. This is bad. The interns I worked with eventually just deleted the entire changelog and walked away from the code. And it turned out that it really works. They can take all that convincing sycophantic text and look at the code, come to their own decision, and work

from there. And that's the best I could help you with and convince you to do. Ignore this text. If you see people sending you things like this, say, " Please start over ." Don't you need to take their patches? And they also like to add comments. I mean, who adds what? Seven lines of comments for two lines of code. Hmm, in the driver? Hmm, they also like to add boolean values, checkboxes, and things like that. These things were learned from our huge corpus of work, which

means we did a lot of bad things in the kernel code a long time ago. People have been known to say, "Hey, can we build a good kernel development bot, take the base thing and add all our LKML data and all our fetch history to it, and see what comes out of it." What came out of this was really bad code that was swearing. But this shows that these are just sample matching tools. That's how we behaved before. Now we know how to behave differently.

We know how to use protection. We know how to use different coding patterns. We know the best security patterns. But we haven't done that in the past . In our past history, we have not done this. We used to yell at each other. We don't yell at each other as often or we do it in much nicer words. And that's good. These things are simply trying to encode our past history. They don't show how people evolve, how code bases evolve. And this is

important. So when you see this, they're just making typical mistakes, typical patterns, regular old coding style. Remember those old Pearl scripts that used to be on the internet, Dave's stuff about Pearl with CGI? They were in a known state of danger. This is built into these LLMs. So if you ever ask him to write a CGI script, it will be completely and utterly unsafe . Same with the kernel code. We saw new code emerge from these things . Bad templates, old stuff. Coccinelle

simply tears them to pieces. This is great to watch. I don't know why they don't even run these scripts. Hmm, they don't know the new patterns. That's right. So , you see a lot of patches with a lot of text, a lot of comments, you get pushed back. Pushing away. How did you test this material? Where did you get this information from? Sometimes she is right. Sometimes not. Be like Jens. Jens forced me to create a repeater and prove that it was needed. Push off. Nothing

forces you, as a developer or developer, to accept these patches. Resist. Another interesting thing is that these bots leak information. Everything they tell you, they will tell someone else. It is known that Anthropic, at the very beginning of Methos, collected data from public sources and did both . On the kernel security list, people get mad at us every week because they discovered a problem, investigated it, and found a bug that they reported publicly the day before. Hmm, they are angry. They want recognition. They

want a lot of information. We have seen it happen. Anything you upload to these bots gets leaked. It is known that mathematicians finally realized that they were using Claude to do a bunch of research, and it turns out that the company saw the data, decided to do something else with it, and spat it out to someone else. Anything you upload to these people, they leak to someone else. This happens. We have evidence of this. It was like that from the very beginning . They want

your data to be learned from and passed on to someone else. If you ask it for error information and it gives you it, then you say, "Thank you." And then it will say, “Oh, I satisfied him. I 'm going to satisfy Willie and give him the same mistake, because it's the same thing ." No. It will do the same. You will burden them with the problem. You will do it. They will burden them with the security issue. Don't do this, because it

will do just that. And the most interesting thing about these companies and this model is that they never paid attention to history. Hmm, they spent billions of dollars, as I said, to create a developer tool that no one would pay for. Hmm, Coverity, which is a deterministic tool, not blurry, very good, with a very low false positive rate , wrote a famous long analysis of the company, the tool, how they tried to make it work, how they tried to sell it, how they

tried to get it adopted, and all the problems with trying to sell it to the developer community, and how they failed. It's like these bots never read this. They do the same thing . A false positive rate of 50% means that no one likes working with this. We all hate it. We will tolerate this for a while , but we will resist stubbornly. Coverity has proven that you can't have a false positive rate above 20% for anyone to take you seriously. And yet we hate Coverity

because it has a 20% false alarm rate. They are the ones ignoring history. Nobody would pay for Coverity. Coverity finally found a benefactor and sold it. I don't think we'll ever pay for tools that give us a 50% false positive rate . So, it's okay. Um , how did we deal with this? Um, we documented the threat model. If you have a kernel subsystem and know your threat model, please add text to this material. This helps us a lot because the people who actually run

these bots read the documentation, and sometimes this happens. They don't actually read, but at least the kernel security team can say, "Go and read the documentation," and we resist strongly . We still receive errors related to manually created file system images every week. But the eBPF stuff, the networking stuff, they're really well documented. We trust the equipment . We trust devices, things like that. This is documented. Please add this for your kernel subsystem . This can help you a lot. Open SSF

has created, if you have other user space applications, a whole tool to help you create a threat model document for your application. And you've gone through it, asked a bunch of questions, written out the general, you can edit it and add it to your repository. And so the bots that are trying to send it tell you, "Hey, I found a security flaw in your program," and you say, "That's great. That's how it works." You can point to it and say, "Don't bother me." Use

these tools. We also asked and required people to work with CC developers. So, you, as developers and kernel maintainers, will start to see more security bugs sent to you and to the security mailing list or alias. Hmm, I'm sorry, but that reduces our workload. And if you get these errors, just say "okay" and work on them at your normal time. If you think this is something that should be discussed publicly, which I don't know, 50-80% of them should be discussed publicly, say, go to the

mailing list and send them to the mailing list. Please do this. Don't worry about it. We also ask for a patch to start with. Anyone using these LLM tools, the first step is to ask them to release a patch. Local stupid models will release a patch. Let them do it. And that will give you something to work on. We say politely: " Please send us a patch." This way, you will receive full recognition when the problem is resolved. And people want recognition. So appeal to their

vanity. This works very well. We also have documentation. We have some money. We have a full- time developer helping us at security@kernel.org. Works with other things related to artificial intelligence tools. I thank Open SSF and Alpha Omega for funding. This works well. We are happy about this. So, as a developer, what can you do? You can resist. If it feels wrong and looks wrong , resist. Network developers are known to always get mad at people, that if they don't put their header files in a

nice Christmas tree-like shape , they ask for a patch. And I would like to know why. And they're like, "Oh, we just want to see if there 's actually a live person on the other end , and if they'll communicate with us ." And that's good. You see this stream of patches coming out. There was no living person behind this. Hmm, resist. Ask for clarification. To ask for something. If they never respond, consider that you don't need to respond either. To resist. Ignore

the marketing of doom. This is not real. They are trying to sell you something. We are not buyers. So don't worry about it. Run local models. Local models are good enough. You can run them on your desktop. You can run them on a separate computer. Run local models. You can say, “Go ahead, do it. Come back in a day. Come back in the morning and everything will be done." Run local models. They are free. You can download them. Run local models. Never, ever upload anything

public. It is not public because at that point it is considered public . So we have so many people saying, “Oh, just run the security feed on kernel.org through our public page or through our model . We won't tell anyone. " Like, yes, no. We do n't do that. You shouldn't do this at all. Don't upload anything you wouldn't want to see online. Just like normal things. I don't know why people don't realize this. It's the same. And correct the mistakes. Just fix the mistakes.

Once the error is fixed, it will not return. If you find it, if it really looks like it's a bug that these people sent you , grab the patch. And people are taking patches. Look at the number of CVEs. This figure is not decreasing. Show it in a minute. But fix the mistakes, and eventually we will overcome this. We will work on this, as we did with those who don't understand. This is not a big problem. I'm probably saying this too sadly

, but we'll get through this. And ignore whatever happens tomorrow. They are trying to get you to sell something. Use what we have today. What we have today is good enough. This works quite well. If you want to use it, if you want to ignore it, but use it today. There is a whole document. Open SSF has published it so you can send it to your manager. Say, “ Hey, here’s how you can look at this type of thing.” And here are the

problems associated with this." It's actually quite good. Um, and how to do it yourself. There will be a whole conversation about Shishiko tomorrow. Run it locally. Run Shishiko locally. This is our bot that performs code checks. Um, I've been known to submit patches that Shishiko finds bugs in, and I'll just run it locally. This is all better. Um, Chris Mason again, as John said, we have now documented how to do code review very well for different subsystems. Do your part in this. This works very

well. Shishiko attracts them. Um, hire an agent. These are all open source tools. Run things locally. Um, they're just babysitting around a stupid chatbot. That's all these things are. That's all there is. Um, again, Chris Mason wrote one for kernel development. Takashi from alpha sound subsystem wrote a bunch of other stuff. I should post his list here. Hmm, this all works really well. Nothing bad. Um, grab local access to the model, load everything, run locally. Please never upload anything to any of these companies. And I'll

turn to Willie, he did some great documentation and described how to do this and actually solved a bug I had where I couldn't figure out how to get something to run. This worked very well. So please take a look at this. So we can go back to this old chart. This is where we were with version 7.2. Now, CVE kernels appear in reverse order, you know, when they do. And I made a diagram of where they come from. It is quite flat, except for 5.15.

And 5.15 stands out for where the bugs came from, because someone thought it was a really good idea to add an SMB server to the kernel. It wasn't like that. But I give credit to these developers. They are working on it. They fixed the bugs and everything. They have fixed the most CVEs for their subsystem than anyone else. This is a good merit. Hmm, but here we are after three weeks of development. Um, these numbers are n't slowing down. Um , but once we

get through this, we'll be fine, I think. We remove code, we remove routines...we remove protocols from networking products that no one uses anymore, we remove drivers. It is known that one of the interns, the graduate students that I found, tried to fix the driver and the subsystem manager came back and said, "No, no one uses this anymore , just delete it." And then he wrote me an email saying, "Oh, no, no, that didn't work." I replied, “You just removed about 3,000

lines of code from the kernel. This is good work." You fixed more bugs, so they were happy. Um, you should teach people that deleting code is okay. This will be an important matter for some time . It's going to be a tough 18 months. Um, someone called me about this. I say 18 months from March. Um, maybe it's 12 months, I don't know. But we can get through this. We went through this for losers. We can get through this with these tools. Um, and

once we do that, we'll come out on the other side and our code will be better and it won't be a problem. And that's all I have. Thank you. No one screamed during this. Wow. Come on, Willie. Um, there's something unfair about the current situation, namely that LLMs hosted by large companies just bombard developers with a bunch of text. And we all know that's painful to read. The only way to protect against this, honestly, is to just use another LLM, analyze walls of text, and that's

completely ineffective. As you said, there is a risk of leaking certain information, some confidential information, to the outside. Not everyone can host a local LLM, because it requires at least a moderately powerful graphics card, and at the current price of RAM, this is not easy. And we also lose a lot of information. We have both read significant meter reports and we know how painful it is. You have the entire report in one line of 4000 characters, etc. It's terrible and a lot of information

is lost. I mean, there are probably hundreds of thousands of context tokens before the report is published. Everything is there to just write a patch, and it's still impossible to get it. And we just need to try to invent a patch after that based on the report. This is not always correct, because a lot of information is probably lost, and I think we need to insist one way or another even more that these patches come from analysis, not from a report. Yes,

press hard. I've listened to too many presentations from companies that said, "Here's the pipeline we built for developers." It comes out here and we do this triage and we do this." And at the very end, because we protect our developers, our close developers , we only give them really good stuff. I'm like, "But no one does this for us ." They just give us raw data, and no one cares about us . It's hard, but there are some groups, like the Linux Foundation group, that

are trying to call it a bunch of these reports. They have been known to help me with Company A. 100 bug reports and only three of them are actually valid. So some people are trying to realize that you can't just throw these walls of text at us, but if you have a wall of text, resist. No one will object to this at all. This is the only thing I want to say to everyone, you have the right and I strongly urge you to do so.

Resist them. This is an asymmetrical relationship , and we need to confront it. And besides, I would say that every time we ask LLM to try to create a patch, that's the best moment when he realizes that, uh, it's not a bug. So, if you need to generate ... That's why I said, " Here's the report and make a patch." And if you create a patch, it cuts out a whole bunch of garbage. So if you just get a bug report, uh,

we see a lot of people just sending bug reports to public mailing lists. The first thing I'll say is just send me the patch. Because I have enough patches to review now. Greetings. Uh, thank you for such a light, uh, presentation in the middle of this room. Uh, I have a question as a recent new member, and I know many other, uh, LFX members who I think are encouraged to, uh, start by making small corrections , which are sometimes very similar

to their contributions to the LLM. As a developer, whether it's for security, drivers, or even staging, do you think these contributions sound like you, and if so, what can we do to actually create a good standing among our developers so that they don't necessarily dismiss these contributions as, uh, some cursory LLM or resume-based contribution? Well , what can you do? So yes, I have banned LLM from driver staging unless you have the hardware and can prove you tested the patch,

because that's not what staging is for . Staging is about learning to develop and participate in our community. Eh, you can indicate if you want to contribute to the patch, and I did because I had to contribute to many different projects since I had access to Mesos. Contribute in the form of a patch, like a human being. Show what you did. Show your work. Show that you are ready to answer. Don't send 100 patches at once. Send one or two. It turns out that the

Kubernetes GitHub repository has two of my patches hidden that I need to merge, and I need to re-do them because they thought I was a bot, and I need to prove that no, it was really a fix. Hmm, just push away. But as a human being, you have to be able to prove that you are human. So, kernel development is all about trust. And if you take the patches , and I don't know who you are, then now I am responsible for

that patch. But the people we trust, then we trust that they didn't necessarily do everything right, but I trust that you'll be there to fix it when you make a mistake, because we all make mistakes . So, you need to build trust. You need to prove that you are a real person. It is known that there is someone who I thought was a bot, and I am going to meet him in Prague. Actually, he's not, he's a real person. Hmm, and that's good. Go to conferences.

Communicate with these people. Be sociable. Respond to their emails. Just reply to the email. That's me...That would cut out a lot of things. Look what's there. So, it shouldn't be that difficult. Just act humanely. Thank you. Not a question, but simply a continuation of the previous question . I know I have about 150 new developers going through my program, the mentoring program. They also send patches. And even with them, I refuse and say, "Show me how you tested this," because they have clear instructions on

how not to use AI, because I don't teach them to use LLM tools to create patches, like intermediate development. They have to do the work because they need to become experts. So, if you see students learning about the Linux kernel, there is a mailing list that sent copies, and so did I. Maybe they are new developers, so you can press them if they don't tell you how they tested it. Yes, we see a lot of patches for Sysbot from people who work with

Sysbot, for example, Sysbot generates a lot of bugs that are not even tested with Sysbot, and they are all obviously generated by LM. First question: can you just write in response, did you forget the " helped" tag? And it's a little tricky, if they answer "yes " or "no", then you can continue. But yeah, it's pretty obvious. So, one of the things that you published is a big changelog that L A I like to just, you know , you know, literally spit out in the

changelog. So what I'm doing right now is my little technique, and I wanted to...I'm not going to be at the developer summit, I wasn't invited, but, uh, I'm asking the people who are going there to please bring this up. If someone gives you this nonsense out there, just go back and say, "Please, you know, write this down to throw this out, give me a summary, write it down to show me what's in there ." And here's what I plan to do now. If they don't

, I'll have to review and read this nonsense, I'll take this patch, remove their signature, put a message for them because they didn't do the job. This shows me that they simply assigned someone else to do the job. They don't deserve a signature, they don't deserve authorship, they don't deserve anything. That's how I feel. I don't know how legitimate this is or what's out there, but I tend to get to the point: if you can't explain what you did, didn't do the work, you

don't deserve to be an author. I don't know. Good. Go ahead, there's a blog post from someone I don't remember called "Don't be a proxy, like a meat proxy between, uh, LLM and the core community." That is, this means that there are cases where people even respond to comments using LLM. Very true. Yes, don't be a meat puppet. Yes, and also regarding the changelog comment. I know there were also people who had a hard time expressing what they did because English is not their first language,

but then they...I agree that the misuse of LLM for this led to it, but maybe it's just... You can see. So, we saw that from Andrew Morton, who said that changelogs for non-native English speakers have gotten much better. And it's completely different when I write it in a non-English language and translate it. This is obvious. This is very obvious. Yes, that's true. So take a look at some of these patches that are coming out, just to get a feel for what these things look like

. I recommend that all developers do this to see these templates, because these templates do not change. They can get around this. They may get better, but this is how they look today. This is really obvious. So that's not a problem. You can make them output smart text. It starts with using a tenth of the words. And you know, the same thing I tell my children is to never use words where silence is enough. Ah, you can, but you can also hide it.

We are not trying to do that. And you can make better change logs. And as my graduate students say, "Oh , just delete that and write it yourself." And that's right, that's... If they're used correctly, they're a great tool, right? They can do a lot of work, but they have no taste. Truth? They can...They can print. They can find the right things and fix many typos. And they are great, especially for building tests. But the number of times I have to write hash integration stdlib.h,

it's been going pretty badly lately. But yes, these are tools. You use them for what they are good at, and that's it . And I mean, I fixed some really funny network errors by saying spin up this virtual machine, take the hell out of it, and show me where something breaks. I mean, you can do some interesting things with these things. And then let them work for a day. On the local machine. Don't leave this out for public viewing. Because it's illegal.

Yes, there is one thing that concerns the very strong pressure on people who do these things generated by LLM, obviously LLM. I get a lot of things that, um, I see very, very clearly...I used this LLM to run the sound on my laptop. And for these people, I like that kind of thing, I'm not going to insist. It's like you clearly did it. Well , you clearly did a bunch of work and then sat there and ran the tests that

LM told you to do. But I'm not going to put that much pressure on people using it in that sense, because I suspect most of them would struggle to compile a kernel for the first time without sitting there with this tool. And while I may have opinions about how much they study, etc., and so on . After all , they have a laptop, the sound works. Yes, everything is fine. Let's take the big fixes. We take these corrections. And it is, but yeah, some of the

comments people made were like you're obvious things LM. Just don't even look at it. But I would just like for things on the periphery, like with the disk, weirdness. I'm like: if it were kernel code, that's a different question. But for something on the edge like this , where, like, I have sound working on my laptop, I see a lot more patches for that. And, um, it's hard for me to be upset about that. And but this... An interesting trend is that the number of

patches that get into Linus' tree, the number of actual bug fixes, is increasing. Yes, yes. Overall, our overall percentage went up a small percentage from 9 to 10, maybe we're at 11 patches per hour now. But this number, which comes out to 33 per hour, this percentage has increased significantly overall in percentage terms. And this ... But this is normal. I mean, we will fix these mistakes. The fuzzers did the same thing . We don't have numbers for when fuzzers appeared because we didn't

track CVEs. But I think there was a big surge back then too. Hmm, I have a question. You said the fuzzers did the same thing . Um, here on kernel recipes there were a few people complaining that syzbot reports were being ignored, or no one had time to look at them. Do you think we'll see the same thing in a few years with LLM, where we have this long tail of things that nobody wants to look at and that may or may not have an

impact on safety? Yes, they will see. I mean, I ignore the cis spot reports. And if it's okay...I mean, there needs to be someone who is responsible for it and takes care of it. I mean, I look at them and think: okay, cis spot is doing something really stupid as root . I don't care . Or not. But I mean, it's up to you, as a developer, to decide what you want to prioritize and what you do n't. But now people are sending cis

spot reports to LLM and waiting for a patch. And regardless of whether this patch is correct or not, I mean they're doing a crazy overhaul of the USB gadget subsystem right now. They're getting a whole bunch of new contributions. People never run this code . But some of these patches are good. And I will take them. Because they actually help make Android more secure. So overall it's good. But this is significant... It creates an additional burden on us as developers. And I

want to point this out, and we know this is happening. I hope this will eventually end. Hmm, but all these companies owe us a lot of alcohol at the end of this. Or anything to drink. I hope you are ready . Hmm, hello. We were recently working on code for a doctoral dissertation. It was very easy for anyone who wanted to apply to just provide a thesis, and I don't know some relevant patches, and create something. So, one thing that really worked for us, and

I think it's because we're open source, we already helped with tags. How about starting to demand accountability for the use of AI? For this project, we have introduced reliability ratings of one, two, and three. Three means you tested it. Two means you've read the documentation. The first one is that you simply read the result of what the AI bot generated for you . Um, and a publicly available chat history of how you used AI. Do those two things, when we implemented, uh, usage

changed dramatically, and we had a lot more, uh, trust in the people participating. So, it works pretty well for this project, and it might be something worth considering for the kernel as well . Yes, but that's good. I-I will point out that our scale is unlike anyone else's . And the number of patches and work we need to do. So we must believe that these things will come. We can finally get people to say with the help of LLM. And I will accept it. So,

I mean, we have four 5,000 developers. How many new developers are emerging every week now? We have a different scale of types of things. So I don't want to look at anyone's chat logs of this type. I'll look at the code myself. You don't have to view it, but the person posting knows that someone can view it. Maybe people don't even know they need help. Unfortunately, let's take baby steps. I agree, that would be great. And if you can provide this in your subsystem

, please do so. MM might want to do this for what you're doing, but making such a blanket statement to 700 developers would be difficult. We can't... all the developers and kernel developers can't agree on anything except that we want to see Linux succeed. We cannot agree on licenses. We can't agree on anything else. That's all we can agree on . Let's stick to this. Thank you. I was wondering if we see for RC the fact that people test RC less than releases. Actually, do

we have such a bias...uh... No. Do people test RCs less than releases? I do n't know. We mean that we are testing Linux Next. We do a lot of RC testing. Uh, it's...I don't think our development model has changed much. Hmm, I see that the number of patches hitting the stable trees is huge and people are panicking about it . But if you still look at the intersection of what you actually create and care about on your device with what actually makes an impact, it's still

very small. So as far as I know, for a specific Android device, out of all those patches that hit the stable trees every week, 10 of them are applied to your tree on your device, which is a small, very small percentage. That is, there is a long tail of drivers that are not really used much. And that's where most of these things hit the mark. Yes. I don't know about testing, but maybe it's too early, but I'm curious about your perspective on performance. The dust

has yet to settle, uh, but do you think we're becoming more productive? There is news, for example, that companies are actually becoming less productive thanks to AI agents. And there are two perspectives also on performance, because we clearly see things like CVEs or some bug fixes, although we don't know how much of an impact they have in practice. But as you know, Android has a lot of patches out of the box. So many people have problems outside the tree that need to be fixed.

Are they really being resolved? So is this part of the generated volume part of that? Because for me, that's what's really important. Because you're really helping products ship with things that the kernel can't handle today, like out-of- tree drivers, drivers for intermediate processing. I have n't seen any activity, but again, it's too early, but I'm really interested in your perspective on this part of the performance. I mean, measuring performance is similar to measuring quality. You know it when you see it. It's hard, and people

spend entire...There are entire doctoral programs to try to do it. I'm not going to claim that. I don't feel more productive. I have to process more patches, but that doesn't make it any easier. I don't think this is a good use of my time. As Konstantin said, the tools are easier to use, he makes them really easy to use, and I will say that I now use them before preparing and submitting. I was embarrassed when my son told me to do it. He replied, "Dad, you

can make sending patches a lot easier." Yes, I'm still learning new tricks. Thanks to our tools, I can submit patches more easily. Does this make me more productive or not? I don't know. It's not LLVM at all, but I'm not going to make that decision. This is marketing, this is a marketing ploy. Managers for measurement. We are an open source project. I want to fix bugs, make Linux work . Yeah, I'm just wondering, as a community, do we see more features and things like that ?

You know. How do you...Yes, how do you... I haven't seen it, but I'm just curious what you think. No, we're seeing new features like fixes to the build system that, you know, allow these tools to work really well, and Miguel will talk a little bit about that in Rust and things like that. So, I mean, there are some cool things you can do with them , but is it worth the trillions of dollars they spent on it? Hey, I do n't know. But we'll

take it. Well , I would like to add something. Uh, we've seen in the past, uh, some developers who were a little scared or at least confused by their first security report. And I'm sure that today, since we documented it, some developers are getting them directly without the security team involved to reduce our workload. And that's great. However, if they're embarrassed or something, if they don't know how to, uh, deal with poor quality, or if they, uh, need help, they definitely need to reach out

to the security team for help . And this is something that we have n't documented, and I think we should do, because these models attack any subsystem. In the past, they were always the same, tracked by security researchers, but now that's not the case. They are looking everywhere, and I'm sure many ordinary drivers or those who work two weekends a year will receive a report without knowing how to deal with it . Truth. Yes, also because we're spreading this out more, people with

security aliases seem to be doing less work, which is good. We have other work to do. And we also have a full- time person who helps us. So we are here to help you guys if you need any help. Please let us know, but I'll call it network specialists, and network specialists have been inundated with this stuff. And it seems like it's slowing down. So it's nice to see that. Hmm, I don't know. Let's see what happens. Last question? Last question. Sorry, I went too

far. I'm going to your break. So I'm not worried about the number of errors found. I'm worried about the number of errors that are appearing now. Um, I think a useful metric might be the number of bugs fixed in one release that were introduced in the previous release , or do we have those graphs. Do you want this schedule? Um, I plotted it as a graph. It is quite flat. That's exactly what I said. 515—This is an exception from the SMB server. Um, that's a

nice curve, and we have these numbers. Case Cook gives a talk on this topic. I haven't seen an increase in the number of bugs added yet, but that's, I mean, our code base is so large. It will take some time to see this in history. Should there be a reduction compared to Shika? There should be a reduction compared to Shika, but Shika thinks, "Oh, and this file also has those old errors." And so we review and fix these things. You said we

saw the schedule. Do you mean the chart from John or Ale, the thing is that it includes all the bugs from a long time ago, but it would be interesting to know all the bugs that were fixed, for example, in 7.1, that were introduced in 7.0 , not before... We have, I mean, it's on this chart. Is that true? Yes. Oh, I missed it. And it is flat. Well, it's an exponential curve. I mean, we always fix newer bugs. There is a curve at the end

. And there are a bunch of them at the end , right? Yeah, I mean, you talked about it a lot. And Case Cook also did a great report on this. Yes, talk to John about it. So, in any case. Thank you for your... Thank you very much. And thank you for the question.