The people spamming curl did step one, "write me a vulnerability report on X" but skipped step two, "verify for me that it's actually exploitable". Tack on a step three where a reasonably educated user in the field of security research does a sanity check on the vulnerability implementation as well and you'll have a pipeline that doesn't generate a ton of false positives. The question then will rather be how cost-effective it is for the tokens and the still-non-zero human time involved.
That was then, this is now. The new models are scarily good. If you're skeptical, just take an hour to replicate the strategy the article references. Point Claude at any open-source codebase you find interesting and instruct it to find exploitable vulnerabilities. Give it a well-defined endpoint if you want (e.g., "You must develop a Python script that triggers memory corruption via a crafted request") and see how well it does.
No, what we were seeing with curl was script kiddies. It wasn't about the quality of the models at all. They were not filtering their results for validity.
It was definitely partially about model quality. The frontier models are capable of producing valid findings with (reasonably) complex exploit chains on the first pass (or with limited nudging) and are much less prone to making up the kinds of nonsensical reports that were submitted to curl. Compared to now, the old models essentially didn't work for security.
If those script kiddies had been using today's models instead and _still_ didn't do any filtering, a lot more of those bugs would have been true positives.
It sounds like what makes the pipeline in the article effective is the second stage, which takes in the vulnerability reports produced by the first level and confirms or rejects them. The article doesn't say what the rejection rate is there.
I don't think the spammers would think to write the second layer, they would most likely pipe the first layer (a more naive version of it too, probably) directly to the issue feed.
* Carlini's team used new frontier models that have gotten materially better at finding vulnerabilities (talk to vulnerability researchers outside the frontier labs, they'll echo that). Stenberg was getting random slop from people using random models.
* Carlini's process is iterated exhaustively over the whole codebase; he's not starting with a repo and just saying "find me an awesome bug" and taking that and only that forward in the process.
* And then yes, Carlini is qualifying the first-pass findings with a second pass.
I guess the broader point I wanted to make is about the people responsible for the deluge of LLM-reported bugs and security vulnerabilities on countless open-source projects (not only on curl): they weren't considerate or thoughtful security researchers, they were spammers looking to raise their profile with fully automated, hands-off open source "contributions". I would expect that the spammers would continue to use whatever lowest common denominator tooling is available, and continue to cause these headaches for maintainers.
That doesn't mean frontier models and tooling built around them aren't genuinely useful to people doing serious security research: that does seem to be the case, and I'm glad for it.
The problem is that you have all kind of "security spam" in the same way that social media is flooded by automatic, but on-topic, content. This doesn't mean that some very few reports are not correct.
One way to filter that out could be to receive the PoC of the exploit, and test it in some sandbox. I think what XBOW and others are doing is real.
To some extent, that’s just the nature of the incentive structure — you pay money for exploits with no penalty for false reports? Just rely on monkeys typing reports till one happens to be correct.
I don't see much motivation for fixing that when I can purchase a nrf52xx Bluetooth Beacon on aliexpress for €4 and flash it with firmware that pretends to be 50 different airtags, rotating every 10 minutes, and therefore bypassing all tracker detections.
First of all this is not technically distillation, it is more imitation learning.
Second, you could do something like asking Claude to create 1 million prompt, offensive response, non offensive response triplets. Then train a model with DPO to prefer the offensive responses.
This sounds sick, I looked for a photo and couldn't find one that seemed to be this specifically, just lots of the flexible ones, do you know if there's a photo ref somewhere?
> why ... didn't help answer this simple question?
How do you know it didn't?
Someone may have observed it and fixed it someplace else, but from the point of OP it doesn't matter to them - their system still works "as desired". The world wasn't meant to be perfect for everyone, but you can certainly carve out your own "perfect" piece of it.
It seems you have only surface knowledge and little practice of using Emacs. You're not offering any evidence to your arguments and not suggesting any alternatives. Because there isn't a practical alternative that exists today to effectively replace Emacs. You're frantically trying to grasp for reasons not to use Emacs. Why are you looking for validation on an Emacs-themed thread? If you don't want to use Emacs, just don't. If you want to use Emacs but observing obstacles, then get more concrete, instead of complaining just for the sake of it.
reply