146 Commits

Include renames

Author	SHA1	Message	Date
Marius CiepluchandClaude Opus 5	b5d3cd6ab6	docs(roadmap): a v0.1 release definition, ordered toward libpng and WordPress ... The named release targets are, today, exactly the two cases that do not work -- and they fail for different reasons, which is what makes them a good pair rather than one. WordPress fails because `taint_specs(language="php")` returns zero families: Joern's php2cpg parses it and we say nothing. That is the shortest path from here to a named target, and the case this arbiter suits best, because PHP web bugs are the classic taint families. libpng fails because c/memory models string functions while libpng overflows through arithmetic, and its library never calls a sink we model. That needs either a second abstraction -- size arithmetic, a constraint problem -- or an honest stated limit. So "ready for" means two different things and the criteria say which applies where. Where a family can decide the class, find the known vulnerability. Where it cannot, run inside the envelope, report nothing false, and STATE the limit. The second is not a consolation prize: a tool that says "I do not model this class" is usable, and one that silently finds nothing is what teaches developers to distrust SAST in the first place. Seven v0.1 criteria, each tied to a requirement that already exists: unattended runs inside a stated envelope; zero known-false entailed findings on every pinned checkout; the known vulnerability found wherever a family can decide it (Req 4.6); every family's limit stated in the report (Req 5.6); a proportionate report (Req 5.5); installs without the user having Joern, a JVM or PHP; and a committed regression baseline per target. That reorders the middle of the roadmap. PHP moves up, and the c/memory decision moves up with it, because criterion 4 needs that family either earning its place or being scoped -- and scoping it is cheap. The vibe-code repository stays valuable as the guard against tuning everything on vampi, but drops below PHP because it is not a named target. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:06:11 +02:00
Marius CiepluchandClaude Opus 5	9a65c998b4	docs(steering): an overview of what is true, and a roadmap ordered by one test ... Fourteen specs have accumulated and the roadmap still described a pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI stub", "Joern-as-required deferred" -- which is stale enough to mislead someone reading it to decide what to do next. It is kept as roadmap-2026-09-04.md rather than deleted. **overview.md** is the synthesis above the specs: what the tool is, how one pass works, what has been measured, and what is not true yet. Every number cites a committed artifact. The thesis is stated plainly -- the differentiator is not the dataflow engine, since CodeQL and Semgrep have one and developers do not turn it on, but the rung: a claim you can act on without writing a query, and an explicit admission when the engine could not decide. It says what does not work as clearly as what does. The C story is structurally broken, not undertuned. One language is demonstrated. PHP, Ruby and Go have zero families, so Joern would parse them and the engine would say nothing. The evidence base is two repositories and one has never produced a true positive. Control dependence is unavailable in both CPGs. execution_confirmed has no production caller. And it records the lesson that shaped the current specs: nine defects found by pointing the tool at two real repositories, three of them the same bug in three matchers, with the three gates byte-identical through every fix -- so a corpus of function-level pairs could not have caught any of them. **roadmap.md** is ordered by the standing test, that the tool must find bugs or it is SAST noise. Which gives the ordering principle: make the output worth reading, then make it find more. A better arbiter behind an unreadable report changes nothing, and a second language before the report is fixed doubles the noise. It also says what is NOT being worked and why -- authorization-obligations and reachability-flow-model were largely delivered by other routes and should be reconciled or retired rather than carried as approved-not-started, which misrepresents both. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:50:46 +02:00
Marius CiepluchandClaude Opus 5	290ecff9de	docs(finding-feedback-loop): requirements approved, design generated ... The design turns on an asymmetry the session measured: two of the three matcher bugs needed no judgement at all -- once the first was understood, the others were the same fix carried across. So the automation that pays is propagation and memory, and the model is confined to the one step that needs inference: naming what a group of counter-examples has in common. Two prohibitions are enforced by tests rather than by convention. No module under feedback/ may import a writer for semantic/facts, semantic/obligations or ruleset/. The boundary between evolving the QUESTION and authoring the FACTS is the whole design, and a comment saying so is not a boundary. And `evaluation_set(recipe, dismissals, *, split)` has no "all" -- no argument returns both splits, and no code path returns holdout rows during optimisation. The missing argument IS the design: the 2026-09-05 leak was possible because reading the holdout was merely discouraged, and five of eleven candidate shapes were taught by the pairs they then recovered. A dismissal is recorded AT a rung. "Not a defect at suspicion" must not silence the same site when the graph later entails it -- that is a promotion and deserves a fresh look, which is also what keeps a dismissal file from becoming a graveyard. Risk recorded rather than glossed: the evidence base is two repositories, one of which has never produced a true positive. The ledger and the dismissals are useful immediately; the evolution should not be trusted until contributor-scan 2.15 or 5.2 lands a checkout that can produce one. Phase: design, generated, not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:46:10 +02:00
Marius CiepluchandClaude Opus 5	bccc89be52	docs(specs): approve the group-2 amendments, generate finding-feedback-loop requirements ... **Amendments approved and folded into the numbered requirements** rather than left as a proposal appended to the file: - Req 5.5 — a report proportionate to the evidence behind a finding. A rule whose only evidence is a text match is summarised past a stated number of sites; a finding a proposer reasoned about keeps its detail whatever its rung; nothing leaves findings.json, which is the record a baseline is compared against. - Req 5.6 — a family that cannot decide a class states the limit, and no scan presents an absence of findings as evidence of absence. - Req 4.6 — at least one pinned checkout carries a vulnerability a family can decide in principle, so a scan producing nothing is distinguishable from an engine that cannot see. Each carries the evidence that produced it, and the amendment record keeps what group 2 confirmed rather than changed: Req 8's overfitting warning earned its place twice, Req 3.3's honesty counters made an invisible failure visible, and Req 11.2's gates held through all nine defect fixes. Req 5.5 is half implemented. Task 2.11 made pattern matches proportionate; task 2.17 now carries the model layer's own suspicion band, which is 38 of vampi's 60 findings and is an enumerated candidate the graph could NOT decide with a judge's opinion attached -- a pattern match with an opinion, not a reasoned claim, and weaker evidence than the obligations checker's structural output. **finding-feedback-loop requirements generated**, seven of them: a dismissal file the developer owns and the tool never writes, rejecting one without a reason because a suppression list without reasons is a graveyard nobody can audit; a counter-example ledger whose entries are runnable regression cases, grouped by shared cause so a recurring bug is visible as one; a model that proposes a cause into a review queue and decides nothing, validated by reverting the three known matcher fixes and measuring what it recovers; dismissals plus a recipe's known vulnerabilities as one split evaluation set with the holdout unread and asserted; evolution scoped to the judge and residual questions with the fact tables off limits to any automated means; an adoption that reports what it traded and reverts itself when net-negative; and the loop inheriting the corpus blind spot, so its evidence base is repositories and it states how few it has. Phase: requirements, generated, not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:40:45 +02:00
Marius CiepluchandClaude Opus 5	72932ed879	docs(specs): group 2 feedback as amendments, and lift out what was never specified ... Following the Kiro workflow rather than continuing to write tasks. **A workflow error, corrected.** Two task groups -- a counter-example ledger, and per-repository evolutionary prompting so developers can commit dismissals and evolve the harness -- were appended to contributor-scan's tasks.md on 2026-09-09 without passing requirements or design. They are not refinements of scan integration; they are a different feature, and writing them as tasks skipped the two phases that decide whether a feature is shaped right. They are lifted to .kiro/specs/finding-feedback-loop/, at brief, unapproved. That brief states the boundary the design turns on: an optimiser may evolve the QUESTION (the judge and residual prompts, which are how we ask) and never the FACTS (tables, rulesets, the declared policy file, which are what we claim). The closed-loop leak measured on 2026-09-05 is why -- five of eleven candidate shapes were taught by the holdout pairs they then recovered, reporting a -0.059 holdout Youden as +0.059. It also states the acceptance test for the model's role: revert the three matcher fixes and see whether the grouping rediscovers the known causes. **Three requirements amendments, drafted and unapproved**, at the end of contributor-scan/requirements.md: - Req 5 should state what a report may repeat. libpng emitted 198 findings and nothing in the spec forbade it. Task 2.11 fixed pattern noise; the model layer's own suspicion band is still 38 of vampi's 60, and it is the same epistemic class -- an enumerated candidate the graph could not decide and an LLM affirmed. - A family that cannot decide a class must say so. c/memory scored zero true positives on 89k lines of C for a structural reason, and a silent scan reads as a clean bill of health. Worse than noise because it is quiet, and the one failure mode the rung ladder does not already guard. - Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all -- libpng's CVE is out of family and no version of it can be otherwise. It should require one that can. And what group 2 confirmed rather than changed: Req 8's overfitting warning stopped a rule being extended until one repository was clean and forced the access-control work onto pairs nobody tuned against; Req 3.3's honesty counters are what made a budget stopping free work visible at all; and Req 11.2's byte-identical gates held through all nine defect fixes, which says from the other side that the corpus could not have caught any of them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:36:33 +02:00
Marius CiepluchandClaude Opus 5	00e178d69e	feat(reports): a report proportionate to what the engine can say ... contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting that `memcpy` and `strcpy` are PRESENT in a library that uses them correctly throughout. That is the noise developers have learned to skip in every other tool, and shipping it costs the findings beside it their credibility. **Filtering on rung was the obvious rule and it was wrong.** The obligations checker emits at `suspicion` too, so hiding that rung would have hidden an entire detector -- six tests failed and they were right to. The discriminator is evidence_level: on libpng 163 of 198 rows are `static_corroboration`, on vampi 3 of 60 are, and the model layer and the obligations checker carry `suspicion` there instead. That is exactly the distinction the ladder exists to make -- an undecided CLAIM about a site is not an observation that an API exists. So a pattern-only rule is grouped once it has more than three sites, and keeps its full sections under that. One `python-flask-debug` is a finding a contributor can act on and must not be summarised away; fifty-four `c-unsafe-copy` are a property of the codebase. libpng 198 findings, 198 sections -> 39, 161 grouped into three lines vampi 60 findings, 61 sections -> 61, nothing grouped Counted, never dropped: findings.json stays complete because it is the record a baseline is compared against, and the grouped section says the model reached none of these and that its silence means no fact table covered the site rather than that the site is safe. What it does not fix, recorded rather than glossed: vampi still emits 61 sections for a 520-line application, 51 of them reasoned claims this rule does not touch. It fixes pattern noise, not volume. Also records what group 2 established about the three corpus asks, so none is started on a wrong assumption. Joern ships php2cpg but our fact tables have zero PHP families, so WordPress would parse and produce nothing -- fact tables first, corpus second. An older vulnerable libpng will not help for a reason no version can fix. Vibe-code repositories are the best fit and the cheapest route to a second true-positive checkout, because Python and JavaScript are the two languages whose fact tables are complete. And Group 8, for per-repository state a developer owns: a committed dismissal file the tool reads and never writes, dismissals as a split evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION -- the judge and residual prompts -- never to the fact tables. That boundary is the whole design: a prompt is how we ask, a fact table is what we claim, and the closed-loop leak measured on 2026-09-05 is why the distinction is not negotiable. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:29:26 +02:00
Marius CiepluchandClaude Opus 5	d232f03597	feat(mcp): the findings list carries the rung, and leads with what was decided ... The stated final objective is the rung plus an LLM way of diving in without being forced to rethink the code. The second half is largely plumbing, because the material already exists: the witness (`username -> db.session.execute(...) in models/user_model.py (line 73, 8 steps)`) is the reasoning, already written down and already attached to the finding. So this session's recurring shape was worth checking rather than assuming, and it was there. `openultrasast.get_finding` returns the whole record including the witness, but `openultrasast.findings` dropped the rung: an LLM listing libpng's 198 rows could not tell the one the graph decided from the 197 an enumerator proposed without fetching all 198 one at a time. That is exactly the re-derivation the list exists to spare it. The list now carries the rung, orders strongest first, and returns a per-rung tally so a caller can triage before it fetches anything. The brief records what still stands between here and the objective: the witness is one line and the query already walks the path elements before discarding all but the first; the suspicion band will be reasoned about at length by an LLM, which is worse than a human ignoring it; and a family that cannot decide its class must say so, or the model will confidently explain a silence. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:15:54 +02:00
Marius CiepluchandClaude Opus 5	20edd9354b	docs(contributor-scan): what the evidence says, and what it changes ... Spec feedback after group 2, recorded in the brief where the goal is stated rather than in a task where it would be lost. The standing test is that the tool must find bugs or it is SAST noise, and that most developers already have dataflow in CodeQL and Semgrep and never use it. That settles what the differentiator is: not the engine, which they already have, but the RUNG -- a claim the graph decided, reported without configuration. Everything below model_entailed is on trial. The scoreboard, kept honest: vampi finds 3 of 3 known vulnerabilities at model_entailed, including a second BOLA the engine found before it was in the recipe, with 2 false positives that are endpoints the OpenAPI spec declares public. libpng produced 198 rows and zero valid ones, for a structural reason rather than a tuning one -- the library never calls a sink the c/memory family models. Nine defects this session, each found by a measurement and none by review; three were one bug class in three matchers, two were a capability present but unreachable. The teachable finding is not any one of them, it is that a corpus of function-level pairs cannot see any of them, and every one appeared within minutes of pointing the tool at a real repository. The gates stayed byte-identical through all nine, which says the same thing from the other side. Two priorities change. 2.11, the suspicion band, becomes the critical path rather than cleanup: libpng's 163 "API is present" rows are exactly the noise the quote is about, and no arbiter accuracy survives shipping them. And a family that cannot decide its class has to say so, because a silent C scan reading as a clean bill of health is worse than noise -- it is quiet. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:13:11 +02:00
Marius CiepluchandClaude Opus 5	2589c3d47a	docs(contributor-scan): what c/memory covers, and what SMT covers instead ... libpng's library code never calls strcpy, strcat, sprintf or gets -- all twelve files that do are under contrib/. It overflows through array and pointer arithmetic and memcpy sizing, which is a different question from the one this family asks: "can this expression exceed this allocation" rather than "does tainted data reach this call". That is a constraint problem. It is what SMT is for, and what this project deliberately did not adopt -- the architecture decision was abstract interpretation as arbiter, taken because a model layer can be built from code where execution cannot be assumed. That decision was right for the flow families and it does not reach this one. The maintainer has attacked libpng exactly this way, using SMT to lay out memory and then guiding a fuzzer to the inputs that reach it, which is evidence about the shape of the problem rather than a plan to reimplement it: the honest routes to C memory safety are constraint solving or execution, and the ladder already has a rung for the second. So 2.15 now separates the cheap route (pin a C project whose CVE runs through a string function, giving the family its first true positive) from the expensive one (model the arithmetic, a different engine), instead of implying a sink-table fix would cover both. And 2.16 asks the question this spec's predecessor would have asked: c/memory has scored zero true positives and three separate classes of false one -- a substring match, a bound it could not see, an allocation sized to fit. Every one was a real engine defect and every one is fixed, but the family has never been right. Keep it scoped and labelled, or retire it; either way with a number behind it, and never leaving a silent C scan to read as a clean bill of health. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:11:30 +02:00
Marius CiepluchandClaude Opus 5	c1191b3a6b	feat(model): an access-control finding names a line, and what the PDG does not give us ... contributor-scan 2.7. Taint and config verdicts have always carried a `path:line:function` location; dominance never did, so every access-control finding reached a contributor as `api_views/users.py:?:update_password` -- the right file by accident, because the region was asked about that file, and no line at all. The query now reports the operation's file and the verdict carries the location, with an engine that cannot report one getting no location rather than an invented one. Two questions answered by measurement while doing it. **Does this factor in the PDG?** The data half, centrally: `reachableByFlows` is the entire taint arbiter and `joern-parse` applies the `dataflowOss` overlay. The control half, not at all -- and it is not available to us. Measured: 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in the vampi CPG have a controlling control structure, so `controlledBy` returns nothing in either. Everything guard-shaped therefore runs on CFG dominance or on syntactic conditions, which is exactly why 2.12's first attempt failed: libpng's bound reaches its sink through an `error` flag, and that is a control-dependence fact and nothing else. Recorded as 2.14, including the possibility that it cannot be enabled -- which would bound what guard reasoning this engine can ever do. **Would checking out a vulnerable libpng find a known CVE?** No, and not because of the version. Every one of the twelve files using `strcpy`/`strcat`/`sprintf`/`gets` is under `contrib/`; the library itself uses none of them. libpng overflows through array and pointer arithmetic and `memcpy` sizing, and five of its buffer-overflow fixes touch no modelled sink at all. Pinning an older commit would produce the same silence for the same reason. That leaves two routes of very different size, recorded as 2.15: pin a C project whose CVE genuinely runs through a string function, which is cheap and would give the memory family its first true positive; or model how C actually overflows, which is size arithmetic rather than sink matching and should not be started by accident. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:09:47 +02:00
Marius CiepluchandClaude Opus 5	9221ba8e40	feat(model): a bound on the data lowers the rung, it does not clear it ... contributor-scan 2.12. c/injection and c/memory carried zero sanitizers, so no C flow could ever be discharged: every `argv -> strcpy` entailed however carefully it was bounded. Which sinks can be discharged this way is read from the sink fact's own CWE rather than declared: CWE-121 is a stack buffer overflow, so gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the fact tables and it is C-only, so no Python or JavaScript sink is touched and the pair corpus is unaffected -- gates byte-identical. Not a sanitizer list, because `strlen` appears in the guarded and the unguarded case alike; the discharge is a guard, not a call. **The first implementation required the bound to DOMINATE the sink, and it found nothing.** libpng guards `strcpy(outname+len, ".png")` with `(len = strlen(inname)) > 250`, but that check sits on an else-if chain whose sibling branch skips it entirely, and the protection reaches the sink through an `error` flag: `++error` in the guarded branch, `if (!error)` around the copy. Joern's dominator set for that sink has 89 calls and correctly excludes the bound; `controlledBy` returns none either. Verified against the CPG before the rule was changed rather than after it failed. So the bound is searched in the method, and the consequence is corroborated rather than silence. That is the honest rung: a bound on this variable means the graph has NOT shown the copy is unguarded, and it has not shown it is guarded either -- off-by-one being how bounds fail. The judge is asked. Proving the flag-mediated case needs path-sensitive reasoning this arbiter does not do. libpng entailed 3 -> 1 The survivor is also a false positive: `strcpy(output + len, ".icc")` in iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That is a different discharge shape -- allocated to size, not bounds-checked -- and it is recorded as 2.13 rather than chased. Both shapes were found on one repository, and extending a rule until that repository is clean is the overfitting Req 8 exists to prevent; 2.13 requires a second C checkout with a genuine CWE-121 so the change can be measured for what it silences as well as what it clears. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:01:30 +02:00
Marius CiepluchandClaude Opus 5	6dc25d77e4	feat(model): let the project say what it ships, and rank that first ... contributor-scan 2.10. Every one of libpng's twenty-five entry-point regions was a `main()` -- twenty-three under contrib/, plus example.c and pngtest.c -- and those regions carried the fifty interprocedural taint requests that dominated a 300s query phase. A library has no entry points, so entry-point-driven analysis had nothing to anchor to except its own example programs. The task said to derive this rather than hardcode `contrib/`, and a project already declares it. Makefile.am names installed targets in `lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES` names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own word for "not installed" -- exactly the distinction wanted. Taking every `_SOURCES` would have called libpng's whole test suite shipped code, since contrib/libtests declares its sources the same way the library does. interprocedural regions 25 -> 1 taint query 300.1s -> 125.0s total query 311.9s -> 138.5s regions 121 -> 121, unchanged by design The surviving interprocedural region is contrib/tools/pngfix.c:main, which libpng genuinely installs. Two rules keep it honest. **Silence is not exclusion**: a project with no build file this can read gets None back and nothing changes, so vampi's numbers are untouched. **Not shipped is not unscanned**: those regions are still arbitrated function-locally, because a contributor editing a test must still get an answer -- what they lose is the expensive treatment and first claim on the budget. Entailed findings went 0 -> 3, and all three are false positives at contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe: outname is char[256] and line 325 guards with `if ((len = strlen(inname)) > 250)`. They were not introduced so much as uncovered -- a different callDepth meant a different sink set, and the 26 fgets false positives were masking them. The structural cause is recorded as 2.12: c/injection and c/memory carry ZERO sanitizers, so no C flow can ever be discharged and every `argv -> strcpy` entails however carefully it is bounded. The discharge here is not a call but a guard that dominates the sink, which is precisely what dominance.sc already does for access control and is wired to that one family. Also adds Group 7, five tasks for stopping the same false positive twice. Two of today's three false-positive classes were one bug in three matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in `fgets` -- each fixed locally and each next one surviving because nobody carried the fix across. A model may group counter-examples by cause and draft an edit; only a split-respecting measurement may adopt one, because the predecessor's improve lever admitted shapes taught by the holdout pairs they then recovered and turned a -0.059 Youden into +0.059. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 12:45:21 +02:00
Marius CiepluchandClaude Opus 5	67497c4d68	fix(cpg): a sink is matched as a word, not as a substring ... `fgets` contains `gets`. The sink matcher's last clause was an unbounded `methodFullName.contains(n)`, so every bounded read in libpng matched the sink for the unbounded one. All 26 entailed findings on that repository were the same false positive: *argv -> fgets(pnmline, 256, wpng_info.infile) (line 335, 8 steps) which is the safe API being reported as the dangerous one -- and `fgets` is in the fact tables as a SOURCE, never a sink. Verified against the CPG: methodFullName is exactly "fgets", and .contains("gets") is true. Third instance of one bug class in this codebase, after the sanitizer that matched `resolve` inside `resolveUrl` and the discharger that matched `user` inside `users`. The sink matcher had never been given the same treatment. All four of its clauses now run through the shared bounded matcher, and that matcher is the punctuation-tolerant version written for task 2.6 -- so `input(`, which an unconditional lookahead had made unmatchable in `input(x)`, works for the first time as well. libpng entailed 26 -> 0 Zero is the right answer for a PNG library whose only known CVE is out of family. Gates byte-identical, so no pair outcome depended on the substring match. What remains on that scan is recorded, not claimed: 113 `c-unsafe-*` rows assert an API is present rather than misused, and 50 overlay-coverage rows are annotations rather than findings. 82 of the 113 are under `contrib/`. Task 2.11 covers whether that band is worth putting in front of a contributor at all; 2.10 already covers not scanning example code. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:43:00 +02:00
Marius CiepluchandClaude Opus 5	68b4e551f3	feat(model): the envelope checkout, and no to Joern server mode ... contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform. 89,082 lines of C across 93 files, under a 2GB heap cap: CPG build 16.1s (25s cold) Joern queries 250.0s arbitration 122.0s (105 calls, $0.0122) whole scan 446s peak RSS 911MB regions 121, all arbitrated, 32 unasked CVE-2025-66293 out_of_scope, as the recipe declares Memory is not the constraint a fleet runner would hit: 911MB leaves room for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen while building this was concurrent JVMs on one desktop, not repository size. **Server mode: no.** The query phase splits taint 243.3s / config 10.7s over two invocations, and fixed overhead measured on the same CPG is ~13s per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL evaluation: a warm server removes 6% of the scan and none of the part that costs. The driver now records query seconds per kind, because a single total cannot tell startup from evaluation and the two answer this oppositely. The small repository pointed the other way. vampi's 34.3s over three kinds is roughly three startups and almost no evaluation, which made a warm JVM look worth having; the ratio inverts with size. That inversion is why the task was sequenced after a real repository rather than decided on a 520-line file. On whether one whole-repository CPG is the only shape for a horizontal fleet: no, and it is also not where the pressure is. The build parallelises and costs 16s; the query does not shard and costs 250s, so splitting a repository across runners divides the cheap phase -- and after 2.4 a shard boundary hides exactly the cross-module flows that matter. Scoping to a contributor's diff cuts the expensive phase instead. 911MB for 89k lines extrapolates to roughly 5GB at a million lines, which would exceed a standard runner, so the ceiling is real and further out than this checkout can locate. Two defects recorded rather than fixed, both invisible at vampi scale: 2.9, all 26 entailed findings are the same site reported 26 times, because regions that share a sink each emit their own; and 2.10, 22 of the 30 highest-ranked regions are libpng's `contrib/` example programs rather than the library. Also ignores Joern's `workspace/`, which a maintainer querying a CPG by hand from the repository root will create. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:30:02 +02:00
Marius CiepluchandClaude Opus 5	d15985f695	fix(model): a guard discharges the obligation it is a guard of ... contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers, including three that do authorize and three the OpenAPI spec declares public. Five separate defects, each invisible to a corpus of one-function excerpts and each found by asking the query what it actually matched. **A trailing segment is not a match.** `query.get` was matched by its last segment, so every `.get(` in the repository became a protected database operation -- `request.headers.get('Authorization')` and `request_data.get('username')` included. Four of the seven false positives came from this, and so did the apparent detection of the known BOLA: it was entailed because fetching the Authorization header had no dominating guard, not because of the defect. A qualified token now requires its qualifier. **Configuration is not authorization.** dominance_specs took every obligation fact. `security_setting` operations are discharged by a VALUE, which config_specs already claims them for; pooled in here they made `app.run(...)` an operation requiring an authorization guard and put thirteen tokens in BOTH lists -- an obligation that is its own discharge, the same shape as the taint sink that used to sanitize itself. **Input validation is not authorization.** `jsonschema.validate` was counted as a guard, so handlers with no authorization check at all read as guarded. No operation in the table requires validated_input. **A guard discharges the obligation it is a guard of.** The largest one. `orm-read` requires an identity_constraint or an ownership_check; `token_validator` is a path_guard. Flattened into one list, authenticating the caller discharged an object-level obligation -- a handler that authenticates and then looks the record up by a PATH parameter read as fully guarded. The facts state that relation and the spec was discarding it; it is now carried and the engine asks per operation. **A word boundary needs a word.** Found while fixing the above. Guard matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`. The identity token `['sub']` begins with `[`, and the character before it in `resp['sub']` is `p` -- so the assertion could never hold. Every identity token starting with punctuation was unmatchable, and the identity constraint that separates a correct lookup from an IDOR was invisible. Also: dominance rows were repository-wide, so the model's unguarded `delete_user` was attributed to the API handler of the same name. Rows are scoped to the region's file, which is what the rung's own justification says -- the evidence is a file contradicting itself. Measured against VAmPI's OpenAPI `security` blocks, which declare per operation whether auth is required, and agree with the code: vampi precision 28.6% (2/7) -> 50.0% (2/4) known found 3 of 3, all model_entailed And on the labelled access-control pairs, which nobody tuned this against: all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1% 7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0% Every detection lost was on a pair whose FIXED side was flagged too, which scores nothing. No correct detection was traded for the precision. get_by_title was flagged before it was in the recipe: its code carries the comment "# Broken Object Level Authorization" and the README lists it, so a real finding was being scored as a false positive. It is now the recipe's third known vulnerability. Both remaining false positives are endpoints the spec declares public. Recorded as 2.8, with the reason the mapper's access_level cannot simply be trusted: for decorator frameworks "public" means "no decorator found", which is the bug itself. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:02:48 +02:00
Marius CiepluchandClaude Opus 5	0fcd5feade	feat(model): a region for the code that is in no function ... contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly once, at app.py:17:<module>. Before task 2.2 it was reported eight times in eight files that do not contain it; after 2.2 it was reported nowhere. Two causes, one shape: nothing could ask about module-level code. **The main-guard marker is not a function.** `__main__` is what the mapper emits for `if __name__ == '__main__':`. Sent to the engine as a function name it matched nothing, because a CPG calls a file's top-level code `<module>` -- verified against the VAmPI CPG rather than assumed. The region model translates the marker, which is the one place that turns what the mapper meant into what the scanner asks. **A function-scoped file had no region for its module body.** Settings, route registration and globals live in no function, and every region's query filters by function name. A file-level region beside the handlers would report each handler's findings a second time, which is why the earlier rule dropped it as "a second full-file pass". A module-scope region PARTITIONS the file instead of overlapping it. That only became cheap after batching: before it, a region was a JVM. Two things this needed to be correct rather than merely present: - A module method spans its whole file, so the taint query's line-range containment made `<module>` the parent of every function in it. Under module scope, membership is direct -- otherwise the module region reports every handler's sinks again, which is the duplication the old rule was avoiding. - A module body is not a handler. The marker-derived region inherited the handler families and entailed a missing authorization check on the main guard: a finding with no possible remediation. Caught by the first repository run that had the region, and now covered by a test. bind address 8 wrong files -> 0 -> 1, at app.py:17:<module> regions 22 -> 25 entailed 8 -> 9 query time 37.4s -> 41.3s peak RSS 827MB -> 823MB cost $0.0067 -> $0.0068 Three more regions for four seconds and no more model calls: the budget is a total and was already spent, so regions_unasked rose 17 -> 20 -- those regions got the free pass and are reported as unasked, not as clean. `<module>` is Joern's Python name for a module body; JavaScript uses `:program`, so this is verified for Python only. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:43:53 +02:00
Marius CiepluchandClaude Opus 5	ca7b5ba370	feat(model): a region that spans the flow, not just a function ... contributor-scan 2.4, closing the no-go task 2.2 recorded. VAmPI's SQL injection is a request parameter reaching `get_by_username` in api_views/users.py, passed to `User.get_user` in models/user_model.py, interpolated into a query string and executed there. Source and sink are in different functions and different modules. Regions were per-function, so neither region saw both ends and the engine was silent on the most ordinary shape a real injection bug has. `reachableByFlows` was never the limitation -- it follows calls already. The limitation was the QUESTION. Sinks were filtered to the labeled function's own body, so a crossing flow had no sink to end at. Three changes, each with a reason it is sound rather than convenient: **An entry point is the trust boundary.** parameter_sources was off for repository scans because a repository's parameters are not all attacker input. An entry point's are, by definition -- that is what makes it an entry point. The driver turns it on only for regions the mapper found and named, and leaves it off for walker fallbacks and for named functions that are not entry points. **Sinks may sit below the handler.** taint.sc admits sinks in methods reachable from the labeled method through the call graph, bounded at ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is where these bugs live. Unbounded would make the question about most of the repository rather than about this entry point, and make it unaffordable. **The finding names the sink, not the asker.** A crossing flow ends in another file and another function, so Verdict carries the evidence's own `path:line:function` and the site uses it. Reporting `models/user_model.py:73:get_by_username` would name a place that does not exist: line 73 is inside get_user. Also: labeled methods are now resolved by name AND file. `get_user` exists in more than one module of a real repository, which a corpus of one-file excerpts could never surface. Measured on the same checkout and budget as 2.2: VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now entailed 7 -> 8 query time 40.9s -> 37.4s peak RSS 869MB -> 827MB cost $0.0067, unchanged Asking the bigger question cost nothing measurable here: file scoping and call-graph following pull in opposite directions and roughly cancel at this size. That is one 520-line repository, and VAmPI is now the repository this change was developed against -- the exact overfitting shape Req 8 exists to catch. The depth bound and the entry-point rule mean nothing general until a checkout nobody tuned against says so. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:27:19 +02:00
Marius CiepluchandClaude Opus 5	177c7f3719	feat(model): first repository measurement, and the four things it found ... contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than as excerpts, with benchmarks/repos/measure.py recording build time, query time, arbitration time, peak memory, cost, unjudged regions, and whether each known vulnerability was found and at which rung. The corpus could not have found any of this. Every defect below needs a repository to become visible, and three of them were invisible behind a passing test suite. **The batch path was unreachable.** query_batch was implemented, tested and committed while CpgResult exposed only `run`, so the driver's capability check found nothing and fell back to one JVM per region-family: the first attempt at this measurement ran eighteen minutes on 520 lines, issuing `--param function=` one at a time. Every test called query_batch directly, so none of them noticed. CpgResult now carries run_batch and a test asserts the capability is reachable through the object the driver is handed. The earlier 275s->106s improvement attributed to batching came from region deduplication alone. **The budget stopped free work.** It bounds model calls, but was enforced by breaking the region loop -- spending a paid limit to stop unpaid CPG arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the graph entails for nothing, was never reached. It now withholds the judge only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent because unaffordable is not silent because decided), and the BOLA is entailed by the model layer. **A region with no function asked about the whole repository.** Sending function="" matched every hit in the tree and the driver attributed that answer to whoever asked: one `host='0.0.0.0'` in app.py became eight identical entailed findings in eight files that do not contain it. Queries now carry the region's file. Entailed findings 14 -> 7, query time 70.9s -> 40.9s. The single-request form stays unscoped so the committed pair measurements remain reproducible. **The JVM had no heap bound.** Joern claimed a quarter of physical RAM and runs were killed under memory pressure at different points, so the numbers described the host rather than the engine. -J-Xmx2048m by default, OPENULTRASAST_CPG_HEAP_MB to raise it. Also: arbitrated findings carried `function:line` where the report layer read `path:line`, so every entailed finding reached a contributor with a function name where its file belonged; and the witness line was parsed by splitting on a comma, which produced sites like `config.py:17): permissive literal '0.0.0.0':?`. Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end, 869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the only shape: VAmPI's SQL injection runs from a handler parameter into a function in another module, and a per-function region sees neither end. That is the ordinary shape of a real injection bug, so it is recorded as task 2.4 along with three smaller defects this run exposed. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:46:21 +02:00
Marius CiepluchandClaude Opus 5	8c387306be	feat(repos): pinned known-vulnerable checkouts, offline by default ... The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:10:59 +02:00
Marius CiepluchandClaude Opus 5	c35426e186	perf(model): one Joern invocation per query kind, not one per region ... JVM startup dominated a scan: a call per region per family put a ten-line file at 275s over three regions, and would have put a thousand regions at roughly fifty hours. The queries themselves are milliseconds once the CPG is loaded, so the cost was almost entirely process starts. scan_repository() is now collect -> batch -> arbitrate. The three .sc scripts accept a `requests` JSON map and answer all of them in one run, keeping the single-request form so the committed measurements stay reproducible byte-for-byte. Arbiters are untouched: each is handed a CpgResult whose `run` returns its own prefetched rows. The batcher builds parameters by calling request_params() extracted from taint.py, dominance.py and config_value.py rather than reconstructing them. A second copy that drifts from the arbiter's is the exact shape of the five family-assumption bugs the predecessor spec spent a session finding; there is now one definition per family and the batcher borrows it. Regions are also deduplicated by (path, function), collapsing the nameless entry-point rows that were producing `model:injection:?:10`. Measured on the same ten-line file: 275s -> 106s, three duplicate findings -> one `model:injection:ping:10`, rung unchanged at model_entailed. Gates byte-identical. Server mode is recorded as task 2.3, deliberately sequenced after the repository measurement: whether the remaining ~100s floor is worth a process model, a weakened subprocess seam and a Scala-evaluating HTTP endpoint depends on a build-time-to-query-time ratio only 2.2 produces. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:57:35 +02:00
Marius CiepluchandClaude Opus 5	e67f3ee2f0	feat(contributor-scan): 1.3 the model layer reaches a user ... `model-grounded-detection` finished 16/16 with the model layer having zero production callers. This closes that. A real scan of a Flask file now reports: by_rung: {"model_entailed": 3, "model_corroborated": 0, "suspicion": 0} model:injection:ping:10 <- os.system("ping -c 1 " + request.args["host"]) The stage is the fifth MAP proposer, mirroring _check_obligations, and is additive: pattern, overlay and obligation findings all keep working, and without a CPG engine it records cpg_unavailable and changes nothing else. Two scan integrity tests pin the exact degradation set and both broke -- correctly. Their `requested == baseline` assertion, that findings are byte-identical, still passed, so the only change is that the scan now SAYS the arbiter was absent instead of being silently without one. The guards are updated to assert the model degradation specifically rather than being loosened. ModelLayerConfig, not ModelConfig: that name is already the [models] block, and the type checker caught the collision immediately. Two meanings of one name is how a scan silently reads the wrong settings. Every payload carries the same shape, skipped ones included. A payload that is sometimes a full record and sometimes a single "skipped" key is how a report ends up quietly omitting a section. unjudged_sample names the lowest-ranked regions the budget did not reach. Counting them is the honesty requirement; naming them is what lets a maintainer write a large-repository exclusion list from evidence rather than guesswork. The run also produced the finding behind new task 1.4: 275 seconds for a ten-line file. cost_usd is 0.0, so those 21 invocations were not model calls -- they were Joern query launches, one per region per family, each a JVM start. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:42:44 +02:00
Marius CiepluchandClaude Opus 5	8663c1a5b3	feat(contributor-scan): 1.1 regions from entry points, 1.2 the repository driver ... Pair scoring could rely on a labelled function; a contributor cannot supply one -- which code is risky is the question they are asking. Regions now come from entry points, with a file-level fallback that says it is one, and each carries only the families its shape and language admit. Ranking uses what EntryPointRecord already decided (access_level, trust_boundary) rather than a second opinion computed from the same evidence. A family is offered only where its arbiter could answer: access_control needs a handler, so a library file with no entry point is not asked about obligations it has no siblings to be inconsistent with. The driver makes the two costs that were free at pair scale explicit. ONE CPG per repository, reused across regions -- a build is 5-30s on a forty-line excerpt and per-region builds would make this unusable. And a budget that is a TOTAL for the run rather than the enumerator's per-region cap of eight, spent highest-rank-first, with whatever it does not reach counted into regions_unjudged. A tool that silently analyses a tenth of a repository and reports a clean result is worse than one that says what it skipped. Two flaws the tests caught, both mine and both about ownership: the budget counted calls by reading a client's own usage log, so it silently never tripped for a client that keeps none -- the driver now counts its own spend through a wrapper; and the driver trusted its caller to have sorted, when the order the budget is spent in belongs to whoever holds the budget. The third was worse and only surfaced because a test failed for the right reason: the driver passed candidates=() and never enumerated, so a repository scan would have reported only what the arbiters establish unaided and never populated the suspicion band -- which supplied 15 of 26 findings in the pair measurement. It enumerates now. Sibling audit (Req 8.5) done before writing: family routing lives in exactly one production place, pipeline._arbitrate, dispatching on spec type. regions_for returns family names and the driver resolves specs, so that stays true. 750 passing, mypy clean over 96 files, gates byte-identical, zero orphans. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:28:30 +02:00
Marius CiepluchandClaude Opus 5	64a27c4e30	docs(model-grounded-detection): record what 16/16 does and does not mean ... Asked whether the prior spec is finished. Audited rather than counted, and the answer is: all 16 tasks are complete as written, and not all requirements are satisfied in the product. Four are built but unreachable, so the requirement is not met where it matters: model/report.py implements the per-slice and overfitting-gap discipline (Req 11.1) and has no production caller; resolve_execution_backend returns NullExecutionBackend on BOTH branches so the execution_confirmed rung (Req 10.1) can never be reached; harvest_recipe (Req 9.4) is a seam with nothing plugged into it; and the pipeline that measures 90.0% pair-correct is called only by a measurement harness, never by ousast scan. That is a spec defect rather than an execution one. No task ever said "make this reachable", and Req 11.2 -- the gates stay byte-identical -- is satisfied PRECISELY BECAUSE nothing is wired. I read that as reassurance for a whole session when it was evidence. The spec is not reopened; it remains the complete record of building and measuring the arbiter. contributor-scan takes all four: its Req 1 closes 8.x and 11.1 in the product, Req 7.1 forces the memory decision 10.1 defers, and Req 9 covers the harvest 9.4 left as a seam. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:19:41 +02:00
Marius CiepluchandClaude Opus 5	4e4681f162	docs(contributor-scan): design and tasks -- wire it, then measure on a real repo ... Requirements approved, design and tasks generated. 47 criteria, all traced; 13 tasks in 6 groups. The design is deliberately conservative. The model layer joins as ONE STAGE INSIDE MAP, beside the obligations checker it most resembles -- which already takes entry points, produces findings and merges them. Findings are added, never substituted; the pattern and overlay paths keep working; a machine without Joern scans exactly as it does today. Nothing here is allowed to make the tool worse for someone who does not have the engine. Two decisions carry the most weight. One CPG per scan rather than per region: a build is 5-30s on an excerpt and is the dominant cost at repository scale, so per-region builds would make this unusable. And one total budget rather than the enumerator's per-region cap of eight, which is meaningless across thousands of regions; the budget is spent highest-rank-first and whatever it does not reach is counted and reported, never silently truncated. Task 2.2 is a go/no-go and comes before shipping: the first scan of a pinned known-vulnerable checkout is a measurement nobody has taken, and if one CPG per repository proves infeasible the region and budget design changes shape. Better to learn that before groups 3-6 are built on it. Group 5 (corpus) is last on purpose. The binding constraint today is reachability, not corpus size -- 198 of 238 OWASP pairs went unvendored because volume without readability makes every number worse. The task preamble carries the predecessor's two hard-won habits: audit sibling paths in the same commit as any routing or matching change (five bugs shared that shape, each caught by a measurement rather than a check), and run the whole CI list rather than the parts one remembers (CI was red for a session because `ruff check` passing was taken as evidence for `ruff format --check`). Not approved for implementation. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:17:20 +02:00
Marius CiepluchandClaude Opus 5	14ca0a96ce	docs(contributor-scan): requirements, including the corpus asks I first missed ... 10 requirements, 46 criteria. A NEW spec rather than an amendment to model-grounded-detection, which is complete and closed. Req 1-8 cover the gap that prompted it: the model layer has zero production callers, so ousast scan must use it (1), regions must come from entry points rather than a labelled function (2), ranking and budget must work at repository scale where the enumerator's per-region cap of eight is meaningless (3), the performance envelope must be measured on a real checkout (4), output must be actionable (5), packaging must be built and verified rather than merely defined (6), the two deferred decisions must close (7), and the honesty disciplines carry forward (8). Req 9 and 10 were added after the maintainer asked whether their requests had been captured. They had not been -- I folded the corpus asks implicitly into "run against a known-vulnerable repository", which is not the same thing. Now explicit: a PHP/WordPress slice (php2cpg works once an interpreter is present, which the image provides, and plugin CVEs are overwhelmingly the families the model arbitrates well); real-world C library CVEs -- libpng, image libraries, OpenSSH, libpam, kernel modules -- gated on the memory decision, because until it closes they would join 173 pairs already idle; more vibe-code pairs, since that slice is where the stated audience works and it is only 35 pairs; and regression tuning over a pinned large-repository baseline, which is the thing pair scoring structurally cannot see. Req 9.4 carries today's lesson: verify on a sample that the model can READ a slice before vendoring it. The OWASP harvest found 238 pairs and vendored 40 for exactly that reason. Not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:42:53 +02:00
Marius CiepluchandClaude Opus 5	41266ce472	docs(contributor-scan): brief -- the model layer is complete and unshipped ... Asked what the spec state was and whether anything was forgotten. It was, and it is the largest thing outstanding: model-grounded-detection is 16/16 tasks complete and its arbiter measures 86.7% pair-correct at 6.7% leak, but the model layer has ZERO production callers. `ousast scan` still runs preprocess -> map -> overlay -> obligations -> regress -> report and touches none of it. Every number quoted today came from a measurement harness in a scratch directory. That is a scope gap in the predecessor spec rather than a slip in executing it: there was never a task saying "make this reachable", and its Req 11.2 (gates stay byte-identical) is satisfied PRECISELY BECAUSE nothing is wired. The spec is complete and the feature is unshipped, both at once. A new spec rather than an amendment, because the predecessor is coherent and closed -- it built an arbiter and measured it -- while this is a different question: making it something a developer runs on a repository they are about to contribute to. What it covers, in the order it bites: scan integration; whole-repo scanning (entry-point region selection, ranking across thousands of candidates, a budget policy -- the enumerator caps at 8 per region, meaningless at repo scale -- and a performance envelope never measured); running against a known-vulnerable checkout, since the 176 real-world CVE pairs are single-function excerpts that exercise none of it; the deferred memory decision, with 173 of those pairs idle behind a stub tier; and retiring evidence_level. It also records what was built outside any spec: the pipeline itself, the ops work (one image, compose, shell wrappers, pyinfra -- with the Docker image never actually built), and the corpus additions. And it carries the known limits forward rather than losing them: the flat permissive_values list that cannot tell debug=True from httponly=True, interprocedural resolution as the most repeated miss cause, and the five bugs that shared one shape -- a taint-specific assumption applied to every family, each caught by a measurement rather than a check. Not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:40:38 +02:00
Marius CiepluchandClaude Opus 5	0c71c956fd	feat(model-grounded-detection): group 6, the execution tier as a seam not a build ... The rung this project deliberately does not build. Clearwing already runs sandbox lifecycles, container pools, sanitizer images, PoC replay and stability classification, and the corrective brief concluded -- from this project's own defect history -- that reimplementing that machinery is where we go wrong. So model/execution.py decides eligibility, calls an adopted tool, and records what happened. A test asserts structurally that it imports no container or subprocess machinery and defines no build/run/compile/patch function, because the failure mode to prevent is this file quietly growing a sandbox. Eligibility is narrow in both directions: only a family a static model cannot arbitrate (memory safety in C, where the bug turns on runtime layout -- Req 10.3 forbids any web/logic family depending on this tier, and each now has taint, dominance or constant abstraction instead), and only a finding the model left undecided. The tier escalates uncertainty; it never re-checks a verdict. One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox that does not trigger a bug has not shown the bug is absent, only that this attempt did not trigger it, so the candidate stays where the model left it. Reading a failed reproduction as a clean bill of health is the same error class as reading an absent engine as "no vulnerability". With no adopted tool present -- the normal case -- the rung is never reached and clearwing_unavailable is recorded. All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:58:03 +02:00
Marius CiepluchandClaude Opus 5	86f434ab89	docs(model-grounded-detection): tick 5.1/5.2 and record the closure-scoping fix ... The boxes were left unticked and the notes unwritten when group 5 was committed. Records the closure fix and the three bugs it uncovered, each of which would have raised the number while making the answer untrue, and the correction that req.url was not a wrong diagnosis but an untestable one -- the scoping bug blocked everything downstream of it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:39:13 +02:00
Marius CiepluchandClaude Opus 5	91c0931a05	measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2% ... Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter that makes no model calls at all. Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10, path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the class no crash oracle can reach, arbitrated deterministically. A hypothesis tested and discarded. From the group-2 misses I diagnosed path's 0/8 as a missing req.url source token and said so. The probe -- the same 8 pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real cause is that the sink sits lexically inside an anonymous callback, so its enclosing method is the closure (<lambda>0) and the sinkMethod == function filter drops it; verdict(function="") on the same CPG returns model_entailed. And the obvious fix for that is wrong. Relaxing the filter raises the number with a FALSE witness: the flows found in the closure are `res -> readFileSync` and `this -> ...`, neither of which is the vulnerability, whose real path crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a miss into a wrong entailment is worse than the miss -- it is the arbiter asserting something untrue, the one failure mode this architecture exists to prevent. The correct fix is two-part and recorded: scope a sink to any method lexically nested in the labeled function, and scope parameter sources to that function's own parameters, then re-measure counting only real witnesses. Every miss is classified and none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source unmodelled with the idiom named. Two honesty items in the artifact. 16 of 92 pairs carry no function label and are arbitrated over the whole file, so a verdict can credit a different finding than the labeled one -- owasp-python-hash (CWE-327) is credited for a permissive set_cookie elsewhere in its file, on both sides. And config_secrets conflates three abstractions: permissive setting values (implemented), weak algorithm choice (facts already carry weak_literals, unwired) and hardcoded secrets (needs a credential-literal check). 694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:46:05 +02:00
Marius CiepluchandClaude Opus 5	b8f8f727c6	feat(model-grounded-detection): group 3, the evidence ladder and the judge ... 3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The existing evidence_level carries `static_corroboration`, emitted by the flat IR overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as model_corroborated would claim a CPG arbitrated something it never saw, and rewriting every legacy finding to suspicion would be a product-visible regression nobody asked for. So `rung` defaults to suspicion and is raised only by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4: nothing rises above suspicion without a Verdict behind it. The witness travels with the rung, because a rung without its evidence is just a louder assertion. It reaches markdown, JSON and SARIF. 3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive outcomes: the model entails (report it, never call the model), the model contradicts (drop the claim, record why), the model cannot decide (report suspicion). The LLM gets one bounded typed question with tools=[] so it cannot search for the site, asked only about the residual the graph cannot settle -- whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow the CPG already found; it can never conjure one. Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while loop, or an import from the deleted round machinery. That test first matched the word "votes" in the module's own docstring, which is why it now inspects code rather than text. Verified live against a real CPG, not only scripted rows: vuln rung=model_entailed request.args["name"] -> db.execute(...) fixed rung=suspicion no source reaches this sink model calls made: 0 Both sides arbitrated by the graph with the LLM never consulted -- the thing the prior architecture could not do at any price. 678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:49:55 +02:00
Marius CiepluchandClaude Opus 5	7387d2f190	measure(model-grounded-detection): task 2.3 reads GO on Joern ... Injection slice, 50 pairs, ~30 minutes of real engine time. model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2% ... on minimal-fix pairs only 61.5% (16/26) pairs distinguished 32% overall, 38.5% on minimal-fix determinism (Req 6.4) every verdict identical on re-run cost median 32s/pair, max 58s CPG build failures 1 The engine is not the constraint: all 23 misses trace to our own data model -- 11 sinks absent from the fact tables (Django ORM objects.raw, self.query, GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and one C translation unit that does not parse standalone, counted separately so it can never read as "no vulnerability". That is exactly the result that justifies adopting a CPG rather than building one. The finding that changes the design: classifying what the 50 injection FIXES actually do gives other_or_validation 37, bound_parameters 10, argv_list 2, sanitizer_call 1. The dominant fix is a validation guard on the path, not a sanitizer and not a sink shape -- and taint reachability can never separate those pairs, because the flow still reaches the sink and what changed is that a guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is the same primitive task 4.1 builds for access_control, with the guard vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms collapse into one, and 4.1 should now precede the remaining TaintSpecs. Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs: 12 are different programs (local-python-vulnerable is 7% similar with 83 changed lines) and 12 are large rewrites. A raw distinguished_rate would measure corpus quality and report it as model quality. Also corrected in the notes: the Flask-shaped source table is a real gap but a minor lever -- adding Django patterns would gain a source for exactly one pair. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:26:59 +02:00
Marius CiepluchandClaude Opus 5	1765e51da1	feat(model-grounded-detection): the safe-shape test, without which the model arbitrates nothing ... The first live run produced the finding that shapes this design: taint reachability alone does not distinguish a fix from a bug. On the canonical SQL pair the flow request.args["name"] -> name -> execute exists on BOTH sides -- vuln db.execute("select ... '" + name + "'") one argument, interpolated fixed db.execute("select ... = %s", (name,)) two arguments, bound -- so the model entailed the fixed twin exactly as readily as the vulnerable one, and arbitrated nothing. What separates them is the SHAPE of the sink call, which is precisely what safe_shape_sinks preserved from the earlier sanitizer fix. The query now reports sinkArity and sinkArg0Literal, and a flow ending at a sink written in its declared-safe form is not a vulnerability. Two bugs in that shape test, both found only against the real engine: Joern's `call.argument` includes the RECEIVER at argumentIndex 0, so `db` was counted as an argument -- making a one-argument interpolated call look like a two-argument bound one, the exact inversion of the test -- and the "first argument" whose literalness decides a format-string sink was the receiver rather than the format. Both fixed with argumentIndexGt(0). The rule is narrow on purpose: only sinks named in safe_shape_sinks are subject to it (os.system(x) is not safe merely because it takes one argument), and a row carrying no shape fields is judged on the flow alone -- an unknown shape must never be read as a safe one. End to end on the pair, deterministic and with no model call: vuln -> model_entailed request.args["name"] -> db.execute(...) (line 4) fixed -> None Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:51:42 +02:00
Marius CiepluchandClaude Opus 5	9ed0e45c5f	feat(model-grounded-detection): group 1, subtract the noise architecture ... 99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16 modules, ~4,100 lines) and six semantic/ modules removed, with the three gates byte-identical to 2026-09-06-gate-baseline.txt at every step. The sequence is what made it safe: vocabularies ported first (1.1), importers detached second (1.2-1.3), deletion third (1.4). Every import break during the deletion was therefore a detachment miss in the importer and was fixed there -- no module was restored to satisfy one. Both misses were the same shape and were fixed as a class: operations.py and shapes.py each imported trailing_name from the deleted variants.py, so it moved to semantic/ir.py, its natural home, and both importers changed in one edit. Ported before deleting: the closed guard vocabulary (verbatim, nothing else holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from the retained ruleset facts rather than copying them. Re-homed: families -> model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint, plus Region, labeled_family and labeled_spans. labeled_family no longer falls back to the LLM classifier -- an unlabeled pair is honestly unknown. Four deviations, each recorded in the spec notes with its reason. improve/ evolve.py and validator.py were stripped, not deleted: each mixes the mechanism lever (gone) with the deterministic rule-status loop `ousast improve` still runs, which has no LLM in it; a test now asserts the lever's absence by symbol so it cannot creep back. The learning CLI group became `model candidates`, because deleting it outright would have orphaned the retained enumerator by removing its only entry point. thinking moved from [learning] to [models]. A gap named rather than papered over: no retained sink fact routes by CWE to path, output_encoding, untrusted_destination or prototype -- the same hole the entailment ceiling diagnosed. The test asserts the seeded set and the gap set exactly, so task 4.2 must update it when it authors those sinks. The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness with no production caller and no CLI, alive only through its own test. The five named subsystems each carry an explicit decision. Manifest committed, with a test asserting the tree still matches it. Group 2 (the Joern go/no-go) has not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:15:54 +02:00
Marius CiepluchandClaude Opus 4.8	fc4dab97fb	docs(model-grounded-detection): attach Req 7.4 (verifier boundary) to task 1.1 ... The prior commit's edit did not match and 7.4 was left unclaimed. model/specs.py is read-only data -- no LLM and no optimiser writes it -- and task 1.1 now carries that criterion with a test asserting the module exposes no writer. All 49 criteria trace to a task. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:47:29 +02:00
Marius CiepluchandClaude Opus 4.8	6fe8682993	docs(model-grounded-detection): 16 tasks in 6 groups, with a coupling map so Opus does not crash-discover ... Design approved. All 49 criteria trace to a task. The tasks carry an agent-guidance preamble aimed squarely at the escalations this session hit: the eighteen-defect lesson (fix the class, not the instance), the gate-baseline as the safety net (the whole learning/ package is off the gate path, so its deletion cannot move a gate -- a moved gate means you touched a retained path), the zero-dep capability-detection pattern to copy verbatim, and -- most important for the subtraction -- a full coupling map naming every touchpoint of the removed code (cli, pairs hunter path, config, reports, semantic seed/loo, the mechanism loop across improve/ and semantic/). The subtraction is not discovery by import error. Group 1 is pure gate-protected subtraction, sequenced so vocabularies are ported (1.1) and importers detached (1.2-1.3) before anything is deleted (1.4), then tests reduced in the same spirit (1.5) and the tree audited for orphans (1.6). Group 2 is a Joern go/no-go: task 2.3 reads go or stop, and groups 3-6 do not begin until it is green -- building past a red 2.3 is the mistake the whole spec exists to prevent. Parallelism is deliberately absent; the subtraction is sequential and the spike gates the rest. Not approved for implementation. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:47:10 +02:00
Marius CiepluchandClaude Opus 4.8	0a8cc9b5a7	docs(model-grounded-detection): design, subtraction first and a Joern spike that can stop it ... Requirements approved. All 49 criteria trace to a component. The shape: a Code Property Graph (Joern) is the arbiter; taint reachability and guard dominance produce evidence-ladder verdicts (suspicion / corroborated / entailed / execution_confirmed); the LLM answers one bounded question per candidate and its claim is checked against the CPG, never averaged. Two design disciplines carried from the requirements: Phase 1 is subtraction, and it removes ~5,000 lines before a line of Joern is written. The learning/ noise architecture is absent from the gate path, so rounds/acceptance/proposer/journal, the K-run/floor/sign-test in scoring, and the mechanism loop delete without moving a gate -- proven by the committed gate-baseline run before and after, and by an import-graph check that fails the build if a removed module is still imported. The guard and sink vocabularies are ported into model/specs.py *before* their modules are deleted. Tests are reduced in the same change, with a test->requirement map. A module-audit manifest classifies all 99 modules and names fusion/mcp/skills/harness_ext/ hunter_harness for an explicit keep-or-remove decision. Phase 2 is a Joern spike on the injection family alone -- the CpgBackend seam (subprocess to joern-parse + a CPGQL query emitting JSON, out of process, one isolated touch-point) and one TaintSpec, end to end, with a determinism test. It can end the design before the integration cost is paid, the same way the candidate ceiling could have in the prior spec. Joern is capability-detected on PATH like the semantic extra; without it every verdict degrades to suspicion and the core stays zero-dependency. What survives re-homes under model/: the taxonomy, the candidate enumerator (suspicion band), the endpoint client, and the obligation facts feeding the dominance queries. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:39:49 +02:00
Marius CiepluchandClaude Opus 4.8	1ef578f793	docs(model-grounded-detection): requirements, subtraction first ... 11 requirements, 49 criteria. The outgrowth is addressed head-on and it leads, because building on a superseded 99-module / 832-test surface is how this session logged eighteen defects. Req 1-3 are subtraction and come first: 1 remove the noise architecture by name -- rounds, acceptance, proposer, journal, the K-run/floor/sign-test in scoring, the mechanism lever and shape-search loop -- preserving the security *models* they carried. The whole learning/ noise layer is absent from the gate path, so it deletes without moving a gate. 2 reduce the 832 tests to what protects surviving behavior: a test that pinned a deleted mechanism dies with it, every survivor maps to a surviving requirement, and the "does nothing / greps --help / passes with the capability absent" tests are removed outright. 3 audit all 99 modules -- load-bearing / standalone / orphaned -- and delete the orphans, with fusion/mcp/skills/harness_ext/hunter_harness named for an explicit keep-or-remove decision. Req 4-8 adopt the CPG arbiter: Joern behind an optional extra (core stays zero-dep, degrades to suspicion without it), the evidence ladder as the verdict type, deterministic taint-reachability and guard-dominance verdicts, our taxonomy and source/sink/sanitizer/guard models ported to CPG queries, and the LLM reduced to a proposer whose claim the model checks -- no K-runs, no floor, no acceptance rule, no evolve loop. Req 9 makes the corpus the model's calibration set (a pair the model can't split is a named model gap). Req 10 keeps execution as a deferred, adopted-from-Clearwing tier for memory/C only. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:36:58 +02:00
Marius CiepluchandClaude Opus 4.8	a3945427ab	docs(model-grounded-detection): IL not AST, and adopt the CPG engine (Joern) ... Two design questions answered by the entailment measurement. IL, not AST. Our semantic/ir.py is a lossy AST projection -- flat call/bind lists, no CFG, no PDG -- which is *why* entailment is 2.2%. Guard dominance is a dominator-tree question and path-sensitive taint needs branch structure; a flat bind list expresses neither. Real abstract interpretation is a fixpoint over a CFG, so the substrate is a Code Property Graph: AST + CFG + PDG + call graph in one queryable graph, taint as source->sink-minus-sanitizer reachability, absence as dominance. Adopt, not build. Building a multi-language CFG/PDG/taint engine on tree-sitter is reinventing CodeQL/Semgrep/Joern -- the maximal case of this session's failure mode. Joern is Apache-2.0, covers our exact corpus (C/C++/Java/JS/Python/Kotlin/binary), has a taint engine, is embeddable, and a CPG+LLM MCP bridge already exists. The 2.2% on our flat IR is the measured distance to a real CPG, and closing it by hand is reimplementing Joern. The flow: source -> Joern CPG -> taint + dominance -> deterministic entailed/corroborated verdict -> LLM judges the residual, checked against the CPG. What stays ours is the contribution: the taxonomy and the source/sink/sanitizer/guard models for the web/logic CWEs, ported to CPG queries, plus the LLM-checked-against-the-model orchestration. Joern is a workbench with no opinion on these classes; that opinion is our layer. Named cost: Joern is JVM/Scala, so server-mode or the MCP bridge, not in-process. Three-layer adoption now: Joern CPG for the web/logic arbiter, Clearwing for the memory/C execution tier, and our security modelling + LLM orchestration as the layer that binds them. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:22:44 +02:00
Marius CiepluchandClaude Opus 4.8	9e811c1baf	measure(model-grounded-detection): the entailment ceiling is 2.2%, and it is depth not a wall ... The analogue of the candidate ceiling, one rung up: not "can a site be enumerated" (96.6%) but "can a deterministic abstract-interpretation model DECIDE the vulnerability without trusting the LLM". Offline, no model calls. model_entailed today 2.2% (2/92 web/logic pairs) model_corroborated, one-hop taint 24% (22/92) taint reachable, multi-hop 35% (26/74 flow families) candidate site enumerable 96.6% (for comparison) The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating, and it is the finding. The model cannot be a pure arbiter at v1. But the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed literal sink table (misses project wrappers and path/xss/prototype sinks), and a function-naming gap that eats 20/50 injection pairs. Deepening the taint to the candidate enumerator's depth-4 chain already moved injection 24% -> 48%. So the course is a layered arbiter, not a pure one: report model_entailed where the model proves it (deterministic, no LLM); let the LLM judge but check its claim against the resolved taint on the ~35% corroboration band (noise removed there); honest suspicion for the rest, which is where the deferred execution/differential tier earns its place. Modeling confirmed as abstract interpretation; SMT reserved for the memory/C tier; concolic avoided. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:10:47 +02:00
Marius CiepluchandClaude Opus 4.8	091303c7c0	docs(model-grounded-detection): the arbiter is a program model, not a sandbox ... Maintainer correction: execution isn't possible for all SAST targets, and the bugs this project owns -- IDOR, missing authz, injection -- don't crash anything. Execution is the easy arbiter only for memory-safety-in-C, which is not our world. What execution provided was an independent, deterministic arbiter of whether a claim is real -- not the same LLM that made it, not a noisy vote. That arbiter can be built statically, from the code, as a program model the claim is checked against. LLM proposes; the model disposes. Same architecture, same repair of the root cause, no runnable target required. The evidence ladder becomes static and deterministic: suspicion -> model_corroborated (claim consistent with resolved dataflow / a guard its siblings carry is absent) -> model_entailed (the model alone establishes it) -> execution_confirmed (deferred tier, C memory-safety only). model_corroborated and above are deterministic -- run the checker twice, same verdict -- which removes the run-to-run noise the same way a crash would, with no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling), binding chains and obligation checker stop being prompt context and become the arbiter itself. The course: promote those assets from inputs to arbiter, delete the noise layer, make the LLM a proposer the model audits, keep the corpus as the model's calibration set (the fix diff is what the model should see), and defer execution to Clearwing for the memory-safety classes only. Build the arbiter we can build from code; adopt the one we cannot, when a class demands it. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 10:41:07 +02:00
Marius CiepluchandClaude Opus 4.8	81a63cf5ac	docs(execution-grounded-detection): corrective brief that supersedes the project ... Written after five evolve rounds produced zero decisions distinguishable from noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals. The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to average (K runs), floor (noise budget), gate (sign test) and slowly improve (evolve loop). 120 runs produced 120 distinct trajectories -- the variance is the loop's freedom to search, not the model's uncertainty. And the premise is empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the 2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The lever is architecture, not prompt optimization. What the winners share, and Clearwing is the open instance of: the LLM is a judgement layer on an execution substrate, and truth comes from the sandbox -- a crash, a sanitizer report, a patch that stops the crash -- not from a score against a labeled corpus. Clearwing's poc_runner (apply diff, recompile, re-run, does-the-crash-still-happen) replaces our entire noise architecture. We built K-runs because we had no ground truth; they generate their own. The course: adopt Clearwing as the substrate (the maintainer already forks it), delete the noise architecture rather than fix it, and contribute the one thing Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator (12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker reach the dangerous web bugs that never crash a sandbox, which Clearwing's crash-oracle model cannot. The open research problem is a differential oracle for those classes: the vulnerable-parent/fix-commit pair as ground truth, with a request the fixed version rejects and the vulnerable one serves. Recommendation is explicitly not to rebuild: eighteen defects this session, all ours, against a 19k-line battle-tested MIT substrate. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 10:04:43 +02:00
Marius CiepluchandClaude Opus 5	66a51c674c	docs(learning-harness): round 5, and what it says about the experiment itself ... Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered to the detector as "Worked examples" rather than "Known non-issues, do not report these" -- and was rejected because one pair of eight moved, at p = 1.000. So the experiment I designed to give the demo lever a fair test could not give it one. Correcting my authoring error changed the reason for the verdict and not the verdict, because the rule cannot tell a lever that helps from a lever that does nothing. Five rounds, $2.17, zero decisions distinguishable from chance. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 08:09:50 +02:00
Marius CiepluchandClaude Opus 5	78924316be	docs(learning-harness): the reading of the first evolve rounds ... Four rounds on injection, $1.76. Not one decision carried evidence, in either direction. Round 2 rejected a change because one pair of eight moved, at p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, both readings are what changing nothing looks like. I corrected an overstatement in the notes: after two rejections I wrote that the rule was near-incapable of accepting. Round 3 accepted. The accurate statement is symmetric and worse -- it accepts on noise as readily as it rejects on it. Escalated, not applied: Req 9.5's threshold rule should be replaced by an acceptance that survives the measured variance. The Pareto archive and the sign test both exist and neither decides anything. The honest caveat is that at a minibatch of 8 a reliability-gated rule would accept almost nothing either -- the corpus is the binding constraint, not the rule. And the finding that outranks the rounds, in the maintainer's words: the error is not the model, the error is the elsewhere. Eighteen defects this session, every one in this repository, three of them the same shape of fixing an instance and missing the class. The default assumption for an unexplained number is now that we broke it -- which puts pro's 46.7% against flash's 83.3% on our side of the ledger until shown otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 07:39:48 +02:00
Marius CiepluchandClaude Opus 5	84b1ba7a72	fix(learning-harness): the published table lost a whole slice ... `publish` keyed baselines by model alone, so the vibe-py run silently replaced the agent-vfc run and the table showed hand-written teaching code as if it were the corpus. That is precisely the failure the constrained-detector requirements exist to prevent, live in a published artifact. Third time today that a key was fixed in one place and not another -- `compare_baselines` got the model+slice label and `_baselines` did not, the same way the exception path got checkpointing and the timeout path did not. The table now carries a slice column and one row per slice, and the artifact is keyed by what identifies a run. Republished: agent-vfc's six families are back, including access_control at 0.000 recall on four pairs, which is the number that was missing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 20:44:56 +02:00
Marius CiepluchandClaude Opus 5	78b4f33a8e	feat(constrained-detector): the ceiling, measured offline across every slice ... Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning candidates`, and the stop-or-go reading. No model is called anywhere in this group, so the condition that could have ended the feature was reachable for free. Verdict: go. agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection vfc (deciding) 84.7%, baseline still running, non-gating vfc-js (deciding) 52.9% -- see below github (deciding) 100.0% vibe-py (development)100.0% against 83.3% The margin is largest on agent-vfc, which is the slice the tool is actually for. The measurement forced a shape correction before the detector existed. config_secrets first read 63.6% with a bind-and-literal shape, because a permissive default is as often an argument to a call -- cors({origin: '*'}), app.use(session({secure: false})) -- as it is a binding. Adding call sites to that family's shape took it to 100% and recovered four agent-vfc rows. Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because eight of seventeen rows label a function the JavaScript CST names `<anon>` -- `module.exports.publish = function () {}` and its relatives. That is a function-naming gap, it caps the existing hunter on that slice exactly as much, and it is the same class as the `_declarator_start` defects found in learning-harness task 4.1. The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a measurement rather than a remembered number. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 18:44:19 +02:00
Marius CiepluchandClaude Opus 5	d9d6dfdbcf	docs(constrained-detector): 15 tasks in 5 groups, phase 1 able to end the feature ... Design approved in session. Every one of the 36 criteria is claimed by a task. Group 1 calls no model at all. The candidate enumerator, the ceiling report and the stop-or-go reading are offline, so the condition that ends this feature -- a deciding slice whose ceiling sits below its recorded detection rate -- is reachable before a cent is spent. The ruleset variant of this same design would have failed that condition at 12.1% against 64.9%. Group 2 is the only work that touches scoring.py, and its observable is that nothing changes: the recorded round-zero artifacts are the pin, and if they cannot be reproduced exactly then the refactor is wrong, not the numbers. Group 3 builds the detector, and task 3.3 is the test that would have caught the defect this spec exists for -- two runs must make an identical sequence of model calls. It lives in the suite and runs repeatedly, not in a one-off measurement. Group 4 is the maintainer's emphasis as code: deciding slices, development slices, the cross-slice rejection by name, and the overfitting gap printed beside every headline number. Group 5 compares the two detectors per slice, re-measures the trajectory count and the noise floor, and names -- without amending -- the learning-harness requirements that a collapsed floor would retire. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 17:32:13 +02:00
Marius CiepluchandClaude Opus 5	eaf9e3e370	docs(constrained-detector): design, with a phase that can kill it before it costs anything ... Requirements approved in session. All 36 criteria trace to a component. The shape: a deterministic pass enumerates candidate sites from the semantic IR, shaped by family -- call sites for injection and path, operations with their guards for access control, bindings and literals for configuration. The model answers one bounded typed question per candidate with no tools and no choice of what to look at. The predicates that decide whether a finding counts are applied while it is being made, with one retry carrying the violation, and they come from a single definition that the scorer also uses, so the two cannot drift. Determinism is a design property, not a hope: candidates are ordered by (path, line, name) and batched at a fixed size, so two runs make an identical sequence of model calls. That is the direct answer to 120 runs producing 120 distinct trajectories. Phase 1 is the candidate ceiling, offline, no model call, measured on the real-project slices as well as the teaching one. If the ceiling on a deciding slice comes in below that slice's current detection rate, the design stops there having spent nothing. Building that in as the first phase is deliberate: the ruleset variant of this design would have capped recall at 12.1% against a 64.9% baseline, and only a measurement caught it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:12:35 +02:00
Marius CiepluchandClaude Opus 5	c97dcc656d	docs(constrained-detector): requirements, with real-world code as the decider ... Renamed from detector-optimizer: the brief's own conclusion is that the optimizer is not the next thing to build, and a spec should be named for what it delivers. Seven requirements, 37 criteria. The load-bearing ones: Req 1 -- candidates come from the semantic IR, never the pattern ruleset, and the enumerator's ceiling is published as the limit on recall. Measured before building: ruleset 12.1%, IR 96.6%. Req 2 -- one bounded typed question per candidate, no tools, no choice of what to look at, and a test asserts two runs make an identical sequence of model calls. Req 3 is the maintainer's emphasis and it decides acceptance. The slices harvested from real projects decide; hand-written vibe-py is a development slice that cannot on its own justify a change; a change that improves vibe-py and regresses a deciding slice is rejected by name; and the overfitting gap -- 83.3% against 44.4% at baseline -- is published beside every headline number. The candidate ceiling is measured separately on real-world slices, because their functions are larger and the enumerator's behaviour there is not implied by its behaviour on teaching code. Req 5 -- the K-run and noise-floor machinery is re-measured against the constrained detector and deleted if the floor collapses, naming the learning-harness requirements that would be amended. Not approved; design not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:08:17 +02:00
Marius CiepluchandClaude Opus 5	4e6afa862b	docs: reflect the trajectory measurement back onto the approved spec ... learning-harness divides cleanly. The measurement layer stands and is worth more than before -- it is what made a $0.23 experiment able to answer 'is our own instability structural?'. What was answering the wrong question: Req 3.5's K (an ensemble of distinct search procedures, not a repeated measurement), Req 8.2's noise floor (it measures how often an unconstrained search reaches the sink, so the regression budget grows with the search's own wandering), Req 9.5's acceptance rule, and the sign test added this session. And the highest-leverage change in either spec turns out not to be a prompt, a lever or an optimizer: stop asking the model to locate the sink. 555 grep calls against 71 entry_points, while the CST and the entry-point mapper already know where the sinks are. Limits stated in the brief: twelve pairs, one family, one model; the high-similarity bucket is two run pairs and its 0% is decoration. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:50:44 +02:00
Marius CiepluchandClaude Opus 5	5e2070bc17	measure(detector-optimizer): the detector never runs the same investigation twice ... Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled, recording every tool call. $0.23. The statistic I designed for could not be computed: across 120 runs there was not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side out of 5, and 78 distinct tool-name sequences ignoring arguments entirely. The detector is not sampling one measurement five times; it is running five different investigations and voting on the results. As a gradient the relationship is monotone: runs whose trajectories were similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%, low-similarity 30.9%. Three more facts. The search is grep: 555 grep_repo calls against 181 read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is guessing patterns to locate sinks the CST and the entry-point mapper already know. All the instability is on the vulnerable side; every pair produced exactly one finding set on the fixed side, so the noise is in detection and none of it in false positives. And findings converge much more than trajectories do: 78 trajectories, one to three finding sets per pair. The floor is therefore not measuring uncertainty about a question. It is measuring how often an unconstrained search happens to reach the sink. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:49:54 +02:00