100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Marius CiepluchandClaude Opus 5	60d2a1a92c	feat(model): an endpoint its contract declares open carries no obligation ... contributor-scan 2.8. VAmPI's last two false positives were register_user and login_user, both declared open by its OpenAPI spec: the arbiter was reporting a missing guard on endpoints whose contract says there is nothing to miss. **The fix keys on provenance, not on the value, and that distinction is the whole task.** The mapper says "public" in two very different situations. An OpenAPI operation with no `security` block is DECLARED open, because the spec is a complete contract. A Flask handler with no `@login_required` is "public" only because nothing was found -- which is precisely the bug this family exists to report. Suppressing on the value alone would have silenced the detector on its primary case, so EntryPointRecord now carries `access_declared` and only a declaration removes the obligation. WordPress supplied the second source of declared access on the same day -- wp_ajax_nopriv_ for a logged-out handler, wp_ajax_ for a logged-in one, permission_callback => __return_true for an open REST route. A rule resting on one framework's convention would be a special case; two make it a shape. vampi entailed 6 -> 4 access-control precision 2 of 4 -> 2 of 2 known vulnerabilities 3 of 3 still entailed access_control offered to 6 of 14 handlers, exactly the six the spec declares authenticated **And the guard that decides whether it is a real improvement:** the access-control pair corpus, which nobody tuned this against, is unmoved -- pair_correct 3 and leak 1 across all nine, pair_correct 2 and leak 0.0% across the seven from projects unrelated to VAmPI, identical to task 2.6. The precision was not bought by silencing the family. Pair excerpts carry no route manifest and no hook, so nothing in them is declared open and the rule correctly does not fire. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 15:39:58 +02:00
Marius CiepluchandClaude Opus 5	3e1d13e4b6	feat(reports): the model's own suspicion band is a proposal too ... contributor-scan 2.17, completing Req 5.5. Task 2.11 made static pattern matches proportionate and left the model layer's own suspicions alone, reasoning that a proposer had reasoned about them. That does not survive inspection. A model suspicion is what the enumerator proposed at a site the graph could NOT decide, affirmed by a judge with no coverage there -- and its own rationale says exactly that: "the model resolved no flow here; its sink table may not cover this API". It is a pattern match with an opinion attached, and it reads like a reasoned claim only because a model produced it. vampi 60 findings, 61 sections -> 26, report 1085 lines -> 446 libpng 198 findings, 39 sections -> 30 **What it costs, which is the number that decides whether the rule is right.** For each known vulnerability the recipe declares, take every finding row at its location and ask whether any survives the rule. All three of vampi's -- the SQL injection and both documented BOLAs -- are still found at model_entailed and all three survive. Nothing known is hidden. Three vulnerabilities in one repository is also a statement about how small the evidence base is, and the artifact says so. The obligations checker sits at `suspicion` too and is NOT grouped: its output comes from a structural analysis, an obligated operation with no discharging guard, rather than from a guess. An earlier attempt at this rule filtered on rung alone and would have collapsed it; six tests failed and they were right to. "Pattern matches" is now "Proposed, not corroborated", since it holds two shapes. Model rows group by FAMILY rather than by rule, because their rule is always `model`. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 15:23:57 +02:00
Marius CiepluchandClaude Opus 5	af7e33ef3e	feat(reports): say what was looked for, and what cannot be decided ... contributor-scan 2.16, implementing the approved Req 5.6. `c/memory` scored zero true positives across 89k lines of C and three distinct classes of false one, for a structural reason: it models string CALLS and libpng overflows through arithmetic. The task allowed retiring the family. Kept, scoped and stated instead -- `strcpy` misuse is real in plenty of C and the family earns its place for that. Retiring it would have removed the disclosure along with the coverage, leaving a C scan silent for a reason nobody could read. **Its own description was the quiet-failure risk.** It claimed "bounds, lifetime, arithmetic", which is exactly what it does not decide. It now names the APIs it models, and a new `limits` field in families.toml says what it cannot: an overflow expressed as an index, a pointer offset, or a memcpy whose size is computed -- which is how most real C overflows are written, and how every one of libpng's is. A limit recorded in a fact table nobody reads is no better than silence, so the report renders it. **What was analysed** lists every family offered to a region whether or not it found anything, because a family that ran and reported nothing is the case a reader is most likely to misread, and it states plainly that silence means no fact table covered the code rather than that the code is safe. The counts are derived from the fact tables the arbiter actually used, so they cannot drift from what the engine looked for; only the limit sentence is authored, because "cannot decide an overflow expressed as an index" is a statement about an abstraction and no table can produce it. A C scan that finds nothing now says: | c | `memory` | 121 | 9 | **`memory` (c)** — Models CALLS, not arithmetic... Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 15:08:01 +02:00
Marius CiepluchandClaude Opus 5	f488e625c2	fix(specs): the php2cpg "blocker" was the shim, not the frontend ... Task 5.5 recorded that php2cpg emits a CPG containing only one PHP file -- the last alphabetically -- and called it a blocker for the WordPress target. Rerun with a native php-cli, as asked, and it is not true. php 8.3.6, no shim, same joern 4.0.623, same two files: files: plugin.php, vuln.php methods: mp_fetch_report, mp_save_report, mp_render_box, mp_helper, get_user, safe_user, show, run Both files, all eight methods, and no PHP-Parser banner warning at all -- the banner that looked like the thread to pull was a symptom of the same thing and does not occur natively. The cause was the workaround. This machine has no PHP interpreter, so the shim ran each `php` invocation in its own container; php2cpg drives PHP-Parser once per file, and that per-invocation isolation kept only one file's JSON. **The lesson is worth more than the fix.** Two independent reproductions agreed, and both were measured through the same broken instrument. A consistent result is not a correct one when every measurement shares an apparatus -- the same trap as a corpus that cannot see the defect it is scoring, which is this session's other recurring finding. And with the instrument fixed, 5.4 is demonstrably done. A two-file PHP tree yields 10 function-scoped regions and 7 entailed findings in 15 seconds: SQL injection in both WordPress handlers, XSS in the shortcode renderer, and all three of the original probe's. safe_user is correctly absent, its mysqli_real_escape_string dropping it to corroborated. The wp_ajax_nopriv_ handler ranks 1.0 and the plain helper 0.3. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:45:36 +02:00
Marius CiepluchandClaude Opus 5	0ffa99b5e6	feat(model): one finding per sink site, and WordPress's registration model ... contributor-scan 5.4. Both halves, and the second found a frontend blocker. **One verdict per region was right for a pair and wrong for a file.** A pair is one labelled function with one bug, so one answer is correct. A region spanning a PHP file is not: the probe's SQL injection on line 5 and command injection on line 13 produced ONE injection verdict, `system` won it on flow length, and the SQL injection went unreported by construction rather than by any failure of the analysis. `taint.verdicts()` now returns one verdict per distinct sink site, strongest first, and its first element is exactly what `verdict()` returns -- so the pair path and every committed measurement resting on it are unchanged, and the gates are byte-identical. Both entry points share `_usable_flows` so neither can drift from the other. php probe 2 of 3 -> 3 of 3 vampi 6 entailed, unchanged, now with lines on access-control findings from task 2.7 (users.py:187, books.py:51 -- the two documented BOLAs at their exact lines) libpng 1 -> 2 entailed; the second was hidden by the collapse. Neither is validated and both are in contrib/. **WordPress registers handlers rather than decorating them, and the hook name is the declaration.** `wp_ajax_nopriv_*` is WordPress saying logged-out callers reach this, so it is public; `wp_ajax_*` fires only for logged-in ones; `permission_callback => __return_true` declares a REST route open. That is the declared-provenance signal task 2.8 needs, arriving from a second framework. Plus every top-level PHP function as its own region, without which a file is one region however many functions it holds: 1 file-level region became 10 function-scoped ones. **And a blocker, recorded as 5.5.** php2cpg emits a CPG containing only ONE PHP file, the last alphabetically. Reproduced twice on independent samples; both files parse cleanly under `php -l` and under PHP-Parser directly, and joern-parse reports success either way. php2cpg logs "Found warning in PHP-Parser JSON output" naming the survivor, and that warning is PHP-Parser's own `====> File X:` banner appearing in what php2cpg expects to be pure JSON. Not isolated yet: this machine has no PHP interpreter, so php2cpg was driven through a container shim. The banner comes from PHP-Parser rather than the shim, which makes a shim-only cause unlikely but not excluded, and rerunning with the native php-cli in the shipped image is the first step. It blocks WordPress outright -- a plugin is many files, and a scan that silently covers one of them and reports success is the quiet failure Req 5.6 was approved for. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:30:48 +02:00
Marius CiepluchandClaude Opus 5	61811512a8	feat(facts): PHP semantic facts, validated against a real PHP CPG ... contributor-scan 5.1, the first step toward the v0.1 WordPress target. Written against a PHP CPG rather than against PHP source, because the two do not look alike. Verified with joern-parse 4.0.623 / php2cpg on a probe: $_GET["name"] -> call `<operator>.indexAccess`, CODE $_GET["name"] mysqli_query($conn, $sql) -> call NAMED mysqli_query echo $_GET["msg"] -> call NAMED echo So sources match on code and sinks match on name, and every entry the probe did not exercise is marked unverified in the file -- `include` and `require` among them, which is PHP's most characteristic path bug and the first thing to check on a real checkout. Five families derive (injection, output_encoding, path, deserialization, untrusted_destination) with no sink that is its own sanitizer -- asserted by a test, because that bug class has now appeared three times in three matchers. `php` is added to the region model's languages, without which the facts would have reached nothing. The arbiter decides 4 of 4 on the probe: the SQL concat entails, its `mysqli_real_escape_string` twin drops to corroborated, `echo` XSS entails and `system` command injection entails. **A whole scan finds 2 of those 3, and the gap is the finding.** PHP files yield FILE-LEVEL regions because no PHP entry points are mapped, and the taint arbiter returns one verdict per region per family -- so a file-level region reports at most one injection finding however many it holds. `system` won on flow length and the SQL injection went unreported. That is the mirror image of task 2.9: that one is many regions collapsing to one site, this is many sites collapsing to one verdict, and it bites hardest exactly where PHP needs it not to, since a WordPress file is large and has no decorator for the mapper to find. Recorded as task 5.4 with WordPress's registration model (add_action, add_filter, admin-ajax, REST routes), which the mapper already handles in principle for other frameworks. A 13-line probe is not a WordPress result. What is established is that the seam works end to end for PHP; everything about scale and coverage is unmeasured, and there are still zero PHP pairs in a corpus of 263 python, 351 c_cpp, 47 javascript and 49 java. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:18:57 +02:00
Marius CiepluchandClaude Opus 5	b5d3cd6ab6	docs(roadmap): a v0.1 release definition, ordered toward libpng and WordPress ... The named release targets are, today, exactly the two cases that do not work -- and they fail for different reasons, which is what makes them a good pair rather than one. WordPress fails because `taint_specs(language="php")` returns zero families: Joern's php2cpg parses it and we say nothing. That is the shortest path from here to a named target, and the case this arbiter suits best, because PHP web bugs are the classic taint families. libpng fails because c/memory models string functions while libpng overflows through arithmetic, and its library never calls a sink we model. That needs either a second abstraction -- size arithmetic, a constraint problem -- or an honest stated limit. So "ready for" means two different things and the criteria say which applies where. Where a family can decide the class, find the known vulnerability. Where it cannot, run inside the envelope, report nothing false, and STATE the limit. The second is not a consolation prize: a tool that says "I do not model this class" is usable, and one that silently finds nothing is what teaches developers to distrust SAST in the first place. Seven v0.1 criteria, each tied to a requirement that already exists: unattended runs inside a stated envelope; zero known-false entailed findings on every pinned checkout; the known vulnerability found wherever a family can decide it (Req 4.6); every family's limit stated in the report (Req 5.6); a proportionate report (Req 5.5); installs without the user having Joern, a JVM or PHP; and a committed regression baseline per target. That reorders the middle of the roadmap. PHP moves up, and the c/memory decision moves up with it, because criterion 4 needs that family either earning its place or being scoped -- and scoping it is cheap. The vibe-code repository stays valuable as the guard against tuning everything on vampi, but drops below PHP because it is not a named target. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:06:11 +02:00
Marius CiepluchandClaude Opus 5	9a65c998b4	docs(steering): an overview of what is true, and a roadmap ordered by one test ... Fourteen specs have accumulated and the roadmap still described a pre-Joern world -- "adjudication is still Python-ast plus a tree-sitter CLI stub", "Joern-as-required deferred" -- which is stale enough to mislead someone reading it to decide what to do next. It is kept as roadmap-2026-09-04.md rather than deleted. **overview.md** is the synthesis above the specs: what the tool is, how one pass works, what has been measured, and what is not true yet. Every number cites a committed artifact. The thesis is stated plainly -- the differentiator is not the dataflow engine, since CodeQL and Semgrep have one and developers do not turn it on, but the rung: a claim you can act on without writing a query, and an explicit admission when the engine could not decide. It says what does not work as clearly as what does. The C story is structurally broken, not undertuned. One language is demonstrated. PHP, Ruby and Go have zero families, so Joern would parse them and the engine would say nothing. The evidence base is two repositories and one has never produced a true positive. Control dependence is unavailable in both CPGs. execution_confirmed has no production caller. And it records the lesson that shaped the current specs: nine defects found by pointing the tool at two real repositories, three of them the same bug in three matchers, with the three gates byte-identical through every fix -- so a corpus of function-level pairs could not have caught any of them. **roadmap.md** is ordered by the standing test, that the tool must find bugs or it is SAST noise. Which gives the ordering principle: make the output worth reading, then make it find more. A better arbiter behind an unreadable report changes nothing, and a second language before the report is fixed doubles the noise. It also says what is NOT being worked and why -- authorization-obligations and reachability-flow-model were largely delivered by other routes and should be reconciled or retired rather than carried as approved-not-started, which misrepresents both. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:50:46 +02:00
Marius CiepluchandClaude Opus 5	290ecff9de	docs(finding-feedback-loop): requirements approved, design generated ... The design turns on an asymmetry the session measured: two of the three matcher bugs needed no judgement at all -- once the first was understood, the others were the same fix carried across. So the automation that pays is propagation and memory, and the model is confined to the one step that needs inference: naming what a group of counter-examples has in common. Two prohibitions are enforced by tests rather than by convention. No module under feedback/ may import a writer for semantic/facts, semantic/obligations or ruleset/. The boundary between evolving the QUESTION and authoring the FACTS is the whole design, and a comment saying so is not a boundary. And `evaluation_set(recipe, dismissals, *, split)` has no "all" -- no argument returns both splits, and no code path returns holdout rows during optimisation. The missing argument IS the design: the 2026-09-05 leak was possible because reading the holdout was merely discouraged, and five of eleven candidate shapes were taught by the pairs they then recovered. A dismissal is recorded AT a rung. "Not a defect at suspicion" must not silence the same site when the graph later entails it -- that is a promotion and deserves a fresh look, which is also what keeps a dismissal file from becoming a graveyard. Risk recorded rather than glossed: the evidence base is two repositories, one of which has never produced a true positive. The ledger and the dismissals are useful immediately; the evolution should not be trusted until contributor-scan 2.15 or 5.2 lands a checkout that can produce one. Phase: design, generated, not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:46:10 +02:00
Marius CiepluchandClaude Opus 5	bccc89be52	docs(specs): approve the group-2 amendments, generate finding-feedback-loop requirements ... **Amendments approved and folded into the numbered requirements** rather than left as a proposal appended to the file: - Req 5.5 — a report proportionate to the evidence behind a finding. A rule whose only evidence is a text match is summarised past a stated number of sites; a finding a proposer reasoned about keeps its detail whatever its rung; nothing leaves findings.json, which is the record a baseline is compared against. - Req 5.6 — a family that cannot decide a class states the limit, and no scan presents an absence of findings as evidence of absence. - Req 4.6 — at least one pinned checkout carries a vulnerability a family can decide in principle, so a scan producing nothing is distinguishable from an engine that cannot see. Each carries the evidence that produced it, and the amendment record keeps what group 2 confirmed rather than changed: Req 8's overfitting warning earned its place twice, Req 3.3's honesty counters made an invisible failure visible, and Req 11.2's gates held through all nine defect fixes. Req 5.5 is half implemented. Task 2.11 made pattern matches proportionate; task 2.17 now carries the model layer's own suspicion band, which is 38 of vampi's 60 findings and is an enumerated candidate the graph could NOT decide with a judge's opinion attached -- a pattern match with an opinion, not a reasoned claim, and weaker evidence than the obligations checker's structural output. **finding-feedback-loop requirements generated**, seven of them: a dismissal file the developer owns and the tool never writes, rejecting one without a reason because a suppression list without reasons is a graveyard nobody can audit; a counter-example ledger whose entries are runnable regression cases, grouped by shared cause so a recurring bug is visible as one; a model that proposes a cause into a review queue and decides nothing, validated by reverting the three known matcher fixes and measuring what it recovers; dismissals plus a recipe's known vulnerabilities as one split evaluation set with the holdout unread and asserted; evolution scoped to the judge and residual questions with the fact tables off limits to any automated means; an adoption that reports what it traded and reverts itself when net-negative; and the loop inheriting the corpus blind spot, so its evidence base is repositories and it states how few it has. Phase: requirements, generated, not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:40:45 +02:00
Marius CiepluchandClaude Opus 5	72932ed879	docs(specs): group 2 feedback as amendments, and lift out what was never specified ... Following the Kiro workflow rather than continuing to write tasks. **A workflow error, corrected.** Two task groups -- a counter-example ledger, and per-repository evolutionary prompting so developers can commit dismissals and evolve the harness -- were appended to contributor-scan's tasks.md on 2026-09-09 without passing requirements or design. They are not refinements of scan integration; they are a different feature, and writing them as tasks skipped the two phases that decide whether a feature is shaped right. They are lifted to .kiro/specs/finding-feedback-loop/, at brief, unapproved. That brief states the boundary the design turns on: an optimiser may evolve the QUESTION (the judge and residual prompts, which are how we ask) and never the FACTS (tables, rulesets, the declared policy file, which are what we claim). The closed-loop leak measured on 2026-09-05 is why -- five of eleven candidate shapes were taught by the holdout pairs they then recovered, reporting a -0.059 holdout Youden as +0.059. It also states the acceptance test for the model's role: revert the three matcher fixes and see whether the grouping rediscovers the known causes. **Three requirements amendments, drafted and unapproved**, at the end of contributor-scan/requirements.md: - Req 5 should state what a report may repeat. libpng emitted 198 findings and nothing in the spec forbade it. Task 2.11 fixed pattern noise; the model layer's own suspicion band is still 38 of vampi's 60, and it is the same epistemic class -- an enumerated candidate the graph could not decide and an LLM affirmed. - A family that cannot decide a class must say so. c/memory scored zero true positives on 89k lines of C for a structural reason, and a silent scan reads as a clean bill of health. Worse than noise because it is quiet, and the one failure mode the rung ladder does not already guard. - Req 4.1 is satisfied by a checkout that cannot demonstrate detection at all -- libpng's CVE is out of family and no version of it can be otherwise. It should require one that can. And what group 2 confirmed rather than changed: Req 8's overfitting warning stopped a rule being extended until one repository was clean and forced the access-control work onto pairs nobody tuned against; Req 3.3's honesty counters are what made a budget stopping free work visible at all; and Req 11.2's byte-identical gates held through all nine defect fixes, which says from the other side that the corpus could not have caught any of them. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:36:33 +02:00
Marius CiepluchandClaude Opus 5	00e178d69e	feat(reports): a report proportionate to what the engine can say ... contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting that `memcpy` and `strcpy` are PRESENT in a library that uses them correctly throughout. That is the noise developers have learned to skip in every other tool, and shipping it costs the findings beside it their credibility. **Filtering on rung was the obvious rule and it was wrong.** The obligations checker emits at `suspicion` too, so hiding that rung would have hidden an entire detector -- six tests failed and they were right to. The discriminator is evidence_level: on libpng 163 of 198 rows are `static_corroboration`, on vampi 3 of 60 are, and the model layer and the obligations checker carry `suspicion` there instead. That is exactly the distinction the ladder exists to make -- an undecided CLAIM about a site is not an observation that an API exists. So a pattern-only rule is grouped once it has more than three sites, and keeps its full sections under that. One `python-flask-debug` is a finding a contributor can act on and must not be summarised away; fifty-four `c-unsafe-copy` are a property of the codebase. libpng 198 findings, 198 sections -> 39, 161 grouped into three lines vampi 60 findings, 61 sections -> 61, nothing grouped Counted, never dropped: findings.json stays complete because it is the record a baseline is compared against, and the grouped section says the model reached none of these and that its silence means no fact table covered the site rather than that the site is safe. What it does not fix, recorded rather than glossed: vampi still emits 61 sections for a 520-line application, 51 of them reasoned claims this rule does not touch. It fixes pattern noise, not volume. Also records what group 2 established about the three corpus asks, so none is started on a wrong assumption. Joern ships php2cpg but our fact tables have zero PHP families, so WordPress would parse and produce nothing -- fact tables first, corpus second. An older vulnerable libpng will not help for a reason no version can fix. Vibe-code repositories are the best fit and the cheapest route to a second true-positive checkout, because Python and JavaScript are the two languages whose fact tables are complete. And Group 8, for per-repository state a developer owns: a committed dismissal file the tool reads and never writes, dismissals as a split evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION -- the judge and residual prompts -- never to the fact tables. That boundary is the whole design: a prompt is how we ask, a fact table is what we claim, and the closed-loop leak measured on 2026-09-05 is why the distinction is not negotiable. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:29:26 +02:00
Marius CiepluchandClaude Opus 5	d232f03597	feat(mcp): the findings list carries the rung, and leads with what was decided ... The stated final objective is the rung plus an LLM way of diving in without being forced to rethink the code. The second half is largely plumbing, because the material already exists: the witness (`username -> db.session.execute(...) in models/user_model.py (line 73, 8 steps)`) is the reasoning, already written down and already attached to the finding. So this session's recurring shape was worth checking rather than assuming, and it was there. `openultrasast.get_finding` returns the whole record including the witness, but `openultrasast.findings` dropped the rung: an LLM listing libpng's 198 rows could not tell the one the graph decided from the 197 an enumerator proposed without fetching all 198 one at a time. That is exactly the re-derivation the list exists to spare it. The list now carries the rung, orders strongest first, and returns a per-rung tally so a caller can triage before it fetches anything. The brief records what still stands between here and the objective: the witness is one line and the query already walks the path elements before discarding all but the first; the suspicion band will be reasoned about at length by an LLM, which is worse than a human ignoring it; and a family that cannot decide its class must say so, or the model will confidently explain a silence. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:15:54 +02:00
Marius CiepluchandClaude Opus 5	20edd9354b	docs(contributor-scan): what the evidence says, and what it changes ... Spec feedback after group 2, recorded in the brief where the goal is stated rather than in a task where it would be lost. The standing test is that the tool must find bugs or it is SAST noise, and that most developers already have dataflow in CodeQL and Semgrep and never use it. That settles what the differentiator is: not the engine, which they already have, but the RUNG -- a claim the graph decided, reported without configuration. Everything below model_entailed is on trial. The scoreboard, kept honest: vampi finds 3 of 3 known vulnerabilities at model_entailed, including a second BOLA the engine found before it was in the recipe, with 2 false positives that are endpoints the OpenAPI spec declares public. libpng produced 198 rows and zero valid ones, for a structural reason rather than a tuning one -- the library never calls a sink the c/memory family models. Nine defects this session, each found by a measurement and none by review; three were one bug class in three matchers, two were a capability present but unreachable. The teachable finding is not any one of them, it is that a corpus of function-level pairs cannot see any of them, and every one appeared within minutes of pointing the tool at a real repository. The gates stayed byte-identical through all nine, which says the same thing from the other side. Two priorities change. 2.11, the suspicion band, becomes the critical path rather than cleanup: libpng's 163 "API is present" rows are exactly the noise the quote is about, and no arbiter accuracy survives shipping them. And a family that cannot decide its class has to say so, because a silent C scan reading as a clean bill of health is worse than noise -- it is quiet. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:13:11 +02:00
Marius CiepluchandClaude Opus 5	2589c3d47a	docs(contributor-scan): what c/memory covers, and what SMT covers instead ... libpng's library code never calls strcpy, strcat, sprintf or gets -- all twelve files that do are under contrib/. It overflows through array and pointer arithmetic and memcpy sizing, which is a different question from the one this family asks: "can this expression exceed this allocation" rather than "does tainted data reach this call". That is a constraint problem. It is what SMT is for, and what this project deliberately did not adopt -- the architecture decision was abstract interpretation as arbiter, taken because a model layer can be built from code where execution cannot be assumed. That decision was right for the flow families and it does not reach this one. The maintainer has attacked libpng exactly this way, using SMT to lay out memory and then guiding a fuzzer to the inputs that reach it, which is evidence about the shape of the problem rather than a plan to reimplement it: the honest routes to C memory safety are constraint solving or execution, and the ladder already has a rung for the second. So 2.15 now separates the cheap route (pin a C project whose CVE runs through a string function, giving the family its first true positive) from the expensive one (model the arithmetic, a different engine), instead of implying a sink-table fix would cover both. And 2.16 asks the question this spec's predecessor would have asked: c/memory has scored zero true positives and three separate classes of false one -- a substring match, a bound it could not see, an allocation sized to fit. Every one was a real engine defect and every one is fixed, but the family has never been right. Keep it scoped and labelled, or retire it; either way with a number behind it, and never leaving a silent C scan to read as a clean bill of health. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:11:30 +02:00
Marius CiepluchandClaude Opus 5	c1191b3a6b	feat(model): an access-control finding names a line, and what the PDG does not give us ... contributor-scan 2.7. Taint and config verdicts have always carried a `path:line:function` location; dominance never did, so every access-control finding reached a contributor as `api_views/users.py:?:update_password` -- the right file by accident, because the region was asked about that file, and no line at all. The query now reports the operation's file and the verdict carries the location, with an engine that cannot report one getting no location rather than an invented one. Two questions answered by measurement while doing it. **Does this factor in the PDG?** The data half, centrally: `reachableByFlows` is the entire taint arbiter and `joern-parse` applies the `dataflowOss` overlay. The control half, not at all -- and it is not available to us. Measured: 0 of 13,184 calls in the libpng CPG and 0 of 1,152 in the vampi CPG have a controlling control structure, so `controlledBy` returns nothing in either. Everything guard-shaped therefore runs on CFG dominance or on syntactic conditions, which is exactly why 2.12's first attempt failed: libpng's bound reaches its sink through an `error` flag, and that is a control-dependence fact and nothing else. Recorded as 2.14, including the possibility that it cannot be enabled -- which would bound what guard reasoning this engine can ever do. **Would checking out a vulnerable libpng find a known CVE?** No, and not because of the version. Every one of the twelve files using `strcpy`/`strcat`/`sprintf`/`gets` is under `contrib/`; the library itself uses none of them. libpng overflows through array and pointer arithmetic and `memcpy` sizing, and five of its buffer-overflow fixes touch no modelled sink at all. Pinning an older commit would produce the same silence for the same reason. That leaves two routes of very different size, recorded as 2.15: pin a C project whose CVE genuinely runs through a string function, which is cheap and would give the memory family its first true positive; or model how C actually overflows, which is size arithmetic rather than sink matching and should not be started by accident. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:09:47 +02:00
Marius CiepluchandClaude Opus 5	9221ba8e40	feat(model): a bound on the data lowers the rung, it does not clear it ... contributor-scan 2.12. c/injection and c/memory carried zero sanitizers, so no C flow could ever be discharged: every `argv -> strcpy` entailed however carefully it was bounded. Which sinks can be discharged this way is read from the sink fact's own CWE rather than declared: CWE-121 is a stack buffer overflow, so gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the fact tables and it is C-only, so no Python or JavaScript sink is touched and the pair corpus is unaffected -- gates byte-identical. Not a sanitizer list, because `strlen` appears in the guarded and the unguarded case alike; the discharge is a guard, not a call. **The first implementation required the bound to DOMINATE the sink, and it found nothing.** libpng guards `strcpy(outname+len, ".png")` with `(len = strlen(inname)) > 250`, but that check sits on an else-if chain whose sibling branch skips it entirely, and the protection reaches the sink through an `error` flag: `++error` in the guarded branch, `if (!error)` around the copy. Joern's dominator set for that sink has 89 calls and correctly excludes the bound; `controlledBy` returns none either. Verified against the CPG before the rule was changed rather than after it failed. So the bound is searched in the method, and the consequence is corroborated rather than silence. That is the honest rung: a bound on this variable means the graph has NOT shown the copy is unguarded, and it has not shown it is guarded either -- off-by-one being how bounds fail. The judge is asked. Proving the flag-mediated case needs path-sensitive reasoning this arbiter does not do. libpng entailed 3 -> 1 The survivor is also a false positive: `strcpy(output + len, ".icc")` in iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That is a different discharge shape -- allocated to size, not bounds-checked -- and it is recorded as 2.13 rather than chased. Both shapes were found on one repository, and extending a rule until that repository is clean is the overfitting Req 8 exists to prevent; 2.13 requires a second C checkout with a genuine CWE-121 so the change can be measured for what it silences as well as what it clears. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:01:30 +02:00
Marius CiepluchandClaude Opus 5	6dc25d77e4	feat(model): let the project say what it ships, and rank that first ... contributor-scan 2.10. Every one of libpng's twenty-five entry-point regions was a `main()` -- twenty-three under contrib/, plus example.c and pngtest.c -- and those regions carried the fifty interprocedural taint requests that dominated a 300s query phase. A library has no entry points, so entry-point-driven analysis had nothing to anchor to except its own example programs. The task said to derive this rather than hardcode `contrib/`, and a project already declares it. Makefile.am names installed targets in `lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES` names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own word for "not installed" -- exactly the distinction wanted. Taking every `_SOURCES` would have called libpng's whole test suite shipped code, since contrib/libtests declares its sources the same way the library does. interprocedural regions 25 -> 1 taint query 300.1s -> 125.0s total query 311.9s -> 138.5s regions 121 -> 121, unchanged by design The surviving interprocedural region is contrib/tools/pngfix.c:main, which libpng genuinely installs. Two rules keep it honest. **Silence is not exclusion**: a project with no build file this can read gets None back and nothing changes, so vampi's numbers are untouched. **Not shipped is not unscanned**: those regions are still arbitrated function-locally, because a contributor editing a test must still get an answer -- what they lose is the expensive treatment and first claim on the budget. Entailed findings went 0 -> 3, and all three are false positives at contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe: outname is char[256] and line 325 guards with `if ((len = strlen(inname)) > 250)`. They were not introduced so much as uncovered -- a different callDepth meant a different sink set, and the 26 fgets false positives were masking them. The structural cause is recorded as 2.12: c/injection and c/memory carry ZERO sanitizers, so no C flow can ever be discharged and every `argv -> strcpy` entails however carefully it is bounded. The discharge here is not a call but a guard that dominates the sink, which is precisely what dominance.sc already does for access control and is wired to that one family. Also adds Group 7, five tasks for stopping the same false positive twice. Two of today's three false-positive classes were one bug in three matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in `fgets` -- each fixed locally and each next one surviving because nobody carried the fix across. A model may group counter-examples by cause and draft an edit; only a split-respecting measurement may adopt one, because the predecessor's improve lever admitted shapes taught by the holdout pairs they then recovered and turned a -0.059 Youden into +0.059. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 12:45:21 +02:00
Marius CiepluchandClaude Opus 5	67497c4d68	fix(cpg): a sink is matched as a word, not as a substring ... `fgets` contains `gets`. The sink matcher's last clause was an unbounded `methodFullName.contains(n)`, so every bounded read in libpng matched the sink for the unbounded one. All 26 entailed findings on that repository were the same false positive: *argv -> fgets(pnmline, 256, wpng_info.infile) (line 335, 8 steps) which is the safe API being reported as the dangerous one -- and `fgets` is in the fact tables as a SOURCE, never a sink. Verified against the CPG: methodFullName is exactly "fgets", and .contains("gets") is true. Third instance of one bug class in this codebase, after the sanitizer that matched `resolve` inside `resolveUrl` and the discharger that matched `user` inside `users`. The sink matcher had never been given the same treatment. All four of its clauses now run through the shared bounded matcher, and that matcher is the punctuation-tolerant version written for task 2.6 -- so `input(`, which an unconditional lookahead had made unmatchable in `input(x)`, works for the first time as well. libpng entailed 26 -> 0 Zero is the right answer for a PNG library whose only known CVE is out of family. Gates byte-identical, so no pair outcome depended on the substring match. What remains on that scan is recorded, not claimed: 113 `c-unsafe-*` rows assert an API is present rather than misused, and 50 overlay-coverage rows are annotations rather than findings. 82 of the 113 are under `contrib/`. Task 2.11 covers whether that band is worth putting in front of a contributor at all; 2.10 already covers not scanning example code. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:43:00 +02:00
Marius CiepluchandClaude Opus 5	68b4e551f3	feat(model): the envelope checkout, and no to Joern server mode ... contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform. 89,082 lines of C across 93 files, under a 2GB heap cap: CPG build 16.1s (25s cold) Joern queries 250.0s arbitration 122.0s (105 calls, $0.0122) whole scan 446s peak RSS 911MB regions 121, all arbitrated, 32 unasked CVE-2025-66293 out_of_scope, as the recipe declares Memory is not the constraint a fleet runner would hit: 911MB leaves room for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen while building this was concurrent JVMs on one desktop, not repository size. **Server mode: no.** The query phase splits taint 243.3s / config 10.7s over two invocations, and fixed overhead measured on the same CPG is ~13s per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL evaluation: a warm server removes 6% of the scan and none of the part that costs. The driver now records query seconds per kind, because a single total cannot tell startup from evaluation and the two answer this oppositely. The small repository pointed the other way. vampi's 34.3s over three kinds is roughly three startups and almost no evaluation, which made a warm JVM look worth having; the ratio inverts with size. That inversion is why the task was sequenced after a real repository rather than decided on a 520-line file. On whether one whole-repository CPG is the only shape for a horizontal fleet: no, and it is also not where the pressure is. The build parallelises and costs 16s; the query does not shard and costs 250s, so splitting a repository across runners divides the cheap phase -- and after 2.4 a shard boundary hides exactly the cross-module flows that matter. Scoping to a contributor's diff cuts the expensive phase instead. 911MB for 89k lines extrapolates to roughly 5GB at a million lines, which would exceed a standard runner, so the ceiling is real and further out than this checkout can locate. Two defects recorded rather than fixed, both invisible at vampi scale: 2.9, all 26 entailed findings are the same site reported 26 times, because regions that share a sink each emit their own; and 2.10, 22 of the 30 highest-ranked regions are libpng's `contrib/` example programs rather than the library. Also ignores Joern's `workspace/`, which a maintainer querying a CPG by hand from the repository root will create. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:30:02 +02:00
Marius CiepluchandClaude Opus 5	d15985f695	fix(model): a guard discharges the obligation it is a guard of ... contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers, including three that do authorize and three the OpenAPI spec declares public. Five separate defects, each invisible to a corpus of one-function excerpts and each found by asking the query what it actually matched. **A trailing segment is not a match.** `query.get` was matched by its last segment, so every `.get(` in the repository became a protected database operation -- `request.headers.get('Authorization')` and `request_data.get('username')` included. Four of the seven false positives came from this, and so did the apparent detection of the known BOLA: it was entailed because fetching the Authorization header had no dominating guard, not because of the defect. A qualified token now requires its qualifier. **Configuration is not authorization.** dominance_specs took every obligation fact. `security_setting` operations are discharged by a VALUE, which config_specs already claims them for; pooled in here they made `app.run(...)` an operation requiring an authorization guard and put thirteen tokens in BOTH lists -- an obligation that is its own discharge, the same shape as the taint sink that used to sanitize itself. **Input validation is not authorization.** `jsonschema.validate` was counted as a guard, so handlers with no authorization check at all read as guarded. No operation in the table requires validated_input. **A guard discharges the obligation it is a guard of.** The largest one. `orm-read` requires an identity_constraint or an ownership_check; `token_validator` is a path_guard. Flattened into one list, authenticating the caller discharged an object-level obligation -- a handler that authenticates and then looks the record up by a PATH parameter read as fully guarded. The facts state that relation and the spec was discarding it; it is now carried and the engine asks per operation. **A word boundary needs a word.** Found while fixing the above. Guard matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`. The identity token `['sub']` begins with `[`, and the character before it in `resp['sub']` is `p` -- so the assertion could never hold. Every identity token starting with punctuation was unmatchable, and the identity constraint that separates a correct lookup from an IDOR was invisible. Also: dominance rows were repository-wide, so the model's unguarded `delete_user` was attributed to the API handler of the same name. Rows are scoped to the region's file, which is what the rung's own justification says -- the evidence is a file contradicting itself. Measured against VAmPI's OpenAPI `security` blocks, which declare per operation whether auth is required, and agree with the code: vampi precision 28.6% (2/7) -> 50.0% (2/4) known found 3 of 3, all model_entailed And on the labelled access-control pairs, which nobody tuned this against: all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1% 7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0% Every detection lost was on a pair whose FIXED side was flagged too, which scores nothing. No correct detection was traded for the precision. get_by_title was flagged before it was in the recipe: its code carries the comment "# Broken Object Level Authorization" and the README lists it, so a real finding was being scored as a false positive. It is now the recipe's third known vulnerability. Both remaining false positives are endpoints the spec declares public. Recorded as 2.8, with the reason the mapper's access_level cannot simply be trusted: for decorator frameworks "public" means "no decorator found", which is the bug itself. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:02:48 +02:00
Marius CiepluchandClaude Opus 5	0fcd5feade	feat(model): a region for the code that is in no function ... contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly once, at app.py:17:<module>. Before task 2.2 it was reported eight times in eight files that do not contain it; after 2.2 it was reported nowhere. Two causes, one shape: nothing could ask about module-level code. **The main-guard marker is not a function.** `__main__` is what the mapper emits for `if __name__ == '__main__':`. Sent to the engine as a function name it matched nothing, because a CPG calls a file's top-level code `<module>` -- verified against the VAmPI CPG rather than assumed. The region model translates the marker, which is the one place that turns what the mapper meant into what the scanner asks. **A function-scoped file had no region for its module body.** Settings, route registration and globals live in no function, and every region's query filters by function name. A file-level region beside the handlers would report each handler's findings a second time, which is why the earlier rule dropped it as "a second full-file pass". A module-scope region PARTITIONS the file instead of overlapping it. That only became cheap after batching: before it, a region was a JVM. Two things this needed to be correct rather than merely present: - A module method spans its whole file, so the taint query's line-range containment made `<module>` the parent of every function in it. Under module scope, membership is direct -- otherwise the module region reports every handler's sinks again, which is the duplication the old rule was avoiding. - A module body is not a handler. The marker-derived region inherited the handler families and entailed a missing authorization check on the main guard: a finding with no possible remediation. Caught by the first repository run that had the region, and now covered by a test. bind address 8 wrong files -> 0 -> 1, at app.py:17:<module> regions 22 -> 25 entailed 8 -> 9 query time 37.4s -> 41.3s peak RSS 827MB -> 823MB cost $0.0067 -> $0.0068 Three more regions for four seconds and no more model calls: the budget is a total and was already spent, so regions_unasked rose 17 -> 20 -- those regions got the free pass and are reported as unasked, not as clean. `<module>` is Joern's Python name for a module body; JavaScript uses `:program`, so this is verified for Python only. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:43:53 +02:00
Marius CiepluchandClaude Opus 5	ca7b5ba370	feat(model): a region that spans the flow, not just a function ... contributor-scan 2.4, closing the no-go task 2.2 recorded. VAmPI's SQL injection is a request parameter reaching `get_by_username` in api_views/users.py, passed to `User.get_user` in models/user_model.py, interpolated into a query string and executed there. Source and sink are in different functions and different modules. Regions were per-function, so neither region saw both ends and the engine was silent on the most ordinary shape a real injection bug has. `reachableByFlows` was never the limitation -- it follows calls already. The limitation was the QUESTION. Sinks were filtered to the labeled function's own body, so a crossing flow had no sink to end at. Three changes, each with a reason it is sound rather than convenient: **An entry point is the trust boundary.** parameter_sources was off for repository scans because a repository's parameters are not all attacker input. An entry point's are, by definition -- that is what makes it an entry point. The driver turns it on only for regions the mapper found and named, and leaves it off for walker fallbacks and for named functions that are not entry points. **Sinks may sit below the handler.** taint.sc admits sinks in methods reachable from the labeled method through the call graph, bounded at ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is where these bugs live. Unbounded would make the question about most of the repository rather than about this entry point, and make it unaffordable. **The finding names the sink, not the asker.** A crossing flow ends in another file and another function, so Verdict carries the evidence's own `path:line:function` and the site uses it. Reporting `models/user_model.py:73:get_by_username` would name a place that does not exist: line 73 is inside get_user. Also: labeled methods are now resolved by name AND file. `get_user` exists in more than one module of a real repository, which a corpus of one-file excerpts could never surface. Measured on the same checkout and budget as 2.2: VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now entailed 7 -> 8 query time 40.9s -> 37.4s peak RSS 869MB -> 827MB cost $0.0067, unchanged Asking the bigger question cost nothing measurable here: file scoping and call-graph following pull in opposite directions and roughly cancel at this size. That is one 520-line repository, and VAmPI is now the repository this change was developed against -- the exact overfitting shape Req 8 exists to catch. The depth bound and the entry-point rule mean nothing general until a checkout nobody tuned against says so. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:27:19 +02:00
Marius CiepluchandClaude Opus 5	177c7f3719	feat(model): first repository measurement, and the four things it found ... contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than as excerpts, with benchmarks/repos/measure.py recording build time, query time, arbitration time, peak memory, cost, unjudged regions, and whether each known vulnerability was found and at which rung. The corpus could not have found any of this. Every defect below needs a repository to become visible, and three of them were invisible behind a passing test suite. **The batch path was unreachable.** query_batch was implemented, tested and committed while CpgResult exposed only `run`, so the driver's capability check found nothing and fell back to one JVM per region-family: the first attempt at this measurement ran eighteen minutes on 520 lines, issuing `--param function=` one at a time. Every test called query_batch directly, so none of them noticed. CpgResult now carries run_batch and a test asserts the capability is reachable through the object the driver is handed. The earlier 275s->106s improvement attributed to batching came from region deduplication alone. **The budget stopped free work.** It bounds model calls, but was enforced by breaking the region loop -- spending a paid limit to stop unpaid CPG arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the graph entails for nothing, was never reached. It now withholds the judge only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent because unaffordable is not silent because decided), and the BOLA is entailed by the model layer. **A region with no function asked about the whole repository.** Sending function="" matched every hit in the tree and the driver attributed that answer to whoever asked: one `host='0.0.0.0'` in app.py became eight identical entailed findings in eight files that do not contain it. Queries now carry the region's file. Entailed findings 14 -> 7, query time 70.9s -> 40.9s. The single-request form stays unscoped so the committed pair measurements remain reproducible. **The JVM had no heap bound.** Joern claimed a quarter of physical RAM and runs were killed under memory pressure at different points, so the numbers described the host rather than the engine. -J-Xmx2048m by default, OPENULTRASAST_CPG_HEAP_MB to raise it. Also: arbitrated findings carried `function:line` where the report layer read `path:line`, so every entailed finding reached a contributor with a function name where its file belonged; and the witness line was parsed by splitting on a comma, which produced sites like `config.py:17): permissive literal '0.0.0.0':?`. Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end, 869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the only shape: VAmPI's SQL injection runs from a handler parameter into a function in another module, and a per-function region sees neither end. That is the ordinary shape of a real injection bug, so it is recorded as task 2.4 along with three smaller defects this run exposed. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:46:21 +02:00
Marius CiepluchandClaude Opus 5	8c387306be	feat(repos): pinned known-vulnerable checkouts, offline by default ... The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:10:59 +02:00
Marius CiepluchandClaude Opus 5	c35426e186	perf(model): one Joern invocation per query kind, not one per region ... JVM startup dominated a scan: a call per region per family put a ten-line file at 275s over three regions, and would have put a thousand regions at roughly fifty hours. The queries themselves are milliseconds once the CPG is loaded, so the cost was almost entirely process starts. scan_repository() is now collect -> batch -> arbitrate. The three .sc scripts accept a `requests` JSON map and answer all of them in one run, keeping the single-request form so the committed measurements stay reproducible byte-for-byte. Arbiters are untouched: each is handed a CpgResult whose `run` returns its own prefetched rows. The batcher builds parameters by calling request_params() extracted from taint.py, dominance.py and config_value.py rather than reconstructing them. A second copy that drifts from the arbiter's is the exact shape of the five family-assumption bugs the predecessor spec spent a session finding; there is now one definition per family and the batcher borrows it. Regions are also deduplicated by (path, function), collapsing the nameless entry-point rows that were producing `model:injection:?:10`. Measured on the same ten-line file: 275s -> 106s, three duplicate findings -> one `model:injection:ping:10`, rung unchanged at model_entailed. Gates byte-identical. Server mode is recorded as task 2.3, deliberately sequenced after the repository measurement: whether the remaining ~100s floor is worth a process model, a weakened subprocess seam and a Scala-evaluating HTTP endpoint depends on a build-time-to-query-time ratio only 2.2 produces. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:57:35 +02:00
Marius CiepluchandClaude Opus 5	e67f3ee2f0	feat(contributor-scan): 1.3 the model layer reaches a user ... `model-grounded-detection` finished 16/16 with the model layer having zero production callers. This closes that. A real scan of a Flask file now reports: by_rung: {"model_entailed": 3, "model_corroborated": 0, "suspicion": 0} model:injection:ping:10 <- os.system("ping -c 1 " + request.args["host"]) The stage is the fifth MAP proposer, mirroring _check_obligations, and is additive: pattern, overlay and obligation findings all keep working, and without a CPG engine it records cpg_unavailable and changes nothing else. Two scan integrity tests pin the exact degradation set and both broke -- correctly. Their `requested == baseline` assertion, that findings are byte-identical, still passed, so the only change is that the scan now SAYS the arbiter was absent instead of being silently without one. The guards are updated to assert the model degradation specifically rather than being loosened. ModelLayerConfig, not ModelConfig: that name is already the [models] block, and the type checker caught the collision immediately. Two meanings of one name is how a scan silently reads the wrong settings. Every payload carries the same shape, skipped ones included. A payload that is sometimes a full record and sometimes a single "skipped" key is how a report ends up quietly omitting a section. unjudged_sample names the lowest-ranked regions the budget did not reach. Counting them is the honesty requirement; naming them is what lets a maintainer write a large-repository exclusion list from evidence rather than guesswork. The run also produced the finding behind new task 1.4: 275 seconds for a ten-line file. cost_usd is 0.0, so those 21 invocations were not model calls -- they were Joern query launches, one per region per family, each a JVM start. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:42:44 +02:00
Marius CiepluchandClaude Opus 5	8663c1a5b3	feat(contributor-scan): 1.1 regions from entry points, 1.2 the repository driver ... Pair scoring could rely on a labelled function; a contributor cannot supply one -- which code is risky is the question they are asking. Regions now come from entry points, with a file-level fallback that says it is one, and each carries only the families its shape and language admit. Ranking uses what EntryPointRecord already decided (access_level, trust_boundary) rather than a second opinion computed from the same evidence. A family is offered only where its arbiter could answer: access_control needs a handler, so a library file with no entry point is not asked about obligations it has no siblings to be inconsistent with. The driver makes the two costs that were free at pair scale explicit. ONE CPG per repository, reused across regions -- a build is 5-30s on a forty-line excerpt and per-region builds would make this unusable. And a budget that is a TOTAL for the run rather than the enumerator's per-region cap of eight, spent highest-rank-first, with whatever it does not reach counted into regions_unjudged. A tool that silently analyses a tenth of a repository and reports a clean result is worse than one that says what it skipped. Two flaws the tests caught, both mine and both about ownership: the budget counted calls by reading a client's own usage log, so it silently never tripped for a client that keeps none -- the driver now counts its own spend through a wrapper; and the driver trusted its caller to have sorted, when the order the budget is spent in belongs to whoever holds the budget. The third was worse and only surfaced because a test failed for the right reason: the driver passed candidates=() and never enumerated, so a repository scan would have reported only what the arbiters establish unaided and never populated the suspicion band -- which supplied 15 of 26 findings in the pair measurement. It enumerates now. Sibling audit (Req 8.5) done before writing: family routing lives in exactly one production place, pipeline._arbitrate, dispatching on spec type. regions_for returns family names and the driver resolves specs, so that stays true. 750 passing, mypy clean over 96 files, gates byte-identical, zero orphans. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:28:30 +02:00
Marius CiepluchandClaude Opus 5	64a27c4e30	docs(model-grounded-detection): record what 16/16 does and does not mean ... Asked whether the prior spec is finished. Audited rather than counted, and the answer is: all 16 tasks are complete as written, and not all requirements are satisfied in the product. Four are built but unreachable, so the requirement is not met where it matters: model/report.py implements the per-slice and overfitting-gap discipline (Req 11.1) and has no production caller; resolve_execution_backend returns NullExecutionBackend on BOTH branches so the execution_confirmed rung (Req 10.1) can never be reached; harvest_recipe (Req 9.4) is a seam with nothing plugged into it; and the pipeline that measures 90.0% pair-correct is called only by a measurement harness, never by ousast scan. That is a spec defect rather than an execution one. No task ever said "make this reachable", and Req 11.2 -- the gates stay byte-identical -- is satisfied PRECISELY BECAUSE nothing is wired. I read that as reassurance for a whole session when it was evidence. The spec is not reopened; it remains the complete record of building and measuring the arbiter. contributor-scan takes all four: its Req 1 closes 8.x and 11.1 in the product, Req 7.1 forces the memory decision 10.1 defers, and Req 9 covers the harvest 9.4 left as a seam. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:19:41 +02:00
Marius CiepluchandClaude Opus 5	4e4681f162	docs(contributor-scan): design and tasks -- wire it, then measure on a real repo ... Requirements approved, design and tasks generated. 47 criteria, all traced; 13 tasks in 6 groups. The design is deliberately conservative. The model layer joins as ONE STAGE INSIDE MAP, beside the obligations checker it most resembles -- which already takes entry points, produces findings and merges them. Findings are added, never substituted; the pattern and overlay paths keep working; a machine without Joern scans exactly as it does today. Nothing here is allowed to make the tool worse for someone who does not have the engine. Two decisions carry the most weight. One CPG per scan rather than per region: a build is 5-30s on an excerpt and is the dominant cost at repository scale, so per-region builds would make this unusable. And one total budget rather than the enumerator's per-region cap of eight, which is meaningless across thousands of regions; the budget is spent highest-rank-first and whatever it does not reach is counted and reported, never silently truncated. Task 2.2 is a go/no-go and comes before shipping: the first scan of a pinned known-vulnerable checkout is a measurement nobody has taken, and if one CPG per repository proves infeasible the region and budget design changes shape. Better to learn that before groups 3-6 are built on it. Group 5 (corpus) is last on purpose. The binding constraint today is reachability, not corpus size -- 198 of 238 OWASP pairs went unvendored because volume without readability makes every number worse. The task preamble carries the predecessor's two hard-won habits: audit sibling paths in the same commit as any routing or matching change (five bugs shared that shape, each caught by a measurement rather than a check), and run the whole CI list rather than the parts one remembers (CI was red for a session because `ruff check` passing was taken as evidence for `ruff format --check`). Not approved for implementation. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:17:20 +02:00
Marius CiepluchandClaude Opus 5	fe45081b43	fix(ci): run ruff format, and record run 5 at 90% pair-correct ... CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every change this session and never once ran `ruff format --check`, though CI runs both. Lint passing is not formatting passing, and I treated one as evidence for the other for the whole session. Formatting applied. The full CI sequence now reproduces locally: format, lint, mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the semantic job's 16 marked tests. Also records run 5 of the pipeline measurement, which the vampi fix lifted as predicted: hunter (K=5) pipeline (K=1) pair-correct 76.7% 90.0% leak 12.3% 6.7% detection -- 96.7% cost ~$0.69 $0.0068 27 of 30 pairs. Five runs, each step from a measurement contradicting an assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which exposed the discarded verdict) -> 80.0 (residual question matched to its arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer contradicting its own evidence). One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default rather than a security-setting call. Out of reach before the settings work and still is; contorting the arbiter around one pair would be worse than the miss. The caveat is recorded beside the number: 30 pairs of one slice, and the slice iterated against all session. The +16% overfitting gap against vfc-js is why contributor-scan Req 8.1/8.2 exist. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:46:58 +02:00
Marius CiepluchandClaude Opus 5	14ca0a96ce	docs(contributor-scan): requirements, including the corpus asks I first missed ... 10 requirements, 46 criteria. A NEW spec rather than an amendment to model-grounded-detection, which is complete and closed. Req 1-8 cover the gap that prompted it: the model layer has zero production callers, so ousast scan must use it (1), regions must come from entry points rather than a labelled function (2), ranking and budget must work at repository scale where the enumerator's per-region cap of eight is meaningless (3), the performance envelope must be measured on a real checkout (4), output must be actionable (5), packaging must be built and verified rather than merely defined (6), the two deferred decisions must close (7), and the honesty disciplines carry forward (8). Req 9 and 10 were added after the maintainer asked whether their requests had been captured. They had not been -- I folded the corpus asks implicitly into "run against a known-vulnerable repository", which is not the same thing. Now explicit: a PHP/WordPress slice (php2cpg works once an interpreter is present, which the image provides, and plugin CVEs are overwhelmingly the families the model arbitrates well); real-world C library CVEs -- libpng, image libraries, OpenSSH, libpam, kernel modules -- gated on the memory decision, because until it closes they would join 173 pairs already idle; more vibe-code pairs, since that slice is where the stated audience works and it is only 35 pairs; and regression tuning over a pinned large-repository baseline, which is the thing pair scoring structurally cannot see. Req 9.4 carries today's lesson: verify on a sample that the model can READ a slice before vendoring it. The OWASP harvest found 238 pairs and vendored 40 for exactly that reason. Not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:42:53 +02:00
Marius CiepluchandClaude Opus 5	41266ce472	docs(contributor-scan): brief -- the model layer is complete and unshipped ... Asked what the spec state was and whether anything was forgotten. It was, and it is the largest thing outstanding: model-grounded-detection is 16/16 tasks complete and its arbiter measures 86.7% pair-correct at 6.7% leak, but the model layer has ZERO production callers. `ousast scan` still runs preprocess -> map -> overlay -> obligations -> regress -> report and touches none of it. Every number quoted today came from a measurement harness in a scratch directory. That is a scope gap in the predecessor spec rather than a slip in executing it: there was never a task saying "make this reachable", and its Req 11.2 (gates stay byte-identical) is satisfied PRECISELY BECAUSE nothing is wired. The spec is complete and the feature is unshipped, both at once. A new spec rather than an amendment, because the predecessor is coherent and closed -- it built an arbiter and measured it -- while this is a different question: making it something a developer runs on a repository they are about to contribute to. What it covers, in the order it bites: scan integration; whole-repo scanning (entry-point region selection, ranking across thousands of candidates, a budget policy -- the enumerator caps at 8 per region, meaningless at repo scale -- and a performance envelope never measured); running against a known-vulnerable checkout, since the 176 real-world CVE pairs are single-function excerpts that exercise none of it; the deferred memory decision, with 173 of those pairs idle behind a stub tier; and retiring evidence_level. It also records what was built outside any spec: the pipeline itself, the ops work (one image, compose, shell wrappers, pyinfra -- with the Docker image never actually built), and the corpus additions. And it carries the known limits forward rather than losing them: the flat permissive_values list that cannot tell debug=True from httponly=True, interprocedural resolution as the most repeated miss cause, and the five bugs that shared one shape -- a taint-specific assumption applied to every family, each caught by a measurement rather than a check. Not approved. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:40:38 +02:00
Marius CiepluchandClaude Opus 5	adbb824eac	feat(ops): a sourceable wrapper so docker is nobody's problem ... source ops/shell/ousast.sh # bash / zsh . ops\shell\ousast.ps1 # powershell ousast scan . ousast scan src/api The container becomes an implementation detail. The engine is a JVM and ~2 GB of Joern, and a contributor evaluating this tool should not have to install either to try it -- what they get is a command. Path rewriting is what makes it feel local: arguments under the working directory are mapped to their mounted equivalents, so `src/api` means what the user expects. Anything outside the mount is passed through unchanged WITH A WARNING rather than silently rewritten into a path the container cannot see -- a wrong path that looks like it worked is worse than one that complains. A native install wins when it is present. Someone who installed Joern deliberately should not be quietly routed through a container, so the wrapper uses a local joern-parse and venv when both exist; OUSAST_PREFER_NATIVE=0 forces the container. Verified end to end from an unrelated directory. The image builds itself on first use and says why it is pausing. A first run that prints a wall of build output and then works is worse than one that explains itself. Defaults stay conservative: the working directory mounts READ-ONLY, findings go to ./.ousast as the only writable mount, and the network is off unless `ousast-with-judge` asks for it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:37:58 +02:00
Marius CiepluchandClaude Opus 5	46232fe3e6	feat(ops): ship the tool and its engine in one image ... A contributor should be able to run this with nothing installed but Docker: docker compose build TARGET=/path/to/repo docker compose run --rm ousast scan /target One image rather than two services, and the reason is the seam. The model layer reaches Joern by subprocess; putting both in the same image means Joern is simply on PATH, so there is no container-to-container call, no docker socket to mount, no host/container path translation, and no second code path in cpg/backend.py to get wrong. Sibling services would need the tool to exec into the engine's container, which means handing a linter the docker socket -- a large amount of privilege for something that reads source code. The image is big (a JRE plus ~2 GB of Joern); that is the honest cost of bundling an engine. I had started building the two-service version with a docker-wrapper mode in the backend and dropped it, tests included: once the engine is on PATH the wrapper is complexity with no purpose. Defaults chosen so an unfamiliar analyser is safe to try: the target mounts READ-ONLY, /work is a tmpfs so CPG scratch never outlives the run or touches the host, network_mode is `none` unless the LLM judge is wanted, and API keys pass through the environment rather than into an image layer. mem_limit is set because two Joern JVMs at once is what triggered an out-of-memory kill on a 7.7 GB machine during this feature's measurements. pyinfra deploy kept for persistent host installs, with the same version pin and the same checksum verification -- done as an explicit `sha512sum -c` step because pyinfra's files.download offers sha384/sha256/sha1/md5 and Joern publishes sha512. That mismatch was caught by validating against the installed pyinfra rather than trusting the API from memory. Both include php-cli: php2cpg drives PHP-Parser and shells out to a real interpreter, and without it a PHP tree fails with an opaque "Process exited with code 1" -- measured here, which is what makes a WordPress corpus feasible. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:36:10 +02:00
Marius CiepluchandClaude Opus 5	3fe7b9ded7	fix(model-grounded-detection): the dominance question contradicted its own evidence ... vampi-users-update-password scored both_silent while the arbiter returned model_corroborated on the vulnerable side and nothing on the fixed one -- so the finding existed and the pipeline threw it away. The residual question was written for the ENTAILED case, but entailed returns before the judge is ever consulted, so the question is only ever asked for CORROBORATED. It asserted "in a file where sibling handlers are guarded" and asked about "the guard its siblings have", printed directly above a witness reading "and no guarded sibling" -- because corroborated means precisely that no sibling is guarded. The judge was handed a prompt contradicting its own evidence line and reasonably answered no. The question now describes what corroborated actually is: an obligated operation no guard governs, with no sibling evidence either way, so the model cannot tell from consistency alone whether the guard is missing or the module is deliberately public. That is the real open question and the one worth asking. Verified end to end against a live endpoint: vuln model_corroborated, fixed silent, the pair splits. The taint and config questions were checked for the same defect and are correct -- each describes the corroborated case it is actually asked about. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:22:55 +02:00
Marius CiepluchandClaude Opus 5	c74e101d10	measure(model-grounded-detection): 86.7% pair-correct at half the hunter's leak ... hunter (K=5) pipeline (K=1) pair-correct 76.7% 86.7% leak 12.3% 6.7% detection -- 93.3% cost ~$0.69 $0.0067 Ten points of recall over the hunter, half the leak, a fifth of the model calls, ~100x cheaper. 26 of 30 pairs correct. Four runs, and each step came from a measurement contradicting an assumption -- twice one I had stated: 1 73.3% the absence and config families had no pipeline path at all 2 73.3% routing fixed and the result IDENTICAL, which was the clue: the arbiter was reached and its verdict thrown away, because every corroboration was put to the judge as a taint question 3 80.0% residual question phrased for the arbiter that raised it 4 86.7% aiohttp Application(debug=True) and RedisStorage(httponly=False) added to the settings table The division of labour keeps moving the right way: model_entailed 9 -> 11 while suspicion held at 15, so both findings gained in run 4 came from the MODEL, not the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free deterministic path. That is the payoff curve for fact-table coverage, measured rather than argued: closing a gap moves findings out of the LLM's column into the model's, improving precision and cost at once. Recorded corrections and limits, because the numbers are only worth what the caveats are. The three silent config pairs were called corpus defects on a similarity threshold borrowed from the OWASP minimal-fix analysis; it does not apply to a slice that deliberately pairs a vulnerable function with a different safe one, where 36% and 34% pairs both score pair_correct. Two were a settings gap and now split. And permissive_values is a flat literal list with no keyword association, holding both True and False -- debug=True is dangerous where httponly=True is safe, and the model cannot tell them apart. It did not bite here, and it is a real limit of the fact schema. Four remaining misses, each with a named cause. Four runs cost about $0.04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:18:06 +02:00
Marius CiepluchandClaude Opus 5	950a4c7ac4	fix(corpus): the three silent config pairs were a settings gap, not corpus defects ... I was wrong, and the correction matters more than the fix. I reported these three pairs as corpus defects on a similarity measure -- 23%, 45%, 53% -- and said no settings table could make them split. That threshold came from the OWASP minimal-fix analysis and does not apply to this slice. The vibe-py corpus pairs a vulnerable function with a DIFFERENT, SAFE function from the same repository; low similarity is the design, not a defect. djangoat-views-user-view sits at 36% and dsvpwa-handlers-do-bdr at 34%, and both score pair_correct. Reading the actual code instead of the metric shows what the settings table was missing: aiohttp's Application(debug=True) and RedisStorage(httponly=False). Adding them, two of the three now split -- vuln model_entailed, fixed silent: dvpwa-app-init Application(debug = True, middlewares = [...]) dvpwa-middlewares RedisStorage(app['redis'], httponly = False) The third, dsvpwa-dsvpwa-main, stays silent for the reason predicted before the change: its setting is an argparse default (--ssl defaulting to 0), which is a CLI default rather than a security-setting call and is not reachable by this arbiter. One weakness this exposes and does not fix: permissive_values is a flat list of literals with no keyword association, so it holds both True and False. debug=True is dangerous while httponly=True is safe, and the model cannot currently tell them apart. It happens not to bite on these pairs; it is a real limit of the fact schema and worth naming. 734 passing, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 18:50:09 +02:00
Marius CiepluchandClaude Opus 5	db3db6056d	feat(model-grounded-detection): widen the settings table, wire the weak-algorithm arm ... Two things, and a finding that says neither fixes the pairs that prompted them. The settings table was Flask/FastAPI-shaped (CORS, set_cookie, app.run) while the corpus contains aiohttp and Django applications, so three config_secrets pairs contained no modelled security setting at all. Added aiohttp's session_setup and cookie storages and Django's SECURE_* / DEBUG settings: 8 calls to 24. And config_secrets spans three abstractions while only one was wired -- a permissive setting VALUE. Weak algorithm CHOICE (CWE-326/327) is a different thing: the danger is the algorithm NAMED, not a flag left open. Its data has been sitting on the sink facts as `weak_literals` since before this feature and nothing read it. Both abstractions read the same literal, so the arm is small: hashlib.new("md5") is now entailed and hashlib.new("sha256") is not. The finding: the three silent config pairs are CORPUS DEFECTS, not model gaps. Measured similarity 23%, 45% and 53% -- dsvpwa-dsvpwa-main pairs `def main()` against `def run(self, handler)`, a different function entirely. No settings table can make those split, and it is worth saying so rather than letting the work imply otherwise. The table gap was real and worth closing on its own merits; it just is not why those pairs are silent. 734 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 18:43:33 +02:00
Marius CiepluchandClaude Opus 5	4e7fe65f17	measure(model-grounded-detection): the pipeline beats the hunter on both axes ... hunter (K=5) pipeline (K=1) pair-correct 76.7% 80.0% leak 12.3% 6.7% cost ~$0.69 $0.0076 determinism none entailed band deterministic Higher recall, half the leak, a fifth of the model calls, ~90x cheaper. Three runs, and the middle one earned its keep by changing nothing. Run 2 fixed family routing and returned numbers IDENTICAL to run 1 -- which was the clue that the dominance arbiter was being reached and its verdict thrown away. Every corroboration was put to the judge as a taint question ("a sanitizer appears on the path -- does it suffice?"), meaningless for a missing guard, so the judge said no and the finding was dropped. Run 3 phrases the residual question for the arbiter that raised it and converts 2 of 3 access_control pairs from silent to pair-correct, arriving at model_corroborated -- a rung that existed but had never reached a finding. This settles a question open since the start of the session, and reverses an earlier answer of mine. "Detection has regressed" was measured with the proposer disconnected: an arbiter compared against a detector, before four wiring bugs were found. Division of labour: 15 suspicion, 9 model_entailed, 2 model_corroborated. The model arbitrates 11 of 26 findings, and every conversion from suspicion to a model rung removes a leak and a model call at once -- the payoff curve for sink coverage and interprocedural resolution, now measured rather than assumed. Still silent: 3 config_secrets (the settings table is Flask/FastAPI-shaped while the pairs are aiohttp) and 1 access_control still to diagnose. Three full runs cost about $0.03 in total. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 18:40:51 +02:00
Marius CiepluchandClaude Opus 5	52f2ab5b4d	fix(model-grounded-detection): ask the residual question the arbiter actually raised ... The corrected re-measure returned numbers IDENTICAL to the buggy one, which was the clue. Routing was fixed, the dominance arbiter was reached, and the finding was discarded anyway -- because every corroboration was put to the judge as a TAINT question: "a sanitizer appears on the path: yes, does it suffice?". For a missing authorization guard that question is meaningless. The judge answers no to something that does not apply and the claim is dropped. All three vibe-py access_control pairs scored both_silent this way even though the dominance arbiter had returned model_corroborated for each of them, which I had verified by hand earlier in the session. Fourth instance in this feature of one shape: taint-specific behaviour applied to every family. Sanitizer matching, config routing, family dispatch, now the residual question. Each was exposed by a measurement rather than by a check. The question is now phrased for the arbiter that raised it -- a guard asymmetry for dominance, a permissive setting for constant abstraction, sanitizer sufficiency only for taint. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 18:19:18 +02:00
Marius CiepluchandClaude Opus 5	6dbb0e6422	measure(model-grounded-detection): the wired pipeline halves the leak at 1/55th the cost ... The first like-for-like comparison with the LLM hunter, on the same 30 vibe-py pairs and the same metric. hunter (K=5) pipeline (K=1) pair-correct 76.7% 73.3% leak 12.3% 6.7% cost ~$0.69 $0.0126 determinism none entailed band deterministic Half the leak, 55x cheaper, a fifth of the model calls, one pair short of the hunter's recall (22 against 23 of 30). And understated: all six both_silent pairs are access_control and config_secrets, which had no pipeline path to their arbiters when this ran. A corrected run supersedes those rows. The rung split says what is doing the work: 15 suspicion to 9 model_entailed on the vulnerable side. The LLM supplies most of the recall; the model supplies the precision, halving the leak by dropping claims it could contradict. That is the division of labour the architecture was designed around and the first evidence it pays. It also revises an earlier conclusion of mine. "Detection has regressed" was measured with the proposer disconnected -- an arbiter compared against a detector. With the enumerator wired in, the comparison reverses on precision and cost while holding recall. And it makes the strategy concrete: the model arbitrates about a third of findings today, and every suspicion converted to model_entailed is leak removed for free and one fewer model call. Sink coverage and interprocedural resolution now have a measured payoff rather than a hoped-for one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:59:01 +02:00
Marius CiepluchandClaude Opus 5	7a76fcb028	fix(model-grounded-detection): the pipeline arbitrated only flow families ... scan_region took a TaintSpec and called taint_verdict, so access_control and config_secrets -- whose dominance and constant-abstraction arbiters have worked since group 4 -- had no pipeline path at all. On the first end-to-end vibe-py run every one of their pairs scored `both_silent`, which reads as the model failing when in fact no code reached the arbiter. This is the same class as the ceiling harness routing config_secrets through taint: family routing fixed in one place and not carried to the other. Third time this shape has appeared in this feature, which is worth saying plainly. Dispatch is on the spec TYPE rather than a family name, so a family without an arbiter cannot be silently routed to the wrong one -- it simply has no spec and is skipped, visibly. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:53:56 +02:00
Marius CiepluchandClaude Opus 5	cdedb372dd	feat(model-grounded-detection): wire the detector -- the enumerator proposes, the model disposes ... Everything before this built an ARBITER: something that judges a flow taint reachability had already found unaided. Nothing proposed to it. That is why its recall was never comparable to a detector's, and why comparing 41.3% entailment against the LLM hunter's 76.7% pair-correct was comparing two different kinds of thing. model/judge.py and enumerate_candidates were called by no production code at all; every number so far came from a measurement harness. The wiring turns on a distinction judge() alone conflated: the model CONTRADICTS it has coverage at this site and says no -- a sanitizer on the path, or a declared-safe sink shape. The claim is dropped (Req 8.2). the model CANNOT DECIDE no modelled sink or source reaches the site at all. The model is SILENT, not negative, so the claim is reported at `suspicion` (Req 8.4). Treating silence as refutation is the tempting mistake and would be wrong here: 23 of the 39 misses in the entailment ceiling were sinks simply absent from the fact tables, so "no flow" usually means the model cannot see it rather than that nothing is there. The rung carries that uncertainty on the finding's face instead of a confidence score nobody can act on. An entailed site still costs zero model calls. The candidate budget is bounded so a dense region cannot become an unbounded spend. Smoke-tested end to end against a live endpoint: 2 pairs, detection 100%, $0.0024. The full vibe-py measurement -- the first like-for-like comparison with the hunter's 76.7% pair-correct and 12.3% leak -- is running. 727 passing, mypy/ruff clean, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:41:45 +02:00
Marius CiepluchandClaude Opus 5	1af3db37f6	feat(corpus): vendor 40 OWASP xss pairs as the owasp slice ... output_encoding had exactly ONE pair in the whole corpus. It now has 41. 238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets, GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to the safe test of the same category it most resembles -- adjacent tests come from one template, so they differ in the sink or the sanitiser rather than being unrelated programs. Median similarity 0.91, against the 64% of the older `sast` OWASP entries that the group-4 stratification flagged as different programs. Only 40 are vendored. One representative of each of the seven categories was run through the model BEFORE vendoring, and only xss distinguished: its fix adds ESAPI encoding, which the Java sanitizer facts recognise, so the vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli, ldapi and xpathi put the sink in a helper class outside the servlet and resolved no flow at all; pathtraver found the flow but its "safe" twin carries no distinguishing call, so text similarity had matched a twin that is not semantically the fix -- a subtler form of the very defect the harvest was designed to avoid. Those 198 pairs are deliberately left out: adding pairs the model cannot read would inflate the corpus while making every number less interpretable. Evidence for the vendored slice is 3 of 3 distinguished, which is a small sample and is stated as such in the catalog header rather than implied to be more. The slice is marked SYNTHETIC in its header: it measures breadth across a weakness class and must never stand in for real-world performance. The corpus already shows a +16% overfitting gap, and a large generated slice blended into a headline would widen it invisibly. Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0 while the older sast entries declare GPL-3.0 for OWASP, which looks like an error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled the train/holdout split before these pairs (owasp-java-cmdi train, owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing straddle rather than create one. The two vendored-count guards did their job -- a corpus change could not land silently -- and are updated as the acknowledgement. Hygiene artifact regenerated: 277 -> 317 vendored pairs. 720 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:28:11 +02:00
Marius CiepluchandClaude Opus 5	fcb12e116a	feat(model-grounded-detection): Java web sinks, and the post-closure-fix ceiling ... Java shipped with injection, deserialization and crypto sinks only, so the path and output_encoding families had nothing routing to them by CWE and any Java pair in either was unreadable -- the gap hutool-cve-2018-17297 reported as "no path spec for java". Adds the JDK filesystem entry points, the servlet response writers, the outbound/redirect APIs, and the ESAPI / canonical-path sanitizers the OWASP suite itself uses. Documented APIs, provenance per entry. Post-closure-fix ceiling: 41.3% entailed (38/92) and 26.1% distinguished, up from 39.1% / 23.9%. path went 0/8 to 1/8 with its first distinguished pair; injection 29->31 entailed and 18->19 distinguished. Two points -- real, and the mechanism is understood, but the remaining gaps are data rather than machinery. Carrying a correction: the measurement harness routed config_secrets through TAINT rather than the constant-abstraction arm, so its 0/10 in this artifact measures the wrong arbiter and did so in both full runs. The harness is fixed; that family's real number is the earlier targeted re-measure, and this artifact's config row should not be read. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:16:00 +02:00
Marius CiepluchandClaude Opus 5	0c71c956fd	feat(model-grounded-detection): group 6, the execution tier as a seam not a build ... The rung this project deliberately does not build. Clearwing already runs sandbox lifecycles, container pools, sanitizer images, PoC replay and stability classification, and the corrective brief concluded -- from this project's own defect history -- that reimplementing that machinery is where we go wrong. So model/execution.py decides eligibility, calls an adopted tool, and records what happened. A test asserts structurally that it imports no container or subprocess machinery and defines no build/run/compile/patch function, because the failure mode to prevent is this file quietly growing a sandbox. Eligibility is narrow in both directions: only a family a static model cannot arbitrate (memory safety in C, where the bug turns on runtime layout -- Req 10.3 forbids any web/logic family depending on this tier, and each now has taint, dominance or constant abstraction instead), and only a finding the model left undecided. The tier escalates uncertainty; it never re-checks a verdict. One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox that does not trigger a bug has not shown the bug is absent, only that this attempt did not trigger it, so the candidate stays where the model left it. Reading a failed reproduction as a clean bill of health is the same error class as reading an absent engine as "no vulnerability". With no adopted tool present -- the normal case -- the rung is never reached and clearwing_unavailable is recorded. All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:58:03 +02:00
Marius CiepluchandClaude Opus 5	86f434ab89	docs(model-grounded-detection): tick 5.1/5.2 and record the closure-scoping fix ... The boxes were left unticked and the notes unwritten when group 5 was committed. Records the closure fix and the three bugs it uncovered, each of which would have raised the number while making the answer untrue, and the correction that req.url was not a wrong diagnosis but an untestable one -- the scoping bug blocked everything downstream of it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:39:13 +02:00
Marius CiepluchandClaude Opus 5	f1c341df6c	feat(model-grounded-detection): group 5, corpus calibration and honest reporting ... 5.1 inverts what the corpus is for. Previously a pair the detector missed was a DETECTOR failure, to be averaged away over K runs and chased with a prompt change. The arbiter is deterministic: it will miss that pair the same way every time, and the only thing that moves it is better modelling. So a pair the model cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every gap carries one even when the caller supplies none -- a bare gap teaches nothing. `covered` means the model reached a strictly higher rung on the vulnerable side than on its fix. Flagging both sides equally is not coverage: it distinguishes nothing, which is exactly what group 2 measured taint reachability doing on every parameterised SQL pair. The harvest seam takes the fix commit and derives the vulnerable side as its parent, with the fix diff as the oracle -- nothing hand-labelled. 5.2 keeps three disciplines from the deleted publish layer, each because its absence caused a real error here: * per-slice rows always -- `render` RAISES without them. A mixed headline once hid agent-written code scoring 44% against the development slice's 83%. * the overfitting gap beside the headline, and ABSENT rather than zero when a slice is missing: reading an unmeasured slice as "no overfitting" is the flattering failure, not a neutral one. * the rung on every finding, so a suspicion cannot be read as a result. Run end to end against the committed ceiling artifact, the report says things worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an overfitting gap of +16%, and gaps grouped by family with their reasons -- injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no verdict), untrusted_destination 5 (4 entailed on BOTH sides). 713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. A full re-measure with the closure-scoping fix is in flight; these numbers are the pre-fix baseline and will move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:28:10 +02:00
Marius CiepluchandClaude Opus 5	d651455d23	fix(model-grounded-detection): closure scoping, and the three bugs it uncovered ... The group-4 finding: JavaScript is callback-heavy, so the method containing a sink routinely is not the labeled function. `fs.readFileSync` inside the callback passed to `fs.stat` has enclosing method `<lambda>0`, and filtering on `sinkMethod == function` dropped it -- the whole path family measured 0/8. Scope is now decided by line-range containment within the same file (astParentFullName is empty for lambdas, so nesting is not readable there), and reported as `inLabeledScope`; the Python side trusts that flag and keeps the name comparison only as a fallback, so an engine that cannot report scope never has its sinks silently treated as scoped. Parameter sources stay bound to the labeled function's OWN parameters -- a callback's `err`/`stats` are not attacker input. Fixing the scope exposed three further bugs, each of which would have raised the number while making the answer untrue: * Shortest-flow-wins picked the wrong witness. With closures in scope the same sink has flows from `req.url...` (10 steps, the vulnerability), `res` (3) and `this` (7). A flow whose source matches the source model now outranks a shorter one from a bare captured name; shortest still breaks ties, so the order stays total and the verdict deterministic. * The JS fact file had NO sanitizers at all, so no JavaScript fix of any family could ever be recognised. Added the documented path-normalisation and HTML-escaping APIs, with provenance. * Adding them regressed the witness, because `resolveUrl` CONTAINS `resolve` and the substring test marked the vulnerable flow sanitized. This is the same bug class already fixed in dominance.sc and not carried across -- the exact fix-the-instance-miss-the-class failure. taint.sc now matches on word boundaries too. And the principled rule the fix turns on: parameter sources are a FALLBACK, never an override. When a modelled source reaches the sink, those flows are the evidence; otherwise a captured `res` reaching a sanitized sink reports the fix as still vulnerable on a flow that names no attacker input. Also recorded: req.url was NOT wrong as a diagnosis, it was untestable in isolation -- the scoping bug masked it. Both were needed. On the pair that motivated all of this: vuln model_entailed req.url.split('?')[0] -> readFileSync(possibleFilename) fixed model_corroborated same flow, sanitized by path.normalize distinguished: true 700 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:24:23 +02:00
Marius CiepluchandClaude Opus 5	91c0931a05	measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2% ... Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter that makes no model calls at all. Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10, path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the class no crash oracle can reach, arbitrated deterministically. A hypothesis tested and discarded. From the group-2 misses I diagnosed path's 0/8 as a missing req.url source token and said so. The probe -- the same 8 pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real cause is that the sink sits lexically inside an anonymous callback, so its enclosing method is the closure (<lambda>0) and the sinkMethod == function filter drops it; verdict(function="") on the same CPG returns model_entailed. And the obvious fix for that is wrong. Relaxing the filter raises the number with a FALSE witness: the flows found in the closure are `res -> readFileSync` and `this -> ...`, neither of which is the vulnerability, whose real path crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a miss into a wrong entailment is worse than the miss -- it is the arbiter asserting something untrue, the one failure mode this architecture exists to prevent. The correct fix is two-part and recorded: scope a sink to any method lexically nested in the labeled function, and scope parameter sources to that function's own parameters, then re-measure counting only real witnesses. Every miss is classified and none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source unmodelled with the idiom named. Two honesty items in the artifact. 16 of 92 pairs carry no function label and are arbitrated over the whole file, so a verdict can credit a different finding than the labeled one -- owasp-python-hash (CWE-327) is credited for a permissive set_cookie elsewhere in its file, on both sides. And config_secrets conflates three abstractions: permissive setting values (implemented), weak algorithm choice (facts already carry weak_literals, unwired) and hardcoded secrets (needs a credential-literal check). 694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:46:05 +02:00
Marius CiepluchandClaude Opus 5	eebca3f663	fix(model-grounded-detection): the absence arbiter was silencing its own family ... Two bugs, together making 4.1 report "no bug" on a textbook IDOR. Both found by running it against a real corpus pair rather than a fixture. 1. constraint_params were treated as dischargers. A field named `user_id` is what an IDOR is MADE OF -- `filter_by(id=user_id)` with user_id off the request is the bug, not the fix. The deleted checker only counted an identity constraint whose value's PROVENANCE was the authenticated context, so the parameter name alone evidences nothing. This is the mirror of group 2's sanitizer bug: a thing that must never count as a discharge, counted. 2. Guard tokens matched as substrings. With `user` in the token list, the string "SELECT * FROM users WHERE id = ?" contains `user`, so the query reported every operation in the handler as guarded. Matching is word-boundary now. On threatbyte-api-v1-delete the arbiter previously found three operations, all "guarded" by evidence like '"SELECT * FROM users WHERE id = ?"' and '(user_id,)', and returned no verdict at all. After the fix, on the real pairs: threatbyte-api-v1-delete vuln=model_corroborated fixed=none vampi-users-update-password vuln=model_corroborated fixed=none Both distinguished, deterministically, with no model call. Corroborated rather than entailed is right for these: they are single-handler excerpts with no guarded sibling to be inconsistent with, which is the asymmetry the rung distinction exists to express. identity_sources still discharge; request_sources and permissive_values remain excluded because they describe the vulnerable shape rather than the fix. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:10:51 +02:00
Marius CiepluchandClaude Opus 5	4f8f590b93	fix(model-grounded-detection): type the config literals list ... mypy could not see that row["literals"] is a list on an object-valued dict. Narrowed at the use site rather than widened at the boundary. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:01:08 +02:00
Marius CiepluchandClaude Opus 5	d9ceeb6a09	feat(model-grounded-detection): 4.2 constant abstraction, the third arbiter form ... A configuration bug has no flow to follow and no guard to dominate. CORS(app, origins="*") is dangerous because of the value it is SET TO, so the arbiter evaluates the setting's arguments against the closed permissive set in the obligation facts -- the third form beside taint reachability and guard dominance, and the one Req 6.3 names. ENTAILED a security setting is passed a literal in the permissive set CORROBORATED the argument is computed, so constant abstraction cannot decide it -- that residual is the judge's, never silently "safe" (none) every literal is outside the set, or the call is not a setting Two Joern node-shape bugs, both found by running it and both of the same class as group 2's receiver bug -- the graph's indexing is not what the source text suggests: * a keyword argument (`origins="*"`) is an assignment-shaped node, so isLiteral is false on the argument itself and the literal sits in its subtree; literals are collected from the argument's AST now. * Joern indexes the receiver at 0, positional arguments from 1, and NAMED arguments at -1. Settings are configured overwhelmingly by keyword, so argumentIndexGt(0) -- correct when counting arity for the taint shape test -- dropped exactly the arguments that matter and every keyword-configured setting read as "computed". Live, deterministic, zero model calls: create_app -> model_entailed CORS(app, origins = "*"): permissive literal "*" safe_app -> no finding (origins = "https://app.example") All three arbiter forms now work against a real CPG: taint reachability for the flow families, guard dominance for absence, constant abstraction for config. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:00:41 +02:00
Marius CiepluchandClaude Opus 5	fec67253f3	feat(model-grounded-detection): 4.1 guard dominance, 4.2 the missing sink facts ... 4.1 is the arbiter for the class no crash oracle reaches. An access-control bug is the ABSENCE of a guard: no flow to follow, no crash to reproduce. What the model can establish is a consistency violation -- an obligated operation no discharging guard governs, in a file where its siblings are governed by one. "Governed" turned out not to be one relation, and asking pure CFG dominance of the operation gets two of three shapes wrong. Measured against the real engine: identity constraint the guard is INSIDE the operation's arguments filter_by(owner_id = current_user.id) identity branch abort(403).controlledBy == [note.owner_id != current_user.id] -- the check runs AFTER the fetch, so it never dominates the operation, but it controls whether the value escapes dominating guard a discharger-naming call that dominates the operation That is why the query uses controlledBy rather than dominatedBy for shape 2. Two bugs found by running it, both the same class as the sanitizer bug in group 2 -- part of a fact flattened, the part carrying the signal dropped: * dominance_specs used only `calls` and `decorators`, omitting identity_sources (current_user, request.user) and constraint_params (owner_id, user_id) -- which is how an identity constraint is actually written. The arbiter reported "no guard" on handlers that plainly had one. request_sources and permissive_values stay excluded: they describe the vulnerable shape, and including them would let the arbiter read a bug as its own fix. * matching operations by code containment yielded four or five desugared rows per operation (the bare `tmp1.filter_by` receiver among them, carrying none of the guards the real call has), so a guarded operation could be flagged. Matching is by call name now. Live, deterministic, zero model calls: leaky -> model_entailed no discharging guard, while 2 sibling handlers are guarded edit_note -> no finding (ownership check controls the denial) list_notes -> no finding (identity constraint in the query) 4.2 authors the sink facts the go/no-go measured as its largest miss class (11 of 23): Django ORM raw SQL, and the four families that previously had no fact routing to them by CWE at all -- path, output_encoding, untrusted_destination, prototype. Each entry is a documented dangerous API of the named library with its provenance recorded inline, and rule_ids left empty because a sink fact is not a detection rule. Flagged for maintainer review as data, given the standing "no LLM-authored patterns" constraint. The exact-set assertion written in task 1.1 to hold this gap open is updated rather than deleted, so it still fails if a family ever loses its seed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 14:45:12 +02:00
Marius CiepluchandClaude Opus 5	b8f8f727c6	feat(model-grounded-detection): group 3, the evidence ladder and the judge ... 3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The existing evidence_level carries `static_corroboration`, emitted by the flat IR overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as model_corroborated would claim a CPG arbitrated something it never saw, and rewriting every legacy finding to suspicion would be a product-visible regression nobody asked for. So `rung` defaults to suspicion and is raised only by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4: nothing rises above suspicion without a Verdict behind it. The witness travels with the rung, because a rung without its evidence is just a louder assertion. It reaches markdown, JSON and SARIF. 3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive outcomes: the model entails (report it, never call the model), the model contradicts (drop the claim, record why), the model cannot decide (report suspicion). The LLM gets one bounded typed question with tools=[] so it cannot search for the site, asked only about the residual the graph cannot settle -- whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow the CPG already found; it can never conjure one. Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while loop, or an import from the deleted round machinery. That test first matched the word "votes" in the module's own docstring, which is why it now inspects code rather than text. Verified live against a real CPG, not only scripted rows: vuln rung=model_entailed request.args["name"] -> db.execute(...) fixed rung=suspicion no source reaches this sink model calls made: 0 Both sides arbitrated by the graph with the LLM never consulted -- the thing the prior architecture could not do at any price. 678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:49:55 +02:00
Marius CiepluchandClaude Opus 5	7387d2f190	measure(model-grounded-detection): task 2.3 reads GO on Joern ... Injection slice, 50 pairs, ~30 minutes of real engine time. model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2% ... on minimal-fix pairs only 61.5% (16/26) pairs distinguished 32% overall, 38.5% on minimal-fix determinism (Req 6.4) every verdict identical on re-run cost median 32s/pair, max 58s CPG build failures 1 The engine is not the constraint: all 23 misses trace to our own data model -- 11 sinks absent from the fact tables (Django ORM objects.raw, self.query, GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and one C translation unit that does not parse standalone, counted separately so it can never read as "no vulnerability". That is exactly the result that justifies adopting a CPG rather than building one. The finding that changes the design: classifying what the 50 injection FIXES actually do gives other_or_validation 37, bound_parameters 10, argv_list 2, sanitizer_call 1. The dominant fix is a validation guard on the path, not a sanitizer and not a sink shape -- and taint reachability can never separate those pairs, because the flow still reaches the sink and what changed is that a guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is the same primitive task 4.1 builds for access_control, with the guard vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms collapse into one, and 4.1 should now precede the remaining TaintSpecs. Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs: 12 are different programs (local-python-vulnerable is 7% similar with 83 changed lines) and 12 are large rewrites. A raw distinguished_rate would measure corpus quality and report it as model quality. Also corrected in the notes: the Flask-shaped source table is a real gap but a minor lever -- adding Django patterns would gain a source for exactly one pair. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:26:59 +02:00
Marius CiepluchandClaude Opus 5	e127e430ed	feat(model-grounded-detection): parameter sources, so the measurement is like-for-like ... Only 11 of the 50 injection pairs contain a framework source token at all. The other 39 receive untrusted input as a FUNCTION PARAMETER, because SVEN-style pairs are function-level extractions: def add_user(username, password): os.system("useradd ... -p " + encPass + " " + username) A framework-anchored source model finds nothing there, which caps entailment on this slice near 22% by construction -- a property of the source model, not of the engine. More to the point, the flat-IR baseline this feature is measured against already counted these (source_kinds: ["parameter"]), so a run without them is not comparable to the 2.2% it is supposed to beat. The flag is opt-in and off by default, because the two settings are right in different places: in a function-level pair the function boundary IS the trust boundary, while across a whole repository most parameters carry internal values and treating them all as untrusted is not sound. The corpus harness opts in; a repository scan does not. On the pair that exposed this the model now resolves both flows, including one through an intermediate binding: username -> os.system(...) 2 steps password -> crypt.crypt(...) -> encPass -> ... 7 steps Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:56:13 +02:00
Marius CiepluchandClaude Opus 5	1765e51da1	feat(model-grounded-detection): the safe-shape test, without which the model arbitrates nothing ... The first live run produced the finding that shapes this design: taint reachability alone does not distinguish a fix from a bug. On the canonical SQL pair the flow request.args["name"] -> name -> execute exists on BOTH sides -- vuln db.execute("select ... '" + name + "'") one argument, interpolated fixed db.execute("select ... = %s", (name,)) two arguments, bound -- so the model entailed the fixed twin exactly as readily as the vulnerable one, and arbitrated nothing. What separates them is the SHAPE of the sink call, which is precisely what safe_shape_sinks preserved from the earlier sanitizer fix. The query now reports sinkArity and sinkArg0Literal, and a flow ending at a sink written in its declared-safe form is not a vulnerability. Two bugs in that shape test, both found only against the real engine: Joern's `call.argument` includes the RECEIVER at argumentIndex 0, so `db` was counted as an argument -- making a one-argument interpolated call look like a two-argument bound one, the exact inversion of the test -- and the "first argument" whose literalness decides a format-string sink was the receiver rather than the format. Both fixed with argumentIndexGt(0). The rule is narrow on purpose: only sinks named in safe_shape_sinks are subject to it (os.system(x) is not safe merely because it takes one argument), and a row carrying no shape fields is judged on the flow alone -- an unknown shape must never be read as a safe one. End to end on the pair, deterministic and with no model call: vuln -> model_entailed request.args["name"] -> db.execute(...) (line 4) fixed -> None Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:51:42 +02:00
Marius CiepluchandClaude Opus 5	f59e0f5b5a	fix(model-grounded-detection): two modelling bugs the first live Joern run exposed ... Both were found by running the real engine, and neither was visible offline. 1. A sink was its own sanitizer. The shipped facts mark some sanitizers with a *shape* qualifier -- parameterized ("execute(sql, params) binds rather than interpolates") and literal_format_arg ("printf with a constant format string is safe"). These describe a safe FORM OF THE SINK CALL, not a call that cleans a value on the path. Flattening them into the sanitizer list made `execute` both the injection sink and its own sanitizer, so every SQL flow would have reported as already-clean and nothing could ever be entailed. Fixed as a class, not an instance: any fact carrying a shape qualifier is excluded from `sanitizers` and kept in `safe_shape_sinks` for the shape test task 4.2 will model properly. Fixing only the Python `parameterized` case would have left C's entire printf family -- which the generalised test then caught immediately. 2. labeled_family lost its deterministic derivation in group 1. The deleted classifier resolved a pair's family by declared label, else obligation, else mechanism, else CWE, and only asked an LLM when every link failed -- which for a labeled corpus was never. Group 1 reduced this to the declared label alone, which silently emptied every family whose pairs are labeled by CWE: the injection slice went to zero pairs. The chain is restored, still with no model call, and the family census again matches the committed entailment ceiling exactly (injection 50, memory 176, access_control 8, ...). Also: the Joern seam now runs in the CPG's own scratch directory, because `joern --script` writes a `workspace/` beside the working directory and would otherwise leave one inside the repository being analysed. The taint query itself was corrected against the live engine: the sink is the call's ARGUMENTS (data flows into arguments, not into the call node) and reachableByFlows is called on the sink with the source. With that, the three-hop chain our flat IR could not resolve is found: request.args["cmd"] -> cmd -> "echo " + cmd -> full -> os.system(full) Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:45:42 +02:00
Marius CiepluchandClaude Opus 5	2b973dc0db	feat(model-grounded-detection): tasks 2.1-2.2, the Joern seam and the taint verdict ... The whole engine integration is one subprocess boundary: joern-parse builds a cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between fence markers, because Joern's stdout carries a banner and a REPL prompt around whatever a script emits. Nothing is imported in-process, so the core install keeps dependencies = [] -- a test walks the seam's AST to assert it imports only the standard library and names no JVM binding. Fail closed everywhere. A build or query that errors, times out, or prints something unparseable yields no verdict and the caller reports suspicion. The taint verdict distinguishes this from absence deliberately: a query returning None means "the engine could not decide", which is not "there is no flow", and conflating them is how a missing engine silently becomes a clean bill of health. The verdict is where the ladder earns its keep. An unsanitized source->sink path is model_entailed -- the graph establishes it and the LLM's claim is no longer load-bearing. A path a sanitizer may break is model_corroborated: the flow is real, but whether the sanitizer suffices is the semantic residual the LLM answers, checked against this verdict rather than averaged over runs. The chosen flow is picked by a total order (shortest, then lexicographic), so two runs over one CPG cannot disagree -- Req 6.4 asserted directly. The query itself is the part our flat IR could not do: reachableByFlows walks the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is exactly the one-hop limit that held measured entailment at 2.2%. Capability detection reuses semantic/engines.joern_available rather than growing a second notion of "installed". The audit manifest is regenerated; the new modules are named by Req 4/5/6 rather than counted as orphans. Task 2.3, the go/no-go measurement, needs Joern installed and has not run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:23:40 +02:00
Marius CiepluchandClaude Opus 5	9ed0e45c5f	feat(model-grounded-detection): group 1, subtract the noise architecture ... 99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16 modules, ~4,100 lines) and six semantic/ modules removed, with the three gates byte-identical to 2026-09-06-gate-baseline.txt at every step. The sequence is what made it safe: vocabularies ported first (1.1), importers detached second (1.2-1.3), deletion third (1.4). Every import break during the deletion was therefore a detachment miss in the importer and was fixed there -- no module was restored to satisfy one. Both misses were the same shape and were fixed as a class: operations.py and shapes.py each imported trailing_name from the deleted variants.py, so it moved to semantic/ir.py, its natural home, and both importers changed in one edit. Ported before deleting: the closed guard vocabulary (verbatim, nothing else holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from the retained ruleset facts rather than copying them. Re-homed: families -> model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint, plus Region, labeled_family and labeled_spans. labeled_family no longer falls back to the LLM classifier -- an unlabeled pair is honestly unknown. Four deviations, each recorded in the spec notes with its reason. improve/ evolve.py and validator.py were stripped, not deleted: each mixes the mechanism lever (gone) with the deterministic rule-status loop `ousast improve` still runs, which has no LLM in it; a test now asserts the lever's absence by symbol so it cannot creep back. The learning CLI group became `model candidates`, because deleting it outright would have orphaned the retained enumerator by removing its only entry point. thinking moved from [learning] to [models]. A gap named rather than papered over: no retained sink fact routes by CWE to path, output_encoding, untrusted_destination or prototype -- the same hole the entailment ceiling diagnosed. The test asserts the seeded set and the gap set exactly, so task 4.2 must update it when it authors those sinks. The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness with no production caller and no CLI, alive only through its own test. The five named subsystems each carry an explicit decision. Manifest committed, with a test asserting the tree still matches it. Group 2 (the Joern go/no-go) has not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:15:54 +02:00
Marius CiepluchandClaude Opus 4.8	fc4dab97fb	docs(model-grounded-detection): attach Req 7.4 (verifier boundary) to task 1.1 ... The prior commit's edit did not match and 7.4 was left unclaimed. model/specs.py is read-only data -- no LLM and no optimiser writes it -- and task 1.1 now carries that criterion with a test asserting the module exposes no writer. All 49 criteria trace to a task. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:47:29 +02:00
Marius CiepluchandClaude Opus 4.8	6fe8682993	docs(model-grounded-detection): 16 tasks in 6 groups, with a coupling map so Opus does not crash-discover ... Design approved. All 49 criteria trace to a task. The tasks carry an agent-guidance preamble aimed squarely at the escalations this session hit: the eighteen-defect lesson (fix the class, not the instance), the gate-baseline as the safety net (the whole learning/ package is off the gate path, so its deletion cannot move a gate -- a moved gate means you touched a retained path), the zero-dep capability-detection pattern to copy verbatim, and -- most important for the subtraction -- a full coupling map naming every touchpoint of the removed code (cli, pairs hunter path, config, reports, semantic seed/loo, the mechanism loop across improve/ and semantic/). The subtraction is not discovery by import error. Group 1 is pure gate-protected subtraction, sequenced so vocabularies are ported (1.1) and importers detached (1.2-1.3) before anything is deleted (1.4), then tests reduced in the same spirit (1.5) and the tree audited for orphans (1.6). Group 2 is a Joern go/no-go: task 2.3 reads go or stop, and groups 3-6 do not begin until it is green -- building past a red 2.3 is the mistake the whole spec exists to prevent. Parallelism is deliberately absent; the subtraction is sequential and the spike gates the rest. Not approved for implementation. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:47:10 +02:00
Marius CiepluchandClaude Opus 4.8	0a8cc9b5a7	docs(model-grounded-detection): design, subtraction first and a Joern spike that can stop it ... Requirements approved. All 49 criteria trace to a component. The shape: a Code Property Graph (Joern) is the arbiter; taint reachability and guard dominance produce evidence-ladder verdicts (suspicion / corroborated / entailed / execution_confirmed); the LLM answers one bounded question per candidate and its claim is checked against the CPG, never averaged. Two design disciplines carried from the requirements: Phase 1 is subtraction, and it removes ~5,000 lines before a line of Joern is written. The learning/ noise architecture is absent from the gate path, so rounds/acceptance/proposer/journal, the K-run/floor/sign-test in scoring, and the mechanism loop delete without moving a gate -- proven by the committed gate-baseline run before and after, and by an import-graph check that fails the build if a removed module is still imported. The guard and sink vocabularies are ported into model/specs.py *before* their modules are deleted. Tests are reduced in the same change, with a test->requirement map. A module-audit manifest classifies all 99 modules and names fusion/mcp/skills/harness_ext/ hunter_harness for an explicit keep-or-remove decision. Phase 2 is a Joern spike on the injection family alone -- the CpgBackend seam (subprocess to joern-parse + a CPGQL query emitting JSON, out of process, one isolated touch-point) and one TaintSpec, end to end, with a determinism test. It can end the design before the integration cost is paid, the same way the candidate ceiling could have in the prior spec. Joern is capability-detected on PATH like the semantic extra; without it every verdict degrades to suspicion and the core stays zero-dependency. What survives re-homes under model/: the taxonomy, the candidate enumerator (suspicion band), the endpoint client, and the obligation facts feeding the dominance queries. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:39:49 +02:00
Marius CiepluchandClaude Opus 4.8	1ef578f793	docs(model-grounded-detection): requirements, subtraction first ... 11 requirements, 49 criteria. The outgrowth is addressed head-on and it leads, because building on a superseded 99-module / 832-test surface is how this session logged eighteen defects. Req 1-3 are subtraction and come first: 1 remove the noise architecture by name -- rounds, acceptance, proposer, journal, the K-run/floor/sign-test in scoring, the mechanism lever and shape-search loop -- preserving the security *models* they carried. The whole learning/ noise layer is absent from the gate path, so it deletes without moving a gate. 2 reduce the 832 tests to what protects surviving behavior: a test that pinned a deleted mechanism dies with it, every survivor maps to a surviving requirement, and the "does nothing / greps --help / passes with the capability absent" tests are removed outright. 3 audit all 99 modules -- load-bearing / standalone / orphaned -- and delete the orphans, with fusion/mcp/skills/harness_ext/hunter_harness named for an explicit keep-or-remove decision. Req 4-8 adopt the CPG arbiter: Joern behind an optional extra (core stays zero-dep, degrades to suspicion without it), the evidence ladder as the verdict type, deterministic taint-reachability and guard-dominance verdicts, our taxonomy and source/sink/sanitizer/guard models ported to CPG queries, and the LLM reduced to a proposer whose claim the model checks -- no K-runs, no floor, no acceptance rule, no evolve loop. Req 9 makes the corpus the model's calibration set (a pair the model can't split is a named model gap). Req 10 keeps execution as a deferred, adopted-from-Clearwing tier for memory/C only. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:36:58 +02:00
Marius CiepluchandClaude Opus 4.8	a3945427ab	docs(model-grounded-detection): IL not AST, and adopt the CPG engine (Joern) ... Two design questions answered by the entailment measurement. IL, not AST. Our semantic/ir.py is a lossy AST projection -- flat call/bind lists, no CFG, no PDG -- which is *why* entailment is 2.2%. Guard dominance is a dominator-tree question and path-sensitive taint needs branch structure; a flat bind list expresses neither. Real abstract interpretation is a fixpoint over a CFG, so the substrate is a Code Property Graph: AST + CFG + PDG + call graph in one queryable graph, taint as source->sink-minus-sanitizer reachability, absence as dominance. Adopt, not build. Building a multi-language CFG/PDG/taint engine on tree-sitter is reinventing CodeQL/Semgrep/Joern -- the maximal case of this session's failure mode. Joern is Apache-2.0, covers our exact corpus (C/C++/Java/JS/Python/Kotlin/binary), has a taint engine, is embeddable, and a CPG+LLM MCP bridge already exists. The 2.2% on our flat IR is the measured distance to a real CPG, and closing it by hand is reimplementing Joern. The flow: source -> Joern CPG -> taint + dominance -> deterministic entailed/corroborated verdict -> LLM judges the residual, checked against the CPG. What stays ours is the contribution: the taxonomy and the source/sink/sanitizer/guard models for the web/logic CWEs, ported to CPG queries, plus the LLM-checked-against-the-model orchestration. Joern is a workbench with no opinion on these classes; that opinion is our layer. Named cost: Joern is JVM/Scala, so server-mode or the MCP bridge, not in-process. Three-layer adoption now: Joern CPG for the web/logic arbiter, Clearwing for the memory/C execution tier, and our security modelling + LLM orchestration as the layer that binds them. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:22:44 +02:00
Marius CiepluchandClaude Opus 4.8	9e811c1baf	measure(model-grounded-detection): the entailment ceiling is 2.2%, and it is depth not a wall ... The analogue of the candidate ceiling, one rung up: not "can a site be enumerated" (96.6%) but "can a deterministic abstract-interpretation model DECIDE the vulnerability without trusting the LLM". Offline, no model calls. model_entailed today 2.2% (2/92 web/logic pairs) model_corroborated, one-hop taint 24% (22/92) taint reachable, multi-hop 35% (26/74 flow families) candidate site enumerable 96.6% (for comparison) The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating, and it is the finding. The model cannot be a pure arbiter at v1. But the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed literal sink table (misses project wrappers and path/xss/prototype sinks), and a function-naming gap that eats 20/50 injection pairs. Deepening the taint to the candidate enumerator's depth-4 chain already moved injection 24% -> 48%. So the course is a layered arbiter, not a pure one: report model_entailed where the model proves it (deterministic, no LLM); let the LLM judge but check its claim against the resolved taint on the ~35% corroboration band (noise removed there); honest suspicion for the rest, which is where the deferred execution/differential tier earns its place. Modeling confirmed as abstract interpretation; SMT reserved for the memory/C tier; concolic avoided. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:10:47 +02:00
Marius CiepluchandClaude Opus 4.8	091303c7c0	docs(model-grounded-detection): the arbiter is a program model, not a sandbox ... Maintainer correction: execution isn't possible for all SAST targets, and the bugs this project owns -- IDOR, missing authz, injection -- don't crash anything. Execution is the easy arbiter only for memory-safety-in-C, which is not our world. What execution provided was an independent, deterministic arbiter of whether a claim is real -- not the same LLM that made it, not a noisy vote. That arbiter can be built statically, from the code, as a program model the claim is checked against. LLM proposes; the model disposes. Same architecture, same repair of the root cause, no runnable target required. The evidence ladder becomes static and deterministic: suspicion -> model_corroborated (claim consistent with resolved dataflow / a guard its siblings carry is absent) -> model_entailed (the model alone establishes it) -> execution_confirmed (deferred tier, C memory-safety only). model_corroborated and above are deterministic -- run the checker twice, same verdict -- which removes the run-to-run noise the same way a crash would, with no K-runs and no target to build. Our IR candidate enumerator (96.6% ceiling), binding chains and obligation checker stop being prompt context and become the arbiter itself. The course: promote those assets from inputs to arbiter, delete the noise layer, make the LLM a proposer the model audits, keep the corpus as the model's calibration set (the fix diff is what the model should see), and defer execution to Clearwing for the memory-safety classes only. Build the arbiter we can build from code; adopt the one we cannot, when a class demands it. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 10:41:07 +02:00
Marius CiepluchandClaude Opus 4.8	81a63cf5ac	docs(execution-grounded-detection): corrective brief that supersedes the project ... Written after five evolve rounds produced zero decisions distinguishable from noise, and after reading Clearwing, Big Sleep, and the AIxCC 2025 finals. The premise we got wrong: OpenUltraSAST treats the LLM as a noisy oracle to average (K runs), floor (noise budget), gate (sign test) and slowly improve (evolve loop). 120 runs produced 120 distinct trajectories -- the variance is the loop's freedom to search, not the model's uncertainty. And the premise is empirically false as a claim about LLMs: Big Sleep found a real SQLite zero-day in Nov 2024; AIxCC finalists uncovered 18 real 0-days across 54M LOC, with the 2nd-place system using non-reasoning LLMs at 90% accuracy and $181/point. The lever is architecture, not prompt optimization. What the winners share, and Clearwing is the open instance of: the LLM is a judgement layer on an execution substrate, and truth comes from the sandbox -- a crash, a sanitizer report, a patch that stops the crash -- not from a score against a labeled corpus. Clearwing's poc_runner (apply diff, recompile, re-run, does-the-crash-still-happen) replaces our entire noise architecture. We built K-runs because we had no ground truth; they generate their own. The course: adopt Clearwing as the substrate (the maintainer already forks it), delete the noise architecture rather than fix it, and contribute the one thing Clearwing lacks -- the web/logic detection path. Our IR candidate enumerator (12.1% ruleset vs 96.6% IR, measured) and our obligation/absence-bug checker reach the dangerous web bugs that never crash a sandbox, which Clearwing's crash-oracle model cannot. The open research problem is a differential oracle for those classes: the vulnerable-parent/fix-commit pair as ground truth, with a request the fixed version rejects and the vulnerable one serves. Recommendation is explicitly not to rebuild: eighteen defects this session, all ours, against a 19k-line battle-tested MIT substrate. Not approved. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 10:04:43 +02:00
Marius CiepluchandClaude Opus 5	66a51c674c	docs(learning-harness): round 5, and what it says about the experiment itself ... Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered to the detector as "Worked examples" rather than "Known non-issues, do not report these" -- and was rejected because one pair of eight moved, at p = 1.000. So the experiment I designed to give the demo lever a fair test could not give it one. Correcting my authoring error changed the reason for the verdict and not the verdict, because the rule cannot tell a lever that helps from a lever that does nothing. Five rounds, $2.17, zero decisions distinguishable from chance. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 08:09:50 +02:00
Marius CiepluchandClaude Opus 5	78924316be	docs(learning-harness): the reading of the first evolve rounds ... Four rounds on injection, $1.76. Not one decision carried evidence, in either direction. Round 2 rejected a change because one pair of eight moved, at p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, both readings are what changing nothing looks like. I corrected an overstatement in the notes: after two rejections I wrote that the rule was near-incapable of accepting. Round 3 accepted. The accurate statement is symmetric and worse -- it accepts on noise as readily as it rejects on it. Escalated, not applied: Req 9.5's threshold rule should be replaced by an acceptance that survives the measured variance. The Pareto archive and the sign test both exist and neither decides anything. The honest caveat is that at a minibatch of 8 a reliability-gated rule would accept almost nothing either -- the corpus is the binding constraint, not the rule. And the finding that outranks the rounds, in the maintainer's words: the error is not the model, the error is the elsewhere. Eighteen defects this session, every one in this repository, three of them the same shape of fixing an instance and missing the class. The default assumption for an unexplained number is now that we broke it -- which puts pro's 46.7% against flash's 83.3% on our side of the ledger until shown otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 07:39:48 +02:00
Marius CiepluchandClaude Opus 5	3ebcc6729a	fix: a body that stops mid-read is transient, and a round that dies still says so ... Round 5 died on `http.client.IncompleteRead(0 bytes read)` and left no record at all. Two defects, both ours. The retry classified only connection-level failures. IncompleteRead and RemoteDisconnected arrive while the *body* is being read, after the connection succeeded, so they are not URLError -- they escaped both the retry and the error wrapper and walked out of the round. The network is allowed to be unreliable; losing a paid round to it is not the provider's defect. And the exception path was wrapped around the stages that come *after* the proposal is applied, because those are the ones with something to revert. The stages before it had already spent money, and a round that leaves no record is a round nobody can account for. Every outcome is journalled now, with the exception type in the reason. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 07:37:41 +02:00
Marius CiepluchandClaude Opus 5	cde57e4293	fix(learning-harness): say why the proposer had nothing, and stop hard-coding its vendor ... The meta-agent round journalled `no_proposal`. The real cause was an authentication failure: `_model_config` hard-coded the anthropic provider while the detector was talking to DeepSeek and no ANTHROPIC_API_KEY exists here. The composition itself was fine -- MetaAgent constructed, evolve started, budget $5 -- and the trace was thrown away. `no_proposal` covered two different worlds: a proposer that considered the facts and declined, and one that could not start at all. The journal now carries the proposer's own reason, and the vendor comes from `[harnessx] provider` instead of being assumed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 22:24:40 +02:00
Marius CiepluchandClaude Opus 5	84b1ba7a72	fix(learning-harness): the published table lost a whole slice ... `publish` keyed baselines by model alone, so the vibe-py run silently replaced the agent-vfc run and the table showed hand-written teaching code as if it were the corpus. That is precisely the failure the constrained-detector requirements exist to prevent, live in a published artifact. Third time today that a key was fixed in one place and not another -- `compare_baselines` got the model+slice label and `_baselines` did not, the same way the exception path got checkpointing and the timeout path did not. The table now carries a slice column and one row per slice, and the artifact is keyed by what identifies a run. Republished: agent-vfc's six families are back, including access_control at 0.000 recall on four pairs, which is the number that was missing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 20:44:56 +02:00
Marius CiepluchandClaude Opus 5	506bbe5f5c	fix(learning-harness): round zero writes down every pair it has paid for ... The 176-pair vfc baseline hit its four-hour timeout and wrote nothing. Scores were accumulated in memory and persisted once at the end, so roughly four hours of paid API calls were lost. I fixed the exception path this morning -- a provider hiccup costs one pair -- and missed that a timeout, a kill or a machine going away cost all of them. Each pair is now appended to `scores.jsonl` and flushed as it is measured, and `--resume` keeps what a previous run of that exact baseline already bought. This also gives the per-pair outcomes the round-3 review asked for, which make the noise floor auditable after the fact instead of only in aggregate. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 19:53:49 +02:00
Marius CiepluchandClaude Opus 5	78b4f33a8e	feat(constrained-detector): the ceiling, measured offline across every slice ... Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning candidates`, and the stop-or-go reading. No model is called anywhere in this group, so the condition that could have ended the feature was reachable for free. Verdict: go. agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection vfc (deciding) 84.7%, baseline still running, non-gating vfc-js (deciding) 52.9% -- see below github (deciding) 100.0% vibe-py (development)100.0% against 83.3% The margin is largest on agent-vfc, which is the slice the tool is actually for. The measurement forced a shape correction before the detector existed. config_secrets first read 63.6% with a bind-and-literal shape, because a permissive default is as often an argument to a call -- cors({origin: '*'}), app.use(session({secure: false})) -- as it is a binding. Adding call sites to that family's shape took it to 100% and recovered four agent-vfc rows. Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because eight of seventeen rows label a function the JavaScript CST names `<anon>` -- `module.exports.publish = function () {}` and its relatives. That is a function-naming gap, it caps the existing hunter on that slice exactly as much, and it is the same class as the `_declarator_start` defects found in learning-harness task 4.1. The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a measurement rather than a remembered number. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 18:44:19 +02:00
Marius CiepluchandClaude Opus 5	8d771901bf	feat(constrained-detector): candidates from the semantic IR, shaped by family ... Task 1.1. The detector stops searching: something deterministic decides what it looks at, ordered by (path, line, name) and bounded by a constant, so the batches a later task forms are determined by the input alone. A call candidate carries what the model used to grep for -- the arguments, and the binding chain the IR resolved for them, nearest first. An access control candidate is the operation with the guards in scope, because an absence bug has no sink. Two things the fixture taught while writing it. The IR emits a chained call twice at one line, once bare and once carrying the keyword arguments, so deduplication keeps the richest: a bare `filter_by` and `filter_by(user_id=owner)` are the same site and only one of them says what the constraint was. And `leaky` and `constrained` in the shared obligations fixture both discharge through `filter_by` -- what separates them is that one chain ends at `request.args.get('owner')` and the other at `request.user.id`. That contrast is now the test, and it is exactly the judgement the model is being narrowed down to make. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 18:23:54 +02:00
Marius CiepluchandClaude Opus 5	d9d6dfdbcf	docs(constrained-detector): 15 tasks in 5 groups, phase 1 able to end the feature ... Design approved in session. Every one of the 36 criteria is claimed by a task. Group 1 calls no model at all. The candidate enumerator, the ceiling report and the stop-or-go reading are offline, so the condition that ends this feature -- a deciding slice whose ceiling sits below its recorded detection rate -- is reachable before a cent is spent. The ruleset variant of this same design would have failed that condition at 12.1% against 64.9%. Group 2 is the only work that touches scoring.py, and its observable is that nothing changes: the recorded round-zero artifacts are the pin, and if they cannot be reproduced exactly then the refactor is wrong, not the numbers. Group 3 builds the detector, and task 3.3 is the test that would have caught the defect this spec exists for -- two runs must make an identical sequence of model calls. It lives in the suite and runs repeatedly, not in a one-off measurement. Group 4 is the maintainer's emphasis as code: deciding slices, development slices, the cross-slice rejection by name, and the overfitting gap printed beside every headline number. Group 5 compares the two detectors per slice, re-measures the trajectory count and the noise floor, and names -- without amending -- the learning-harness requirements that a collapsed floor would retire. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 17:32:13 +02:00
Marius CiepluchandClaude Opus 5	eaf9e3e370	docs(constrained-detector): design, with a phase that can kill it before it costs anything ... Requirements approved in session. All 36 criteria trace to a component. The shape: a deterministic pass enumerates candidate sites from the semantic IR, shaped by family -- call sites for injection and path, operations with their guards for access control, bindings and literals for configuration. The model answers one bounded typed question per candidate with no tools and no choice of what to look at. The predicates that decide whether a finding counts are applied while it is being made, with one retry carrying the violation, and they come from a single definition that the scorer also uses, so the two cannot drift. Determinism is a design property, not a hope: candidates are ordered by (path, line, name) and batched at a fixed size, so two runs make an identical sequence of model calls. That is the direct answer to 120 runs producing 120 distinct trajectories. Phase 1 is the candidate ceiling, offline, no model call, measured on the real-project slices as well as the teaching one. If the ceiling on a deciding slice comes in below that slice's current detection rate, the design stops there having spent nothing. Building that in as the first phase is deliberate: the ruleset variant of this design would have capped recall at 12.1% against a 64.9% baseline, and only a measurement caught it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:12:35 +02:00
Marius CiepluchandClaude Opus 5	c97dcc656d	docs(constrained-detector): requirements, with real-world code as the decider ... Renamed from detector-optimizer: the brief's own conclusion is that the optimizer is not the next thing to build, and a spec should be named for what it delivers. Seven requirements, 37 criteria. The load-bearing ones: Req 1 -- candidates come from the semantic IR, never the pattern ruleset, and the enumerator's ceiling is published as the limit on recall. Measured before building: ruleset 12.1%, IR 96.6%. Req 2 -- one bounded typed question per candidate, no tools, no choice of what to look at, and a test asserts two runs make an identical sequence of model calls. Req 3 is the maintainer's emphasis and it decides acceptance. The slices harvested from real projects decide; hand-written vibe-py is a development slice that cannot on its own justify a change; a change that improves vibe-py and regresses a deciding slice is rejected by name; and the overfitting gap -- 83.3% against 44.4% at baseline -- is published beside every headline number. The candidate ceiling is measured separately on real-world slices, because their functions are larger and the enumerator's behaviour there is not implied by its behaviour on teaching code. Req 5 -- the K-run and noise-floor machinery is re-measured against the constrained detector and deleted if the floor collapses, naming the learning-harness requirements that would be amended. Not approved; design not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:08:17 +02:00
Marius CiepluchandClaude Opus 5	309faad266	measure: the candidate generator must be the IR, not the ruleset ... A candidate-driven detector can only judge sites something else enumerates, so the enumerator caps recall. Measured over the 58 scorable web pairs before building anything: pattern ruleset: 12.1% ceiling -- and the misses are not near-misses, the ruleset finds nothing at all in most of these files semantic IR: 96.6% ceiling, a median of ten call sites and bindings inside the labeled function The ruleset would have capped us at 12.1% against the 64.9% the unconstrained loop already reaches. That design was one commit from being built. Two corpus defects found on the way, both mine from task 4.1. The overleaf excerpt started mid-import, because `_declarator_start` walks back to the nearest `}` and Overleaf writes JavaScript without semicolons, so the nearest one was inside `import { expressify }`. It then ended mid-object-literal, because registration mentions were appended as bare lines. Neither parsed, and a pair that cannot be parsed cannot be scored. Registration mentions now expand to the statement that contains them; the ceiling moved 94.8% -> 96.6%. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:04:07 +02:00
Marius CiepluchandClaude Opus 5	4e6afa862b	docs: reflect the trajectory measurement back onto the approved spec ... learning-harness divides cleanly. The measurement layer stands and is worth more than before -- it is what made a $0.23 experiment able to answer 'is our own instability structural?'. What was answering the wrong question: Req 3.5's K (an ensemble of distinct search procedures, not a repeated measurement), Req 8.2's noise floor (it measures how often an unconstrained search reaches the sink, so the regression budget grows with the search's own wandering), Req 9.5's acceptance rule, and the sign test added this session. And the highest-leverage change in either spec turns out not to be a prompt, a lever or an optimizer: stop asking the model to locate the sink. 555 grep calls against 71 entry_points, while the CST and the entry-point mapper already know where the sinks are. Limits stated in the brief: twelve pairs, one family, one model; the high-similarity bucket is two run pairs and its 0% is decoration. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:50:44 +02:00
Marius CiepluchandClaude Opus 5	5e2070bc17	measure(detector-optimizer): the detector never runs the same investigation twice ... Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled, recording every tool call. $0.23. The statistic I designed for could not be computed: across 120 runs there was not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side out of 5, and 78 distinct tool-name sequences ignoring arguments entirely. The detector is not sampling one measurement five times; it is running five different investigations and voting on the results. As a gradient the relationship is monotone: runs whose trajectories were similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%, low-similarity 30.9%. Three more facts. The search is grep: 555 grep_repo calls against 181 read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is guessing patterns to locate sinks the CST and the entry-point mapper already know. All the instability is on the vulnerable side; every pair produced exactly one finding set on the fixed side, so the noise is in detection and none of it in false positives. And findings converge much more than trajectories do: 78 trajectories, one to three finding sets per pair. The floor is therefore not measuring uncertainty about a question. It is measuring how often an unconstrained search happens to reach the sink. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:49:54 +02:00
Marius CiepluchandClaude Opus 5	bd3e2fe50f	docs(detector-optimizer): the noise is structural, and the optimizer is not the next thing to build ... The harness treats the detector as a black box emitting a noisy scalar, and K runs, majority voting, the measured floor, the regression budget and the sign test all follow from that. It is the stance you take when you cannot see inside what you are measuring, and we can. Temperature is already 0 and thinking is disabled, and 11 of 20 injection pairs still disagree with themselves. The freedom is in the trajectory -- which files get read, which greps get run -- and we built statistics to average over it instead of removing it. Three DSPy constructs attack the source: constraints enforced as predicates at generation time (we already compute every one of them, after the fact, at scoring time), typed decomposition of the single ReAct loop into steps whose deterministic parts we already answer without a model, and best-of-N against a checkable reward instead of a majority vote that discards the disagreement. The cheaper the detector model, the more of the return sits there and the less in prompt evolution -- the opposite of the order this brief first proposed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:29:36 +02:00
Marius CiepluchandClaude Opus 5	0db9b378d9	docs(detector-optimizer): SIMBA, the reflection meta-prompt, and the corpus ceiling ... SIMBA ranks instances by max-to-min score gap across variants, which turns our noise floor from an obstacle into an instance sampler: the 11 injection pairs that disagree with themselves are the pairs where a rule change can flip an outcome. Its two alternating strategies -- append a demo, append a rule -- are the split our levers already have. GEPA's reflection meta-prompt is a specification for the feedback function, not just a prompt: it asks the reflector to extract domain facts the model cannot re-derive later, which means feedback must carry which sink, which span and which side leaked, not a verdict. And the finding that outranks both: GEPA splits a trainset, SIMBA batches 32. Injection has 12 train teachers. These algorithms are sample-efficient in rollouts, not in instances, and they will not shortcut a corpus of twelve. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:24:37 +02:00
Marius CiepluchandClaude Opus 5	78f20df167	docs(detector-optimizer): brief for a published optimizer behind a DSPy-shaped interface ... The learning harness built the measurement honestly and then hill-climbed on it by hand. Four review rounds' worth of defects in that loop were defects of its shape, not its code. Read from GEPA's Algorithm 1 and 2 rather than from a summary, three things bear directly on ours: the acceptance gate is deliberately cheap and noisy, because the Pareto frontier -- not the gate -- is what protects against a bad accept; selection is stochastic and diversity-preserving, which is what stops the search collapsing onto one lineage; and the expensive full evaluation is spent only on candidates that already passed the cheap gate. Two consequences for us are honesty findings, not performance ones. Our acceptance rule reads the holdout every round, so repeated rounds overfit the test set -- the same leak class this project already measured once, one level up. And the proposer is handed a score where the published work hands it textual feedback from the evaluation trace, which our evaluator produces and then discards. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:16:29 +02:00
Marius CiepluchandClaude Opus 5	b57b1a35d8	feat(learning-harness): thinking is a measured condition, and every round ... records whether its decision beat noise Two things the maintainer was right about. Thinking was disabled so `temperature: 0` would not be silently ignored -- a trade that only pays if it buys determinism. Round zero says it did not: 11 of 20 injection pairs disagreed with themselves across five runs, so the variance is in the agentic path, not the decoder. We paid the reasoning cost on a reasoning model and got nothing. It is now `[learning] thinking`, recorded in the artifact and part of its key so the two modes cannot be read as one number. The default stays off only because that is what round zero was measured under. And the deeper point: the spec built `sign_test` and `reliable_change` and the acceptance rule calls neither -- `decide` compares raw deltas. On a detector with that floor a raw delta is mostly noise, and a rule that brittle is exactly what makes a deterministic decoder look necessary. It is not; the alternative is an acceptance rule that models the score as noisy, the way reflective-evolution and Bayesian prompt optimizers do. Changing Req 9.5 is a maintainer's call, so this does not change the decision: every round now journals the two-sided sign test over the pairs that moved, and after four rounds the journal will say whether any decision could be told from a coin flip. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:12:09 +02:00
Marius CiepluchandClaude Opus 5	1e4f212868	fix(learning-harness): logprobs are on the choice, not the message ... OpenAI-shaped providers put `logprobs` beside the message rather than inside it, so the confidence signal the classifier is meant to abstain on was always None -- a knob that reads 'never unsure' whatever the model said. Measured against DeepSeek while checking: it returns the structure but not the values. Every chosen token comes back at logprob 0.0 and every alternative at -9999.0, on a deliberately uncertain prompt as much as a certain one. The field is there; the information is not. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:05:56 +02:00
Marius CiepluchandClaude Opus 5	ce89b0dbe0	fix(learning-harness): a slice is part of what identifies a baseline ... Keyed by model alone, the memory-safety run over the vfc slice silently replaced the run over the web families, and the published table would have shown one of them as if it were everything. The artifact key and the comparison label now carry the slice selection when it is not the default. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:39:43 +02:00
Marius CiepluchandClaude Opus 5	c1ef3ca539	fix(learning-harness): a provider hiccup costs one pair, not the whole run ... Round zero is the most expensive thing this harness does. An error two thirds of the way through raised straight out of `run_baseline` and took everything already measured and paid for with it, so the only honest response to a flaky endpoint was to run the whole thing again and hope. The pair is now recorded as unscorable with the reason `detector_unreachable` and the run finishes. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:20:52 +02:00
Marius CiepluchandClaude Opus 5	fc98a48579	docs(learning-harness): record the two in-session amendments in spec.json ... Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:19:03 +02:00
Marius CiepluchandClaude Opus 5	d2ebb58007	fix(learning-harness): a hunt that ends in prose is asked to format it ... Measured live against DeepSeek on one pair: the model spends its step budget on tools and then answers in a paragraph that names the sink, the argument and the line -- which parses as nothing. A detector that found the bug and reported zero is indistinguishable from one that found nothing, and every family in round zero would have baselined at zero for this reason alone. The loop now takes one more turn when the hunt ends in prose or runs out of steps, asking the provider for JSON and handing back the reasoning the model already produced. It formats an answer that was reached; it never asks for a new one, and a hunt that called no tool never gets there. Same pair, same model, before and after: recall 0.00 -> 1.00, Youden 0.00 -> 1.00 over three runs per side. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:18:09 +02:00
Marius CiepluchandClaude Opus 5	a1e7b1a20f	fix(learning-harness): the run reads the meter it publishes ... The chat client meters itself and nothing passed the meter to the baseline or the round, so every published cost would have been 0.00 -- and the round's cost cap, which reads the same meter, could never have fired. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:12:14 +02:00
Marius CiepluchandClaude Opus 5	5fbb0494a7	docs(learning-harness): record the offline dress rehearsal of round zero ... Round zero and one evolve round run end to end with the scripted client over the whole vibe-py slice: per-family and per-slice blocks, twins out by reason, a floor each, a proposal applied and reverted byte for byte, the sweep reporting zero negative flips, the rejected version archived. The only thing the live run adds is a detector that can see. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:06:46 +02:00
Marius CiepluchandClaude Opus 5	32f16f78ff	fix(learning-harness): a baseline over a named slice measures that slice ... `--slice vfc` selected the vfc rows and `run_baseline` filtered them all out again against its own default of the two web slices, so the memory-safety baseline Req 8.4 asks for produced an empty report and said nothing about it. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:05:23 +02:00
Marius CiepluchandClaude Opus 5	3acad97297	docs(learning-harness): 5.3 targets injection, and 5.2 waits on the key ... Measured on the corpus under the split and review-tier rules, access_control has 1 train teacher and 4 holdout pairs: every agent-vfc row sits at tier `title` and the vibe-py absence rows that could teach are trap pairs whose two sides are different handlers, so the train split teaches zero obligation shapes. injection has 12 teachers and 10 holdout pairs, the only family with enough train evidence to tell a real change from noise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:03:35 +02:00
Marius CiepluchandClaude Opus 5	2dad25d312	fix(learning-harness): the round-4 review findings ... Three defects, two of them the same shape as ones already rejected once: - The meta-agent proposer still could not propose. `MetaAgent(inner_model=)` takes a ModelConfig; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting AttributeError into `meta_agent_failed` -- which reads exactly like the model declining. The composition is now a seam a test can pin, and a construction error is re-raised rather than reported as a quiet round. - `ousast pairs --hunter` crashed in its documented form: the K floor landed with `--k-runs` still defaulting to 1. The covering test always passed `--k-runs 4`, which is why nothing noticed. - 28 rows (12% of the corpus) resolved a span on the vulnerable side and none on the fixed side, so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said. `fix_function` now reaches the catalog and the scorer, and the generator derives it from the fixed excerpt for the 25 trap rows that never declared one. Measured after: 0 rows. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:03:10 +02:00
Marius CiepluchandClaude Opus 5	75ae9ee47b	test(learning-harness): the gate baseline and the degradation matrix ... The three gate outputs are captured at 71efc9c -- the commit before the first line of this feature -- and a test reproduces them byte for byte in both matrices. Byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI. Six absences are each exercised once with the reason they record: HarnessX, the chat endpoint (the run still writes its artifact), the family configurations, the network for pointer pairs, the semantic extra (named, not guessed), and the sandbox -- so a claim nothing could run is not confused with a claim that failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 13:20:23 +02:00
Marius CiepluchandClaude Opus 5	3e71e9e88b	feat(learning-harness): fix dates and the post-cutoff slice ... The catalog generator fills `fix_date` from the recipe at the precision the recipe has -- a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row counts as post-cutoff only when its earliest possible date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it. Each family now reports the same numbers over the rows fixed after the cutoff, beside the whole number and never instead of it, with the undated count on its own line. Without a configured cutoff every scoreboard says so rather than inventing one: guessing a model's training cutoff and then publishing a contamination number from the guess would be worse than reporting none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 13:15:20 +02:00