67 Commits

Include renames

Author	SHA1	Message	Date
Marius CiepluchandClaude Opus 5	435c14f3ec	docs(contributor-scan): 5.8 -- WordPress class by class, including what it cannot ask ... Three plugins, each pinned at a version whose advisory was read out of the code, each measured against its own fixed commit: unauth injection, sink in the receiving function pmpro FOUND arbitrary file operation mwwpform FOUND unauth injection across a WordPress hook wpstatistics MISSED missing capability check -- cannot be asked unescaped output -- cannot be trusted Both hits are entailed by the graph alone -- no judge, no spend -- and both disappear on the fixed side, because both fixes are calls the fact tables carry: `$wpdb->prepare` and `realpath()`. The engine is not recognising a bug; it is failing to find a cleansing call on a path it resolved, then finding one. Precision is stated rather than flattered. PMPro's vulnerable side reports 15 entailed, 4 from a named parameter and 11 from the receiver, and one of those survives into 2.9.8 at a query whose every fragment is `esc_sql`-wrapped. The miss is the useful half, and it is checkable rather than inferred. WP Statistics' CVE has a modelled source, a modelled sink and `esc_sql` as its fix, and reports nothing on either side, because the file holding the sink has zero superglobals, makes no call into the class holding the source, and is joined to it only by add_filter('wp_statistics_current_page', array($this, 'set_current_page')); return apply_filters('wp_statistics_current_page', $current_page); Both ends name the callback with a string, so no frontend can draw the edge. Most plugin data travels this way. What the engine reports there instead is two other sites, the same class at the wrong lines, identical on both sides -- and not noise: upstream annotates both `// phpcs:ignore WordPress.DB.PreparedSQL` and later rewrote one with `$wpdb->prepare`. Two classes could not be asked at all, which is not the same as a miss. `dominance_specs(language="php")` is empty, so missing-capability is never asked; and `output_encoding` shares the one flat sanitizer list, which holds `esc_sql` and `intval` but not `esc_html` -- wrong in both directions, so no XSS verdict from this engine currently means anything. 5.10's root cause is recorded too: php2cpg's parser bridge, a race rather than a scale limit, and the mitigation is per-file exclusion. Adds 5.11 (the hook edge, with CVE-2022-25148 as its observable) and 5.12 (taint a field where it is assigned, with CVE-2023-6559 as its guard). Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 18:36:00 +02:00
Marius CiepluchandClaude Opus 5	e0442d146c	fix(cpg): a graph the frontend emptied must not read as a clean repository ... Scanning a two-file WordPress slice returned 64 regions examined, zero findings and zero degradations. The graph it examined had no methods in it. php2cpg 4.0.623 feeds several files to one `php-parse` process and reads its stdout and stderr merged. php-parse writes a `====> File <next>:` banner to stderr the moment it starts the next file, while the previous file's JSON -- megabytes of it -- is still draining from a block-buffered stdout. The banner lands inside the JSON, ujson reports expected json value or } got "=" at index 5011645 and the frontend drops that file. Reproducible on real plugin code, and the size that matters is one file's AST dump, not the repository's: class.memberorder.php alone (1834 lines) -> 142KB CPG, no failures the same file + a two-line file -> 5.7KB CPG, BOTH dropped One oversized file empties the whole graph. `joern-parse` then does not propagate any of the frontend's warnings: 13 lines of output, exit status 0, "Successfully wrote graph". This is what made a 637-file plugin unscannable and reported it as clean -- it was never a scale limit. Two fixes, because the failure arrives on two different paths: * `_unparsed_files` reads the per-file warnings the frontend does emit, and the scan reports them as `files_unparsed`. This covers the direct-frontend path where the warnings are visible. * The batched query now carries a `__census__` of the graph -- method and file counts -- under a key no request id can collide with, and a scan over a graph with no methods degrades as `cpg_empty`. This covers joern-parse, which shows nothing. It rides the batch rather than costing its own invocation, because avoiding a JVM start is the entire reason batching exists. Neither invents a verdict. There is no rung for "the graph was empty" because the ladder labels verdicts and none was produced, which is exactly why it has to be a degradation. pmpro two-file slice before: 0 findings, degradations=[] after: 0 findings, degradations=[cpg_empty] Gates byte-identical; 840 tests pass. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 18:07:28 +02:00
Marius CiepluchandClaude Opus 5	37a6594703	fix(mapping): a PHP method with a visibility modifier is a region ... `^\s*function` matched a bare declaration and nothing else, so every method written the way modern PHP writes them -- `public function`, `protected function`, `private static function` -- was invisible to the region model. The effect is not subtle on class-based plugins. MW WP Form's `class.mail.php` produced ONE file-level region spanning 272 lines instead of 16 method-scoped ones, and its `protected function _delete_files()` -- which holds CVE-2023-6559 -- was not a region at all. Paid Memberships Pro happened to work only because that particular file omits the modifiers. class.mail.php 1 file-level region -> 16 method-scoped regions Also adds the MW WP Form checkout as the PATH family's plugin, pinned at 3bdd33d (5.0.3) with CVE-2023-6559 verified in the code: protected function _delete_files() { foreach ( $this->attachments as $file ) { if ( file_exists( $file ) ) { unlink( $file ); and the fix at 02ebc67 replacing the existence check with `realpath()`, `is_file()` and a directory-confinement test. `unlink` is a modelled path sink and `realpath` a modelled sanitizer, so this is a pair the engine should separate rather than merely flag -- which is what makes it worth more than a single hit. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 17:33:36 +02:00
Marius CiepluchandClaude Opus 5	96e7476c6d	feat: the first real CVE, found in production plugin code ... CVE-2023-23488 is detected at classes/class.memberorder.php:936:getMemberOrderByCode -- the exact line verified in the source, in unmodified Paid Memberships Pro 2.9.7. A two-file slice carrying both ends of the CVE, 2885 lines, loads and queries in 122 seconds and reports 16 entailed injection findings. **The other fifteen are not noise.** The file holds 26 `$wpdb` query calls and **zero** of them use `->prepare`; all 26 interpolate or concatenate a variable. The engine reports 16 of 26, so it under-reports rather than over-reports, and every sampled finding is a genuine unprepared query -- `get_row("... WHERE id = '$id'")`, `get_var("... WHERE ID = '" . $this->user_id . "'")`. CVE-2023-23488 is simply the instance that got an identifier, because it is reachable unauthenticated; the rest are the same defect with different reachability. That answers the question that prompted this: whether the system can find one real CVE. It can, in production code, at the exact line, in two minutes, for a cent. What it could not do was survive its own plumbing at 637 files -- four separate quiet failures on this one checkout, all between the scan and the arbiter, none of them in the analysis. What this does NOT settle is recorded with it: the full checkout still does not load and that boundary is unmeasured; whether php2cpg resolves `new X()` and `$this->method()` is still unknown, since this slice reached the sink through a same-function flow rather than across the object; and one CVE on one plugin is not the class-by-class table over several plugins that 5.8 asks for. On cutting the input with a model, proposed when size looked like the blocker: the evidence argues for scoping rather than triage. Two of 637 files found the CVE, task 2.10 already scopes by what a project declares it ships, and diff-scoping is this spec's thesis. A model choosing what to cut adds a fifth quiet-failure surface -- drop the file holding the bug and the scan reports clean with nothing recording why, which is the shape of all four failures found today. The reverse direction is where it fits: ranking which regions get the budget, not choosing what to remove. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 17:24:37 +02:00
Marius CiepluchandClaude Opus 5	21fdb554aa	fix(cpg): a query that could not be asked is not an empty answer ... contributor-scan 5.7, and it was the wrong question. **The 117k-line pmpro CPG does not load.** Importing it throws in a ForkJoinParallelCpgPassWithAccumulator during overlay application, so every query against it returns nothing. It is not a call-graph question at all. Isolated by running the ARBITER against two prebuilt CPGs rather than writing a sixth ad-hoc Scala probe, four of which had already failed on my own mistakes: small wpdb CPG get_order_by_code -> 1 model_entailed pmpro CPG getMemberOrderByCode -> 0 Both have their source and sink in ONE function, both were asked with parameter sources and depth 3, same arbiter and same spec. The CPG is the only difference. That clears the facts, which is worth stating: `$wpdb` sinks work, the parameter-source rule works, and a PHP parameter reaching `$wpdb->get_var` through a concat is entailed. Whether php2cpg resolves `new X()` and `$this->method()` is still unmeasured -- it cannot be measured on a CPG that will not load -- and is recorded as 5.10 along with finding the size boundary. **And the defect underneath it, which is the fourth quiet failure found on this one checkout.** `query_batch` returned `{}` for a failure and `{}` for a genuine empty answer, so a caller could not tell "could not ask" from "nothing found". Every query on this scan failed at load and it reported 500 regions examined, no findings, and no degradation: a vulnerable plugin, clean. A failed batch now returns None, unparseable output counts as a failure rather than an empty answer, and the driver records a `query_failed` degradation naming the kind and the request count. This is the failure mode the rung ladder structurally cannot catch. Every other honesty mechanism here labels a VERDICT -- suspicion, regions_unjudged, regions_unasked, stated family limits -- and when no verdict is produced there is nothing to label. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 17:15:51 +02:00
Marius CiepluchandClaude Opus 5	8579c10e7c	docs: the WordPress checkout scans, and does not find its CVE ... Reporting the result rather than the intention. Paid Memberships Pro 2.9.7 with CVE-2023-23488 is pinned, verified in the code, and scans cleanly: 637 files, 117k lines, 1577 regions, build 47s, query 74s, 140s total, no degradations. **The CVE is not detected.** Both ends are inside the budget -- the REST handler at position 290 as a recognised entry point, the `$wpdb->get_var` sink at 472 -- and the taint query runs for 23 seconds. So the gap is the middle: the flow crosses `new MemberOrder( $code )` into a constructor and then `$this->getMemberOrderByCode( $code )`, both dynamic in PHP. Whether php2cpg resolves either into a callee edge is UNVERIFIED; two attempts to query the call graph failed on script errors and were abandoned rather than guessed at. Recorded as 5.7, with the answer required before anything is concluded about the fact tables. Three engine defects had to be fixed before the scan could produce even this result, and each had previously reported the plugin clean for a different reason: 1577 regions, 0 scanned joern-parse failed where php2cpg succeeded 500 scanned, taint 0.05s a 1.49MB batch over Linux's 128KB argv cap 500 scanned, taint 21.7s the entry point was not recognised as one The middle one is the one to remember. Every honesty mechanism in this codebase labels a verdict -- suspicion, regions_unjudged, regions_unasked, stated family limits -- and there no verdict was produced to label, so a vulnerable plugin was reported clean with every counter agreeing. What is established: the WordPress engine path works end to end. What is not: detection on a real plugin. One checkout that does not find its CVE is not yet evidence either way about the facts, and 5.7 has to answer whether the call graph can carry the flow at all. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 16:43:10 +02:00
Marius CiepluchandClaude Opus 5	922ed84053	feat(repos): a WordPress plugin with a verified CVE, and the WordPress DB API ... contributor-scan 5.1's detection half. Paid Memberships Pro 2.9.7 pinned at 1a900be3, carrying CVE-2023-23488 -- unauthenticated SQL injection through the `code` parameter of the /pmpro/v1/order REST route. **Verified in the code, not taken from the advisory.** At this commit classes/class.memberorder.php:936 reads $id = $wpdb->get_var("SELECT id FROM $wpdb->pmpro_membership_orders WHERE code = '" . $code . "' LIMIT 1"); and tag 2.9.8 replaces it with `$wpdb->prepare( "... code = %s ...", $code )`. 637 PHP files and 117k lines, so it is also the first PHP checkout large enough to say anything about an envelope. **WordPress never calls mysqli; it goes through $wpdb**, so the facts could not have found this. Added qualified: `$wpdb->get_var`, `->get_results`, `->get_row`, `->get_col`, `->query`, `->replace`. Qualified because a bare `get_var` or `query` matches any method of that name anywhere -- the generic-name trap that made `query.get` match every `.get(` in a repository this morning. `->insert`, `->update` and `->delete` are deliberately absent: they take arrays and WordPress prepares them itself. **And the thing that nearly went wrong.** The obvious next step when writing PHP facts is to add WordPress's sanitizers -- sanitize_text_field, esc_html, esc_attr. Doing so would have made this CVE invisible, because its handler calls `sanitize_text_field( $params['code'] )` and then concatenates the result into a query. sanitize_text_field is a real sanitizer that does nothing whatsoever about SQL injection. `TaintSpec.sanitizers` is one flat list per language whose comment states the assumption outright -- "a sanitizer that breaks one flow breaks it whatever the sink's family" -- and PHP falsifies it. So only SQL-specific cleansers are listed, which leaves XSS under-sanitized instead, and the real fix is recorded as task 5.6: a discharge counts for the obligation it discharges. That is task 2.6's finding arriving in the taint family, where the dominance arbiter already reads kind from the facts and the taint spec has no equivalent. The end-to-end scan of this checkout is not yet reported: the first run returned 1577 regions with 0 scanned, and the CPG builds standalone at 4.3MB with no errors, so the two do not yet agree. Numbers when they do. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 16:13:17 +02:00
Marius CiepluchandClaude Opus 5	60d2a1a92c	feat(model): an endpoint its contract declares open carries no obligation ... contributor-scan 2.8. VAmPI's last two false positives were register_user and login_user, both declared open by its OpenAPI spec: the arbiter was reporting a missing guard on endpoints whose contract says there is nothing to miss. **The fix keys on provenance, not on the value, and that distinction is the whole task.** The mapper says "public" in two very different situations. An OpenAPI operation with no `security` block is DECLARED open, because the spec is a complete contract. A Flask handler with no `@login_required` is "public" only because nothing was found -- which is precisely the bug this family exists to report. Suppressing on the value alone would have silenced the detector on its primary case, so EntryPointRecord now carries `access_declared` and only a declaration removes the obligation. WordPress supplied the second source of declared access on the same day -- wp_ajax_nopriv_ for a logged-out handler, wp_ajax_ for a logged-in one, permission_callback => __return_true for an open REST route. A rule resting on one framework's convention would be a special case; two make it a shape. vampi entailed 6 -> 4 access-control precision 2 of 4 -> 2 of 2 known vulnerabilities 3 of 3 still entailed access_control offered to 6 of 14 handlers, exactly the six the spec declares authenticated **And the guard that decides whether it is a real improvement:** the access-control pair corpus, which nobody tuned this against, is unmoved -- pair_correct 3 and leak 1 across all nine, pair_correct 2 and leak 0.0% across the seven from projects unrelated to VAmPI, identical to task 2.6. The precision was not bought by silencing the family. Pair excerpts carry no route manifest and no hook, so nothing in them is declared open and the rule correctly does not fire. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 15:39:58 +02:00
Marius CiepluchandClaude Opus 5	3e1d13e4b6	feat(reports): the model's own suspicion band is a proposal too ... contributor-scan 2.17, completing Req 5.5. Task 2.11 made static pattern matches proportionate and left the model layer's own suspicions alone, reasoning that a proposer had reasoned about them. That does not survive inspection. A model suspicion is what the enumerator proposed at a site the graph could NOT decide, affirmed by a judge with no coverage there -- and its own rationale says exactly that: "the model resolved no flow here; its sink table may not cover this API". It is a pattern match with an opinion attached, and it reads like a reasoned claim only because a model produced it. vampi 60 findings, 61 sections -> 26, report 1085 lines -> 446 libpng 198 findings, 39 sections -> 30 **What it costs, which is the number that decides whether the rule is right.** For each known vulnerability the recipe declares, take every finding row at its location and ask whether any survives the rule. All three of vampi's -- the SQL injection and both documented BOLAs -- are still found at model_entailed and all three survive. Nothing known is hidden. Three vulnerabilities in one repository is also a statement about how small the evidence base is, and the artifact says so. The obligations checker sits at `suspicion` too and is NOT grouped: its output comes from a structural analysis, an obligated operation with no discharging guard, rather than from a guess. An earlier attempt at this rule filtered on rung alone and would have collapsed it; six tests failed and they were right to. "Pattern matches" is now "Proposed, not corroborated", since it holds two shapes. Model rows group by FAMILY rather than by rule, because their rule is always `model`. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 15:23:57 +02:00
Marius CiepluchandClaude Opus 5	f488e625c2	fix(specs): the php2cpg "blocker" was the shim, not the frontend ... Task 5.5 recorded that php2cpg emits a CPG containing only one PHP file -- the last alphabetically -- and called it a blocker for the WordPress target. Rerun with a native php-cli, as asked, and it is not true. php 8.3.6, no shim, same joern 4.0.623, same two files: files: plugin.php, vuln.php methods: mp_fetch_report, mp_save_report, mp_render_box, mp_helper, get_user, safe_user, show, run Both files, all eight methods, and no PHP-Parser banner warning at all -- the banner that looked like the thread to pull was a symptom of the same thing and does not occur natively. The cause was the workaround. This machine has no PHP interpreter, so the shim ran each `php` invocation in its own container; php2cpg drives PHP-Parser once per file, and that per-invocation isolation kept only one file's JSON. **The lesson is worth more than the fix.** Two independent reproductions agreed, and both were measured through the same broken instrument. A consistent result is not a correct one when every measurement shares an apparatus -- the same trap as a corpus that cannot see the defect it is scoring, which is this session's other recurring finding. And with the instrument fixed, 5.4 is demonstrably done. A two-file PHP tree yields 10 function-scoped regions and 7 entailed findings in 15 seconds: SQL injection in both WordPress handlers, XSS in the shortcode renderer, and all three of the original probe's. safe_user is correctly absent, its mysqli_real_escape_string dropping it to corroborated. The wp_ajax_nopriv_ handler ranks 1.0 and the plain helper 0.3. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:45:36 +02:00
Marius CiepluchandClaude Opus 5	0ffa99b5e6	feat(model): one finding per sink site, and WordPress's registration model ... contributor-scan 5.4. Both halves, and the second found a frontend blocker. **One verdict per region was right for a pair and wrong for a file.** A pair is one labelled function with one bug, so one answer is correct. A region spanning a PHP file is not: the probe's SQL injection on line 5 and command injection on line 13 produced ONE injection verdict, `system` won it on flow length, and the SQL injection went unreported by construction rather than by any failure of the analysis. `taint.verdicts()` now returns one verdict per distinct sink site, strongest first, and its first element is exactly what `verdict()` returns -- so the pair path and every committed measurement resting on it are unchanged, and the gates are byte-identical. Both entry points share `_usable_flows` so neither can drift from the other. php probe 2 of 3 -> 3 of 3 vampi 6 entailed, unchanged, now with lines on access-control findings from task 2.7 (users.py:187, books.py:51 -- the two documented BOLAs at their exact lines) libpng 1 -> 2 entailed; the second was hidden by the collapse. Neither is validated and both are in contrib/. **WordPress registers handlers rather than decorating them, and the hook name is the declaration.** `wp_ajax_nopriv_*` is WordPress saying logged-out callers reach this, so it is public; `wp_ajax_*` fires only for logged-in ones; `permission_callback => __return_true` declares a REST route open. That is the declared-provenance signal task 2.8 needs, arriving from a second framework. Plus every top-level PHP function as its own region, without which a file is one region however many functions it holds: 1 file-level region became 10 function-scoped ones. **And a blocker, recorded as 5.5.** php2cpg emits a CPG containing only ONE PHP file, the last alphabetically. Reproduced twice on independent samples; both files parse cleanly under `php -l` and under PHP-Parser directly, and joern-parse reports success either way. php2cpg logs "Found warning in PHP-Parser JSON output" naming the survivor, and that warning is PHP-Parser's own `====> File X:` banner appearing in what php2cpg expects to be pure JSON. Not isolated yet: this machine has no PHP interpreter, so php2cpg was driven through a container shim. The banner comes from PHP-Parser rather than the shim, which makes a shim-only cause unlikely but not excluded, and rerunning with the native php-cli in the shipped image is the first step. It blocks WordPress outright -- a plugin is many files, and a scan that silently covers one of them and reports success is the quiet failure Req 5.6 was approved for. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:30:48 +02:00
Marius CiepluchandClaude Opus 5	61811512a8	feat(facts): PHP semantic facts, validated against a real PHP CPG ... contributor-scan 5.1, the first step toward the v0.1 WordPress target. Written against a PHP CPG rather than against PHP source, because the two do not look alike. Verified with joern-parse 4.0.623 / php2cpg on a probe: $_GET["name"] -> call `<operator>.indexAccess`, CODE $_GET["name"] mysqli_query($conn, $sql) -> call NAMED mysqli_query echo $_GET["msg"] -> call NAMED echo So sources match on code and sinks match on name, and every entry the probe did not exercise is marked unverified in the file -- `include` and `require` among them, which is PHP's most characteristic path bug and the first thing to check on a real checkout. Five families derive (injection, output_encoding, path, deserialization, untrusted_destination) with no sink that is its own sanitizer -- asserted by a test, because that bug class has now appeared three times in three matchers. `php` is added to the region model's languages, without which the facts would have reached nothing. The arbiter decides 4 of 4 on the probe: the SQL concat entails, its `mysqli_real_escape_string` twin drops to corroborated, `echo` XSS entails and `system` command injection entails. **A whole scan finds 2 of those 3, and the gap is the finding.** PHP files yield FILE-LEVEL regions because no PHP entry points are mapped, and the taint arbiter returns one verdict per region per family -- so a file-level region reports at most one injection finding however many it holds. `system` won on flow length and the SQL injection went unreported. That is the mirror image of task 2.9: that one is many regions collapsing to one site, this is many sites collapsing to one verdict, and it bites hardest exactly where PHP needs it not to, since a WordPress file is large and has no decorator for the mapper to find. Recorded as task 5.4 with WordPress's registration model (add_action, add_filter, admin-ajax, REST routes), which the mapper already handles in principle for other frameworks. A 13-line probe is not a WordPress result. What is established is that the seam works end to end for PHP; everything about scale and coverage is unmeasured, and there are still zero PHP pairs in a corpus of 263 python, 351 c_cpp, 47 javascript and 49 java. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 14:18:57 +02:00
Marius CiepluchandClaude Opus 5	00e178d69e	feat(reports): a report proportionate to what the engine can say ... contributor-scan 2.11. libpng emitted 198 sections, 163 of them asserting that `memcpy` and `strcpy` are PRESENT in a library that uses them correctly throughout. That is the noise developers have learned to skip in every other tool, and shipping it costs the findings beside it their credibility. **Filtering on rung was the obvious rule and it was wrong.** The obligations checker emits at `suspicion` too, so hiding that rung would have hidden an entire detector -- six tests failed and they were right to. The discriminator is evidence_level: on libpng 163 of 198 rows are `static_corroboration`, on vampi 3 of 60 are, and the model layer and the obligations checker carry `suspicion` there instead. That is exactly the distinction the ladder exists to make -- an undecided CLAIM about a site is not an observation that an API exists. So a pattern-only rule is grouped once it has more than three sites, and keeps its full sections under that. One `python-flask-debug` is a finding a contributor can act on and must not be summarised away; fifty-four `c-unsafe-copy` are a property of the codebase. libpng 198 findings, 198 sections -> 39, 161 grouped into three lines vampi 60 findings, 61 sections -> 61, nothing grouped Counted, never dropped: findings.json stays complete because it is the record a baseline is compared against, and the grouped section says the model reached none of these and that its silence means no fact table covered the site rather than that the site is safe. What it does not fix, recorded rather than glossed: vampi still emits 61 sections for a 520-line application, 51 of them reasoned claims this rule does not touch. It fixes pattern noise, not volume. Also records what group 2 established about the three corpus asks, so none is started on a wrong assumption. Joern ships php2cpg but our fact tables have zero PHP families, so WordPress would parse and produce nothing -- fact tables first, corpus second. An older vulnerable libpng will not help for a reason no version can fix. Vibe-code repositories are the best fit and the cheapest route to a second true-positive checkout, because Python and JavaScript are the two languages whose fact tables are complete. And Group 8, for per-repository state a developer owns: a committed dismissal file the tool reads and never writes, dismissals as a split evaluation set, and GEPA/SIMBA-style evolution scoped to the QUESTION -- the judge and residual prompts -- never to the fact tables. That boundary is the whole design: a prompt is how we ask, a fact table is what we claim, and the closed-loop leak measured on 2026-09-05 is why the distinction is not negotiable. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:29:26 +02:00
Marius CiepluchandClaude Opus 5	9221ba8e40	feat(model): a bound on the data lowers the rung, it does not clear it ... contributor-scan 2.12. c/injection and c/memory carried zero sanitizers, so no C flow could ever be discharged: every `argv -> strcpy` entailed however carefully it was bounded. Which sinks can be discharged this way is read from the sink fact's own CWE rather than declared: CWE-121 is a stack buffer overflow, so gets/sprintf/strcat/strcpy qualify. It is the only overflow CWE in the fact tables and it is C-only, so no Python or JavaScript sink is touched and the pair corpus is unaffected -- gates byte-identical. Not a sanitizer list, because `strlen` appears in the guarded and the unguarded case alike; the discharge is a guard, not a call. **The first implementation required the bound to DOMINATE the sink, and it found nothing.** libpng guards `strcpy(outname+len, ".png")` with `(len = strlen(inname)) > 250`, but that check sits on an else-if chain whose sibling branch skips it entirely, and the protection reaches the sink through an `error` flag: `++error` in the guarded branch, `if (!error)` around the copy. Joern's dominator set for that sink has 89 calls and correctly excludes the bound; `controlledBy` returns none either. Verified against the CPG before the rule was changed rather than after it failed. So the bound is searched in the method, and the consequence is corroborated rather than silence. That is the honest rung: a bound on this variable means the graph has NOT shown the copy is unguarded, and it has not shown it is guarded either -- off-by-one being how bounds fail. The judge is asked. Proving the flag-mediated case needs path-sensitive reasoning this arbiter does not do. libpng entailed 3 -> 1 The survivor is also a false positive: `strcpy(output + len, ".icc")` in iccfrompng.c, where `output = malloc(len + 5)` makes it fit exactly. That is a different discharge shape -- allocated to size, not bounds-checked -- and it is recorded as 2.13 rather than chased. Both shapes were found on one repository, and extending a rule until that repository is clean is the overfitting Req 8 exists to prevent; 2.13 requires a second C checkout with a genuine CWE-121 so the change can be measured for what it silences as well as what it clears. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 13:01:30 +02:00
Marius CiepluchandClaude Opus 5	6dc25d77e4	feat(model): let the project say what it ships, and rank that first ... contributor-scan 2.10. Every one of libpng's twenty-five entry-point regions was a `main()` -- twenty-three under contrib/, plus example.c and pngtest.c -- and those regions carried the fifty interprocedural taint requests that dominated a 300s query phase. A library has no entry points, so entry-point-driven analysis had nothing to anchor to except its own example programs. The task said to derive this rather than hardcode `contrib/`, and a project already declares it. Makefile.am names installed targets in `lib_LTLIBRARIES` and `bin_PROGRAMS`, each target's `<target>_SOURCES` names its files, and `check_PROGRAMS`/`noinst_PROGRAMS` are automake's own word for "not installed" -- exactly the distinction wanted. Taking every `_SOURCES` would have called libpng's whole test suite shipped code, since contrib/libtests declares its sources the same way the library does. interprocedural regions 25 -> 1 taint query 300.1s -> 125.0s total query 311.9s -> 138.5s regions 121 -> 121, unchanged by design The surviving interprocedural region is contrib/tools/pngfix.c:main, which libpng genuinely installs. Two rules keep it honest. **Silence is not exclusion**: a project with no build file this can read gets None back and nothing changes, so vampi's numbers are untouched. **Not shipped is not unscanned**: those regions are still arbitrated function-locally, because a contributor editing a test must still get an answer -- what they lose is the expensive treatment and first claim on the budget. Entailed findings went 0 -> 3, and all three are false positives at contrib/gregbook/wpng.c:374, `strcpy(outname+len, ".png")`. It is safe: outname is char[256] and line 325 guards with `if ((len = strlen(inname)) > 250)`. They were not introduced so much as uncovered -- a different callDepth meant a different sink set, and the 26 fgets false positives were masking them. The structural cause is recorded as 2.12: c/injection and c/memory carry ZERO sanitizers, so no C flow can ever be discharged and every `argv -> strcpy` entails however carefully it is bounded. The discharge here is not a call but a guard that dominates the sink, which is precisely what dominance.sc already does for access control and is wired to that one family. Also adds Group 7, five tasks for stopping the same false positive twice. Two of today's three false-positive classes were one bug in three matchers -- `resolve` in `resolveUrl`, `user` in `users`, `gets` in `fgets` -- each fixed locally and each next one surviving because nobody carried the fix across. A model may group counter-examples by cause and draft an edit; only a split-respecting measurement may adopt one, because the predecessor's improve lever admitted shapes taught by the holdout pairs they then recovered and turned a -0.059 Youden into +0.059. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 12:45:21 +02:00
Marius CiepluchandClaude Opus 5	68b4e551f3	feat(model): the envelope checkout, and no to Joern server mode ... contributor-scan 2.2 for libpng and the 2.3 decision it exists to inform. 89,082 lines of C across 93 files, under a 2GB heap cap: CPG build 16.1s (25s cold) Joern queries 250.0s arbitration 122.0s (105 calls, $0.0122) whole scan 446s peak RSS 911MB regions 121, all arbitrated, 32 unasked CVE-2025-66293 out_of_scope, as the recipe declares Memory is not the constraint a fleet runner would hit: 911MB leaves room for the rest of the job on a 2-vCPU/7GB runner. The memory pressure seen while building this was concurrent JVMs on one desktop, not repository size. **Server mode: no.** The query phase splits taint 243.3s / config 10.7s over two invocations, and fixed overhead measured on the same CPG is ~13s per invocation. So ~26s of the 250s is JVM startup and ~224s is CPGQL evaluation: a warm server removes 6% of the scan and none of the part that costs. The driver now records query seconds per kind, because a single total cannot tell startup from evaluation and the two answer this oppositely. The small repository pointed the other way. vampi's 34.3s over three kinds is roughly three startups and almost no evaluation, which made a warm JVM look worth having; the ratio inverts with size. That inversion is why the task was sequenced after a real repository rather than decided on a 520-line file. On whether one whole-repository CPG is the only shape for a horizontal fleet: no, and it is also not where the pressure is. The build parallelises and costs 16s; the query does not shard and costs 250s, so splitting a repository across runners divides the cheap phase -- and after 2.4 a shard boundary hides exactly the cross-module flows that matter. Scoping to a contributor's diff cuts the expensive phase instead. 911MB for 89k lines extrapolates to roughly 5GB at a million lines, which would exceed a standard runner, so the ceiling is real and further out than this checkout can locate. Two defects recorded rather than fixed, both invisible at vampi scale: 2.9, all 26 entailed findings are the same site reported 26 times, because regions that share a sink each emit their own; and 2.10, 22 of the 30 highest-ranked regions are libpng's `contrib/` example programs rather than the library. Also ignores Joern's `workspace/`, which a maintainer querying a CPG by hand from the repository root will create. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:30:02 +02:00
Marius CiepluchandClaude Opus 5	d15985f695	fix(model): a guard discharges the obligation it is a guard of ... contributor-scan 2.6. Access control entailed 7 of 9 VAmPI handlers, including three that do authorize and three the OpenAPI spec declares public. Five separate defects, each invisible to a corpus of one-function excerpts and each found by asking the query what it actually matched. **A trailing segment is not a match.** `query.get` was matched by its last segment, so every `.get(` in the repository became a protected database operation -- `request.headers.get('Authorization')` and `request_data.get('username')` included. Four of the seven false positives came from this, and so did the apparent detection of the known BOLA: it was entailed because fetching the Authorization header had no dominating guard, not because of the defect. A qualified token now requires its qualifier. **Configuration is not authorization.** dominance_specs took every obligation fact. `security_setting` operations are discharged by a VALUE, which config_specs already claims them for; pooled in here they made `app.run(...)` an operation requiring an authorization guard and put thirteen tokens in BOTH lists -- an obligation that is its own discharge, the same shape as the taint sink that used to sanitize itself. **Input validation is not authorization.** `jsonschema.validate` was counted as a guard, so handlers with no authorization check at all read as guarded. No operation in the table requires validated_input. **A guard discharges the obligation it is a guard of.** The largest one. `orm-read` requires an identity_constraint or an ownership_check; `token_validator` is a path_guard. Flattened into one list, authenticating the caller discharged an object-level obligation -- a handler that authenticates and then looks the record up by a PATH parameter read as fully guarded. The facts state that relation and the spec was discarding it; it is now carried and the engine asks per operation. **A word boundary needs a word.** Found while fixing the above. Guard matching wrapped every token in `(?<![A-Za-z0-9_])...(?![A-Za-z0-9_])`. The identity token `['sub']` begins with `[`, and the character before it in `resp['sub']` is `p` -- so the assertion could never hold. Every identity token starting with punctuation was unmatchable, and the identity constraint that separates a correct lookup from an IDOR was invisible. Also: dominance rows were repository-wide, so the model's unguarded `delete_user` was attributed to the API handler of the same name. Rows are scoped to the region's file, which is what the rung's own justification says -- the evidence is a file contradicting itself. Measured against VAmPI's OpenAPI `security` blocks, which declare per operation whether auth is required, and agree with the code: vampi precision 28.6% (2/7) -> 50.0% (2/4) known found 3 of 3, all model_entailed And on the labelled access-control pairs, which nobody tuned this against: all 9 pairs pair_correct 3 -> 3, leak 44.4% -> 11.1% 7 non-VAmPI pair_correct 2 -> 2, leak 42.9% -> 0.0% Every detection lost was on a pair whose FIXED side was flagged too, which scores nothing. No correct detection was traded for the precision. get_by_title was flagged before it was in the recipe: its code carries the comment "# Broken Object Level Authorization" and the README lists it, so a real finding was being scored as a false positive. It is now the recipe's third known vulnerability. Both remaining false positives are endpoints the spec declares public. Recorded as 2.8, with the reason the mapper's access_level cannot simply be trusted: for decorator frameworks "public" means "no decorator found", which is the bug itself. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 11:02:48 +02:00
Marius CiepluchandClaude Opus 5	0fcd5feade	feat(model): a region for the code that is in no function ... contributor-scan 2.5. VAmPI's `host='0.0.0.0'` is now reported exactly once, at app.py:17:<module>. Before task 2.2 it was reported eight times in eight files that do not contain it; after 2.2 it was reported nowhere. Two causes, one shape: nothing could ask about module-level code. **The main-guard marker is not a function.** `__main__` is what the mapper emits for `if __name__ == '__main__':`. Sent to the engine as a function name it matched nothing, because a CPG calls a file's top-level code `<module>` -- verified against the VAmPI CPG rather than assumed. The region model translates the marker, which is the one place that turns what the mapper meant into what the scanner asks. **A function-scoped file had no region for its module body.** Settings, route registration and globals live in no function, and every region's query filters by function name. A file-level region beside the handlers would report each handler's findings a second time, which is why the earlier rule dropped it as "a second full-file pass". A module-scope region PARTITIONS the file instead of overlapping it. That only became cheap after batching: before it, a region was a JVM. Two things this needed to be correct rather than merely present: - A module method spans its whole file, so the taint query's line-range containment made `<module>` the parent of every function in it. Under module scope, membership is direct -- otherwise the module region reports every handler's sinks again, which is the duplication the old rule was avoiding. - A module body is not a handler. The marker-derived region inherited the handler families and entailed a missing authorization check on the main guard: a finding with no possible remediation. Caught by the first repository run that had the region, and now covered by a test. bind address 8 wrong files -> 0 -> 1, at app.py:17:<module> regions 22 -> 25 entailed 8 -> 9 query time 37.4s -> 41.3s peak RSS 827MB -> 823MB cost $0.0067 -> $0.0068 Three more regions for four seconds and no more model calls: the budget is a total and was already spent, so regions_unasked rose 17 -> 20 -- those regions got the free pass and are reported as unasked, not as clean. `<module>` is Joern's Python name for a module body; JavaScript uses `:program`, so this is verified for Python only. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:43:53 +02:00
Marius CiepluchandClaude Opus 5	ca7b5ba370	feat(model): a region that spans the flow, not just a function ... contributor-scan 2.4, closing the no-go task 2.2 recorded. VAmPI's SQL injection is a request parameter reaching `get_by_username` in api_views/users.py, passed to `User.get_user` in models/user_model.py, interpolated into a query string and executed there. Source and sink are in different functions and different modules. Regions were per-function, so neither region saw both ends and the engine was silent on the most ordinary shape a real injection bug has. `reachableByFlows` was never the limitation -- it follows calls already. The limitation was the QUESTION. Sinks were filtered to the labeled function's own body, so a crossing flow had no sink to end at. Three changes, each with a reason it is sound rather than convenient: **An entry point is the trust boundary.** parameter_sources was off for repository scans because a repository's parameters are not all attacker input. An entry point's are, by definition -- that is what makes it an entry point. The driver turns it on only for regions the mapper found and named, and leaves it off for walker fallbacks and for named functions that are not entry points. **Sinks may sit below the handler.** taint.sc admits sinks in methods reachable from the labeled method through the call graph, bounded at ENTRY_POINT_CALL_DEPTH = 3: handler -> service -> data access, which is where these bugs live. Unbounded would make the question about most of the repository rather than about this entry point, and make it unaffordable. **The finding names the sink, not the asker.** A crossing flow ends in another file and another function, so Verdict carries the evidence's own `path:line:function` and the site uses it. Reporting `models/user_model.py:73:get_by_username` would name a place that does not exist: line 73 is inside get_user. Also: labeled methods are now resolved by name AND file. `get_user` exists in more than one module of a real repository, which a corpus of one-file excerpts could never surface. Measured on the same checkout and budget as 2.2: VAmPI-SQLI missed -> model_entailed at models/user_model.py:73:get_user known found 0 of 2 before 2.2's fixes, 1 of 2 in 2.2, 2 of 2 now entailed 7 -> 8 query time 40.9s -> 37.4s peak RSS 869MB -> 827MB cost $0.0067, unchanged Asking the bigger question cost nothing measurable here: file scoping and call-graph following pull in opposite directions and roughly cancel at this size. That is one 520-line repository, and VAmPI is now the repository this change was developed against -- the exact overfitting shape Req 8 exists to catch. The depth bound and the entry-point rule mean nothing general until a checkout nobody tuned against says so. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 09:27:19 +02:00
Marius CiepluchandClaude Opus 5	177c7f3719	feat(model): first repository measurement, and the four things it found ... contributor-scan 2.2. VAmPI at a pinned commit, scanned whole rather than as excerpts, with benchmarks/repos/measure.py recording build time, query time, arbitration time, peak memory, cost, unjudged regions, and whether each known vulnerability was found and at which rung. The corpus could not have found any of this. Every defect below needs a repository to become visible, and three of them were invisible behind a passing test suite. **The batch path was unreachable.** query_batch was implemented, tested and committed while CpgResult exposed only `run`, so the driver's capability check found nothing and fell back to one JVM per region-family: the first attempt at this measurement ran eighteen minutes on 520 lines, issuing `--param function=` one at a time. Every test called query_batch directly, so none of them noticed. CpgResult now carries run_batch and a test asserts the capability is reachable through the object the driver is handed. The earlier 275s->106s improvement attributed to batching came from region deduplication alone. **The budget stopped free work.** It bounds model calls, but was enforced by breaking the region loop -- spending a paid limit to stop unpaid CPG arbitration. 6 of 22 regions were examined and VAmPI's BOLA, which the graph entails for nothing, was never reached. It now withholds the judge only. 22 of 22 regions arbitrated, 17 counted as regions_unasked (silent because unaffordable is not silent because decided), and the BOLA is entailed by the model layer. **A region with no function asked about the whole repository.** Sending function="" matched every hit in the tree and the driver attributed that answer to whoever asked: one `host='0.0.0.0'` in app.py became eight identical entailed findings in eight files that do not contain it. Queries now carry the region's file. Entailed findings 14 -> 7, query time 70.9s -> 40.9s. The single-request form stays unscoped so the committed pair measurements remain reproducible. **The JVM had no heap bound.** Joern claimed a quarter of physical RAM and runs were killed under memory pressure at different points, so the numbers described the host rather than the engine. -J-Xmx2048m by default, OPENULTRASAST_CPG_HEAP_MB to raise it. Also: arbitrated findings carried `function:line` where the report layer read `path:line`, so every entailed finding reached a contributor with a function name where its file belonged; and the witness line was parsed by splitting on a comma, which produced sites like `config.py:17): permissive literal '0.0.0.0':?`. Reading: GO on one CPG per repository -- 6.2s to build, 177s end to end, 869MB, $0.0067, nothing infeasible. NO-GO on region-scoped taint as the only shape: VAmPI's SQL injection runs from a handler parameter into a function in another module, and a per-function region sees neither end. That is the ordinary shape of a real injection bug, so it is recorded as task 2.4 along with three smaller defects this run exposed. Gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:46:21 +02:00
Marius CiepluchandClaude Opus 5	8c387306be	feat(repos): pinned known-vulnerable checkouts, offline by default ... The pair corpus scores excerpts: a vulnerable function and its fixed twin, handed to the engine with the answer already narrowed down. Nothing in it exercises a repository -- region ranking, the budget, one CPG over thousands of files, or whether a verdict that holds on fifteen lines still holds when the source, the sink and the guard live in three modules. benchmarks/repos/<name>.toml names a real checkout: url, a full 40-char commit SHA, licence, and every known vulnerability with its file, function and line. `ousast repos` lists them; `--fetch` materialises one; CI does neither, and conftest clears the switch and redirects the cache so a developer with warm checkouts cannot read them by accident. Three rules the loader enforces rather than documents: - A branch or tag is rejected. Either would make the same recipe mean different code next month while the baseline it produced kept the old numbers. - `in_scope` must be stated per vulnerability, never defaulted. It decides whether "not found" is a miss or a non-question, which is the most important thing about the result. - A recipe that declares `detection` must name something in scope to detect. `ousast repos` does not trust a recipe either: against a materialised checkout it checks HEAD against the pin, that the licence file is there, and that each declared function appears above its declared line. That earned its place immediately -- the first libpng recipe named LICENSE.md, which is the filename at HEAD today and not at the pinned commit, where it is still LICENSE. Two checkouts, verified against the real repositories: - vampi (MIT, 520 lines) is the DETECTION checkout. Excerpts of it are already in the pair corpus, including the update_password miss diagnosed earlier, so a whole-repository run says directly whether a verdict survives the move off the excerpt. Its routes register through an OpenAPI operationId, so it exercises registration-based entry-point mapping rather than the decorator path. - libpng (libpng-2.0, 91k lines of C) is the ENVELOPE checkout, pinned at the parent of the commit that added the missing clamp. Its CVE is marked out of scope: an out-of-bounds read is not a family this engine arbitrates, and recording "not found" against it as a miss is the easiest way to make a measurement lie. Nothing is vendored; the recipe records the licence so an artifact can say what it ran against. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 08:10:59 +02:00
Marius CiepluchandClaude Opus 5	8663c1a5b3	feat(contributor-scan): 1.1 regions from entry points, 1.2 the repository driver ... Pair scoring could rely on a labelled function; a contributor cannot supply one -- which code is risky is the question they are asking. Regions now come from entry points, with a file-level fallback that says it is one, and each carries only the families its shape and language admit. Ranking uses what EntryPointRecord already decided (access_level, trust_boundary) rather than a second opinion computed from the same evidence. A family is offered only where its arbiter could answer: access_control needs a handler, so a library file with no entry point is not asked about obligations it has no siblings to be inconsistent with. The driver makes the two costs that were free at pair scale explicit. ONE CPG per repository, reused across regions -- a build is 5-30s on a forty-line excerpt and per-region builds would make this unusable. And a budget that is a TOTAL for the run rather than the enumerator's per-region cap of eight, spent highest-rank-first, with whatever it does not reach counted into regions_unjudged. A tool that silently analyses a tenth of a repository and reports a clean result is worse than one that says what it skipped. Two flaws the tests caught, both mine and both about ownership: the budget counted calls by reading a client's own usage log, so it silently never tripped for a client that keeps none -- the driver now counts its own spend through a wrapper; and the driver trusted its caller to have sorted, when the order the budget is spent in belongs to whoever holds the budget. The third was worse and only surfaced because a test failed for the right reason: the driver passed candidates=() and never enumerated, so a repository scan would have reported only what the arbiters establish unaided and never populated the suspicion band -- which supplied 15 of 26 findings in the pair measurement. It enumerates now. Sibling audit (Req 8.5) done before writing: family routing lives in exactly one production place, pipeline._arbitrate, dispatching on spec type. regions_for returns family names and the driver resolves specs, so that stays true. 750 passing, mypy clean over 96 files, gates byte-identical, zero orphans. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-09 07:28:30 +02:00
Marius CiepluchandClaude Opus 5	fe45081b43	fix(ci): run ruff format, and record run 5 at 90% pair-correct ... CI was red on `ruff format --check`: 23 files. I ran `ruff check` after every change this session and never once ran `ruff format --check`, though CI runs both. Lint passing is not formatting passing, and I treated one as evidence for the other for the whole session. Formatting applied. The full CI sequence now reproduces locally: format, lint, mypy over 94 files, 735 tests, three gates byte-identical, compileall, and the semantic job's 16 marked tests. Also records run 5 of the pipeline measurement, which the vampi fix lifted as predicted: hunter (K=5) pipeline (K=1) pair-correct 76.7% 90.0% leak 12.3% 6.7% detection -- 96.7% cost ~$0.69 $0.0068 27 of 30 pairs. Five runs, each step from a measurement contradicting an assumption: 73.3 (no pipeline path) -> 73.3 (routing fixed, IDENTICAL, which exposed the discarded verdict) -> 80.0 (residual question matched to its arbiter) -> 86.7 (aiohttp settings) -> 90.0 (dominance question no longer contradicting its own evidence). One miss left: dsvpwa-dsvpwa-main, whose setting is an argparse --ssl default rather than a security-setting call. Out of reach before the settings work and still is; contorting the arbiter around one pair would be worse than the miss. The caveat is recorded beside the number: 30 pairs of one slice, and the slice iterated against all session. The +16% overfitting gap against vfc-js is why contributor-scan Req 8.1/8.2 exist. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:46:58 +02:00
Marius CiepluchandClaude Opus 5	c74e101d10	measure(model-grounded-detection): 86.7% pair-correct at half the hunter's leak ... hunter (K=5) pipeline (K=1) pair-correct 76.7% 86.7% leak 12.3% 6.7% detection -- 93.3% cost ~$0.69 $0.0067 Ten points of recall over the hunter, half the leak, a fifth of the model calls, ~100x cheaper. 26 of 30 pairs correct. Four runs, and each step came from a measurement contradicting an assumption -- twice one I had stated: 1 73.3% the absence and config families had no pipeline path at all 2 73.3% routing fixed and the result IDENTICAL, which was the clue: the arbiter was reached and its verdict thrown away, because every corroboration was put to the judge as a taint question 3 80.0% residual question phrased for the arbiter that raised it 4 86.7% aiohttp Application(debug=True) and RedisStorage(httponly=False) added to the settings table The division of labour keeps moving the right way: model_entailed 9 -> 11 while suspicion held at 15, so both findings gained in run 4 came from the MODEL, not the LLM, and cost fell from $0.0126 to $0.0067 as work moved onto the free deterministic path. That is the payoff curve for fact-table coverage, measured rather than argued: closing a gap moves findings out of the LLM's column into the model's, improving precision and cost at once. Recorded corrections and limits, because the numbers are only worth what the caveats are. The three silent config pairs were called corpus defects on a similarity threshold borrowed from the OWASP minimal-fix analysis; it does not apply to a slice that deliberately pairs a vulnerable function with a different safe one, where 36% and 34% pairs both score pair_correct. Two were a settings gap and now split. And permissive_values is a flat literal list with no keyword association, holding both True and False -- debug=True is dangerous where httponly=True is safe, and the model cannot tell them apart. It did not bite here, and it is a real limit of the fact schema. Four remaining misses, each with a named cause. Four runs cost about $0.04. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 19:18:06 +02:00
Marius CiepluchandClaude Opus 5	4e7fe65f17	measure(model-grounded-detection): the pipeline beats the hunter on both axes ... hunter (K=5) pipeline (K=1) pair-correct 76.7% 80.0% leak 12.3% 6.7% cost ~$0.69 $0.0076 determinism none entailed band deterministic Higher recall, half the leak, a fifth of the model calls, ~90x cheaper. Three runs, and the middle one earned its keep by changing nothing. Run 2 fixed family routing and returned numbers IDENTICAL to run 1 -- which was the clue that the dominance arbiter was being reached and its verdict thrown away. Every corroboration was put to the judge as a taint question ("a sanitizer appears on the path -- does it suffice?"), meaningless for a missing guard, so the judge said no and the finding was dropped. Run 3 phrases the residual question for the arbiter that raised it and converts 2 of 3 access_control pairs from silent to pair-correct, arriving at model_corroborated -- a rung that existed but had never reached a finding. This settles a question open since the start of the session, and reverses an earlier answer of mine. "Detection has regressed" was measured with the proposer disconnected: an arbiter compared against a detector, before four wiring bugs were found. Division of labour: 15 suspicion, 9 model_entailed, 2 model_corroborated. The model arbitrates 11 of 26 findings, and every conversion from suspicion to a model rung removes a leak and a model call at once -- the payoff curve for sink coverage and interprocedural resolution, now measured rather than assumed. Still silent: 3 config_secrets (the settings table is Flask/FastAPI-shaped while the pairs are aiohttp) and 1 access_control still to diagnose. Three full runs cost about $0.03 in total. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 18:40:51 +02:00
Marius CiepluchandClaude Opus 5	6dbb0e6422	measure(model-grounded-detection): the wired pipeline halves the leak at 1/55th the cost ... The first like-for-like comparison with the LLM hunter, on the same 30 vibe-py pairs and the same metric. hunter (K=5) pipeline (K=1) pair-correct 76.7% 73.3% leak 12.3% 6.7% cost ~$0.69 $0.0126 determinism none entailed band deterministic Half the leak, 55x cheaper, a fifth of the model calls, one pair short of the hunter's recall (22 against 23 of 30). And understated: all six both_silent pairs are access_control and config_secrets, which had no pipeline path to their arbiters when this ran. A corrected run supersedes those rows. The rung split says what is doing the work: 15 suspicion to 9 model_entailed on the vulnerable side. The LLM supplies most of the recall; the model supplies the precision, halving the leak by dropping claims it could contradict. That is the division of labour the architecture was designed around and the first evidence it pays. It also revises an earlier conclusion of mine. "Detection has regressed" was measured with the proposer disconnected -- an arbiter compared against a detector. With the enumerator wired in, the comparison reverses on precision and cost while holding recall. And it makes the strategy concrete: the model arbitrates about a third of findings today, and every suspicion converted to model_entailed is leak removed for free and one fewer model call. Sink coverage and interprocedural resolution now have a measured payoff rather than a hoped-for one. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:59:01 +02:00
Marius CiepluchandClaude Opus 5	cdedb372dd	feat(model-grounded-detection): wire the detector -- the enumerator proposes, the model disposes ... Everything before this built an ARBITER: something that judges a flow taint reachability had already found unaided. Nothing proposed to it. That is why its recall was never comparable to a detector's, and why comparing 41.3% entailment against the LLM hunter's 76.7% pair-correct was comparing two different kinds of thing. model/judge.py and enumerate_candidates were called by no production code at all; every number so far came from a measurement harness. The wiring turns on a distinction judge() alone conflated: the model CONTRADICTS it has coverage at this site and says no -- a sanitizer on the path, or a declared-safe sink shape. The claim is dropped (Req 8.2). the model CANNOT DECIDE no modelled sink or source reaches the site at all. The model is SILENT, not negative, so the claim is reported at `suspicion` (Req 8.4). Treating silence as refutation is the tempting mistake and would be wrong here: 23 of the 39 misses in the entailment ceiling were sinks simply absent from the fact tables, so "no flow" usually means the model cannot see it rather than that nothing is there. The rung carries that uncertainty on the finding's face instead of a confidence score nobody can act on. An entailed site still costs zero model calls. The candidate budget is bounded so a dense region cannot become an unbounded spend. Smoke-tested end to end against a live endpoint: 2 pairs, detection 100%, $0.0024. The full vibe-py measurement -- the first like-for-like comparison with the hunter's 76.7% pair-correct and 12.3% leak -- is running. 727 passing, mypy/ruff clean, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:41:45 +02:00
Marius CiepluchandClaude Opus 5	1af3db37f6	feat(corpus): vendor 40 OWASP xss pairs as the owasp slice ... output_encoding had exactly ONE pair in the whole corpus. It now has 41. 238 pairs were harvested from OWASP BenchmarkJava (2740 generated servlets, GPL-2.0, with a ground-truth CSV) by matching each really-vulnerable test to the safe test of the same category it most resembles -- adjacent tests come from one template, so they differ in the sink or the sanitiser rather than being unrelated programs. Median similarity 0.91, against the 64% of the older `sast` OWASP entries that the group-4 stratification flagged as different programs. Only 40 are vendored. One representative of each of the seven categories was run through the model BEFORE vendoring, and only xss distinguished: its fix adds ESAPI encoding, which the Java sanitizer facts recognise, so the vulnerable side entails and the fixed side drops to corroborated. cmdi, sqli, ldapi and xpathi put the sink in a helper class outside the servlet and resolved no flow at all; pathtraver found the flow but its "safe" twin carries no distinguishing call, so text similarity had matched a twin that is not semantically the fix -- a subtler form of the very defect the harvest was designed to avoid. Those 198 pairs are deliberately left out: adding pairs the model cannot read would inflate the corpus while making every number less interpretable. Evidence for the vendored slice is 3 of 3 distinguished, which is a small sample and is stated as such in the catalog header rather than implied to be more. The slice is marked SYNTHETIC in its header: it measures breadth across a weakness class and must never stand in for real-world performance. The corpus already shows a +16% overfitting gap, and a large generated slice blended into a headline would widen it invisibly. Two things recorded rather than hidden: BenchmarkJava's LICENSE is GPL-2.0 while the older sast entries declare GPL-3.0 for OWASP, which looks like an error and is not copied; and OWASP-Benchmark/BenchmarkJava already straddled the train/holdout split before these pairs (owasp-java-cmdi train, owasp-java-sqli holdout), so the new all-holdout pairs enlarge an existing straddle rather than create one. The two vendored-count guards did their job -- a corpus change could not land silently -- and are updated as the acknowledgement. Hygiene artifact regenerated: 277 -> 317 vendored pairs. 720 passing, mypy/ruff clean, gates byte-identical. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:28:11 +02:00
Marius CiepluchandClaude Opus 5	fcb12e116a	feat(model-grounded-detection): Java web sinks, and the post-closure-fix ceiling ... Java shipped with injection, deserialization and crypto sinks only, so the path and output_encoding families had nothing routing to them by CWE and any Java pair in either was unreadable -- the gap hutool-cve-2018-17297 reported as "no path spec for java". Adds the JDK filesystem entry points, the servlet response writers, the outbound/redirect APIs, and the ESAPI / canonical-path sanitizers the OWASP suite itself uses. Documented APIs, provenance per entry. Post-closure-fix ceiling: 41.3% entailed (38/92) and 26.1% distinguished, up from 39.1% / 23.9%. path went 0/8 to 1/8 with its first distinguished pair; injection 29->31 entailed and 18->19 distinguished. Two points -- real, and the mechanism is understood, but the remaining gaps are data rather than machinery. Carrying a correction: the measurement harness routed config_secrets through TAINT rather than the constant-abstraction arm, so its 0/10 in this artifact measures the wrong arbiter and did so in both full runs. The harness is fixed; that family's real number is the earlier targeted re-measure, and this artifact's config row should not be read. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 17:16:00 +02:00
Marius CiepluchandClaude Opus 5	0c71c956fd	feat(model-grounded-detection): group 6, the execution tier as a seam not a build ... The rung this project deliberately does not build. Clearwing already runs sandbox lifecycles, container pools, sanitizer images, PoC replay and stability classification, and the corrective brief concluded -- from this project's own defect history -- that reimplementing that machinery is where we go wrong. So model/execution.py decides eligibility, calls an adopted tool, and records what happened. A test asserts structurally that it imports no container or subprocess machinery and defines no build/run/compile/patch function, because the failure mode to prevent is this file quietly growing a sandbox. Eligibility is narrow in both directions: only a family a static model cannot arbitrate (memory safety in C, where the bug turns on runtime layout -- Req 10.3 forbids any web/logic family depending on this tier, and each now has taint, dominance or constant abstraction instead), and only a finding the model left undecided. The tier escalates uncertainty; it never re-checks a verdict. One deliberate asymmetry: a failure to reproduce is NOT a refutation. A sandbox that does not trigger a bug has not shown the bug is absent, only that this attempt did not trigger it, so the candidate stays where the model left it. Reading a failed reproduction as a clean bill of health is the same error class as reading an absent engine as "no vulnerability". With no adopted tool present -- the normal case -- the rung is never reached and clearwing_unavailable is recorded. All 16 tasks of model-grounded-detection are now done. 720 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:58:03 +02:00
Marius CiepluchandClaude Opus 5	f1c341df6c	feat(model-grounded-detection): group 5, corpus calibration and honest reporting ... 5.1 inverts what the corpus is for. Previously a pair the detector missed was a DETECTOR failure, to be averaged away over K runs and chased with a prompt change. The arbiter is deterministic: it will miss that pair the same way every time, and the only thing that moves it is better modelling. So a pair the model cannot split is a NAMED GAP IN THE MODEL, recorded with its reason, and every gap carries one even when the caller supplies none -- a bare gap teaches nothing. `covered` means the model reached a strictly higher rung on the vulnerable side than on its fix. Flagging both sides equally is not coverage: it distinguishes nothing, which is exactly what group 2 measured taint reachability doing on every parameterised SQL pair. The harvest seam takes the fix commit and derives the vulnerable side as its parent, with the fix diff as the oracle -- nothing hand-labelled. 5.2 keeps three disciplines from the deleted publish layer, each because its absence caused a real error here: * per-slice rows always -- `render` RAISES without them. A mixed headline once hid agent-written code scoring 44% against the development slice's 83%. * the overfitting gap beside the headline, and ABSENT rather than zero when a slice is missing: reading an unmeasured slice as "no overfitting" is the flattering failure, not a neutral one. * the rung on every finding, so a suspicion cannot be read as a result. Run end to end against the committed ceiling artifact, the report says things worth not hiding: agent-vfc 0/28 covered, vibe-py 40% against vfc-js 24%, an overfitting gap of +16%, and gaps grouped by family with their reasons -- injection 32 (21 no verdict, 11 distinguished nothing), path 8 (all no verdict), untrusted_destination 5 (4 entailed on BOTH sides). 713 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. A full re-measure with the closure-scoping fix is in flight; these numbers are the pre-fix baseline and will move. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 16:28:10 +02:00
Marius CiepluchandClaude Opus 5	91c0931a05	measure(model-grounded-detection): group 4 ceiling, 39.1% entailed against the flat IR's 2.2% ... Full model layer over 92 pairs: 39.1% model_entailed (36/92), 45.6% reaching any verdict, 23.9% distinguished -- an 18x lift in entailment from an arbiter that makes no model calls at all. Per family: untrusted_destination 0.80 entailed, injection 0.58 (0.36 distinguished), deserialization 0.33, prototype 0.14, config_secrets 0.10, path 0.00. access_control reaches 6 of 8 pairs at 0.38 distinguished -- the class no crash oracle can reach, arbitrated deterministically. A hypothesis tested and discarded. From the group-2 misses I diagnosed path's 0/8 as a missing req.url source token and said so. The probe -- the same 8 pairs with and without the token -- returned 0/8 -> 0/8. Falsified. The real cause is that the sink sits lexically inside an anonymous callback, so its enclosing method is the closure (<lambda>0) and the sinkMethod == function filter drops it; verdict(function="") on the same CPG returns model_entailed. And the obvious fix for that is wrong. Relaxing the filter raises the number with a FALSE witness: the flows found in the closure are `res -> readFileSync` and `this -> ...`, neither of which is the vulnerability, whose real path crosses a helper (req.url -> url -> resolveUrl -> possibleFilename). Turning a miss into a wrong entailment is worse than the miss -- it is the arbiter asserting something untrue, the one failure mode this architecture exists to prevent. The correct fix is two-part and recorded: scope a sink to any method lexically nested in the labeled function, and scope parameter sources to that function's own parameters, then re-measure counting only real witnesses. Every miss is classified and none is architectural: 23 sink absent from the excerpt or unmodelled, 13 no recognisable source in the excerpt, 3 source unmodelled with the idiom named. Two honesty items in the artifact. 16 of 92 pairs carry no function label and are arbitrated over the whole file, so a verdict can credit a different finding than the labeled one -- owasp-python-hash (CWE-327) is credited for a permissive set_cookie elsewhere in its file, on both sides. And config_secrets conflates three abstractions: permissive setting values (implemented), weak algorithm choice (facts already carry weak_literals, unwired) and hardcoded secrets (needs a credential-literal check). 694 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:46:05 +02:00
Marius CiepluchandClaude Opus 5	d9ceeb6a09	feat(model-grounded-detection): 4.2 constant abstraction, the third arbiter form ... A configuration bug has no flow to follow and no guard to dominate. CORS(app, origins="*") is dangerous because of the value it is SET TO, so the arbiter evaluates the setting's arguments against the closed permissive set in the obligation facts -- the third form beside taint reachability and guard dominance, and the one Req 6.3 names. ENTAILED a security setting is passed a literal in the permissive set CORROBORATED the argument is computed, so constant abstraction cannot decide it -- that residual is the judge's, never silently "safe" (none) every literal is outside the set, or the call is not a setting Two Joern node-shape bugs, both found by running it and both of the same class as group 2's receiver bug -- the graph's indexing is not what the source text suggests: * a keyword argument (`origins="*"`) is an assignment-shaped node, so isLiteral is false on the argument itself and the literal sits in its subtree; literals are collected from the argument's AST now. * Joern indexes the receiver at 0, positional arguments from 1, and NAMED arguments at -1. Settings are configured overwhelmingly by keyword, so argumentIndexGt(0) -- correct when counting arity for the taint shape test -- dropped exactly the arguments that matter and every keyword-configured setting read as "computed". Live, deterministic, zero model calls: create_app -> model_entailed CORS(app, origins = "*"): permissive literal "*" safe_app -> no finding (origins = "https://app.example") All three arbiter forms now work against a real CPG: taint reachability for the flow families, guard dominance for absence, constant abstraction for config. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 15:00:41 +02:00
Marius CiepluchandClaude Opus 5	fec67253f3	feat(model-grounded-detection): 4.1 guard dominance, 4.2 the missing sink facts ... 4.1 is the arbiter for the class no crash oracle reaches. An access-control bug is the ABSENCE of a guard: no flow to follow, no crash to reproduce. What the model can establish is a consistency violation -- an obligated operation no discharging guard governs, in a file where its siblings are governed by one. "Governed" turned out not to be one relation, and asking pure CFG dominance of the operation gets two of three shapes wrong. Measured against the real engine: identity constraint the guard is INSIDE the operation's arguments filter_by(owner_id = current_user.id) identity branch abort(403).controlledBy == [note.owner_id != current_user.id] -- the check runs AFTER the fetch, so it never dominates the operation, but it controls whether the value escapes dominating guard a discharger-naming call that dominates the operation That is why the query uses controlledBy rather than dominatedBy for shape 2. Two bugs found by running it, both the same class as the sanitizer bug in group 2 -- part of a fact flattened, the part carrying the signal dropped: * dominance_specs used only `calls` and `decorators`, omitting identity_sources (current_user, request.user) and constraint_params (owner_id, user_id) -- which is how an identity constraint is actually written. The arbiter reported "no guard" on handlers that plainly had one. request_sources and permissive_values stay excluded: they describe the vulnerable shape, and including them would let the arbiter read a bug as its own fix. * matching operations by code containment yielded four or five desugared rows per operation (the bare `tmp1.filter_by` receiver among them, carrying none of the guards the real call has), so a guarded operation could be flagged. Matching is by call name now. Live, deterministic, zero model calls: leaky -> model_entailed no discharging guard, while 2 sibling handlers are guarded edit_note -> no finding (ownership check controls the denial) list_notes -> no finding (identity constraint in the query) 4.2 authors the sink facts the go/no-go measured as its largest miss class (11 of 23): Django ORM raw SQL, and the four families that previously had no fact routing to them by CWE at all -- path, output_encoding, untrusted_destination, prototype. Each entry is a documented dangerous API of the named library with its provenance recorded inline, and rule_ids left empty because a sink fact is not a detection rule. Flagged for maintainer review as data, given the standing "no LLM-authored patterns" constraint. The exact-set assertion written in task 1.1 to hold this gap open is updated rather than deleted, so it still fails if a family ever loses its seed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 14:45:12 +02:00
Marius CiepluchandClaude Opus 5	b8f8f727c6	feat(model-grounded-detection): group 3, the evidence ladder and the judge ... 3.1 adds StaticFinding.rung beside evidence_level rather than renaming it. The existing evidence_level carries `static_corroboration`, emitted by the flat IR overlay -- a path Req 4.4 demotes to the suspicion band. Restating that as model_corroborated would claim a CPG arbitrated something it never saw, and rewriting every legacy finding to suspicion would be a product-visible regression nobody asked for. So `rung` defaults to suspicion and is raised only by ladder.at_rung(finding, verdict), which is the structural form of Req 5.4: nothing rises above suspicion without a Verdict behind it. The witness travels with the rung, because a rung without its evidence is just a louder assertion. It reaches markdown, JSON and SARIF. 3.2 is the judge, and it is defined as much by what it lacks. Three exhaustive outcomes: the model entails (report it, never call the model), the model contradicts (drop the claim, record why), the model cannot decide (report suspicion). The LLM gets one bounded typed question with tools=[] so it cannot search for the site, asked only about the residual the graph cannot settle -- whether a sanitizer on a REAL flow suffices. Its answer can only confirm a flow the CPG already found; it can never conjure one. Req 8.5 is asserted structurally: a test parses judge.py and fails on any AST name like k_runs/majority/vote/average/acceptance/floor/budget, any for/while loop, or an import from the deleted round machinery. That test first matched the word "votes" in the module's own docstring, which is why it now inspects code rather than text. Verified live against a real CPG, not only scripted rows: vuln rung=model_entailed request.args["name"] -> db.execute(...) fixed rung=suspicion no source reaches this sink model calls made: 0 Both sides arbitrated by the graph with the LLM never consulted -- the thing the prior architecture could not do at any price. 678 passing, mypy/ruff clean, gates byte-identical, zero orphaned modules. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:49:55 +02:00
Marius CiepluchandClaude Opus 5	7387d2f190	measure(model-grounded-detection): task 2.3 reads GO on Joern ... Injection slice, 50 pairs, ~30 minutes of real engine time. model_entailed on the vulnerable side 54% (27/50) vs the flat IR's 2.2% ... on minimal-fix pairs only 61.5% (16/26) pairs distinguished 32% overall, 38.5% on minimal-fix determinism (Req 6.4) every verdict identical on re-run cost median 32s/pair, max 58s CPG build failures 1 The engine is not the constraint: all 23 misses trace to our own data model -- 11 sinks absent from the fact tables (Django ORM objects.raw, self.query, GraphQL resolvers), 11 where the sink is modelled but no source reaches it, and one C translation unit that does not parse standalone, counted separately so it can never read as "no vulnerability". That is exactly the result that justifies adopting a CPG rather than building one. The finding that changes the design: classifying what the 50 injection FIXES actually do gives other_or_validation 37, bound_parameters 10, argv_list 2, sanitizer_call 1. The dominant fix is a validation guard on the path, not a sanitizer and not a sink shape -- and taint reachability can never separate those pairs, because the flow still reaches the sink and what changed is that a guard now dominates the path. The arbiter must be TAINT + DOMINANCE, which is the same primitive task 4.1 builds for access_control, with the guard vocabulary ported in task 1.1 as its lookup list. Two planned mechanisms collapse into one, and 4.1 should now precede the remaining TaintSpecs. Reported per stratum because only 26 of 50 pairs are genuine minimal-fix pairs: 12 are different programs (local-python-vulnerable is 7% similar with 83 changed lines) and 12 are large rewrites. A raw distinguished_rate would measure corpus quality and report it as model quality. Also corrected in the notes: the Flask-shaped source table is a real gap but a minor lever -- adding Django patterns would gain a source for exactly one pair. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 13:26:59 +02:00
Marius CiepluchandClaude Opus 5	2b973dc0db	feat(model-grounded-detection): tasks 2.1-2.2, the Joern seam and the taint verdict ... The whole engine integration is one subprocess boundary: joern-parse builds a cpg.bin, joern --script runs a shipped CPGQL query that prints its JSON between fence markers, because Joern's stdout carries a banner and a REPL prompt around whatever a script emits. Nothing is imported in-process, so the core install keeps dependencies = [] -- a test walks the seam's AST to assert it imports only the standard library and names no JVM binding. Fail closed everywhere. A build or query that errors, times out, or prints something unparseable yields no verdict and the caller reports suspicion. The taint verdict distinguishes this from absence deliberately: a query returning None means "the engine could not decide", which is not "there is no flow", and conflating them is how a missing engine silently becomes a clean bill of health. The verdict is where the ladder earns its keep. An unsanitized source->sink path is model_entailed -- the graph establishes it and the LLM's claim is no longer load-bearing. A path a sanitizer may break is model_corroborated: the flow is real, but whether the sanitizer suffices is the semantic residual the LLM answers, checked against this verdict rather than averaged over runs. The chosen flow is picked by a total order (shortest, then lexicographic), so two runs over one CPG cannot disagree -- Req 6.4 asserted directly. The query itself is the part our flat IR could not do: reachableByFlows walks the PDG, so `q = "..." + request.args["x"]; execute(q)` is found, which is exactly the one-hop limit that held measured entailment at 2.2%. Capability detection reuses semantic/engines.joern_available rather than growing a second notion of "installed". The audit manifest is regenerated; the new modules are named by Req 4/5/6 rather than counted as orphans. Task 2.3, the go/no-go measurement, needs Joern installed and has not run. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:23:40 +02:00
Marius CiepluchandClaude Opus 5	9ed0e45c5f	feat(model-grounded-detection): group 1, subtract the noise architecture ... 99 source modules -> 81. 832 tests -> 646. The whole learning/ package (16 modules, ~4,100 lines) and six semantic/ modules removed, with the three gates byte-identical to 2026-09-06-gate-baseline.txt at every step. The sequence is what made it safe: vocabularies ported first (1.1), importers detached second (1.2-1.3), deletion third (1.4). Every import break during the deletion was therefore a detachment miss in the importer and was fixed there -- no module was restored to satisfy one. Both misses were the same shape and were fixed as a class: operations.py and shapes.py each imported trailing_name from the deleted variants.py, so it moved to semantic/ir.py, its natural home, and both importers changed in one edit. Ported before deleting: the closed guard vocabulary (verbatim, nothing else holds it) into model/specs.py, which derives TaintSpec and DominanceSpec from the retained ruleset facts rather than copying them. Re-homed: families -> model/taxonomy, candidates -> model/candidates, endpoint -> model/endpoint, plus Region, labeled_family and labeled_spans. labeled_family no longer falls back to the LLM classifier -- an unlabeled pair is honestly unknown. Four deviations, each recorded in the spec notes with its reason. improve/ evolve.py and validator.py were stripped, not deleted: each mixes the mechanism lever (gone) with the deterministic rule-status loop `ousast improve` still runs, which has no LLM in it; a test now asserts the lever's absence by symbol so it cannot creep back. The learning CLI group became `model candidates`, because deleting it outright would have orphaned the retained enumerator by removing its only entry point. thinking moved from [learning] to [models]. A gap named rather than papered over: no retained sink fact routes by CWE to path, output_encoding, untrusted_destination or prototype -- the same hole the entailment ceiling diagnosed. The test asserts the seeded set and the gap set exactly, so task 4.2 must update it when it authors those sinks. The audit found exactly one genuine orphan: vectorstore.py, a bakeoff harness with no production caller and no CLI, alive only through its own test. The five named subsystems each carry an explicit decision. Manifest committed, with a test asserting the tree still matches it. Group 2 (the Joern go/no-go) has not started. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 12:15:54 +02:00
Marius CiepluchandClaude Opus 4.8	9e811c1baf	measure(model-grounded-detection): the entailment ceiling is 2.2%, and it is depth not a wall ... The analogue of the candidate ceiling, one rung up: not "can a site be enumerated" (96.6%) but "can a deterministic abstract-interpretation model DECIDE the vulnerability without trusting the LLM". Offline, no model calls. model_entailed today 2.2% (2/92 web/logic pairs) model_corroborated, one-hop taint 24% (22/92) taint reachable, multi-hop 35% (26/74 flow families) candidate site enumerable 96.6% (for comparison) The gap between 96.6% and 2.2% is the gap between enumerating and arbitrating, and it is the finding. The model cannot be a pure arbiter at v1. But the measurement diagnoses depth, not impossibility: one-hop source resolution (real taint sits 1-3 binds upstream: q = '...'+id; execute(q)), a closed literal sink table (misses project wrappers and path/xss/prototype sinks), and a function-naming gap that eats 20/50 injection pairs. Deepening the taint to the candidate enumerator's depth-4 chain already moved injection 24% -> 48%. So the course is a layered arbiter, not a pure one: report model_entailed where the model proves it (deterministic, no LLM); let the LLM judge but check its claim against the resolved taint on the ~35% corroboration band (noise removed there); honest suspicion for the rest, which is where the deferred execution/differential tier earns its place. Modeling confirmed as abstract interpretation; SMT reserved for the memory/C tier; concolic avoided. Co-Authored-By: Claude Opus 4.8 <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-08 11:10:47 +02:00
Marius CiepluchandClaude Opus 5	66a51c674c	docs(learning-harness): round 5, and what it says about the experiment itself ... Round 5 re-ran round 1's content through the lever it should have used -- counterexamples, rendered to the detector as "Worked examples" rather than "Known non-issues, do not report these" -- and was rejected because one pair of eight moved, at p = 1.000. So the experiment I designed to give the demo lever a fair test could not give it one. Correcting my authoring error changed the reason for the verdict and not the verdict, because the rule cannot tell a lever that helps from a lever that does nothing. Five rounds, $2.17, zero decisions distinguishable from chance. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 08:09:50 +02:00
Marius CiepluchandClaude Opus 5	78924316be	docs(learning-harness): the reading of the first evolve rounds ... Four rounds on injection, $1.76. Not one decision carried evidence, in either direction. Round 2 rejected a change because one pair of eight moved, at p = 1.000 -- the maximum the sign test can return. Round 3 accepted one at p = 0.500 and that acceptance is now versioned as v1. On a family where 11 of 20 pairs disagree with themselves between runs of the unchanged detector, both readings are what changing nothing looks like. I corrected an overstatement in the notes: after two rejections I wrote that the rule was near-incapable of accepting. Round 3 accepted. The accurate statement is symmetric and worse -- it accepts on noise as readily as it rejects on it. Escalated, not applied: Req 9.5's threshold rule should be replaced by an acceptance that survives the measured variance. The Pareto archive and the sign test both exist and neither decides anything. The honest caveat is that at a minibatch of 8 a reliability-gated rule would accept almost nothing either -- the corpus is the binding constraint, not the rule. And the finding that outranks the rounds, in the maintainer's words: the error is not the model, the error is the elsewhere. Eighteen defects this session, every one in this repository, three of them the same shape of fixing an instance and missing the class. The default assumption for an unexplained number is now that we broke it -- which puts pro's 46.7% against flash's 83.3% on our side of the ledger until shown otherwise. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-07 07:39:48 +02:00
Marius CiepluchandClaude Opus 5	84b1ba7a72	fix(learning-harness): the published table lost a whole slice ... `publish` keyed baselines by model alone, so the vibe-py run silently replaced the agent-vfc run and the table showed hand-written teaching code as if it were the corpus. That is precisely the failure the constrained-detector requirements exist to prevent, live in a published artifact. Third time today that a key was fixed in one place and not another -- `compare_baselines` got the model+slice label and `_baselines` did not, the same way the exception path got checkpointing and the timeout path did not. The table now carries a slice column and one row per slice, and the artifact is keyed by what identifies a run. Republished: agent-vfc's six families are back, including access_control at 0.000 recall on four pairs, which is the number that was missing. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 20:44:56 +02:00
Marius CiepluchandClaude Opus 5	78b4f33a8e	feat(constrained-detector): the ceiling, measured offline across every slice ... Group 1 complete. Tasks 1.2 and 1.3: `candidates.ceiling`, `ousast learning candidates`, and the stop-or-go reading. No model is called anywhere in this group, so the condition that could have ended the feature was reachable for free. Verdict: go. agent-vfc (deciding) 89.7% ceiling against 44.4% round-zero detection vfc (deciding) 84.7%, baseline still running, non-gating vfc-js (deciding) 52.9% -- see below github (deciding) 100.0% vibe-py (development)100.0% against 83.3% The margin is largest on agent-vfc, which is the slice the tool is actually for. The measurement forced a shape correction before the detector existed. config_secrets first read 63.6% with a bind-and-literal shape, because a permissive default is as often an argument to a call -- cors({origin: '*'}), app.use(session({secure: false})) -- as it is a binding. Adding call sites to that family's shape took it to 100% and recovered four agent-vfc rows. Carry-forward, and not this feature's to fix: vfc-js sits at 52.9% because eight of seventeen rows label a function the JavaScript CST names `<anon>` -- `module.exports.publish = function () {}` and its relatives. That is a function-naming gap, it caps the existing hunter on that slice exactly as much, and it is the same class as the `_declarator_start` defects found in learning-harness task 4.1. The ruleset arm of `ceiling` is kept so the 12.1% that rejected it stays a measurement rather than a remembered number. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 18:44:19 +02:00
Marius CiepluchandClaude Opus 5	309faad266	measure: the candidate generator must be the IR, not the ruleset ... A candidate-driven detector can only judge sites something else enumerates, so the enumerator caps recall. Measured over the 58 scorable web pairs before building anything: pattern ruleset: 12.1% ceiling -- and the misses are not near-misses, the ruleset finds nothing at all in most of these files semantic IR: 96.6% ceiling, a median of ten call sites and bindings inside the labeled function The ruleset would have capped us at 12.1% against the 64.9% the unconstrained loop already reaches. That design was one commit from being built. Two corpus defects found on the way, both mine from task 4.1. The overleaf excerpt started mid-import, because `_declarator_start` walks back to the nearest `}` and Overleaf writes JavaScript without semicolons, so the nearest one was inside `import { expressify }`. It then ended mid-object-literal, because registration mentions were appended as bare lines. Neither parsed, and a pair that cannot be parsed cannot be scored. Registration mentions now expand to the statement that contains them; the ceiling moved 94.8% -> 96.6%. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 16:04:07 +02:00
Marius CiepluchandClaude Opus 5	5e2070bc17	measure(detector-optimizer): the detector never runs the same investigation twice ... Twelve injection pairs, both sides, K=5, temperature 0, thinking disabled, recording every tool call. $0.23. The statistic I designed for could not be computed: across 120 runs there was not one repeated trajectory -- 4.67 and 4.92 distinct trajectories per side out of 5, and 78 distinct tool-name sequences ignoring arguments entirely. The detector is not sampling one measurement five times; it is running five different investigations and voting on the results. As a gradient the relationship is monotone: runs whose trajectories were similar (>=0.75 Jaccard) disagreed 0% of the time, mid-similarity 4.2%, low-similarity 30.9%. Three more facts. The search is grep: 555 grep_repo calls against 181 read_file, 71 entry_points, 43 find_refs, 13 read_definition -- the model is guessing patterns to locate sinks the CST and the entry-point mapper already know. All the instability is on the vulnerable side; every pair produced exactly one finding set on the fixed side, so the noise is in detection and none of it in false positives. And findings converge much more than trajectories do: 78 trajectories, one to three finding sets per pair. The floor is therefore not measuring uncertainty about a question. It is measuring how often an unconstrained search happens to reach the sink. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 15:49:54 +02:00
Marius CiepluchandClaude Opus 5	2dad25d312	fix(learning-harness): the round-4 review findings ... Three defects, two of them the same shape as ones already rejected once: - The meta-agent proposer still could not propose. `MetaAgent(inner_model=)` takes a ModelConfig; it was handed a model name, a `# type: ignore` hid that from mypy, and a bare `except Exception` turned the resulting AttributeError into `meta_agent_failed` -- which reads exactly like the model declining. The composition is now a seam a test can pin, and a construction error is re-raised rather than reported as a quiet round. - `ousast pairs --hunter` crashed in its documented form: the K floor landed with `--k-runs` still defaulting to 1. The covering test always passed `--k-runs 4`, which is why nothing noticed. - 28 rows (12% of the corpus) resolved a span on the vulnerable side and none on the fixed side, so every fixed-side finding read as outside the labeled function and the pair scored as silence whatever the detector said. `fix_function` now reaches the catalog and the scorer, and the generator derives it from the fixed excerpt for the 25 trap rows that never declared one. Measured after: 0 rows. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 14:03:10 +02:00
Marius CiepluchandClaude Opus 5	75ae9ee47b	test(learning-harness): the gate baseline and the degradation matrix ... The three gate outputs are captured at 71efc9c -- the commit before the first line of this feature -- and a test reproduces them byte for byte in both matrices. Byte-identity was something a reviewer checked by hand against a worktree; it is now mechanical and runs in CI. Six absences are each exercised once with the reason they record: HarnessX, the chat endpoint (the run still writes its artifact), the family configurations, the network for pointer pairs, the semantic extra (named, not guessed), and the sandbox -- so a claim nothing could run is not confused with a claim that failed. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 13:20:23 +02:00
Marius CiepluchandClaude Opus 5	3e71e9e88b	feat(learning-harness): fix dates and the post-cutoff slice ... The catalog generator fills `fix_date` from the recipe at the precision the recipe has -- a full ISO day when one was recorded, otherwise the year. Every harvested recipe carries a year and almost none carries a day, and writing `-01-01` would turn a year into a claim about a month. A row counts as post-cutoff only when its earliest possible date is after the cutoff, so a coarse date can under-claim the clean slice but never over-claim it. Each family now reports the same numbers over the rows fixed after the cutoff, beside the whole number and never instead of it, with the undated count on its own line. Without a configured cutoff every scoreboard says so rather than inventing one: guessing a model's training cutoff and then publishing a contamination number from the guess would be worse than reporting none. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 13:15:20 +02:00
Marius CiepluchandClaude Opus 5	d1faa04771	feat(learning-harness): declared twins and a fixture-difficulty check ... The five byte-identical vibe-py twins now say so in the catalog instead of only hashing to it, and a declared limit that names one of the unscorable vocabulary's own reasons keeps that spelling, so `identical_twin` stays one fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35. The difficulty check measures three axes the corpus varies on: binding hops from the identity source to the constrained use, registration distance, and guard distance. Measured first, and it failed: the shared obligations fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2). Every fixture handler carried its own route decorator, so nothing in the suite exercised a handler registered from another document and no binding chain was longer than one hop -- the checker was passing on a simpler world than the one it meets. `from_spec` closes that with the connexion shape those rows really have. The verdict is committed beside the numbers. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 13:09:56 +02:00
Marius CiepluchandClaude Opus 5	c2d9c4ad01	feat(learning-harness): absence rows travel with their registration ... Eight of the ten vendored access-control rows had no named handler at all, so every one of them would have scored `missing_context` forever rather than as a detector miss. After the re-harvest nine are named and the tenth states its reason. The handler-context mode now carries the class its framework registers, multi-line decorators, and commented-out guards -- `#@token_required` above a ThreatByte handler is the entire bug. Where the registration lives in another document (connexion's `operationId`, an express router), a recipe names `context = [...]`, the harvester writes that stanza per side, and the pair materializes both files at their repository-relative paths so the entry-point mapper sees the layout the project has. `harvest_recipe` fetches every blob before the first write and the driver takes `--report`, so a failed fetch leaves a reviewed excerpt exactly as it was. Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the one pair vendored as an ES module had no file target at all; and `_python_route_access` matched auth words inside decorator prose, reading `@ns.doc(description='... without proper authorization checks')` as a guard. Recorded in the hygiene artifact: nine of the ten duplicate groups sit on both sides of the split, which the repository-level split cannot see. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG	2026-09-06 12:39:55 +02:00