Three defects, two of them the same shape as ones already rejected once:
- The meta-agent proposer still could not propose. `MetaAgent(inner_model=)`
takes a ModelConfig; it was handed a model name, a `# type: ignore` hid
that from mypy, and a bare `except Exception` turned the resulting
AttributeError into `meta_agent_failed` -- which reads exactly like the
model declining. The composition is now a seam a test can pin, and a
construction error is re-raised rather than reported as a quiet round.
- `ousast pairs --hunter` crashed in its documented form: the K floor landed
with `--k-runs` still defaulting to 1. The covering test always passed
`--k-runs 4`, which is why nothing noticed.
- 28 rows (12% of the corpus) resolved a span on the vulnerable side and
none on the fixed side, so every fixed-side finding read as outside the
labeled function and the pair scored as silence whatever the detector
said. `fix_function` now reaches the catalog and the scorer, and the
generator derives it from the fixed excerpt for the 25 trap rows that
never declared one. Measured after: 0 rows.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The catalog generator fills `fix_date` from the recipe at the precision the
recipe has -- a full ISO day when one was recorded, otherwise the year.
Every harvested recipe carries a year and almost none carries a day, and
writing `-01-01` would turn a year into a claim about a month. A row counts
as post-cutoff only when its earliest possible date is after the cutoff, so
a coarse date can under-claim the clean slice but never over-claim it.
Each family now reports the same numbers over the rows fixed after the
cutoff, beside the whole number and never instead of it, with the undated
count on its own line. Without a configured cutoff every scoreboard says so
rather than inventing one: guessing a model's training cutoff and then
publishing a contamination number from the guess would be worse than
reporting none.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
The five byte-identical vibe-py twins now say so in the catalog instead of
only hashing to it, and a declared limit that names one of the unscorable
vocabulary's own reasons keeps that spelling, so `identical_twin` stays one
fact. Leave-one-out over vibe-py consequently runs on 30 rows, not 35.
The difficulty check measures three axes the corpus varies on: binding hops
from the identity source to the constrained use, registration distance, and
guard distance. Measured first, and it failed: the shared obligations
fixture reached (1, 0, 2) where the vibe-py absence rows reach (5, 2, 2).
Every fixture handler carried its own route decorator, so nothing in the
suite exercised a handler registered from another document and no binding
chain was longer than one hop -- the checker was passing on a simpler world
than the one it meets. `from_spec` closes that with the connexion shape
those rows really have. The verdict is committed beside the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Eight of the ten vendored access-control rows had no named handler at all,
so every one of them would have scored `missing_context` forever rather
than as a detector miss. After the re-harvest nine are named and the tenth
states its reason.
The handler-context mode now carries the class its framework registers,
multi-line decorators, and commented-out guards -- `#@token_required`
above a ThreatByte handler is the entire bug. Where the registration lives
in another document (connexion's `operationId`, an express router), a
recipe names `context = [...]`, the harvester writes that stanza per side,
and the pair materializes both files at their repository-relative paths so
the entry-point mapper sees the layout the project has. `harvest_recipe`
fetches every blob before the first write and the driver takes `--report`,
so a failed fetch leaves a reviewed excerpt exactly as it was.
Two defects the re-harvest exposed: `.mjs` was not a known suffix, so the
one pair vendored as an ES module had no file target at all; and
`_python_route_access` matched auth words inside decorator prose, reading
`@ns.doc(description='... without proper authorization checks')` as a
guard. Recorded in the hygiene artifact: nine of the ten duplicate groups
sit on both sides of the split, which the repository-level split cannot
see.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 4.1. A new excerpt mode keeps the labeled handler with its decorators and
every statement outside it that registers the handler by name, so guards living
in decorators, routers and middleware travel with the pair. Comment and string
mentions never count.
The closed mechanism vocabulary gains the three obligation ids, catalog rows
accept an additive obligation field validated against the closed operation
kinds, and the pair scorer counts a finding tagged with that obligation inside
the labeled function as a detection. Rows without the label run the same code
as before.
Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.4 (Req 10.6, 5.1).
- harvest.merge_recipes appends recipes by name and never rewrites
existing rows.
- vibe-py/build_recipes.py --pointers: the 40 LLM-generated repositories
without a license become vendored = false rows (80 pairs, agent,
seeded) appended to recipes.toml; unlicensed human repos stay skipped.
- agent-vfc/build_recipes.py --from ... --pointers [--cache]: unlicensed
agent-authored fixes become title-tier pointer rows whose function
label is derived from the cached excerpt (30 added from a fresh
662-candidate search); TOML booleans rendered correctly.
- catalogs regenerated: vibe-py 115 rows (35 vendored + 80 pointer),
agent-vfc 59 (29 + 30); no excerpt is vendored for any pointer row.
- `pairs --slice vibe-py --pointers` scores both provenances (human 35,
agent 80); the default run skips 80 pairs with pointer_pair_skipped;
suite and gates unchanged. README/docs name the nightly command.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Task 8.3 (Req 10.1-10.5).
- pairs.py: catalog rows with vendored = false carry their harvest recipe
and no excerpts; evaluate_catalog harvests them into
OPENULTRASAST_PAIR_CACHE (default ~/.cache/openultrasast/pairs) only
when `pairs --pointers` or OPENULTRASAST_PAIRS_NETWORK=1 allows the
network, otherwise records pointer_pair_skipped (a failed fetch is
pointer_pair_fetch_failed with detail); a warm cache never scores
without the switch. select_vendored for tests and gates.
- harvest.py: materialize_pointer harvests into the cache without the
license requirement (nothing is redistributed); redaction applies.
- catalog_gen.py: emits pointer rows with recipe fields; training
manifests skip them.
- pair_gate and improve/evolve.evaluate_profiles score vendored pairs
only; cli `pairs --pointers`; .gitignore covers the cache path.
- tests: loader, skip/degradation, cache hit, fetch failure, warm cache
with network off, profile gate exclusion, harvest and generator paths;
conftest keeps every test offline whatever the shell exports.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 8.1, 7.1 (tests) and the re-scoped 5.3.
- pairs.py: review_tier in {seeded, advisory, title, reviewed} with
per-slice defaults (local reviewed, vibe-py seeded, sast/vfc/vfc-js/
github advisory, agent-vfc title); reviewer = "pending" means title, a
named reviewer means reviewed, reviewed without a reviewer is rejected;
per_tier metrics in the payload; select_tier and GATING_TIERS for the
improve loop.
- catalog_gen.py: stamps review_tier (and reviewer) on every generated
row and training manifest row; title rows enter catalog.toml; the
catalog-candidates.toml queue is retired; precedence matches the loader.
- agent-vfc: 29 title-tier rows load and score (0/29, Youden 0.0, the
honest unreviewed baseline); local rows name their reviewer.
- Tests: tier defaults, loader rejections, per_tier payload, the Req 8.1
catalog-on-disk / stage-1 exclusion proof and per-slice CLI checks.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 1.2, 2.1-2.5 and 6.2 (pair-side filters).
- pairs.py: coverage with a source counts as detection and as leak;
function-scoped matching by line-range containment from the vulnerable
side's parse (unresolvable labels are counted loss, never silent
misses); exact or alias sink matching; weak CWE-only labels never match;
side-by-side inventory/overlay/hunter scorers; per_profile,
per_mechanism, achievable, known_limit and loss payload; loader rejects
unknown mechanism/provenance/split and reserved or missing function
labels on harvested slices; candidate rulesets thread through for the
improve loop.
- overlay.py: OverlayRecord.function (additive) filled from the shared
naming helper in one parse; benchmark.py: ExpectedFinding.mechanism.
- cli pairs: --slice over every slice, --profile, --split, --hunter,
--hunter-model.
- catalogs migrated to the new schema (provenance, split, function,
mechanism, known_limit for owasp-java-hash); baseline recorded in the
spec notes.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 1.1, 5.1, 5.2, 5.4 and the data half of 5.3.
- mechanisms.toml: closed vocabulary v1 (20 ids) every expected row cites.
- vibe-py: 35 Real-Vuln-Benchmark pairs (Apache-2.0), human provenance only
because the LLM-generated repositories carry no license (escalated).
- vfc-js: 17 Node fix-commit pairs harvested upstream (MIT/Apache-2.0),
SecBench.js itself is not vendored (no license); sinks derived on 13/17.
- agent-vfc: 29 agent-trailer fixes as catalog-candidates.toml plus
REVIEW.md; the loaded catalog stays empty until a human reviews them.
- datasets.toml and README document every slice and its review status.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 3.3 and 4.1.
- vfc curl and openssl-cve-2021-3449 excerpts re-harvested through the
shared library: whole functions with upstream_start headers, the
vendored key redacted code-safely, no truncated bodies.
- vfc/recipes.toml carries mode, fix_path and function anchors;
training/manifest.jsonl regenerated.
- Juliet CWE-121 strcpy pair gets its buffer declarations back so the
vulnerable side is a real overflow again.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Tasks 3.1, 3.2 (harvest library) and the Scorer-owned naming helper.
- semantic/functions.py: one source of function names for scorer, overlay
and catalog generator (declarator wins over parser name; JS/TS bare-param
arrows and undeclared methods are <anon>; C++ Class::method resolves to
its unqualified tail; Perl sub; statements never name).
- benchmarks/pairs/harvest.py: shared library with name / line_range /
hunk / enclosing modes, comment-blind masking (triple-quote aware),
upstream_start headers, code-safe quoted-only redaction.
- benchmarks/pairs/catalog_gen.py: derive function labels and sinks from
the declarator at the mapped sink location, CWE-family sink tokens,
--prune with refusal when nothing derived, fails loud without the
package; vfc/harvest.py and vfc/generate_catalog.py become shims.
- tests/test_pair_harvest.py replaces test_vfc_harvest.py.
Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
Harvest 176 isolated vuln-vs-fix functions (109 CVEs, 352 training rows)
into benchmarks/pairs/vfc with provenance and a JSONL training manifest.
Score the slice as an honesty dashboard (ousast pairs --slice vfc);
keep the pair gate on the local fixtures only.
Regex inventory stays the proposer. MAP attaches promote/demote/unadjudicated/coverage
from intra-file taint (tree-sitter when a grammar parses, Python ast as last resort).
Deep sandboxes promotions only, ordered by the existing heuristic ranker plus optional
OpenRouter mechanism recall. The SAST pair slice scores overlay outcomes; the stage-1
detection gate and local pair gate stay inventory-only.
