mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
Marius CiepluchandClaude Opus 5.5
8213d3baad
Implemented with Codex; task 1.5 run by the coordinator against the real sources (S3 memory read-only). Co-Authored-By: Claude Opus 5.5 <[email protected]>
63 lines
4.0 KiB
Markdown
63 lines
4.0 KiB
Markdown
# Unseen evaluation, group 1
|
|
|
|
The coordinator runs task 1.5 from the repository root:
|
|
|
|
```sh
|
|
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
|
export PYTHONPATH=$PWD/src
|
|
python -m benchmarks.unseen.eligibility --s3-store s3:// --verify-tests --output benchmarks/measurements/2026-10-04-unseen-01-used-set/record.json
|
|
```
|
|
|
|
This is the networked command; implementation tests use fakes. It loads `.env` with
|
|
`openultrasast.config.load_dotenv` (exported values win), opens the local FileStore and
|
|
S3 MemoryStore through `openultrasast.plane.memory.open_store(..., read_only=True)`,
|
|
and uses `GH_TOKEN` for GitHub GET resolution. `s3://` selects `S3_BUCKET`; pass
|
|
`--s3-store s3://BUCKET/PREFIX` for a prefixed store and `--local-store file:///PATH`
|
|
for a different local store. Read-only S3 opening skips the normal write/Select probe;
|
|
collection uses LIST/GET only. Existing normal store opening is unchanged.
|
|
|
|
Pair catalogs/recipes include local pointer rows (`vendored = false`, including
|
|
`fix_repo`). Additional external manifests can be supplied with repeatable
|
|
`--pointer-manifest PATH`. Registered units in memory and experiment YAML are followed;
|
|
missing units with a registered digest fail. Plans without frozen units are not used data.
|
|
Experiment units identify repositories in `group`. Binary measurement artifacts are
|
|
read and counted; structured JSON, TOML and multi-document YAML are parsed strictly.
|
|
|
|
Each source records files/objects read, bytes read, repository-bearing rows (distinct
|
|
names per input), and distinct repository counts. The digest is SHA-256 of sorted,
|
|
normalized names joined with newlines. Source counts overlap. An empty source fails;
|
|
only after establishing it is empty may the coordinator pass `--known-empty LABEL`.
|
|
The output records that declaration. Missing/unreadable inputs still fail.
|
|
|
|
GitHub redirects add the requested and canonical names, plus parent/source fork
|
|
identities. Used names are also resolved so an old used name excludes a renamed
|
|
candidate. GitHub cannot enumerate all historical names: injected clients may supply
|
|
`former_names`; actual historical aliases come from used references and redirects.
|
|
A 404 for a historical/synthetic used identity retains its literal identity; other
|
|
API failures stop collection. Candidate resolution always fails closed on a 404.
|
|
`resolve(candidate, api)` and `rejection(resolution, used, selected_networks)` are
|
|
available to the later sourcing task. Names stay in memory; output contains only
|
|
counts, labels and SHA-256 digests. Output files are created exclusively, never overwritten.
|
|
|
|
`--verify-tests` runs the three group-1 test modules with both output streams discarded,
|
|
including the existing population guard. Only aggregate test counts and the exit code
|
|
enter the record; a failed test prevents the record. V3 paths are excluded before
|
|
eligibility reads; only the existing guard handles v3 inside its black-box test process.
|
|
|
|
Population reservations retain their existing sweep over `benchmarks/`, `src/`, and
|
|
`tests/`. Pool reservations also cover `plane/`. Each reservation exempts only its own
|
|
directory; population checks include all of `benchmarks/unseen/private/`. This preserves
|
|
the existing population behavior while adding pools (the hard rule for this increment).
|
|
|
|
All `pool-pN.toml` files require a matching `freeze-pN.json`; drafts use `draft-pN.toml`.
|
|
The freeze digest is SHA-256 of parsed public TOML encoded as UTF-8 JSON with sorted
|
|
keys, compact separators and `ensure_ascii=False`. Private entries must be represented
|
|
by opaque digests in that public manifest, as task 3.2 requires.
|
|
|
|
The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `decision`
|
|
(an opaque decision or experiment label), `purpose`, ISO `date`, `result_digest`, and
|
|
`freeze_digest`. Appends lock and validate the existing chain, reject duplicates or a
|
|
changed freeze, and reject qualification of an informed slice for the same decision.
|
|
Keep the published final digest to authenticate history: the chain detects row edits,
|
|
but cannot independently prove that an entire history was not replaced or truncated.
|