mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 20:19:29 +00:00
unseen-eval group 2 (2.1-2.4): advisory sourcing, vulnerability-introducing ranges, ordinary-commit draw, draft manifest and slices; journal with resume
Implemented with Codex. Sourcing run (2.5) next, by the coordinator. Co-Authored-By: Claude Opus 5.5 <[email protected]>
This commit is contained in:
Marius Ciepluch
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
8213d3baad
commit
48ae918dec
@@ -60,3 +60,140 @@ The ledger exposes `append(path, row)` and `read(path)`. Rows contain `slice`, `
|
||||
changed freeze, and reject qualification of an informed slice for the same decision.
|
||||
Keep the published final digest to authenticate history: the chain detects row edits,
|
||||
but cannot independently prove that an entire history was not replaced or truncated.
|
||||
|
||||
## Sourcing and extraction (tasks 2.1–2.4)
|
||||
|
||||
The coordinator's networked task **2.5** is now:
|
||||
|
||||
```sh
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH=$PWD/src
|
||||
python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601 \
|
||||
--cache /home/mc/ousast-results/plane/unseen-draft --max-hours 8
|
||||
```
|
||||
|
||||
Export `GH_TOKEN` first, or supply it through the root `.env` (exported values win).
|
||||
The same read-only local/S3 stores, `--pointer-manifest`, and explicit `--known-empty`
|
||||
options as task 1.5 apply. Missing or unreadable sources fail. The used set is rebuilt
|
||||
and canonicalized before candidate selection, including redirects and fork networks.
|
||||
No population v3 file is opened and no guard/bisection is run by this command; that is
|
||||
reserved for group 3. An existing draft is never overwritten.
|
||||
|
||||
The command writes `--cache/journal/draft-p1.toml` and
|
||||
`--cache/journal/private-draft-p1.toml` only after all sourcing stages complete.
|
||||
Both stay outside the repository, with mode 0600. The public-format file contains permissive entries
|
||||
and, for every private entry, only an opaque ID, slice, private marker, and SHA-256
|
||||
of canonical JSON containing its URL and ordered `[base, head]` pairs. Private entries
|
||||
carry the complete records, including licences, in the private-format file.
|
||||
Licensing is re-read through GitHub's licence endpoint **at the actual extracted
|
||||
vulnerability replay head**; the candidate's earlier fix-parent classification is
|
||||
provisional. An explicit SPDX declaration in the pinned file takes precedence over
|
||||
the API's classification. Missing/unknown/compound licences remain private.
|
||||
|
||||
Three seeded slices each contain 100 repositories, one vulnerability range and twelve
|
||||
ordinary changes per repository. Each ecosystem/post-cutoff stratum differs from its
|
||||
proportional per-slice target by less than one repository. Post-cutoff candidates are
|
||||
attempted first within each ecosystem. Below 300 eligible repositories the command
|
||||
exits 1 with counts and does not write a partial draft or relax any selection bar.
|
||||
The group-2 measured exit and label confirmation remain coordinator tasks 2.5/2.6;
|
||||
passing implementation tests does not assert that 300 eligible repositories exist.
|
||||
|
||||
Clones and checkpoints default to `results_root()/unseen-draft` (normally
|
||||
`~/ousast-results/plane/unseen-draft`), overridable by `--cache /outside/path`.
|
||||
The cache cannot lie inside `--root`. One opaque, disposable clone exists at a time;
|
||||
it is removed on success, rejection, or failure. `--filter=blob:none --no-checkout`
|
||||
keeps the commit graph needed for temporal thirds, blame, and release-tag ancestry,
|
||||
and fetches source blobs on demand. This is a **partial clone**, not a shallow history.
|
||||
The 500 MiB disk cap is checked after Git commands, including lazy blob fetches;
|
||||
one in-flight command can transiently exceed it before rejection and cleanup.
|
||||
Git output is captured, never forwarded. Each repository must read a nonempty source
|
||||
blob before extraction. Summary instrumentation includes source read operations and
|
||||
actual bytes, clone count, cumulative/peak disk bytes, wall time, and logical API calls
|
||||
and HTTP attempts. Zero source bytes or a failed Git read is an instrument failure,
|
||||
not an eligibility drop. Stdout is a JSON summary of counts/digests only; no names or
|
||||
URLs enter stdout or a measurement record. Raw per-change data stays in the external journal and manifests.
|
||||
|
||||
Extraction uses Git and syntax boundaries, with no scanner, engine, or model. Python
|
||||
uses stdlib AST; Java/PHP/JS/TS use conservative named-function brace matching. Ambiguous
|
||||
or unparseable functions can reject a candidate. The first removed code line per
|
||||
fix-touched function is the SZZ sink proxy; pure-addition fixes use its declaration.
|
||||
Normalized text absence, non-root/non-merge ancestry, file/line limits, and the OSV
|
||||
release bracket are checked before accepting `introducing`. Matching tags are `VERSION`
|
||||
or `vVERSION`; an absent/ambiguous tag or missing prior release cannot certify the
|
||||
bracket. After at most five walk-backs, `last_touch` uses `git log -L` with the resolved
|
||||
numeric function extent (also works where Git lacks a language-specific function driver).
|
||||
Method, bias, fallback reasons, head function/range, files/hunks, family, advisory, and
|
||||
cutoff flag are recorded. Task 2.6 must review every third `introducing` label.
|
||||
|
||||
All reviewed advisories are enumerated to exclude **all** known repository fixes from
|
||||
ordinary changes, including old advisories and advisories outside the four candidate
|
||||
ecosystems. Fix URLs are canonicalized before indexing so renames cannot hide them.
|
||||
The ordinary filter uses only the design's narrowed security pattern. It counts
|
||||
rejections (including oversized commits and repositories failing the 40-commit floor),
|
||||
checks both sides for touches to known vulnerable functions, and inspects generated
|
||||
headers and inherited `.gitattributes`. Sampling allocates twelve draws proportionally
|
||||
across three history-time thirds and four source-size buckets, with one per third.
|
||||
|
||||
For offline tests, inject GitHub/OSV clients, the clone context, and `UsedSet` into
|
||||
`draft.run`; HTTP transports and Git subprocess runners also accept fakes. Tests use
|
||||
sanitized REST-shaped data and a temporary synthetic local Git history. No implementation
|
||||
test contacts GitHub, OSV, a model, or a cluster. HTTP retries are bounded to five attempts;
|
||||
rate-limit reset/Retry-After waits and transient 5xx backoff are tested with fake clocks
|
||||
and sleepers. A stale candidate 404 is counted; other service failures fail the run.
|
||||
|
||||
### Checkpoint and resume
|
||||
|
||||
The command above uses journal directory
|
||||
`/home/mc/ousast-results/plane/unseen-draft/journal`. Run the same command to resume.
|
||||
`--max-hours` is a per-invocation budget; changing it does not invalidate checkpoints.
|
||||
Create `journal/STOP` to stop between units, then remove it before resuming. A page,
|
||||
repository resolution, candidate decision, or repository extraction already running
|
||||
finishes first, so the time budget is not a hard timeout on a large repository.
|
||||
Clean stops exit 0 with `status: stopped`; failures and shortfalls exit 1. Completion
|
||||
exits 0 with `status: draft`. No partial manifest is published.
|
||||
|
||||
`inputs.json` binds the seed, source-labelled used-set digest, known-empty declarations,
|
||||
clone cap, and design fingerprints (sourcing/extraction/eligibility/publication code and
|
||||
family taxonomy). A mismatch refuses resume and prints only the changed parameter
|
||||
labels. Use a new external cache for a deliberately changed design or population.
|
||||
The used set is rebuilt each invocation; repository resolutions, including 404s, are
|
||||
reused from the journal. Live GitHub metadata for unfinished units can still change;
|
||||
completed pages and decisions are the durable sourcing snapshot.
|
||||
|
||||
Each completed unit is appended to its stage JSONL and fsynced before work advances:
|
||||
`advisories.jsonl` (pages), `resolution.jsonl` (alias/fork metadata),
|
||||
`eligibility.jsonl` (candidate decisions), `extraction.jsonl` (repository/advisory
|
||||
results, vulnerable range, ordinary draws, pinned licence, method and counts), and
|
||||
`completion.jsonl` (final manifest digest). A lock permits only one writer.
|
||||
`progress.json` is atomically rewritten after every unit with stage counts, last unit,
|
||||
last observed GitHub rate-limit headers, and cumulative active elapsed seconds.
|
||||
These files may contain identities and must stay outside version control. Stdout and
|
||||
repository records contain only counts, digests and fixed labels.
|
||||
|
||||
Replaying completed units makes no GitHub/OSV or clone calls. Final manifests are
|
||||
reconstructed deterministically from journal results; an interrupted publication can
|
||||
be retried, but a different existing manifest is refused. Completed JSONL rows are
|
||||
never rewritten. Malformed or torn records fail closed instead of being interpreted
|
||||
as completed work. Tests interrupt each stage with fakes, then compare resumed and
|
||||
uninterrupted manifests byte for byte.
|
||||
|
||||
### Planning estimate for task 2.5 (not measured)
|
||||
|
||||
Let A be all reviewed advisories, R distinct fix-linked repository identities, U used
|
||||
identities not already cached, C candidate advisories passing early filters, and K
|
||||
repositories surviving extraction. GitHub requests are approximately
|
||||
`ceil(A/100) + R + U + 2*C + K` (enumeration, canonical repositories, fix metadata,
|
||||
provisional licence, final licence), plus retries. OSV requests are at most C. Git
|
||||
fetches and lazy blob transfers are additional network operations, not GitHub API calls.
|
||||
|
||||
For a planning scenario of A=40,000, R=15,000, U≈1,000, C=600–1,500, K=300:
|
||||
about **18,000–20,000 GitHub calls** and **600–1,500 OSV calls**. The all-advisory index
|
||||
makes this substantially more than 300 repository lookups. At an assumed 5,000-request
|
||||
hourly allowance, allow several rate-limit windows. Actual token limits can differ.
|
||||
For 300–1,000 attempted clones averaging 20–100 MB after source hydration, plan roughly
|
||||
**6–100 GB transferred**, with one clone occupying about 500 MiB at most between commands
|
||||
plus the retained journal and manifests after completion. Transfer volume is an estimate; disk bytes
|
||||
are only a proxy. A working planning range is **8–48 hours**, potentially longer on
|
||||
large histories: every first-parent commit is inspected and lazy blobs can dominate.
|
||||
These are assumptions, not evidence from a network run. The command reports actual
|
||||
counts and wall time; task 2.5 establishes the measured estimate.
|
||||
|
||||
Reference in New Issue
Block a user