mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
feat(pre-push-safety-net): enforce evidence eligibility and record blocked qualification
This commit is contained in:
@@ -41,3 +41,26 @@ and its compressed full evidence bundle. All 21 deadline-bound transactions time
|
||||
Every measured artifact matches the frozen installed semantics. No source, engine,
|
||||
query, fact or ranking change was made during this run. The follow-up remains necessary
|
||||
before rollout; the evaluation task does not authorize weakening its gates.
|
||||
|
||||
## VAmPI qualification follow-up, 2026-09-13
|
||||
|
||||
Task 8.3's packaged frozen regression (139,164 total snapshot bytes read per replay)
|
||||
retains raw findings at `models/user_model.py:73:get_user`,
|
||||
`api_views/users.py:187:update_password`, and `api_views/books.py:51:get_by_title`.
|
||||
The SQLI target has no selected exact function question; the two authorization target
|
||||
questions are selected at positions 93 and 63 but have `change_context_incomplete`.
|
||||
Head scope retains 138 selected and 20 deferred questions. Raw findings do not prove
|
||||
completed target questions or a transitive source-to-target witness.
|
||||
|
||||
The first lab run took 112.415s; same-snapshot cached replays took 6.816s and 6.684s,
|
||||
all incomplete. This is diagnostic reuse, not changed-code performance qualification.
|
||||
The 600-second budget is distinct from the hook acceptance profile. Full artifacts are
|
||||
in `2026-09-13-push-qualification.json.gz` under the regression population.
|
||||
|
||||
Reconcile evidence normalization and the unsupported context projection at the shared
|
||||
scan boundary before downstream admission: retain the actual query-origin/path witness
|
||||
when available, resolve supported first-party context, and keep unresolved dynamic/vendor
|
||||
boundaries explicit. Do not fabricate transitivity from a raw finding at the target.
|
||||
Revalidate ordinary VAmPI scan, exact question accounting, push regressions and independent
|
||||
qualification after an approved owner fix. This note records the prerequisite; it does
|
||||
not silently approve an upstream implementation or change detector semantics.
|
||||
|
||||
@@ -106,3 +106,98 @@ retune the ranker or weaken the population/gates to compensate.
|
||||
- CLAIM: Task 8.2 completed its prescribed measurement and owner routing; rollout remains NO-GO.
|
||||
- EVIDENCE: Versioned scorecard and 116-file verified bundle; seven scorer controls and full regression suite pass.
|
||||
- GAPS: No qualified warm graph path or independent capability eligibility established. These failed gates cannot authorize rollout.
|
||||
|
||||
## Task 8.3 brief
|
||||
|
||||
Requirements 3.1, 3.2, 6.1–6.5 and 8.1–8.4; evaluation and actionability integration.
|
||||
Implement a trusted installed eligibility resource whose evidence is recomputed against
|
||||
exact core/engine/runtime/facts/query/policy/semantic identities. Require reviewed
|
||||
operation/advice scope, vulnerable/fixed/benign populations, nonzero actual positive
|
||||
controls and reviewed alerts, untouched per-capability evaluation and all joint gates.
|
||||
The normal runner consumes only eligible declarations; stale or inconsistent resources
|
||||
admit nothing. This is integrity checking of trusted release evidence, not signing or
|
||||
permission to load repository-authored eligibility.
|
||||
|
||||
Freeze the current packaged candidate before the six PMPro/Node development replays.
|
||||
Run readable VAmPI regression snapshots under a separate 600-second laboratory profile;
|
||||
keep their missing independent pairs unresolved. Execute libpng as an unsupported
|
||||
30-second envelope control. Do not inspect or tune on Ghost before its reviewed
|
||||
population exists. Missing required evidence leaves task 8.3 qualification blocked,
|
||||
even when the reproducible NO-GO artifact and admission implementation are delivered.
|
||||
|
||||
### Review scope
|
||||
|
||||
Synthetic controls prove that a matching reviewed population can reach the real runner
|
||||
admission seam, while stale facts/runtime, missing positive observations, absent reviews,
|
||||
wrong untouched workloads and changed consequence/repair semantics cannot. They are
|
||||
contract controls, not real capability qualification. The installed NO-GO resource must
|
||||
recompute identically and enable zero capabilities. The source has no repository- or
|
||||
language-specific rank changes, frontend adaptations or new abstract domains.
|
||||
|
||||
### Task 8.3 measured outcome
|
||||
|
||||
Packaged candidate `9464e229ccfdc873a02e3a0db7c1e83a6d6ec6d3e8234ae7b97677b4aed836fd`
|
||||
executed offline as UID 1000 with a 3 GiB memory limit. Each profile was written before
|
||||
its runs. All six development pushes saved artifacts, timed out in 30.559–30.678s and
|
||||
retained all 11 declared cases. Source snapshot receipts read 57.9–58.1 MB per PMPro
|
||||
comparison and 5.95 MB per Node comparison. These are snapshot bytes, not graph census.
|
||||
The supported introducing-change recall is 0/2, Wilson 95% interval [0, 0.658]; actual
|
||||
reviewed alert precision is undefined (0/0), never 100%. Neither fixed case has a
|
||||
completed witnessed negative. The two benign pushes emitted no alerts; that silence
|
||||
with incomplete coverage does not establish a useful safety net.
|
||||
|
||||
VAmPI's 600-second laboratory profile executed all three pinned regression targets,
|
||||
reading 139,164 snapshot bytes per replay. First run 112.415s; subsequent identical
|
||||
comparisons 6.816s and 6.684s, each with two graph hits and 403 query hits. Raw findings
|
||||
retain the known SQLI and two authorization sites, but the SQLI exact function question
|
||||
is unlocated and authorization questions are `change_context_incomplete`. All three
|
||||
remain unresolved; no transitive witness or completed regression detection is credited.
|
||||
There are 138 selected and 20 deferred head questions. Context/witness normalization
|
||||
is routed to the shared scan owner. A registry-generation provenance probe overlapped
|
||||
startup of the first lab run; these timings are diagnostic, not runtime-gate samples.
|
||||
|
||||
The initial instrument labeled all regression replays cold. The two proved identical
|
||||
cache replays were relabeled `identical_tip` and rescored without changing measurements,
|
||||
profile or population; original run/scorecard/instrument and correction are retained.
|
||||
Future instrument output distinguishes these identical comparisons. None is a warm edit.
|
||||
|
||||
The libpng unsupported envelope saved a partial artifact after 30.590s. Ghost's checkout
|
||||
and reviewed independent cases remain missing and were not scanned. Independent PHP
|
||||
and Python admission populations also remain missing. No ranker, fact, adapter or query
|
||||
retune occurred. The 59-file compressed evidence bundle is verified against originals:
|
||||
`benchmarks/measurements/2026-09-13-push-qualification.json.gz`, SHA-256
|
||||
`79226b4e54042559ba5cc2b94b69b1dcddb550872fd0783b2380368be72f90bd`.
|
||||
|
||||
The installed eligibility resource is reproducible NO-GO, digest
|
||||
`2ddd94d4dc3fd6b84e7de38e5f211d2653fb993c64e4e43cedf7e06062c3265c`, with zero enabled
|
||||
capabilities. Task 8.2 runtime evidence remains failed and predates the eligibility
|
||||
policy addition; it is correctly rejected as admission evidence, not relabeled current.
|
||||
|
||||
## Status Report
|
||||
- STATUS: BLOCKED
|
||||
- TASK: 8.3
|
||||
- RED_PHASE_OUTPUT: `/tmp/group8-3-red.log`: eight failures with eligibility disabled; ON passed, switch removed; final synthetic and installed-resource contracts verified separately.
|
||||
- IMPLEMENTED: Versioned eligibility generation/loading, production admission integration, real offline replay and NO-GO artifact.
|
||||
- BLOCKER: Required independent Node/PHP/Python reviewed populations and verified VAmPI transitive question witness are absent. Runtime/completion gates fail.
|
||||
|
||||
## Review Verdict
|
||||
- VERDICT: REJECTED
|
||||
- TASK: 8.3 completion claim
|
||||
- MECHANICAL_RESULTS: Eligibility and runner implementation inspected; fixed semantic/advice scope, observed controls and exact provenance required. No language-specific ranker changes, placeholders or secrets. RED phase verified.
|
||||
- FINDINGS: Implementation and reproducible NO-GO decision are concrete, but task 8.3's mandatory independent transfer and transitive-regression evidence is not established. Main-context fallback does not provide independent review.
|
||||
- REMEDIATION: Repair the shared context/witness and census/runtime prerequisites; obtain reviewed untouched paired populations, refreeze and rerun all joint gates. Keep capabilities disabled meanwhile.
|
||||
- SUMMARY: Retain the useful implementation, leave task 8.3 unchecked and feature rollout NO-GO.
|
||||
|
||||
## Verification Result
|
||||
- STATUS: NOT_VERIFIED
|
||||
- CLAIM_TYPE: TASK
|
||||
- CLAIM: Full task 8.3 qualification complete.
|
||||
- EVIDENCE: Real frozen replay and installed decision explicitly preserve failed gates and missing evidence.
|
||||
- GAPS: Independent populations, completed positive/fixed controls, VAmPI transitive witness and representative runtime/coverage.
|
||||
- NOTES: Proceed with authorized task 8.4's explicit NO-GO acceptance reconciliation; it does not depend on pretending qualification passed.
|
||||
|
||||
Task 8.3 implementation verification: full suite **1317 passed, 9 skipped in 115.80s**
|
||||
(`/tmp/group8-3-complete-suite.log`, exit 0). After the measurement-label correction,
|
||||
14 focused eligibility/replay-instrument tests passed in 0.95s. Ruff, format, mypy
|
||||
(116 source files) and diff checks pass. These verify the delivered code and recorded
|
||||
NO-GO behavior, not the rejected full qualification claim above.
|
||||
|
||||
@@ -132,7 +132,10 @@
|
||||
"implementation-group-8.md"
|
||||
],
|
||||
"next_group": 8,
|
||||
"blocked_tasks": [],
|
||||
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan."
|
||||
"blocked_tasks": [
|
||||
"8.3"
|
||||
],
|
||||
"feasibility": "NO-GO: all 21 frozen NodeGoat transactions timed out; 0/24 declared target checks completed. Cancellation/reporting remained below 2 seconds. Priming did not qualify graph reuse; named census and engine lifecycle work routed to contributor-scan.",
|
||||
"qualification": "NO-GO: eligibility implementation and evidence delivered; required independent populations and verified VAmPI transitive witness remain missing. Zero capabilities enabled."
|
||||
}
|
||||
}
|
||||
|
||||
@@ -234,6 +234,7 @@ independent capability qualification. Groups 7–8 remain pending; rollout remai
|
||||
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
|
||||
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
|
||||
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||
- _Blocked 2026-09-13:_ Eligibility implementation and reproducible NO-GO artifact delivered; independent reviewed Node/PHP/Python populations and verified VAmPI transitive witness remain missing. Runtime/coverage gates fail. See `implementation-group-8.md` and contributor-scan owner follow-up.
|
||||
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
|
||||
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
|
||||
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
|
||||
|
||||
Reference in New Issue
Block a user