mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
spec(pre-push-safety-net): record M1–M5 release milestones and split M1 into M1a/M1b
Add release-milestones.md as the executable outcome contract: prove one useful Node detection, make it fast, verify PHP/Python transfer, qualify on untouched cases, then enable a narrow advisory rollout. Link it from the roadmap, safety-net steering, requirements, tasks, status and spec.json. Split M1 after review. The raw NodeGoat <lambda>2 witness already exists in the 2026-09-14 post-census bundle; four admission vetoes discard it. M1a runs the vulnerable/fixed/unchanged revisions under a task-local experimental flag that records each veto instead of applying it, with a one-week box and a kill criterion if the fixed twin is not quiet. M1b repairs only the vetoes that fired and reruns through production rules. Existing approvals and thresholds unchanged. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
1ad472c68d
commit
8b66d5cb75
@@ -0,0 +1,200 @@
|
||||
# Release milestones: prove usefulness, then speed and transfer
|
||||
|
||||
Accepted direction: 2026-09-14. The maintainer agreed to the five-step path and asked
|
||||
for it to be recorded in the roadmap as a spec. This amendment defines the order and
|
||||
observable exit criteria for the remaining release work within pre-push-safety-net.
|
||||
It preserves the approved requirements, completed tasks and existing owner boundaries.
|
||||
It does not approve an unreviewed implementation design or enable any capability.
|
||||
|
||||
## Current position
|
||||
|
||||
**Next milestone: M1a. None of M1–M5 is verified yet.** The 26/27 completed executable
|
||||
tasks measure implementation progress, not release readiness. Task 8.3 remains blocked.
|
||||
The published `v1.2.0-alpha.1` is an experimental implementation release, not a qualified
|
||||
advisory rollout. See [current status](status.md) for measured evidence and limitations.
|
||||
|
||||
The immediate goal is one real changed-code defect, a defensible explanation, and a
|
||||
fixed twin that stays quiet. M1a first shows the raw detection with every admission veto
|
||||
recorded rather than applied; M1b repairs only the vetoes that fired. A longer, explicitly
|
||||
named development budget is acceptable for M1; it cannot satisfy the hook's latency gate.
|
||||
Runtime optimization is the next milestone after this useful result is demonstrated.
|
||||
|
||||
## M1 — One real security change works end to end
|
||||
|
||||
Status: **NEXT — not verified**. Dependencies: existing immutable replay and scan machinery.
|
||||
M1 is split into M1a (diagnose with vetoes recorded, not applied) and M1b (repair only
|
||||
the vetoes M1a showed to matter). M1 is verified only when M1b passes.
|
||||
|
||||
Why the split: the detector already reports the NodeGoat contribution injection. The
|
||||
2026-09-14 post-census bundle carries the raw `<lambda>2` witness at contributions.js
|
||||
lines 32–34, and the same bundle carries thousands of `context_incomplete`,
|
||||
`graph_incomplete`, `vendor_semantics_unresolved` and `ambiguous_line_correspondence`
|
||||
reasons that discard it before any developer sees it. Four independent admission vetoes
|
||||
are known: no context producer for dominance/config families, vendor exclusion acting as
|
||||
a global completion veto, line ambiguity inherited by every question, and query identity
|
||||
differing from witness identity. Requiring all four repaired before the first visible
|
||||
result is a full production pass, not one detection.
|
||||
|
||||
### M1a — Show the detection with every veto recorded
|
||||
|
||||
Status: **NEXT — not verified**. Time box: one week of maintainer effort from the first
|
||||
M1a run. Missing the box is a finding, not a reason to extend silently.
|
||||
|
||||
- Use the pinned NodeGoat contribution injection case, its fixed twin and an unchanged
|
||||
vulnerable revision as development data. Record exact base/head revisions, expected
|
||||
operation and outcome before running. Exercise the shipped replay path, existing ranker
|
||||
and physically vendor-free graph; do not substitute a hand-selected slice or hard-coded
|
||||
target function.
|
||||
- Run all three revisions under an explicit experimental evaluation flag that **records**
|
||||
each admission and comparison veto per finding instead of applying it. The flag is
|
||||
task-local, off by default, cannot enable any hook capability, and every output it
|
||||
produces is labeled experimental. A longer, explicitly named development budget applies;
|
||||
it cannot satisfy the hook's latency gate.
|
||||
- Deliverable: the raw finding on the vulnerable revision with location and witness, the
|
||||
fixed twin genuinely analyzed and quiet, the unchanged revision not reported as a new
|
||||
regression, and for every finding the exact list of vetoes that fired and why. Silence
|
||||
from failure is not a pass; a fixed twin that timed out or was not analyzed fails M1a.
|
||||
|
||||
**Exit evidence:** a replayable three-revision record under the experimental flag with
|
||||
exact revision, question and operation provenance, the per-finding veto list, and readable
|
||||
source and census receipts. **Kill criterion:** if the fixed twin is not quiet under the
|
||||
recorded-veto run, M1 has a detector problem, and M1b does not start until that is
|
||||
diagnosed and recorded.
|
||||
|
||||
### M1b — Repair only the vetoes M1a showed to matter
|
||||
|
||||
Status: **PENDING M1a**.
|
||||
|
||||
- From the M1a veto list, repair only the context/family/identity contracts this case
|
||||
actually needs. The owner diagnosis expects query-to-operation provenance and question-
|
||||
owned vendor relevance to be the two that fire here; M1a confirms or corrects that.
|
||||
Vetoes that did not fire on this case stay as they are and their families stay
|
||||
experimental. Resolve vendor and correspondence boundaries through evidence of dependency
|
||||
and ownership; unknown relevance remains unresolved.
|
||||
- Rerun the three revisions **without** the recorded-veto flag through the production
|
||||
evidence and comparison rules. Demonstrate a developer-readable evaluation finding with a
|
||||
supported security consequence, exact location, witness, change attribution and concrete
|
||||
repair direction. Verify the fixed twin is analyzed and produces no actionable defect, and
|
||||
the unchanged vulnerable revision is not reported as a new regression.
|
||||
- Keep qualification eligibility disabled. The demonstration uses explicit experimental
|
||||
evaluation output; it cannot advertise an unqualified normal hook alert.
|
||||
|
||||
**Exit evidence:** a replayable vulnerable/fixed/unchanged comparison record through the
|
||||
production rules, exact question and operation provenance, completed required checks,
|
||||
reviewed explanation, readable source and census receipts, task-local regression
|
||||
verification, and a list of which vetoes were repaired and which remain. Raw findings, rank
|
||||
positions, synthetic-only controls and screenshots alone do not satisfy M1b.
|
||||
|
||||
Owners: contributor-scan for query/operation/context evidence; pre-push-safety-net for
|
||||
comparison, admission, the recorded-veto flag and reporting. Requirements: 1.1–1.4,
|
||||
2.1–2.4, 3.1–3.5, 6.3, 7.1–7.3.
|
||||
|
||||
## M2 — The same useful analysis fits the push budget
|
||||
|
||||
Status: **PENDING M1**.
|
||||
|
||||
- Use the successful M1 case as a correctness baseline for a transaction-owned Joern
|
||||
session within the existing backend. The measured census probe supports investigating
|
||||
this design; it does not establish full-query equivalence or production readiness.
|
||||
- Verify equivalent taint, dominance and configuration answers under matching semantics,
|
||||
fresh request receipts, isolated graph identities, cache invalidation, crash handling
|
||||
and process-group cancellation. Keep one deadline across the complete transaction.
|
||||
- Rerun the frozen seven-class runtime profile with changed-code cases, cold starts,
|
||||
dependency/configuration edits, growth and multi-ref transactions. Retain all timeouts
|
||||
and unresolved checks. Identical-tip reuse cannot substitute for warm changed-code work.
|
||||
|
||||
**Exit evidence:** the M1 outcome remains correct; representative warm changed-code p95
|
||||
is at most 30 seconds, cancellation/reporting at most two seconds, and supported-check
|
||||
completion at least 95%, under the predeclared profile. A fast census alone does not pass.
|
||||
|
||||
Owner: contributor-scan runtime, with pre-push-safety-net deadline/cache integration.
|
||||
Requirements: 4.1–4.4, 6.2–6.5, 7.3.
|
||||
|
||||
## M3 — The shared design transfers across languages
|
||||
|
||||
Status: **PENDING M2**.
|
||||
|
||||
- Repeat the useful vulnerable/fixed comparisons with PMPro/PHP and VAmPI/Python,
|
||||
including the previously unresolved VAmPI SQLI query/path evidence. Preserve explicit
|
||||
limits for authorization and other families whose evidence is still unsupported.
|
||||
- Keep framework knowledge in frontends/adapters and versioned facts. Equivalent normalized
|
||||
evidence must retain the same core ranking, scope and admission behavior.
|
||||
- Record core and adapter changes separately. Any PMPro-driven core change must be frozen
|
||||
and replayed on Node/Express before claiming transfer; the initial Node-first repair
|
||||
does not remove that existing requirement.
|
||||
|
||||
**Exit evidence:** reviewed development/regression results for Node, PHP and Python,
|
||||
including positive, fixed and unchanged controls, exact provenance, coverage and runtime.
|
||||
Do not label these repeatedly inspected projects independent qualification. C/C++ bounds
|
||||
remain a separate unsupported abstraction; adding a new engine is not a prerequisite here.
|
||||
|
||||
Owners: contributor-scan shared evidence and framework adapters; pre-push-safety-net evaluation.
|
||||
Requirements: 6.2–6.4, 8.1–8.4.
|
||||
|
||||
## M4 — Independent examples establish useful detection and low noise
|
||||
|
||||
Status: **PENDING M3**. Population preparation may proceed earlier, but evaluation follows
|
||||
the frozen candidate. Reserve untouched cases before examining their scan results.
|
||||
|
||||
- Freeze reviewed vulnerable, fixed and benign changes from projects not used to develop
|
||||
or tune the particular language/framework/family. Ghost remains reserved until its
|
||||
actual reviewed population exists. Node evidence cannot qualify PHP or Python.
|
||||
- Measure actual actionable alerts, supported recall, fixed-side silence, benign interruptions,
|
||||
completed coverage and runtime together, with counts and uncertainty. Keep unlocated
|
||||
targets, unanswered queries and unsupported cases visible in their declared populations.
|
||||
- Apply the existing acceptance profile: at least 95% observed actionable precision,
|
||||
at least 90% supported recall, at least 95% supported-check completion, the 30s/2s runtime
|
||||
gates, nonzero reviewed emitted alerts and positive controls, and zero known fixed/benign
|
||||
false alerts. Silence on everything fails. A core retune requires new untouched evaluation.
|
||||
|
||||
**Exit evidence:** reproducible, reviewed per-capability qualification with every mandatory
|
||||
gate passing together. Report failures honestly and leave those capabilities experimental;
|
||||
do not require every possible language or family to become eligible for a narrow release.
|
||||
|
||||
Owner: pre-push-safety-net task 8.3, consuming verified contributor-scan semantics.
|
||||
Requirements: 6.1–6.5, 8.1–8.4 and the accepted experimental acceptance profile.
|
||||
|
||||
## M5 — Enable a narrow advisory rollout
|
||||
|
||||
Status: **PENDING M4**.
|
||||
|
||||
- Generate eligibility bound to the exact passing runtime, query, fact, ranking and policy
|
||||
identities. Enable only independently qualified capabilities; stale or missing evidence
|
||||
enables none. Advisory retains the same evidence bar as blocking.
|
||||
- Revalidate the packaged vulnerable/fixed/benign push experience, compact reporting,
|
||||
truthful incomplete coverage, local no-model operation and preservation/removal of hooks.
|
||||
- Publish the supported scope, limitations and evidence with the advisory release.
|
||||
Expand capabilities only when they earn qualification. Blocking deployment is a later,
|
||||
explicit decision, even though opt-in blocking plumbing already exists.
|
||||
|
||||
**Exit evidence:** passing packaged acceptance validation and exact eligibility for the
|
||||
declared advisory scope, with the release decision recorded. Only then mark task 8.3
|
||||
and release readiness complete if their mandatory criteria are satisfied.
|
||||
|
||||
Owner: pre-push-safety-net admission, packaging and release validation.
|
||||
Requirements: 3.1–3.5, 5.1–5.5, 6.5, 7.1–7.4.
|
||||
|
||||
## Execution and reporting contract
|
||||
|
||||
Use M1a → M1b → M2 → M3 → M4 → M5 as the critical path. Preparation may overlap, but later
|
||||
milestones cannot substitute for missing earlier evidence. Necessary runtime repairs
|
||||
that make M1 executable are allowed; a standalone speed benchmark cannot close M1.
|
||||
Keep ranker-owned scope, vendor exclusion, language-agnostic core behavior and all
|
||||
acceptance thresholds unchanged. No new source IR, second detector, differential graph
|
||||
mutation, learned ranker or unrelated backlog is introduced by this amendment.
|
||||
|
||||
Before production edits, make the required owner design/task amendment concrete and
|
||||
apply the existing Kiro approval, review and verification workflow. Prior authorizations
|
||||
remain valid within their scope; this milestone record does not reset them.
|
||||
|
||||
Progress updates shall lead with the current milestone, observed developer outcome,
|
||||
remaining blocker and next concrete action. Report implementation task counts separately.
|
||||
Change a milestone to verified only with linked evidence; do not infer progress from
|
||||
elapsed effort or a completed experimental release tag.
|
||||
|
||||
Immediate next action (M1a): add the task-local recorded-veto experimental flag, run the
|
||||
NodeGoat vulnerable, fixed and unchanged revisions under it, and produce the per-finding
|
||||
veto list alongside the `<lambda>2` witness. Then trace that witness into the declared
|
||||
contribution handler and turn the vetoes that actually fired into the smallest owner fix
|
||||
(M1b). Use [remediation research](../contributor-scan/release-remediation-research.md) as
|
||||
the starting record; do not begin another broad timing campaign before the M1 outcome exists.
|
||||
@@ -2,6 +2,11 @@
|
||||
|
||||
Status: approved for task generation on 2026-09-12 following the maintainer's "ok, next kiro phase". The acceptance profile defines evaluation targets, not measured achievements.
|
||||
|
||||
Release sequencing amendment accepted 2026-09-14: [release milestones](release-milestones.md)
|
||||
defines M1 usefulness → M2 speed → M3 shared-language transfer → M4 independent qualification
|
||||
→ M5 narrow advisory rollout, with observable exit evidence. The requirements and numerical
|
||||
acceptance thresholds below remain unchanged.
|
||||
|
||||
## Introduction and Boundary
|
||||
|
||||
OpenUltraSAST helps developers catch concrete security regressions in the resulting commits they
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-14T11:13:56.805180+00:00",
|
||||
"updated_at": "2026-09-14T11:34:19.788221+00:00",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
@@ -158,5 +158,22 @@
|
||||
"channel": "alpha",
|
||||
"rollout_decision": "NO-GO",
|
||||
"status_record": "status.md"
|
||||
},
|
||||
"release_milestones": {
|
||||
"record": "release-milestones.md",
|
||||
"direction_approved": true,
|
||||
"approval_source": "I agree. please note this into the raodmap as a spec",
|
||||
"current": "M1a",
|
||||
"verified": [],
|
||||
"sequence": [
|
||||
"M1a: NodeGoat detection with admission vetoes recorded, not applied (one-week box)",
|
||||
"M1b: repair only the vetoes M1a showed to matter; rerun through production rules",
|
||||
"M2: equivalent bounded runtime",
|
||||
"M3: shared-language transfer",
|
||||
"M4: independent qualification",
|
||||
"M5: narrow advisory rollout"
|
||||
],
|
||||
"implementation_design_approval": "Existing approvals preserved; this records milestone order and exit criteria, not blanket approval of future design changes.",
|
||||
"m1_split_accepted": "2026-09-14: maintainer asked to split M1 into M1a/M1b after review"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,6 +5,16 @@ approved. **26/27 executable tasks complete; task 8.3 remains blocked.**
|
||||
Rollout: **NO-GO**. The alpha tag publishes experimental implementation and evidence;
|
||||
it does not approve normal hook capabilities or production enforcement.
|
||||
|
||||
## Next release milestone
|
||||
|
||||
**M1a: show the NodeGoat detection with every admission veto recorded rather than
|
||||
applied, fixed twin analyzed and quiet, one-week time box. Then M1b: repair only the
|
||||
vetoes that fired and rerun through production rules.** None of the five release
|
||||
milestones is verified yet.
|
||||
The 26/27 task count is implementation progress, not proximity to rollout. Follow the
|
||||
accepted [release milestone spec](release-milestones.md): usefulness → speed → shared
|
||||
PHP/Python transfer → independent qualification → narrow advisory rollout.
|
||||
|
||||
## Verified
|
||||
|
||||
Immutable pushed snapshots, shared multi-ref deadlines, ranker-owned scope, physical
|
||||
@@ -25,8 +35,8 @@ gates and a packaged native PHP/JavaScript partition smoke. See
|
||||
completed all 21 measurements: all 18 changed/cold/growth/multi-ref transactions timed
|
||||
out; the three identical-tip replays took 18.521–18.874s with valid graph/query reuse.
|
||||
Still 0/24 declared target checks completed. Maximum observed cancellation/report
|
||||
overrun was 0.874s, below the two-second allowance. Amortize engine startup/loading work and rerun the
|
||||
frozen seven-class profile; require warm p95 <=30s and >=95% completed supported checks.
|
||||
overrun was 0.874s, below the two-second allowance. After M1 establishes the useful
|
||||
comparison, amortize engine startup/loading and rerun the frozen seven-class profile; require warm p95 <=30s and >=95% completed supported checks.
|
||||
2. **Target and change-context evidence remains incomplete.** VAmPI preserves raw findings
|
||||
at all three known sites, but the exact SQLI function question/transitive witness and
|
||||
supported authorization context are unresolved. Raw findings or rank positions cannot
|
||||
|
||||
@@ -1,12 +1,14 @@
|
||||
# Implementation Plan: pre-push-safety-net
|
||||
|
||||
Status: task plan approved by the maintainer on 2026-09-12. Groups 1–6 (tasks 1.1–6.4)
|
||||
are implemented, reviewed and verified (20/27 subtasks). Tasks 4.3 and groups 5–6 used the
|
||||
explicit Kiro manual review fallback after fresh agent dispatch reached its thread limit.
|
||||
Group 5's six cold PMPro/Node timeouts remain historical NO-GO evidence. Group 6 verifies
|
||||
compatible reuse with equal evidence/admission in a packaged authored JavaScript control;
|
||||
its identical-revision run completed in 2.00s. This is not representative warm p95 or
|
||||
independent capability qualification. Groups 7–8 remain pending; rollout remains NO-GO.
|
||||
Status: the existing task plan remains approved. Groups 1–7 and tasks 8.1, 8.2 and 8.4
|
||||
are verified (26/27 executable subtasks); task 8.3 and rollout remain NO-GO. Later work
|
||||
used the documented manual review fallback where agent dispatch reached its limit.
|
||||
|
||||
The accepted [release milestone spec](release-milestones.md) now orders remaining work:
|
||||
M1 useful Node comparison → M2 speed → M3 shared-language transfer → M4 independent
|
||||
qualification → M5 narrow advisory rollout. All five milestones remain unverified.
|
||||
These are release outcomes, not additional checked implementation tasks. Prepare bounded
|
||||
owner design/task amendments as needed; preserve all completed task evidence below.
|
||||
|
||||
## Execution contract
|
||||
|
||||
|
||||
+58
-66
@@ -1,84 +1,76 @@
|
||||
# Roadmap
|
||||
|
||||
## Current direction and next review — 2026-09-12
|
||||
## Current release path — 2026-09-14
|
||||
|
||||
The product is an **actionable pre-push security safety net for AI-accelerated development**, across
|
||||
supported languages/frameworks. WordPress is one regression workload. Advisory must be useful and
|
||||
low noise; moving speculative SAST output into a nonblocking hook does not meet the goal. See
|
||||
`safety-net.md` for current steering. The earlier narrative below is retained as dated history and
|
||||
does not override this direction or the latest measurements.
|
||||
Build an actionable pre-push security safety net for AI-accelerated development.
|
||||
The maintainer accepted the following order: **prove one useful detection, make it
|
||||
fast, prove cross-language transfer, qualify independently, then enable narrow advisory
|
||||
rollout**. The executable outcome contract lives in the existing
|
||||
[pre-push-safety-net release milestone spec](../specs/pre-push-safety-net/release-milestones.md).
|
||||
|
||||
### Approach decision
|
||||
**We are at M1a. None of the five release milestones is verified yet.** The published
|
||||
`v1.2.0-alpha.1` and 26/27 completed implementation tasks describe delivered machinery,
|
||||
not a demonstrated useful safety net. Task 8.3 and rollout remain NO-GO.
|
||||
|
||||
Keep the existing ranker as the scope mechanism. Supply it with pushed-change and affected
|
||||
first-party context. Third-party vendor code remains outside both targets and graphs. Reuse
|
||||
compatible graph/results artifacts and apply a total deadline; treat unknown external semantics
|
||||
as a coverage boundary. Require change attribution, witness and repair direction before a finding
|
||||
earns normal hook visibility. A second heuristic scope selector and an assumed incremental Joern
|
||||
engine are out of scope.
|
||||
| Order | Milestone | Observable exit | Current state |
|
||||
|---|---|---|---|
|
||||
| M1a | Show the NodeGoat detection with every admission veto recorded, not applied | Raw finding with witness on the vulnerable revision; analyzed fixed twin and unchanged control quiet; per-finding veto list. One-week time box | NEXT |
|
||||
| M1b | Repair only the vetoes M1a showed to matter | Supported evaluation finding through production rules, exact change/base evidence and repair direction; fixed twin and unchanged control stay quiet | Pending M1a |
|
||||
| M2 | Make that same analysis fast | Equivalent evidence; representative warm changed-code p95 <=30s, cancellation/report <=2s, >=95% supported completion | Pending M1 |
|
||||
| M3 | Prove the shared design transfers | PMPro/PHP and VAmPI/Python vulnerable/fixed evidence; unchanged core rules; Node replay after any PMPro core change | Pending M2 |
|
||||
| M4 | Qualify on independent cases | Reviewed untouched vulnerable/fixed/benign populations pass joint quality, coverage and latency gates per capability | Pending M3 |
|
||||
| M5 | Enable a narrow advisory rollout | Exact passing eligibility and packaged acceptance; only qualified capabilities enabled | Pending M4 |
|
||||
|
||||
### Current evidence and limitations
|
||||
### Immediate next action
|
||||
|
||||
- Whole-PMPro analysis has completed with its known CVE; the earlier "does not finish" diagnosis
|
||||
is superseded. The recorded 72-minute scan is not a pre-push performance result.
|
||||
- Historical Joern 4.0.625 + layout position budgets are PMPro 82, WP Statistics 91, MW WP Form 54.
|
||||
`<50` remains unmet. VAmPI's old 16 omitted SQLI; the corrected instrument retains the unmatched
|
||||
target and reports no full-target budget or detection recall without the required evidence.
|
||||
- Equal-score tie-breaking alone cannot bring PMPro or WP Statistics under 50. The existing
|
||||
ranker is useful, but its measured ordering is not enabled by the ordinary CLI caller.
|
||||
- Docker/Compose/ops now default to 4.0.625; task 1.1 passed local PHP/JavaScript image smoke and independent review. No host deployment or hook latency claim follows from that runtime evidence.
|
||||
- `contributor-scan` has 25/41 checked tasks, including the approved and verified frontend-retention prerequisite 2.18.
|
||||
`flow-aware-ranking` is still an unapproved brief despite implementation history.
|
||||
Add a task-local recorded-veto experimental flag and run the pinned NodeGoat
|
||||
vulnerable, fixed and unchanged revisions under it (M1a). The raw `<lambda>2` witness
|
||||
already exists; the deliverable is that witness plus the exact list of admission vetoes
|
||||
that discard it. Then repair only those vetoes and rerun through production rules (M1b).
|
||||
Normal hook eligibility stays disabled during this experiment.
|
||||
Do not start another broad speed campaign before this outcome exists. See the
|
||||
[owner diagnosis](../specs/contributor-scan/release-remediation-research.md).
|
||||
|
||||
### Approach and boundaries
|
||||
|
||||
The existing ranker owns scope. Vendor code stays out of targets and graphs. Framework
|
||||
facts and frontends supply evidence to a language-agnostic core. Unknown dependencies
|
||||
remain explicit; removing global completeness vetoes requires question-owned evidence,
|
||||
not dropped safety checks. Advisory and blocking share the same actionability bar.
|
||||
|
||||
Graph persistence now works: identical-tip replays take 18.5–18.9s. All 18 other measured
|
||||
transactions still time out and 0/24 target checks complete. A local Joern session census
|
||||
probe is promising, but full-query equivalence and hook performance are unverified.
|
||||
These findings inform M2; they do not complete M1. No second IR/engine, new heuristic
|
||||
scope selector, differential graph mutation or unrelated backlog is part of this path.
|
||||
|
||||
## Existing Spec Updates
|
||||
|
||||
- [ ] `contributor-scan` — review the draft driver/backend integration amendment; reconcile packaging
|
||||
pins, family-specific sanitizer task 5.6, exact scope reporting and repository baseline honesty.
|
||||
Preserve previously approved work; the new amendment is not implicitly approved.
|
||||
- [ ] `flow-aware-ranking` — review the amended experiment contract: ranker owns scope, vendor
|
||||
boundary preserved, missing targets stay unresolved, actual detection and cost accompany rank,
|
||||
untouched evaluation and exploration precede learned-ranking claims.
|
||||
- [ ] `zero-with-a-reason` — reconcile the broader funnel proposal with the hook's consumed coverage
|
||||
contract; it is adjacent work, not a requirement to finish every diagnostics mechanism first.
|
||||
- [ ] pre-push-safety-net — Execute the accepted M1–M5 release milestone amendment.
|
||||
Owns immutable replay, comparison, admission, evaluation and reporting. Preserve the
|
||||
existing completed tasks; task 8.3 remains blocked until qualification passes.
|
||||
- [ ] contributor-scan — Supply the required query/operation/context fixes for M1,
|
||||
followed by equivalent bounded session execution for M2 and shared transfer evidence.
|
||||
Frontend retention 2.18 and named census repair 2.19 are already verified.
|
||||
- [ ] flow-aware-ranking — Preserve ranker-owned scope and normalized cross-language
|
||||
behavior. New ranking research is not a prerequisite for the first useful comparison.
|
||||
- [ ] zero-with-a-reason — Adjacent diagnostics work; finishing its whole backlog is
|
||||
not required before M1.
|
||||
|
||||
## Specs (dependency order)
|
||||
|
||||
- [ ] pre-push-safety-net — Immutable pushed tips, existing-ranker integration, compatible artifact
|
||||
reuse, total deadline, change attribution, actionability admission and compact hook behavior.
|
||||
Dependencies: contributor-scan. Requirements/design approved 2026-09-12; tasks generated
|
||||
(8 groups, 27 executable subtasks), independently reviewed and approved; groups 1–7 are implemented and verified (23/27 tasks), including the resolved frontend-retention prerequisite.
|
||||
- [ ] pre-push-safety-net — Existing approved implementation, dependent on contributor-scan.
|
||||
Groups 1–7 and tasks 8.1, 8.2, 8.4 are verified; 26/27 executable tasks complete.
|
||||
Remaining release work follows [M1–M5](../specs/pre-push-safety-net/release-milestones.md).
|
||||
This amendment creates no duplicate implementation spec and resets no prior approvals.
|
||||
|
||||
### Next concrete step
|
||||
### Progress reporting
|
||||
|
||||
Groups 1–5 now connect immutable replay to the existing ranker, detector, comparison,
|
||||
admission and compact artifacts. The first frozen PMPro/Node experiment is **NO-GO**:
|
||||
all six cold security/fixed/benign cases timed out without completed target checks.
|
||||
After repairing post-deadline semantic planning and a repeated availability probe,
|
||||
every final CLI run saved its artifact and returned in 30.55–30.64 seconds. PMPro spent
|
||||
its deadline in preparation/discovery after materializing both trees; Node reached the
|
||||
driver but did not complete checks. The exact 11-case population remains unresolved.
|
||||
No normal capability is enabled and these six cold samples establish no warm p95 claim.
|
||||
|
||||
Group 6 now provides complete preparation/discovery, graph and query reuse through explicit
|
||||
`--cache-dir`, bounded publication and isolated graph leases. A packaged authored JavaScript
|
||||
control preserved the same evidence/admission as an uncached run; its identical-revision
|
||||
replay took 2.00s. Source/declaration changes invalidate affected units. This correctness
|
||||
control does not establish representative changed-code p95, independent alert precision or
|
||||
eligibility. Rollout remains NO-GO and normal capabilities remain disabled.
|
||||
|
||||
Group 7 now provides the real multi-ref push interface, opt-in hook integration and bounded
|
||||
optional selection among already-admitted witnesses. Group 8 is next and
|
||||
must still pass the joint gates on declared populations. Keep the existing thresholds and
|
||||
profile preparation, graph construction and queries separately when investigating misses;
|
||||
do not substitute identical-tip hits for warm changed-code performance.
|
||||
|
||||
Tasks 4.3 and groups 5–6 used the documented Kiro manual review fallback after the agent
|
||||
thread limit; this is not independent review. See `pre-push-safety-net/implementation-group-5.md`
|
||||
and `benchmarks/measurements/2026-09-13-first-replay-experiment.json` for provenance,
|
||||
initial instrument failures, their repairs and the newly frozen final trial. The NodeGoat
|
||||
teaching workload did not cause a framework/fact adaptation or ranking retune. Ghost remains
|
||||
untouched; independent PHP/Python eligibility populations and C/C++ property support remain
|
||||
separate gaps. Automatic dominance-only context projection is still an upstream limit.
|
||||
Lead with the current milestone, the demonstrated developer outcome, the blocker and
|
||||
the next action. Keep implementation counts separate from release readiness. Preserve
|
||||
prior evidence and review limits in the [implementation records](../specs/pre-push-safety-net/status.md).
|
||||
The earlier PHP-first and runtime-first narrative below is historical and does not
|
||||
supersede this accepted sequence. Blocking deployment follows a later explicit decision.
|
||||
|
||||
## Historical roadmap narrative — September 9–11
|
||||
|
||||
|
||||
@@ -3,6 +3,15 @@
|
||||
Updated 2026-09-12 from the maintainer's explicit direction. This direction supersedes the
|
||||
WordPress-first release framing in older steering; historical measurements remain historical.
|
||||
|
||||
Release order accepted 2026-09-14: first prove one useful Node changed-code comparison
|
||||
with an analyzed quiet fixed twin, then make the same analysis fast, verify PHP/Python
|
||||
transfer, qualify independently, and enable only passing advisory capabilities. See
|
||||
[the milestone spec](../specs/pre-push-safety-net/release-milestones.md). We are at M1a
|
||||
(show the detection with vetoes recorded, one-week box), then M1b (repair only what fired);
|
||||
task counts and an experimental alpha tag do not establish release readiness. PMPro
|
||||
remains a core regression/development case; any PMPro core retune still requires frozen
|
||||
Node transfer evidence. This sequence supersedes older runtime-first next-step wording.
|
||||
|
||||
## Purpose
|
||||
|
||||
AI accelerates code production. OpenUltraSAST should help developers catch concrete security
|
||||
|
||||
Reference in New Issue
Block a user