mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-18 20:49:28 +00:00
docs(pre-push-safety-net): record approved design and implementation plan
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# Brief: pre-push-safety-net
|
||||
|
||||
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation started. Language: English.
|
||||
|
||||
## Problem
|
||||
|
||||
AI increases the rate of code changes. Developers need a bounded pre-push check that catches
|
||||
actionable security regressions without a recurring queue of speculative SAST warnings. The
|
||||
maintainer explicitly rejects noisy advisory output as well as noisy blocking output.
|
||||
|
||||
## Current State
|
||||
|
||||
The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. There is no pre-push
|
||||
snapshot resolver, change context for the ranker, persistent graph cache, or end-to-end deadline.
|
||||
The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking
|
||||
experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually
|
||||
selected two-file slice finding a CVE does not demonstrate automatic incremental analysis.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A developer pushes committed code and receives either a concise actionable security finding or a
|
||||
bounded result with explicit coverage. Unrelated working-tree changes, backlog findings, and model
|
||||
speculation do not become push warnings. The same analysis can be invoked explicitly for diagnosis.
|
||||
|
||||
## Approach
|
||||
|
||||
Resolve pushed snapshots, feed changed and affected first-party context to the existing ranker,
|
||||
reuse compatible results, and run admitted detector capabilities within a total deadline. The ranker
|
||||
remains the scope mechanism. Exclude third-party vendor code from targets AND graphs as already decided.
|
||||
Compare relevant base/head evidence
|
||||
before calling a defect new. Apply a separate actionability policy after arbitration. Evaluate on
|
||||
security and benign changes across repositories/frameworks, with cold and warm caches separated.
|
||||
|
||||
## Scope
|
||||
|
||||
- In: resulting pushed tips, affected context, caching, bounded execution, change-related actionable
|
||||
alerts, coverage status, optional enforcement, evaluation and reproducible local artifacts.
|
||||
- Out: every historical commit introduced by a push, whole-repository certification, style rules,
|
||||
dependency-vulnerability inventory, automatic source edits, mandatory cloud/LLM services,
|
||||
prompt optimization, and new memory-safety abstractions.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
One product boundary owns the pre-push transaction from Git input to user outcome. It consumes the
|
||||
existing ranker's scope and analysis engine through explicit scope/budget/results contracts. Reusable ranking and
|
||||
family semantics remain with their existing specs rather than being reimplemented in the hook.
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- Upstream: `contributor-scan` engine/packaging/family correctness; existing facts and rungs.
|
||||
- Adjacent: `flow-aware-ranking` schedules work; `zero-with-a-reason` diagnoses missing work;
|
||||
`finding-feedback-loop` owns durable human review and later learning.
|
||||
- Downstream: developer pre-push workflow and explicit local/CI replay of the same snapshot contract.
|
||||
|
||||
## Constraints and Initial Evaluation Targets
|
||||
|
||||
- User direction: actionability and low noise are mandatory in every mode.
|
||||
- Proposed rollout: advisory first; blocking is explicit opt-in after per-capability validation.
|
||||
- Proposed latency experiment: 30-second warm-push p95 on a declared reference workload/hardware,
|
||||
with a configurable deadline and 2-second cancellation/report allowance. This is an unproven
|
||||
design target, not a demonstrated SLA. Cold runs use the same deadline and disclose incompleteness.
|
||||
- No promise of all-language coverage; capabilities are admitted per language/framework/family.
|
||||
- Initial work must establish feasibility and noise before investing in an LLM ranker.
|
||||
|
||||
## Cross-language amendment — 2026-09-12
|
||||
|
||||
Keep PMPro as the primary worked optimization example, and require a JavaScript/Express/Node API
|
||||
transfer experiment before claiming the core improvement generalizes. Abstract evidence, ranking,
|
||||
scope and arbitration contracts remain shared. Language/frontend/framework facts supply concrete
|
||||
semantics; missing semantics remain a stated limit. Do not add PHP-specific core ranking shortcuts.
|
||||
|
||||
See `evaluation.md` for the proposed workload matrix: NodeGoat as an Express wiring/teaching
|
||||
candidate, a separate untouched Node API repository, existing Python regressions, and libpng as
|
||||
a C envelope/limit control. C/C++ detection is measured only for a labeled property the chosen
|
||||
abstraction can decide; bounds reasoning is not inferred from successful taint analysis.
|
||||
@@ -0,0 +1,339 @@
|
||||
# Design: pre-push-safety-net
|
||||
|
||||
Status: approved for task generation on 2026-09-12 following the maintainer's "ok, next kiro phase".
|
||||
Requirements and tasks are approved; group 1 implementation was authorized by "approve." on
|
||||
2026-09-12. Runtime and latency claims remain unverified.
|
||||
|
||||
## Overview
|
||||
|
||||
The hook is a bounded transaction around the existing ranker and detector. It analyzes immutable
|
||||
pushed tips, supplies change context to the ranker, and admits only actionable security regressions
|
||||
to the terminal. The ranker remains the scope mechanism. Third-party vendor code stays outside
|
||||
targets and graphs. WordPress is one regression workload, not the architecture's organizing unit.
|
||||
|
||||
The design separates three decisions: the ranker selects questions, the arbiter establishes modeled
|
||||
claims, and hook policy decides whether a supported change-related claim merits an interruption.
|
||||
An entailed rung is necessary for initial admission but is not sufficient by itself: a wrong fact,
|
||||
an irrelevant old defect, or an unactionable witness can still produce noise.
|
||||
|
||||
### Goals and Non-Goals
|
||||
|
||||
- Deliver specific security findings with witness, change relationship, location, and repair direction.
|
||||
- Bound the entire push and retain exact coverage; reuse matching analysis work without stale verdicts.
|
||||
- Measure recall, noise, coverage and latency together across supported languages/frameworks.
|
||||
- Do not build a second ranker/scope heuristic, an incremental CPG engine, a daemon, an automatic fixer,
|
||||
or a mandatory model service. Do not scan intermediate pushed history in this feature.
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
|
||||
- Git push inputs, immutable snapshots, comparison identity, and change context passed to the ranker.
|
||||
- Local artifact reuse, the shared push deadline, delta attribution and hook alert admission.
|
||||
- Concise rendering, explicit finding/coverage/push dispositions, and integration instructions.
|
||||
|
||||
### Out of Boundary
|
||||
|
||||
- Ranking features, weights, tier definitions, and exploration: `flow-aware-ranking` owns these.
|
||||
- Family facts, sanitizers, graph construction semantics and generic driver correctness:
|
||||
`contributor-scan` owns these. Hook integration may extend contracts but may not fork the driver.
|
||||
- Broader telemetry mechanisms: `zero-with-a-reason`; durable dismissals and learning:
|
||||
`finding-feedback-loop`. Initial rollout does not require either to be complete.
|
||||
- Vendor source analysis, new C memory abstractions, dependency CVE inventories, whole-tree assurance.
|
||||
|
||||
### Allowed Dependencies
|
||||
|
||||
Use Git, Python's standard library, `preprocess`, `mapping`, `model/regions`, `model/layout`,
|
||||
`model/scan`, `model/evidence`, `cpg/backend`, existing arbiters, redaction and report records.
|
||||
No new mandatory package or service. Dependency direction is existing engine/types -> push adapters
|
||||
and policy -> push runner -> CLI. The engine never imports the push package; shared deadline and
|
||||
artifact contracts live at the engine boundary. Ranker internals never depend on Git hook objects.
|
||||
|
||||
### Revalidation Triggers
|
||||
|
||||
Changes to engine/frontend/overlay versions, facts/sanitizers, layout, scope rules, ranker mode,
|
||||
query depth, deduplication, finding identity, or alert admission invalidate relevant cached results
|
||||
and require the security/fixed/benign push suite. Ranker changes also require untouched evaluation.
|
||||
Driver contract changes revalidate ordinary `ousast scan`, coverage reports, and repository benchmarks.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
`preprocess_repository` reads the live filesystem and labels it with HEAD; this is not a pushed
|
||||
snapshot. `scan_repository` builds a full-root CPG before selecting regions. `ScanBudget` limits
|
||||
regions/model calls, not total elapsed time. Backend graphs are temporary, with per-command timeouts
|
||||
and cleanup. `order_by_evidence` is implemented but disabled by the ordinary CLI caller.
|
||||
|
||||
Consequently the current ranker does not bound graph-build cost. A faster ordering alone cannot
|
||||
establish a fast pre-push hook, and passing fewer regions must not be described as incremental parsing.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Git ref updates] --> B[Immutable tips and comparison context]
|
||||
B --> C[First-party inputs with vendor exclusions]
|
||||
C --> D[Compatible graph or bounded graph build]
|
||||
B --> E[Changed and affected context]
|
||||
D --> F[Existing ranker selects scope]
|
||||
E --> F
|
||||
F --> G[Existing arbiters and targeted base comparison]
|
||||
G --> H[Change attribution and actionability admission]
|
||||
H --> I[Compact hook result and detailed artifact]
|
||||
D --> J[Coverage and deadline accounting]
|
||||
F --> J
|
||||
G --> J
|
||||
J --> I
|
||||
```
|
||||
|
||||
The deadline surrounds the entire flow. Cache lookup does not bypass identity checks; a miss does
|
||||
not trigger unbounded work. The default path is local and deterministic. An optional LLM may reorder
|
||||
within allowed ranker constraints or explain existing evidence; it cannot mint facts or admit alerts.
|
||||
|
||||
### Language and Framework Contract
|
||||
|
||||
Requirements 8.1, 8.2, 8.3, 8.4 make abstract analysis a transfer contract, not a claim of automatic
|
||||
language coverage. Frontends and versioned facts identify sources, sink argument roles, carried
|
||||
values, dispatch, guards, obligations and configuration properties. The common evidence vector,
|
||||
ranker scope decisions and arbiters consume these representations. Core ranking must not inspect
|
||||
repository names, PHP syntax, WordPress calls or Express API names to assign priority. Equivalent
|
||||
normalized evidence receives the same core tier and score regardless of origin; policy admission
|
||||
also requires the independently evaluated capability for that origin.
|
||||
|
||||
Existing seams are `mapping.py` (route/middleware mapping), `ruleset/semantic/*.toml` and
|
||||
`ruleset/obligations/*.toml` (reviewed language/framework facts), `model/specs.py` (abstract question
|
||||
construction), `model/evidence.py` (shared ranking), and `cpg/queries/` (graph queries). Reuse them;
|
||||
do not add a universal source IR or move framework checks into the hook. Frontend normalization
|
||||
belongs at the backend/query boundary and must document unsupported constructs. Existing
|
||||
language-specific integrations require audit; this design does not claim they already conform.
|
||||
|
||||
For Express, verify route registration and ordered middleware, request input, imported handlers,
|
||||
service calls, callbacks and async boundaries. A function named `authenticate` does not establish
|
||||
that its guard governs an operation. Require the same guard/obligation evidence as in other
|
||||
frameworks. `node_modules` and other declared third-party trees stay outside targets AND graphs;
|
||||
explicit framework/library summaries describe relevant behavior without analyzing vendor source.
|
||||
Do not exclude first-party JavaScript merely because it is browser code: record its partition and
|
||||
capability separately from the server. TypeScript support also needs frontend evidence, not an
|
||||
assumption based on normalizing its language label to JavaScript.
|
||||
|
||||
PMPro is the development workload; Node/Express checks transfer of the frozen core. Framework/fact
|
||||
adaptations are recorded separately from zero-change transfer, and any case used for repairs becomes
|
||||
regression data. C and C++ use the same scheduling/reporting contract but may need different abstract
|
||||
domains: taint transfer, guard dominance and numeric bounds are distinct capability claims. Keep
|
||||
libpng as an envelope/unsupported-bounds control until a suitable domain is implemented and measured.
|
||||
Concrete workload roles and required paired changes are in `evaluation.md`.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice | Role |
|
||||
|---|---|---|
|
||||
| Push input | Installed Git, object-format-aware commands | Resolve refs, trees and immutable blobs |
|
||||
| Runtime | Existing Python >=3.11 CLI and standard library | Transaction, cache, deadline, policy |
|
||||
| Analysis | Existing Joern backend; evaluated baseline 4.0.625 | Graph and arbiters, version recorded |
|
||||
| Storage | Local content-addressed directories and JSON manifests | Atomic reusable artifacts, no database |
|
||||
| Presentation | Existing finding records plus compact hook renderer | Separate findings and coverage |
|
||||
|
||||
Align Docker, Compose and provisioning with the evaluated Joern version before any deployment
|
||||
claim. The old defaults are 4.0.623. Pin alignment is predecessor packaging work, not completed here.
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
New files (proposed, not created by this documentation change):
|
||||
|
||||
| Path | Responsibility |
|
||||
|---|---|
|
||||
| `src/openultrasast/push/snapshot.py` | Parse ref updates, resolve bases, materialize blobs, produce change context |
|
||||
| `src/openultrasast/push/cache.py` | Artifact keys, validity, atomic publication and eviction |
|
||||
| `src/openultrasast/push/policy.py` | Delta identity, actionability admission and push disposition |
|
||||
| `src/openultrasast/push/runner.py` | Shared deadline, ranker/engine orchestration, per-ref result aggregation |
|
||||
| `src/openultrasast/push/report.py` | Compact terminal result and reproducible push manifest |
|
||||
| `benchmarks/push/README.md` and `benchmarks/push/measure.py` | Labeled push recipes and joint quality/latency measurement |
|
||||
| `tests/test_push_snapshot.py`, `tests/test_push_cache.py`, `tests/test_push_policy.py`, `tests/test_push_runner.py` | Contract and integration checks below |
|
||||
|
||||
Modified files: `cli.py` and `config.py` add explicit push command/configuration;
|
||||
`model/regions.py` carries change context without Git-specific logic; `model/scan.py` consumes the
|
||||
ranker's selected questions and returns exact selected/deferred IDs, deadline and reusable-graph
|
||||
contracts; `cpg/backend.py` owns graph artifact validation and process cancellation. Existing
|
||||
`model/evidence.py` changes only through the ranking spec. Packaging pin fixes remain in existing
|
||||
Docker/Compose/ops paths. `README.md` documents opt-in integration without overwriting existing hooks.
|
||||
|
||||
Cross-language validation reuses `tests/test_model_evidence.py` for equivalent normalized evidence,
|
||||
`tests/test_model_scan.py` for shared scope/coverage behavior, and existing mapping/fact tests for
|
||||
frontend adaptation. Reviewed workload recipes use `benchmarks/repos/*.toml` and the proposed
|
||||
`benchmarks/push/` harness. Necessary mapping/fact/query fixes stay within the engine/ranking specs;
|
||||
they do not become JavaScript-specific branches in `push/`.
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### Snapshot Adapter
|
||||
|
||||
`PushUpdate(local_ref: str, local_oid: str, remote_ref: str, remote_oid: str)` records every input
|
||||
line. `PushComparison(head_oid: str, base_oid: str | None, base_reason: str, refs: tuple[str, ...])`
|
||||
records immutable identity. Object IDs are validated through Git; no hard-coded SHA-1 length.
|
||||
|
||||
For existing refs, compare the supplied old and new tips, including force-pushes. For a new branch,
|
||||
use a configured comparison ref resolved to an OID and its unique merge base with the pushed tip;
|
||||
record that this is a chosen baseline, not the previous remote branch. No configured/unique/local
|
||||
base means comparison incomplete. Do not substitute HEAD or a remote-tracking ref silently. No
|
||||
automatic fetch inside the hook. Peel annotated tags to commits; other objects are unsupported.
|
||||
Deletion updates require no target scan. Deduplicate identical target graphs, but retain separate
|
||||
comparisons when the same target has different bases. Multiple refs share one deadline.
|
||||
|
||||
Materialize tracked blobs directly into isolated scratch storage; do not checkout over the user's
|
||||
tree or run filters/build scripts. Use NUL-delimited tree/diff records and argument arrays. Include
|
||||
deletions as impact seeds and retain rename attribution. Symlinks are not followed outside the
|
||||
snapshot; gitlinks, absent LFS content and unsupported objects produce explicit coverage limits.
|
||||
Target-tree content is the analysis population; intermediate history is out of scope.
|
||||
|
||||
### Ranker Scope Contract
|
||||
|
||||
The snapshot adapter provides `ChangeContext`: base/head identities, changed/deleted paths and spans,
|
||||
manifest/config changes, and known affected first-party relationships. It supplies evidence, not
|
||||
an alternative inclusion policy. The existing ranker produces `ScopeDecision`: selected question
|
||||
IDs, their priority/evidence, deferred question IDs with reasons, and unresolved context boundaries.
|
||||
Identifiers include unit/language/path/function/family to avoid collisions across partitions.
|
||||
|
||||
Third-party vendor exclusions apply before both preprocessing targets and graph construction.
|
||||
Tests remain available but unshipped, consistent with the approved layout decision. The ranker
|
||||
cannot reintroduce vendor sources. Known external summaries are versioned facts; without a relevant
|
||||
summary, a claim depending on excluded internals is unresolved. Do not assume a missing external
|
||||
call is safe. Entry/callee, field, hook and removed-guard context must be represented in the ranked
|
||||
question; an unchanged sink may be affected by changed reachability or sanitization.
|
||||
|
||||
Graph construction and question scope are distinct costs. Initial graph input is the first-party
|
||||
repository per supported frontend, or an explicitly declared analysis unit with recorded boundaries.
|
||||
Do not create an additional heuristic source-slicer. A mixed-language repository has explicit
|
||||
frontend partitions and coverage per partition; unsupported cross-language edges remain unresolved.
|
||||
Use old graph relationships only as prioritization hints after a code change, never as current proof.
|
||||
If a ranked question's necessary first-party context is absent, expand its declared unit within the
|
||||
remaining budget or report it unresolved. No claim of exact tier-0 pruning extends beyond the
|
||||
modeled graph/facts/scope; unanswered evidence is never tier 0.
|
||||
|
||||
### Cache and Engine Boundary
|
||||
|
||||
Graph key: digest of every included blob/path and applicable declarations, frontend/engine version,
|
||||
overlay semantics/options, language, and exclusions. Query key additionally includes graph key,
|
||||
question/context/depth, facts/sanitizers, query code and relevant analysis configuration. Result key
|
||||
also includes ranking/admission versions and any model/prompt identity. Comparison results include
|
||||
both base and head identities. Persist manifests with source byte counts and graph census.
|
||||
|
||||
Only validated, complete artifacts can satisfy complete-result lookups. Partial artifacts may be
|
||||
retained for diagnostics but never promoted by cache presence. Publish atomically from private
|
||||
temporary paths under a per-key lock; another push may wait only within its deadline. Apply a
|
||||
configured cache size limit and evict unleased entries. No live graph mutation or dependence on
|
||||
CPG node IDs remaining stable across builds. A changed input invalidates that graph; incremental
|
||||
Joern mutation is neither assumed nor promised. Unchanged declared units and identical snapshots
|
||||
can reuse graphs; changed units rebuild. Measure whether this is enough before claiming scale.
|
||||
|
||||
The maintainer confirmed interest in persistence followed by differential updates on 2026-09-12.
|
||||
Persistence is the first optimization; differential graph mutation remains a separate future
|
||||
experiment. Its acceptance gate must compare against a cold rebuild on identical inputs and
|
||||
semantics, including changed/deleted/renamed declarations, affected call edges and dataflow
|
||||
overlays, and invalidation of dependent query/results artifacts. Updating only changed file nodes
|
||||
is insufficient. No graph partition may claim complete cross-partition analysis without validated
|
||||
relationship handling. This sequencing does not expand the approved implementation into a new
|
||||
incremental engine.
|
||||
|
||||
The backend owns `GraphArtifact` validation/lifecycle. The generic driver accepts an optional
|
||||
validated graph lease and an absolute monotonic deadline, preserving its ordinary fresh-build path.
|
||||
Queries operate on an isolated lease if loading can modify a graph. Every subprocess timeout is
|
||||
bounded by remaining transaction time, including fallback frontend, overlay, shard and retry work.
|
||||
Cancel process groups, close leases, then persist coverage within the cancellation allowance.
|
||||
|
||||
### Delta and Actionability Policy
|
||||
|
||||
Admission requires all of: an enabled/evaluated capability, `model_entailed` or a future actually
|
||||
produced `execution_confirmed` result, a locatable witness, applicable family semantics, no unresolved
|
||||
dependency that undermines the claim, supported change attribution, and a concrete repair direction.
|
||||
No production caller currently reaches `execution_confirmed`; it is not a prerequisite or a promise.
|
||||
|
||||
Findings are keyed by family/mechanism and stable source/operation identity, with rename/span mapping;
|
||||
line numbers alone do not establish novelty. Re-query relevant base counterparts under the same
|
||||
semantics and sufficient comparable context. A missing old finding from a truncated/failed scan is
|
||||
unknown, not proof that the head introduced a bug. A removed guard or newly connected source can
|
||||
establish a worsened flow at an unchanged sink. Unknown novelty stays in the diagnostic artifact
|
||||
with one aggregate comparison-coverage notice. Initial checkout is not automatic acceptance of debt.
|
||||
|
||||
Deduplicate by defect, retaining distinct witnesses and ref associations. Explain input/operation,
|
||||
security consequence, why this change matters and a specific repair locus; do not require automatic
|
||||
patch generation. Templates grounded in facts/witnesses are sufficient. Admission rejects generic
|
||||
"sanitize input" advice without a supported context/operation. An LLM explanation must cite the
|
||||
same witness and cannot raise the rung or invent exploitability.
|
||||
|
||||
### Result and Hook Adapter
|
||||
|
||||
`PushResult` separates `finding_status` (`actionable` / `none`), `coverage_status` (`complete_within_scope`
|
||||
/ `incomplete` / `unavailable` / `not_applicable`) and `push_disposition` (`allow` / `block`). Zero
|
||||
selected questions cannot mean complete when applicable questions were deferred or context is unknown.
|
||||
The detailed manifest retains per-ref/per-capability coverage, supported and unsupported populations,
|
||||
all admitted defects, rejected-candidate reasons, exact scope and timings.
|
||||
|
||||
Initial profile default: advisory, with the same admission policy as blocking. Blocking is opt-in and
|
||||
returns nonzero only for admitted new/worsened findings; incomplete coverage defaults to allow plus
|
||||
a notice, with a separately configurable strict coverage policy. Show up to three concise defect
|
||||
summaries plus the remaining count/artifact path; the cap never changes the exit decision. Do not
|
||||
render the full SAST backlog. No actionable findings: at most one bounded status line. Incomplete:
|
||||
one notice naming the primary reason and a concrete retry/diagnostic action. Unchanged results may
|
||||
be reused, but a still-relevant admitted defect is not dismissed merely because it was printed before.
|
||||
|
||||
Expose a proposed `ousast pre-push` command receiving Git's stdin and remote arguments, plus explicit
|
||||
base/head replay. Hook installation is explicit: preserve `core.hooksPath`, existing hook exit
|
||||
semantics, and stdin for other consumers. Provide an integration snippet and refuse automatic
|
||||
overwrite of an existing hook. Analysis runs no project code and performs no source modification.
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirements | Design elements |
|
||||
|---|---|
|
||||
| 1.1, 1.2, 1.3, 1.4 | Snapshot Adapter: immutable objects, per-ref comparisons, special update cases |
|
||||
| 2.1, 2.2, 2.3, 2.4 | Ranker Scope Contract: change context, vendor exclusion, boundary/guard handling |
|
||||
| 3.1, 3.2, 3.3, 3.4, 3.5 | Delta and Actionability Policy: admission, witnesses, comparison and deduplication |
|
||||
| 4.1, 4.2, 4.3, 4.4 | Cache and Engine Boundary: shared deadline, complete artifact identity and leases |
|
||||
| 5.1, 5.2, 5.3, 5.4, 5.5 | Result and Hook Adapter: separate statuses, compact rendering, explicit policies |
|
||||
| 6.1, 6.2, 6.3, 6.4, 6.5 | Testing Strategy and Rollout: capability admission and joint gates |
|
||||
| 7.1, 7.2, 7.3, 7.4 | Local runtime, model redaction/budget, manifests, non-destructive integration |
|
||||
| 8.1, 8.2, 8.3, 8.4 | Language and Framework Contract; frozen-core transfer and workload matrix in evaluation.md |
|
||||
|
||||
## Error Handling
|
||||
|
||||
Unreadable input or failed census invalidates the affected graph. Missing base invalidates novelty
|
||||
comparison. Missing context invalidates affected checks. Deadline exhaustion preserves completed
|
||||
findings and records deferred work. These states remain visible even when the hook permits the push.
|
||||
If result persistence fails, print one artifact-write notice and retain the real finding/coverage
|
||||
decision in the terminal; do not fabricate a successful cache entry or hang while retrying.
|
||||
|
||||
## Testing Strategy and Rollout
|
||||
|
||||
1. Snapshot contracts (1.1–1.4, 7.1, 7.4): dirty trees, non-HEAD pushes, multiple refs with different
|
||||
bases, force-push/new-branch/missing-object cases, tags, deletion-only pushes, unusual paths,
|
||||
symlinks/gitlinks, and coexistence with an existing hook.
|
||||
2. Ranker/context tests (2.1–2.4): handler-to-unchanged-sink, field/hook flows, removed guards,
|
||||
first-party cross-file context, vendor source absent from both graph and targets, unresolved
|
||||
vendor boundary, polyglot partition coverage. No second selection heuristic in the push adapter.
|
||||
3. Admission tests (3.1–3.5, 5.1–5.5): high rung with wrong/unsupported semantics is rejected;
|
||||
unchanged backlog and missing-base claims stay out; new supported defects include actionable
|
||||
witnesses; one defect reached by many regions prints once; advisory and blocking share admission.
|
||||
4. Deadline/cache tests (4.1–4.4, 7.3): source/context/fact/query/version invalidation, corrupt/partial
|
||||
entries, concurrent pushes, timeout during build/overlay/query/retry, process-group termination,
|
||||
no repeated per-ref budget reset, exact selected/deferred IDs rather than slicing old order.
|
||||
5. Real push evaluation (6.1–6.5, 8.1–8.4): use PMPro for core development, then freeze the core
|
||||
and replay labeled introducing/fixing/benign changes on a JavaScript/Express Node API workload.
|
||||
Preserve Python regressions and a separate untouched Node API repository; training applications
|
||||
and cases used to fix mapping are not fresh holdouts. Include injection, path and authorization/
|
||||
configuration where supported, plus middleware and cross-module/async controls. Test equivalent
|
||||
normalized evidence for identical core decisions. Keep libpng's envelope and unsupported bounds
|
||||
separate from a future C/C++ positive control. See `evaluation.md` for selection and labeling gates.
|
||||
|
||||
Measure the accepted experimental acceptance profile in requirements on declared hardware and fixed workloads.
|
||||
Warm means prior reusable work from the preceding state, not only repeating the identical tip.
|
||||
Report identical-tip hits separately from a one-function change and a dependency/configuration
|
||||
change. Include first installs, cold graphs, growing repository size, multi-ref pushes and incomplete
|
||||
outcomes. Freeze the applicable population before ranker selection; deferred questions count against
|
||||
completion. Report p50/p95 plus timeout counts, observed precision with confidence intervals, recall,
|
||||
benign interruption rate and fixed-side silence. A low-noise score with low recall/coverage fails.
|
||||
|
||||
Rollout order: reconcile engine pins and benchmark honesty; prove one end-to-end change/witness/fix
|
||||
with the existing ranker; establish deadline/cache feasibility; run untouched and benign evaluation;
|
||||
admit passing capabilities in advisory mode; consider opt-in blocking only after these gates pass.
|
||||
LLM ranking is a later experiment, not a prerequisite. If realistic changed-unit rebuilds cannot
|
||||
meet the deadline/coverage gates, keep the hook experimental and revisit the graph strategy with
|
||||
measurements; do not solve the miss by silently shrinking context or lowering alert standards.
|
||||
@@ -0,0 +1,82 @@
|
||||
# Evaluation: core optimization and cross-language transfer
|
||||
|
||||
Status: evaluation contract accepted with requirements/design for task generation, 2026-09-12. Implements requirements 6.1–6.5 and 8.1–8.4.
|
||||
No new repository has been pinned, scanned or admitted by this document.
|
||||
|
||||
## Workload roles
|
||||
|
||||
| Workload | Role | Required evidence |
|
||||
|---|---|---|
|
||||
| Existing pinned PMPro | Primary development and cost-attribution example | Known CVE retained; fixed/benign controls; total build/evidence/query cost |
|
||||
| Existing WP Statistics and MW WP Form | PHP structural regressions | Hook/field flows preserved; vendor exclusion does not become a ranking trick |
|
||||
| JavaScript/Express Node API | Required transfer and hook integration workload | Same frozen core; route/middleware/handler/service context; vulnerable/fixed/benign changes |
|
||||
| Separate untouched Node API repository | Generalization evaluation | Predeclared labels/population; no tuning on its results; all misses and gaps retained |
|
||||
| Existing pinned VAmPI | Python and authorization regression | Actual transitive SQLI witness plus obligation findings; no omitted position targets |
|
||||
| Existing pinned libpng | C graph/cost and capability-limit control | Envelope measured; unsupported arithmetic/bounds explicitly reported |
|
||||
| Future C or C++ positive control | Additional abstract-property transfer | Reviewed vuln/fix for a supported property; C++ frontend behavior verified independently |
|
||||
|
||||
## Recommended JavaScript starting point
|
||||
|
||||
**OWASP NodeGoat is the proposed first Express wiring/teaching candidate.** It is a deliberately
|
||||
vulnerable Node application with remediation teaching material, and its package manifest declares
|
||||
Express. It exercises server-side code without making an entire browser application the primary
|
||||
measurement unit. [Project](https://github.com/OWASP/NodeGoat),
|
||||
[package manifest](https://github.com/OWASP/NodeGoat/blob/master/package.json).
|
||||
|
||||
This is candidate selection, not detection evidence. Before adding an executable recipe, pin an
|
||||
immutable commit, verify readable source/license bytes, identify the exact affected operation and
|
||||
its caller context, and review the vulnerable and fixed forms. An authored repair must be labeled
|
||||
as authored; do not present it as an upstream CVE fix. Do not assume the tutorial's vulnerability
|
||||
categories correspond to supported arbiters: a MongoDB operator-injection case is not automatically
|
||||
covered by SQL string-taint facts. Select a supported positive control and retain other cases as gaps.
|
||||
|
||||
NodeGoat is intentionally educational and will be inspected during integration. It cannot establish
|
||||
production generalization on its own. Its inspected manifest uses an older Express 4 dependency
|
||||
range; it is not evidence of coverage for current module/async/framework patterns. Require those
|
||||
patterns in the additional Node workload and record its exact versions. Reserve an independently selected Node API repository with
|
||||
reviewable security-fix history before the transfer result is used to tune the core. Selection must
|
||||
not depend on getting a favorable scanner score. No npm installation or project execution is needed
|
||||
to establish the static input; runtime controls, when needed, belong in a separate controlled harness.
|
||||
|
||||
## Paired changes and negative controls
|
||||
|
||||
1. Request input crosses route -> imported handler -> service -> supported dangerous operation;
|
||||
a correct family-specific repair stops the finding on the fixed side.
|
||||
2. A removed authorization/ownership guard exposes an unchanged operation. A guard that is present
|
||||
but does not govern the operation must not discharge it; correctly ordered governing middleware must.
|
||||
3. Callback/async data transfer preserves a supported source-to-operation witness. Unresolved dynamic
|
||||
calls are counted as a context gap, not silently treated as no flow.
|
||||
4. Input validation for an unrelated context does not cleanse SQL/command/path operations. Path
|
||||
normalization alone is not assumed to establish confinement to an allowed root. Test both the
|
||||
real repair and a plausible but insufficient repair.
|
||||
5. Benign refactoring, renaming, formatting, parameter binding and unaffected backlog do not generate
|
||||
new hook alerts. Retain benign push counts so low noise cannot be inferred only from security cases.
|
||||
6. `node_modules` and other declared vendor files are absent from both graph and targets, with
|
||||
first-party callers still represented. Missing external semantics remain visible coverage limits.
|
||||
|
||||
Do not force all these shapes into one repository. Keep authored cross-language contract tests
|
||||
separate from reviewed repository changes, and preserve their provenance in the scorecard.
|
||||
|
||||
## Freeze, transfer, classify
|
||||
|
||||
1. Freeze core ranker features/weights, arbiter/query semantics, engine version, budget, and existing
|
||||
framework facts after the PMPro experiment. Run Node with available adapters first; save that result.
|
||||
2. Classify each miss before changing anything: unread input/instrument failure, frontend normalization,
|
||||
missing framework fact, ranker scope/budget, wrong abstract semantics, or unsupported abstract domain.
|
||||
3. If a framework fact or frontend adaptation is needed, review it separately and rerun both languages.
|
||||
Report adapted transfer separately from the unchanged-core result. A core change starts a new
|
||||
version and requires a fresh untouched evaluation; it is not hidden in a "JavaScript config" label.
|
||||
4. Compare actual findings, fixed-side silence, actionable precision, benign interruptions, coverage,
|
||||
and elapsed cost together. Equivalent normalized evidence must produce identical core tier/score
|
||||
results; equivalent syntax across languages is not required and is not presumed.
|
||||
5. Do not generalize a C envelope result to C/C++ detection. Select a positive control only after its
|
||||
property fits a declared arbiter; numerical bounds/overflow needs a suitable domain and separate
|
||||
validation. Do not expand taint sources or sanitizer names to simulate arithmetic reasoning.
|
||||
|
||||
## First bounded implementation experiment
|
||||
|
||||
Replay one supported PMPro change/fix and one Node/Express change/fix through the same ranker,
|
||||
graph exclusion policy, deadline and admission contract. Add at least one benign change per workload.
|
||||
Produce one joint scorecard before any further core optimization. A missing Node frontend, broken
|
||||
middleware mapping or wrong sanitizer model is an explicit finding of this experiment, not a reason
|
||||
to remove Node from the evaluation or silently fall back to a PHP-only release claim.
|
||||
@@ -0,0 +1,159 @@
|
||||
# Requirements: pre-push-safety-net
|
||||
|
||||
Status: approved for task generation on 2026-09-12 following the maintainer's "ok, next kiro phase". The acceptance profile defines evaluation targets, not measured achievements.
|
||||
|
||||
## Introduction and Boundary
|
||||
|
||||
OpenUltraSAST helps developers catch concrete security regressions in the resulting commits they
|
||||
push. Advisory findings must be actionable; nonblocking speculation is still noise. The hook owns
|
||||
the push transaction and its report, using the existing analysis engine. It does not certify the
|
||||
repository, scan every intermediate commit, add new detector semantics, or automatically fix code.
|
||||
|
||||
## Requirement 1: Analyze the code being pushed
|
||||
|
||||
1.1 When a push contains ref updates, the Safety Net shall analyze each distinct resulting commit
|
||||
snapshot and retain its association with every supplied ref update.
|
||||
|
||||
1.2 While analysis runs, the Safety Net shall leave the working tree, index, refs, and existing hook
|
||||
behavior unchanged and shall exclude uncommitted content from the analyzed snapshot.
|
||||
|
||||
1.3 If an update deletes a ref, targets an unsupported object, or lacks a usable comparison base,
|
||||
the Safety Net shall record that disposition without presenting an empty comparison as a successful check.
|
||||
|
||||
1.4 When a new branch, merge, force-push, rename, or deletion changes the comparison, the Safety Net
|
||||
shall record the exact compared revisions and the rule used to choose them.
|
||||
|
||||
## Requirement 2: Analyze affected security behavior
|
||||
|
||||
2.1 When code changes, the Safety Net shall supply changed sites and affected first-party caller,
|
||||
callee, data, guard, dispatch, and configuration context to the ranker, which shall remain the
|
||||
mechanism selecting the regions and questions to analyze.
|
||||
|
||||
2.2 If a dependency cannot be resolved or included within the budget, the Safety Net shall identify
|
||||
the affected checks as incomplete rather than treating omitted context as safe.
|
||||
|
||||
2.3 When constructing analysis targets or graphs, the Safety Net shall exclude third-party vendor
|
||||
code and record those boundaries, along with any test, generated-file, submodule, or language exclusions
|
||||
and their effect on supported coverage; missing vendor semantics shall require an explicit summary
|
||||
or an unresolved result rather than reintroducing vendor code.
|
||||
|
||||
2.4 When a guard or sanitizer is removed, the Safety Net shall evaluate affected unchanged operations
|
||||
where supported rather than requiring the vulnerable operation itself to occur on an added line.
|
||||
|
||||
## Requirement 3: Produce actionable security alerts
|
||||
|
||||
3.1 When the Safety Net emits a normal hook alert, it shall include a supported security consequence,
|
||||
an exact location, a witness connecting the relevant input or obligation to the operation, a supported
|
||||
relationship to the proposed change, and a concrete repair direction.
|
||||
|
||||
3.2 If a candidate is only suspicion, unsupported intent inference, a style concern, or an
|
||||
uncalibrated capability result, the Safety Net shall omit it from normal hook alerts and retain its
|
||||
disposition in an opt-in diagnostic artifact.
|
||||
|
||||
3.3 When multiple regions identify the same defect, the Safety Net shall emit one defect summary
|
||||
with its relevant locations rather than repeated warnings.
|
||||
|
||||
3.4 If an existing defect is unchanged by the push, the Safety Net shall omit it from normal hook
|
||||
alerts; if evidence cannot establish whether a defect is new or worsened, it shall report that
|
||||
comparison gap as coverage information without claiming a new vulnerability.
|
||||
|
||||
3.5 Where a model assists ranking or explanation, the Safety Net shall require the same alert
|
||||
admission evidence as in a run without that model.
|
||||
|
||||
## Requirement 4: Bound cost and reuse valid work
|
||||
|
||||
4.1 When a check begins, the Safety Net shall apply one configurable elapsed-time budget across
|
||||
all ref updates and stages, including preparation, retries, analysis, optional services, and reporting.
|
||||
|
||||
4.2 If the deadline expires, the Safety Net shall cancel outstanding work, persist completed
|
||||
evidence and incomplete coverage, and return within a separately measured cancellation allowance.
|
||||
|
||||
4.3 When reusing an analysis result, the Safety Net shall verify that its input, relevant context,
|
||||
engine, facts, query semantics, scope, and policy remain compatible with the requested check.
|
||||
|
||||
4.4 If a cached artifact is partial, corrupt, incompatible, or unavailable, the Safety Net shall
|
||||
recompute within the remaining budget or return incomplete coverage without reusing a clean verdict.
|
||||
|
||||
## Requirement 5: Keep the push experience proportionate
|
||||
|
||||
5.1 When no actionable defect is found in the completed supported checks, the Safety Net shall
|
||||
emit at most one compact status line that describes the checked scope without calling the repository secure.
|
||||
|
||||
5.2 If analysis is incomplete or unavailable, the Safety Net shall emit a separate compact operational
|
||||
notice with one concrete next action and a detailed artifact reference, without a per-file warning flood.
|
||||
|
||||
5.3 Where advisory mode is selected, the Safety Net shall allow the push while applying the full
|
||||
actionability requirements to every displayed finding.
|
||||
|
||||
5.4 Where blocking mode is explicitly enabled, the Safety Net shall block for admitted new or
|
||||
worsened defects; an incomplete scan shall follow a separately declared coverage policy.
|
||||
|
||||
5.5 When producing a result, the Safety Net shall preserve separate finding, coverage, and push
|
||||
disposition fields so allowing a push cannot be interpreted as complete analysis.
|
||||
|
||||
## Requirement 6: Earn admission through evaluation
|
||||
|
||||
6.1 When evaluating admission of a detector capability to the hook, the project shall evaluate its language,
|
||||
framework, and family on labeled vulnerable/fixed changes and benign pushes, including a repository
|
||||
not used to develop or tune that capability.
|
||||
|
||||
6.2 When reporting hook quality, the project shall report actionable-alert precision, recall on
|
||||
supported labeled changes, fixed-side silence, benign-push interruption rate, coverage completion,
|
||||
and cold/warm latency with sample counts, uncertainty, and all unresolved cases.
|
||||
|
||||
6.3 If a target cannot be located, a query is unanswered, or a transitive path is unproven, the
|
||||
evaluation shall retain that case in the declared population and identify the unresolved result.
|
||||
|
||||
6.4 When comparing rankers, the evaluation shall hold scope, engine, facts, workload, and budget
|
||||
constant, record the actual deployed ranking mode, and evaluate completed detections as well as rank.
|
||||
|
||||
6.5 When evaluating release readiness of the hook, the project shall require passing predeclared precision, recall,
|
||||
coverage, and latency gates together; a policy that achieves silence by examining or alerting on nothing
|
||||
shall fail that evaluation.
|
||||
|
||||
## Requirement 7: Preserve local control and reproducibility
|
||||
|
||||
7.1 When running the default hook, the Safety Net shall use locally available analysis without
|
||||
requiring a model endpoint, downloading an engine, fetching source, or running project build scripts.
|
||||
|
||||
7.2 Where an external model is explicitly configured, the Safety Net shall apply the existing
|
||||
redaction policy, record model/prompt provenance, and include its cost and latency in the budget.
|
||||
|
||||
7.3 When saving an artifact, the Safety Net shall record input identities, scope and exclusions,
|
||||
engine/fact/query/policy versions, exact selected and deferred questions, outcomes, and timings.
|
||||
|
||||
7.4 When installing or removing the hook explicitly, the Safety Net shall preserve existing hooks
|
||||
and document the integration and removal steps.
|
||||
|
||||
## Requirement 8: Transfer the core analysis across languages
|
||||
|
||||
8.1 When equivalent source, sink, guard, obligation and carried-value evidence is supplied from
|
||||
different languages or frameworks, the Safety Net shall apply the same core scope, ranking and
|
||||
admission rules without repository-specific or language-specific priority exceptions.
|
||||
|
||||
8.2 When a frontend or framework cannot supply evidence needed for a supported check, the Safety Net
|
||||
shall identify that capability gap rather than interpreting missing evidence as a safe program.
|
||||
|
||||
8.3 When evaluating a core optimization developed on PMPro, the project shall freeze the core
|
||||
ranking features, weights and decision rules and evaluate transfer on a JavaScript Node API workload
|
||||
with Express-style routes and middleware, including vulnerable, fixed and benign changes.
|
||||
|
||||
8.4 When publishing cross-language results, the project shall distinguish unchanged-core transfer,
|
||||
frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or
|
||||
unsupported-property results separately from demonstrated vulnerability detection.
|
||||
|
||||
## Accepted experimental acceptance profile
|
||||
|
||||
These values are the initial implementation/evaluation profile accepted for task generation, not
|
||||
measured achievements or a demonstrated SLA. Freeze them before evaluating a candidate; report
|
||||
failures without lowering the bar after inspecting holdout results.
|
||||
|
||||
- Advisory first; blocking opt-in; incomplete coverage allows the push with one notice by default.
|
||||
- Warm latency p95 <= 30 seconds, cancellation/report allowance <= 2 seconds; cold latency reported separately.
|
||||
- >= 95% completed supported checks on the declared representative push workload.
|
||||
- >= 95% observed actionable-alert precision with counts and confidence interval; zero known false
|
||||
alerts on the fixed/benign regression suite. Small samples do not establish broad precision.
|
||||
- >= 90% recall on the predeclared supported security-change suite; unsupported classes remain in
|
||||
coverage reporting, not reclassified after a miss. Publish per-capability results, not only an aggregate.
|
||||
- No capability is admitted with zero positive controls, zero reviewed emitted alerts, or no untouched
|
||||
evaluation repository. Failed profiles remain experimental and silent in the normal hook.
|
||||
@@ -0,0 +1,142 @@
|
||||
# Research and design decisions: pre-push-safety-net
|
||||
|
||||
## Summary
|
||||
|
||||
- Date: 2026-09-12. Discovery scope: complex integration of the existing ranker and detector with Git.
|
||||
- User direction: a scalable pre-push safety net as AI accelerates development; WordPress is one case.
|
||||
- Clarifications: advisory must be actionable, not SAST noise; the ranker is the scope mechanism;
|
||||
third-party vendor code must remain outside targets and graphs.
|
||||
- Requirements and design are proposals, not approved or implemented. No hook latency claim is made.
|
||||
|
||||
## Research Log
|
||||
|
||||
### Repository state and measurement limits
|
||||
|
||||
Reviewed through `a8069ac`, including `60963f5` (overlay reuse/scoped seeds), `9f881b4`
|
||||
(closure workaround bounded by upstream version), `60d13d7` (evidence ordering), `3b3b96b`
|
||||
(parser-script readability probe), `b821ac6` (layout), and the latest ranking baselines.
|
||||
|
||||
| Workload | Original position budget | 4.0.625 + layout budget | Build / evidence seconds |
|
||||
|---|---:|---:|---:|
|
||||
| PMPro | 391 | 82 | 129.9 / 295.7 |
|
||||
| WP Statistics | 504 | 91 | 45.4 / 68.7 |
|
||||
| MW WP Form | 513 | 54 | 45.2 / 80.3 |
|
||||
| VAmPI | 16 | 16 | 5.6 / 18.4 |
|
||||
|
||||
Source: `benchmarks/ranking/baselines/2026-09-11-evidence-rank-layout.json`, read and parsed during
|
||||
the review. All four recorded `pairs_unanswered=0`; these are historical artifacts, not fresh runs.
|
||||
WP Statistics's scope shrank after vendor removal. VAmPI's transitive SQLI target is excluded from
|
||||
the position budget, so the value is not verified all-target recall. A 72-minute whole-PMPro scan
|
||||
recorded in `flow-aware-ranking/brief.md` demonstrates a finishing scan, not hook feasibility or
|
||||
precision of its 250 entailed findings. The focused review run passed 87 existing tests; it did not
|
||||
rebuild Joern graphs or evaluate hook behavior.
|
||||
|
||||
The latest ranker head has 67 strictly higher-scored same-tier regions ahead of PMPro's CVE and
|
||||
85 ahead of WP Statistics's. Reordering equal scores alone cannot beat budgets 68 and 86. MW WP
|
||||
Form has 39 strictly higher-scored regions, so its corresponding lower bound is 40. These are
|
||||
arithmetic bounds from the stored head, not a measured new ranking algorithm.
|
||||
|
||||
### Existing seams
|
||||
|
||||
- `preprocess.py:preprocess_repository` reads filesystem bytes and then records HEAD; snapshot
|
||||
identity must be established before calling this path for a push.
|
||||
- `model/scan.py:scan_repository` builds a graph before applying `max_regions`. Its evidence order
|
||||
is experimental and `cli.py` currently does not enable it. Region budget is not graph cost.
|
||||
- `model/layout.py` supplies vendor exclusions and test/shipped classification. Preserve it.
|
||||
- `cpg/backend.py` has process-group cancellation, census and parser readability checks, but fresh
|
||||
temporary graphs and per-command rather than per-push deadlines.
|
||||
- `cli.py` samples unjudged paths by slicing the original region order; the new contract must return
|
||||
exact selected/deferred IDs from the ranker/driver instead of reconstructing them by count.
|
||||
- `model/specs.py:taint_specs` still flattens sanitizers across a language. Contributor task 5.6 is
|
||||
a prerequisite to admitting capabilities whose correctness depends on context-specific cleansing.
|
||||
- `reports.py:scan_exit_code` has broad scan policies; the hook needs an independent admission and
|
||||
coverage decision, without changing the meaning of ordinary scan output.
|
||||
- Docker, Compose and ops default to 4.0.623, whereas the evaluated baseline is 4.0.625.
|
||||
|
||||
### Git contract
|
||||
|
||||
Git supplies ref-update lines to pre-push and uses its exit status to allow or abort the operation.
|
||||
The design consumes the supplied object identities and handles creation/deletion explicitly; it
|
||||
does not infer the pushed tree from the checkout. [Official hook contract](https://git-scm.com/docs/githooks#_pre_push)
|
||||
|
||||
Use machine-readable diffs between recorded snapshots, with rename/deletion attribution and
|
||||
NUL-delimited paths. A chosen new-branch comparison base is recorded separately from a previous
|
||||
remote tip. [Official diff documentation](https://git-scm.com/docs/git-diff)
|
||||
|
||||
### Graph reuse and slicing
|
||||
|
||||
Joern documents data-flow and usage slices extracted from a CPG; data-flow slicing can itself be
|
||||
expensive. This does not establish cheap incremental updates to a changed source tree. The design
|
||||
therefore reuses exact compatible artifacts and rebuilds changed units, with latency as an explicit
|
||||
feasibility gate. [Official CPG slicing documentation](https://docs.joern.io/cpg-slicing/)
|
||||
|
||||
## Architecture Pattern Evaluation
|
||||
|
||||
| Option | Benefit | Limitation | Decision |
|
||||
|---|---|---|---|
|
||||
| Whole scan on every push with a better ranker | Reuses current driver | Graph build precedes rank selection; measured costs exceed a short hook | Baseline/control only |
|
||||
| Changed-file-only graph | Small apparent input | Can lose unchanged callers, guards, fields and dispatch context | Reject as a completeness claim |
|
||||
| Ranker scope + first-party graph reuse + total deadline | Preserves current architecture and explicit vendor boundary | Changed graph units still rebuild; speed must be measured | Proposed |
|
||||
| New incremental graph engine or mandatory daemon | Potential reuse across edits | Unverified backend semantics and additional lifecycle complexity | Defer until measurements require it |
|
||||
|
||||
## Design Decisions
|
||||
|
||||
1. **Preserve scope ownership.** Change context is input to the existing ranker, not a new selector.
|
||||
The push layer executes its decisions and records deferred work. Vendor code never returns as
|
||||
a side effect of context expansion. Unknown vendor behavior uses declared summaries or remains unresolved.
|
||||
2. **Separate claim from interruption.** Existing rungs describe analysis evidence. Hook admission
|
||||
additionally requires evaluated capability, change attribution and actionability. This meets the
|
||||
user's low-noise requirement without pretending an LLM confidence score proves a bug.
|
||||
3. **Compare evidence, not just finding lists.** A missing base finding can mean truncated analysis.
|
||||
Targeted comparable base checks are required for novelty, including removed-guard cases.
|
||||
4. **Adopt Git and the existing engine.** Build only the push transaction, cache and admission adapters.
|
||||
Do not add an IR, database or service merely to describe future scale.
|
||||
5. **Measure honest warm work.** Repeating an identical tip is a cache-hit control; warm changed-code
|
||||
pushes are the product workload. Separate both from cold builds and dependency/configuration changes.
|
||||
|
||||
## Risks and review decisions
|
||||
|
||||
- Proposed 30-second target and enforcement defaults await review; neither is inferred user approval.
|
||||
- Exact graph reuse may help repeated pushes more than everyday edits. The feasibility experiment
|
||||
must expose this and can stop rollout; timeout notices alone do not make a useful safety net.
|
||||
- Initial high-precision admission can lower recall. Joint recall/coverage/noise gates prevent
|
||||
suppressing everything from masquerading as success.
|
||||
- Unknown dynamic and cross-language edges remain real limits; ranking cannot manufacture context.
|
||||
- Existing inspected plugins are not fresh holdouts. Fix scope and evaluation before another tuning cycle.
|
||||
|
||||
## Review record
|
||||
|
||||
### Cross-language clarification
|
||||
|
||||
The maintainer explicitly retained PMPro as the core optimization example and requested transfer
|
||||
to common JavaScript/Node APIs (Express in particular), with C/C++ as a useful additional example.
|
||||
Added requirement 8 and the Language and Framework Contract to make agnostic analysis testable.
|
||||
`evaluation.md` separates development, framework integration, untouched evaluation and C envelope
|
||||
roles. NodeGoat is a proposed Express integration candidate, not a pinned or measured workload.
|
||||
|
||||
Local inspection found existing request sources and eval/exec/query/path/outbound sinks in
|
||||
`ruleset/semantic/javascript.toml`, JavaScript obligation facts, and Express-style route/middleware
|
||||
mapping in `mapping.py`. This establishes reusable implementation seams, not passing end-to-end
|
||||
coverage. The five current repository recipes contain PHP, Python and C workloads but no JavaScript
|
||||
recipe. Existing `libpng.toml` explicitly marks its bounds CVE out of scope and measures only envelope.
|
||||
That distinction is preserved. No new core algorithm, frontend or recipe is implemented here.
|
||||
|
||||
Fresh amendment validation read the six current spec documents, confirmed 35 unique EARS criteria
|
||||
with explicit design traceability, parsed the amended metadata, and checked the cross-language and
|
||||
vendor contracts. The command and `git diff --check` both exited 0. This validates documents only;
|
||||
Node/Express transfer remains the proposed experiment, not a measured result.
|
||||
|
||||
### Earlier validation
|
||||
|
||||
Fresh document validation read all five spec files, parsed metadata, checked all 31 unique EARS
|
||||
criteria against design traceability, and confirmed populated boundary/file/testing sections.
|
||||
An initial mechanical check rejected two criteria starting with "Before"; both were rewritten as
|
||||
event-driven EARS statements and the repeated check passed (exit 0). `git diff --check` passed.
|
||||
Existing contributor/ranker approval fields were compared with HEAD and remained unchanged;
|
||||
new requirements/design approvals remain false. Stored ranking-head arithmetic independently
|
||||
reproduced the equal-score tie bounds of 68, 86 and 40.
|
||||
|
||||
The content review checked boundary ownership, failure handling, vendor exclusion, ranker scope,
|
||||
and independent finding/coverage/push status. Draft defaults remain clearly labeled. Runtime
|
||||
integration, performance and capability admission remain unverified work; no code tests were
|
||||
needed for these documentation-only amendments.
|
||||
@@ -0,0 +1,54 @@
|
||||
{
|
||||
"feature_name": "pre-push-safety-net",
|
||||
"created_at": "2026-09-12T15:35:00Z",
|
||||
"updated_at": "2026-09-12T16:20:40.814596Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true,
|
||||
"implementation_approval_note": "2026-09-12: maintainer explicitly said 'approve.' in response to approval of the generated task plan and starting implementation with group 1. All tasks are approved; this implementation run starts with tasks 1.1\u20131.4. No task is complete until its fresh-evidence and reviewer gates pass.",
|
||||
"depends_on": [
|
||||
"contributor-scan"
|
||||
],
|
||||
"related_specs": [
|
||||
"flow-aware-ranking",
|
||||
"zero-with-a-reason",
|
||||
"finding-feedback-loop"
|
||||
],
|
||||
"approval_note": "2026-09-12: user requested amendments and help moving the design forward for a scalable pre-push security safety net, broader than WordPress; clarified that advisory output must be actionable and not SAST noise. Requirements and a proposed design were drafted together under that explicit request. No requirements, design, tasks, enforcement policy, or latency target are marked approved. Phase remains requirements-generated until human review. Existing predecessor approvals are unchanged. ; 2026-09-12: after the amended requirements/design and cross-language evaluation were presented, the maintainer said \"ok, next kiro phase\". Requirements and design are now approved for task generation; tasks remain unapproved. Earlier unapproved statements above describe the historical drafting state.",
|
||||
"cross_language_note": "2026-09-12: maintainer endorsed the direction and clarified PMPro as core development case, JavaScript/Express Node APIs as required transfer workload, and C/C++ as additional shared-machinery/abstraction-limit evidence. Preserve language-agnostic core decisions; framework facts may differ. Detailed amended requirements/design remain pending review.",
|
||||
"approval_history": [
|
||||
{
|
||||
"at": "2026-09-12T15:56:00Z",
|
||||
"source": "ok, next kiro phase",
|
||||
"action": "Accepted the presented amended requirements and design for task generation; task approval and implementation remain pending."
|
||||
},
|
||||
{
|
||||
"at": "2026-09-12T16:20:40.814596Z",
|
||||
"source": "approve.",
|
||||
"action": "Accepted the generated task plan and authorized starting group 1 (1.1\u20131.4). Implementation started; tasks require independent review and fresh verification before completion."
|
||||
}
|
||||
],
|
||||
"task_review": {
|
||||
"verdict": "PASS",
|
||||
"independent": true,
|
||||
"repair_passes": 1,
|
||||
"major_groups": 8,
|
||||
"executable_subtasks": 27,
|
||||
"requirements_covered": 35,
|
||||
"record": "task-review.md"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
# Task-plan review: pre-push-safety-net
|
||||
|
||||
Date: 2026-09-12. Scope: design readiness and implementation task graph, not runtime validation.
|
||||
|
||||
## Authorization and phase
|
||||
|
||||
After the amended requirements/design and JavaScript transfer plan were presented, the maintainer
|
||||
said **"ok, next kiro phase"**. This was applied as approval of those concrete documents for task
|
||||
generation. Requirements/design are approved; generated tasks remain unapproved and implementation
|
||||
has not begun. The initial acceptance profile is an evaluation target, not a demonstrated SLA.
|
||||
|
||||
## Design readiness assessment
|
||||
|
||||
**GO for task generation.** The design retains the existing ranker as scope owner, excludes vendor
|
||||
code from targets/graphs, preserves language-agnostic evidence contracts, and separates graph
|
||||
claims from actionable change-related alerts. Boundaries, integration paths and all 35 numeric
|
||||
requirements have concrete design counterparts. No fundamental design gap was found in this review.
|
||||
|
||||
Three material risks are explicit work and release gates, not assumed solved:
|
||||
|
||||
1. Whole/changed-unit graph cost may not meet the hook deadline: measure the uncached baseline
|
||||
before cache work, then realistic warm edits rather than identical-tip hits alone (5.2, 8.2).
|
||||
2. Frontend/framework or sanitizer semantics may prevent an actionable claim: retain the gap,
|
||||
keep the affected capability experimental, and route new engine work to its owning spec (3.3, 8.3).
|
||||
3. Repeatedly inspected examples do not prove transfer: freeze the core and retain untouched
|
||||
evidence per language/framework/family, with noise/recall/coverage gates together (1.2, 8.3).
|
||||
|
||||
## Independent task-graph sanity review
|
||||
|
||||
Reviewer received the draft task plan and paths to requirements, design, evaluation and task rules,
|
||||
without a parent-generated coverage summary. The first verdict was **NEEDS_FIXES**:
|
||||
|
||||
- Cache reuse can complete more work within the same deadline. Task 6.4 must require evidence/
|
||||
admission equality for the same completed questions; complete-scope equality belongs to a controlled
|
||||
sufficient-budget comparison, not every deadline-limited run.
|
||||
- Requirement 8.1 needs explicit paired normalized PHP/JavaScript contract tests, including equivalent
|
||||
capability eligibility and an unsupported-origin negative control (3.2, 4.2).
|
||||
- A Node holdout cannot admit PHP or Python capabilities. Untouched evaluation is required for each
|
||||
language/framework/family; missing evidence leaves that capability experimental (1.2, 8.3).
|
||||
|
||||
All three were repaired once. The same independent reviewer read the repaired draft and returned
|
||||
**PASS**, with no remaining important task-graph issue. No return to design was required.
|
||||
|
||||
## Mechanical and scope checks
|
||||
|
||||
- Eight major groups and 27 executable subtasks; one parallel candidate (3.1), otherwise sequential
|
||||
because the work shares backend, ranker, cache and integration contracts.
|
||||
- All 35 requirement IDs mapped; every executable subtask has an observable completion condition,
|
||||
responsibility boundary, and valid explicit dependencies where needed.
|
||||
- No task is checked off; no acceptance-critical tests are optional. No deployment is included.
|
||||
- The initial experiment is 5.2; persistent caching follows it. Normal alert eligibility is gated
|
||||
at 8.3, so experimental diagnostics do not become noisy advisory output during development.
|
||||
- Final file/metadata validation and `git diff --check` are the fresh document checks. No runtime
|
||||
code or tests changed, and no graph/performance measurements were rerun during task generation.
|
||||
|
||||
Next phase: human task approval, then implementation starting with group 1.
|
||||
@@ -0,0 +1,217 @@
|
||||
# Implementation Plan: pre-push-safety-net
|
||||
|
||||
Status: task plan approved by the maintainer's "approve." on 2026-09-12. Implementation has
|
||||
started with group 1 (tasks 1.1–1.4); each task remains pending until independent review and fresh verification pass.
|
||||
|
||||
## Execution contract
|
||||
|
||||
- Preserve the ranker as the scope mechanism and vendor exclusion from targets AND graphs.
|
||||
- Use PMPro for core development and a JavaScript/Express workload for transfer. Keep the core
|
||||
frozen for the first Node measurement; separate adapter repairs from core changes.
|
||||
- Each executable subtask is a bounded work unit targeting 1–3 hours of engineering, excluding
|
||||
unattended measurement time. If a new abstract capability is needed, record the gap and return
|
||||
it to the owning engine/ranking spec; do not hide that implementation in an integration task.
|
||||
- Cross-spec engine changes below are explicit integrations of the accepted push contracts.
|
||||
They do not approve unrelated contributor/ranker backlog or automatically complete its tasks.
|
||||
- Development runs retain experimental candidate/admission diagnostics without enabling normal
|
||||
hook alerts. A capability becomes eligible for normal alerts only after task 8.3 passes its gates.
|
||||
- The first end-to-end milestone is 5.2, before persistent caching and hook installation polish.
|
||||
Negative performance evidence there informs caching; a broken instrument or absent supported
|
||||
positive control requires predecessor remediation before claiming that milestone successful.
|
||||
- Sequential order is the default. Only 3.1 may run alongside group 2 after its explicit prerequisite;
|
||||
shared backend/driver, cache and report edits otherwise remain serial. No optional acceptance tests.
|
||||
|
||||
- [ ] 1. Establish the runtime, evaluation inputs and shared contracts
|
||||
- [ ] 1.1 Reproduce the evaluated engine environment
|
||||
- Align the shipped image, Compose and host-install defaults with Joern 4.0.625; preserve version override and checksum verification.
|
||||
- Exercise PHP with the native interpreter and the installed JavaScript frontend; open a source and the frontend parser where applicable, recording byte counts and graph census.
|
||||
- Done when a local image smoke produces valid small PHP and JavaScript graph/query payloads with recorded versions, and an unreadable source fails explicitly. This builds locally; it does not deploy.
|
||||
- _Boundary: Cache and Engine Boundary, packaging integration_
|
||||
- _Requirements: 7.1, 7.3, 8.2_
|
||||
- [ ] 1.2 Prepare labeled development and reserved evaluation changes
|
||||
- Pin a readable PMPro change/fix and a supported Node/Express change/fix, plus benign controls; verify license and source bytes before accepting recipes.
|
||||
- Use NodeGoat only if its selected mechanism fits a current arbiter; label authored repairs as authored. Reserve an independently selected Node API repository with modern module/async context and reviewable history before tuning on transfer results.
|
||||
- Keep Python regression and libpng envelope recipes, explicit supported/unsupported populations, ref identities and development/holdout roles in machine-readable evaluation inputs.
|
||||
- Reserve untouched evaluation for each language/framework/family intended for admission; the Node holdout cannot qualify PHP or Python. Missing per-capability evaluation leaves it experimental and does not delay the initial PMPro/Node diagnostic experiment.
|
||||
- Done when recipe validation resolves immutable inputs and labels or names an explicit missing prerequisite; no favorable scanner result is required to select a recipe.
|
||||
- _Boundary: Evaluation harness_
|
||||
- _Requirements: 6.1, 6.3, 8.3, 8.4_
|
||||
- [ ] 1.3 Make the baseline and push scorer fail honestly
|
||||
- Retain unmatched targets and unanswered queries in the declared population; require a queried witness before labeling a target transitively detected.
|
||||
- Emit actual vulnerable/fixed outcomes beside rank, selected mode, engine/facts/scope and elapsed stage costs. Freeze a versioned workload/profile before measuring.
|
||||
- Done when a deliberately missing target and failed query cannot improve reported recall, while a verified transitive positive is counted with its witness.
|
||||
- _Boundary: Evaluation harness, repository benchmark integration_
|
||||
- _Requirements: 6.2, 6.3, 6.4, 6.5, 7.3_
|
||||
- [ ] 1.4 Establish typed push and engine integration contracts
|
||||
- Add importable push components and validated configuration for comparison base, shared deadline/cancellation allowance, local cache limit, advisory/blocking and separate coverage policy.
|
||||
- Define ownership of update/comparison, change context, selected/deferred scope, graph identity/lease and separate finding/coverage/push statuses as described in design.
|
||||
- Keep generic engine contracts outside the push package; leave ordinary scan defaults compatible and capabilities experimental until evaluated.
|
||||
- Done when contract/config round trips preserve all identities/statuses, invalid values are rejected, and the engine has no dependency on the push package.
|
||||
- _Boundary: Push runner, Cache and Engine Boundary, CLI/config integration_
|
||||
- _Requirements: 4.1, 4.3, 5.3, 5.4, 5.5, 7.1, 7.3_
|
||||
|
||||
- [ ] 2. Resolve immutable pushed content
|
||||
- [ ] 2.1 Resolve every supplied ref update and comparison
|
||||
- Parse Git's complete update input using object-format-aware validation; resolve existing tips, new-branch configured bases, force-pushes, merges, peeled tags, deleted refs and unsupported objects.
|
||||
- Deduplicate equal targets without merging different base comparisons; retain every ref association. Do not silently substitute HEAD or fetch missing objects.
|
||||
- Done when multi-ref and missing-base fixtures produce exact comparison identities and explicit dispositions, including deletion-only no-analysis results.
|
||||
- _Boundary: Snapshot Adapter_
|
||||
- _Requirements: 1.1, 1.3, 1.4, 7.1_
|
||||
- [ ] 2.2 Materialize isolated object snapshots
|
||||
- Read immutable tracked blobs into bounded scratch storage using machine-readable paths; do not checkout into the live tree or run filters, project scripts or network requests.
|
||||
- Preserve path identity and report symlinks, gitlinks, unavailable LFS blobs and unsupported content; never follow source outside the snapshot.
|
||||
- Done when non-HEAD pushes from a dirty checkout read only the intended object bytes and leave index, files and refs byte-identical after success, cancellation and failure.
|
||||
- _Boundary: Snapshot Adapter_
|
||||
- _Requirements: 1.1, 1.2, 1.3, 2.3, 7.1, 7.3_
|
||||
- [ ] 2.3 Attach changed spans and deletion/rename context
|
||||
- Compare recorded base/head objects using NUL-safe path handling; retain removed spans, renamed identities and declaration/config changes.
|
||||
- Supply change context to the shared ranker contract rather than selecting an independent changed-files-only scope.
|
||||
- Done when renamed functions retain correspondence and a removed guard remains an impact seed for an unchanged operation, with missing correspondence explicitly unresolved.
|
||||
- _Boundary: Snapshot Adapter_
|
||||
- _Requirements: 1.4, 2.1, 2.4, 3.4_
|
||||
|
||||
- [ ] 3. Integrate bounded execution and ranker scope with the existing engine
|
||||
- [ ] 3.1 (P) Enforce one remaining-time budget in the backend
|
||||
- Thread the absolute deadline through readability probes, frontend builds, overlays, census, query batches, fallbacks and retries; no phase gets a fresh full budget.
|
||||
- Bound process-group termination and lease/scratch cleanup by the cancellation allowance, replacing any longer fixed cleanup waits on the push path.
|
||||
- Done when controlled hanging subprocesses at build/overlay/query stages terminate without orphan children inside the declared allowance, while ordinary scans preserve their prior behavior.
|
||||
- _Boundary: Cache and Engine Boundary, generic driver integration_
|
||||
- _Depends: 1.4_
|
||||
- _Requirements: 4.1, 4.2, 7.3_
|
||||
- [ ] 3.2 Return the ranker's exact selected and deferred work
|
||||
- Integrate the existing ranker through explicit mode selection and shared question identity; record selected/deferred IDs, evidence and reasons directly from the final order.
|
||||
- Preserve unanswered evidence as unknown rather than tier zero; report per-family coverage so taint ranking cannot masquerade as authorization/configuration coverage.
|
||||
- Add paired PHP/JavaScript normalized-evidence contract tests asserting identical tier/score and selection decisions for equivalent input and budgets, without repository/language priority exceptions.
|
||||
- Done when changing order produces matching execution and manifest IDs, with no reconstruction from an earlier list, and normalized cross-language equivalence tests pass.
|
||||
- _Boundary: Ranker Scope Contract, generic driver integration_
|
||||
- _Requirements: 2.1, 2.2, 6.4, 7.3, 8.1, 8.2_
|
||||
- [ ] 3.3 Connect change context to supported first-party relationships
|
||||
- Use existing entry/callee, field, hook and guard relationships to supply affected context to ranked questions, including unchanged sinks after source/guard changes.
|
||||
- Preserve current abstract semantics; absent dynamic or external relationships remain unresolved rather than guessed safe. A required new feature or fact is reported to its owning spec.
|
||||
- Done when controlled cross-file and removed-guard cases reach the relevant question/witness, while an unresolved boundary cannot produce a complete negative.
|
||||
- _Boundary: Ranker Scope Contract, Snapshot Adapter integration_
|
||||
- _Depends: 2.3, 3.2_
|
||||
- _Requirements: 2.1, 2.2, 2.4, 8.1, 8.2_
|
||||
- [ ] 3.4 Apply vendor exclusion and explicit language partitions
|
||||
- Reuse declared layout exclusions before target enumeration and graph build; retain unshipped first-party tests and separate browser/server or language capability coverage.
|
||||
- Build supported frontend partitions explicitly; preserve unknown cross-language and vendor semantics without restoring excluded sources.
|
||||
- Done when PHP and Node graphs/targets contain no declared vendor code, first-party callers remain visible, and unsupported partitions appear as coverage limits rather than clean results.
|
||||
- _Boundary: Ranker Scope Contract, Cache and Engine Boundary integration_
|
||||
- _Requirements: 2.2, 2.3, 7.3, 8.2, 8.4_
|
||||
|
||||
- [ ] 4. Implement change attribution and actionable output
|
||||
- [ ] 4.1 Compare targeted base/head evidence for defect novelty
|
||||
- Establish stable mechanism/operation identity with rename/span mapping and collect comparable base evidence under the same semantics and remaining deadline.
|
||||
- Distinguish new, worsened, unchanged and unknown; an absent finding from a failed/truncated base is unknown. Support removed guards and new source connections at unchanged sinks.
|
||||
- Done when benign movement and unchanged backlog do not become new defects, while a supported removed-guard case has a concrete change-to-witness relationship.
|
||||
- _Boundary: Delta and Actionability Policy, engine comparison integration_
|
||||
- _Depends: 2.3, 3.3, 3.4_
|
||||
- _Requirements: 2.4, 3.1, 3.4, 4.1_
|
||||
- [ ] 4.2 Enforce capability and actionability admission
|
||||
- Require evaluated capability provenance, applicable semantics, a sufficient rung, exact witness/location, supported change attribution and a concrete repair direction.
|
||||
- Keep experimental, suspicious, wrongly sanitized and unknown-novelty candidates in diagnostic records; do not let model confidence, a sanitizer name or printed-before state admit/dismiss a defect.
|
||||
- Deduplicate by defect while retaining relevant witnesses and refs; distinguish experimental evaluation from normal hook eligibility.
|
||||
- Add equivalent PHP/JavaScript normalized claims with equally eligible capabilities and assert equal admission; add an unsupported-origin control whose coverage/eligibility differs without changing core scores.
|
||||
- Done when the policy rejects an entailed but unsupported claim and admits fully supported synthetic controls equivalently across origins and advisory/blocking modes; real capabilities remain disabled pending 8.3.
|
||||
- _Boundary: Delta and Actionability Policy_
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 5.3, 5.4, 8.1_
|
||||
- [ ] 4.3 Render compact results and preserve complete artifacts
|
||||
- Render up to three actionable summaries plus remaining count; emit at most one no-finding status and one aggregated incomplete notice with a useful next action.
|
||||
- Persist all defects, candidate dispositions, exact scope, identities, capability/ranker provenance and timings with separate finding/coverage/push status.
|
||||
- Done when the display cap does not change the push decision, incomplete analysis cannot read as secure, and artifact-write failure produces a bounded notice while retaining the real decision.
|
||||
- _Boundary: Result and Hook Adapter_
|
||||
- _Requirements: 3.1, 3.2, 3.3, 5.1, 5.2, 5.5, 7.3_
|
||||
|
||||
- [ ] 5. Prove the first end-to-end experiment before cache optimization
|
||||
- [ ] 5.1 Wire explicit base/head replay through the full transaction
|
||||
- Connect immutable snapshots, ranker decisions, existing arbiter, targeted comparison, admission diagnostics and compact artifacts through an explicit local replay command.
|
||||
- Apply the shared budget to preparation and both revisions; no model/service or installed hook is required. All runtime errors preserve coverage state.
|
||||
- Done when a controlled security change and fixed twin traverse the production runner through the existing ranker and produce replayable results with truthful deadlines.
|
||||
- _Boundary: Push runner, CLI, Snapshot Adapter, engine and policy integration_
|
||||
- _Depends: 1.3, 3.1, 3.4, 4.3_
|
||||
- _Requirements: 1.1, 2.1, 3.1, 3.4, 4.1, 4.2, 5.5, 7.1, 7.3_
|
||||
- [ ] 5.2 Measure PMPro and frozen-core Node transfer
|
||||
- Replay each prepared security/fixed/benign case, first using the declared hook deadline; record timeouts and stage costs. A separately named extended lab run may obtain witnesses but cannot satisfy the hook latency gate.
|
||||
- Freeze the core after PMPro, run Node with existing facts, and classify instrument, mapping, fact, scope and domain gaps without tuning on an untouched holdout.
|
||||
- Done when one joint artifact reports actual evidence and outcomes for both workloads, with unresolved cases retained and an explicit feasibility decision; no normal capability is enabled here.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 1.2, 1.3, 5.1_
|
||||
- _Requirements: 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||
|
||||
- [ ] 6. Reuse valid graphs and results without stale conclusions
|
||||
- [ ] 6.1 Validate graph artifact identity and leases
|
||||
- Have the backend own source/declaration/exclusion digests, frontend/engine/overlay options, graph census and complete/partial identity.
|
||||
- Permit validated graph leases in the generic driver while preserving the fresh-build path; isolate potentially mutating loads and retain bounded cleanup.
|
||||
- Done when identical valid inputs can reuse a graph while any changed graph input or failed census rejects the lease; original cached artifacts survive query loading unchanged.
|
||||
- _Boundary: Cache and Engine Boundary, generic driver integration_
|
||||
- _Requirements: 4.2, 4.3, 4.4, 7.3_
|
||||
- [ ] 6.2 Publish and evict local artifacts safely
|
||||
- Add private temporary publication, per-key bounded locking, atomic completed manifests, cache size enforcement and unleased eviction.
|
||||
- Retain partial records only as diagnostics; never interpret a directory or failed write as a complete result. Do not rebuild graphs incrementally by mutating old node identities.
|
||||
- Done when concurrent writers, corrupt entries and interrupted publication yield either a validated complete artifact or an explicit miss without exceeding the remaining lock budget.
|
||||
- _Boundary: Cache and Engine Boundary, push cache integration_
|
||||
- _Depends: 6.1_
|
||||
- _Requirements: 4.1, 4.3, 4.4, 7.3_
|
||||
- [ ] 6.3 Key query and comparison results by all relevant semantics
|
||||
- Include graph identity, question/context/depth, facts/sanitizers/query versions and configuration; include ranking/admission/model provenance for final results and both revision identities for comparisons.
|
||||
- Keep incomplete query/comparison results out of complete lookups; preserve current eligibility checks after reuse.
|
||||
- Done when source, context, fact, query, mode, policy and optional model changes invalidate the appropriate layer, while an identical compatible request returns the same evidence and provenance.
|
||||
- _Boundary: Push cache, policy integration_
|
||||
- _Requirements: 3.4, 4.3, 4.4, 6.4, 7.2, 7.3_
|
||||
- [ ] 6.4 Integrate cache reuse into the shared push deadline
|
||||
- Reuse identical snapshots/unchanged declared units across refs without merging different comparisons; changed units rebuild and stale graph relationships remain hints only.
|
||||
- Account for lookup, lock wait, lease, build/query and result persistence in the same deadline; preserve completed outcomes on cancellation.
|
||||
- Done when cached and uncached replay produce equal evidence/admission for the same completed questions under compatible semantics, with full equivalence under a sufficiently large controlled budget. Under the hook deadline, verify truthful extra completion/deferred scope from cache reuse rather than requiring identical scope; no stale answer survives a dependency/configuration edit.
|
||||
- _Boundary: Push runner, cache and engine integration_
|
||||
- _Depends: 5.1, 6.2, 6.3_
|
||||
- _Requirements: 1.1, 2.2, 4.1, 4.2, 4.3, 4.4, 7.3_
|
||||
|
||||
- [ ] 7. Connect the real push interface without adding noise
|
||||
- [ ] 7.1 Execute Git's full push input and enforcement policy
|
||||
- Extend the replay command to consume all update lines and remote arguments, aggregating per-ref comparisons under one deadline and preserving independent coverage and finding statuses.
|
||||
- Apply advisory, opt-in blocking and separate incomplete-coverage policy; keep external services disabled by default and normal alerts limited to admitted capabilities.
|
||||
- Done when controlled multi-ref push invocations exercise all allow/block/status combinations, including deletion-only, missing-base and no-work cases, without resetting budgets.
|
||||
- _Boundary: CLI/config, Push runner and Result and Hook Adapter integration_
|
||||
- _Depends: 2.1, 4.2, 4.3, 6.4_
|
||||
- _Requirements: 1.1, 1.3, 1.4, 4.1, 5.1, 5.2, 5.3, 5.4, 5.5, 7.1_
|
||||
- [ ] 7.2 Provide non-destructive hook integration and removal
|
||||
- Supply an executable integration snippet/installation path with explicit invocation, preserving existing hooks, hook path configuration, remote arguments and stdin for other consumers.
|
||||
- Refuse automatic overwrite; retain an existing hook's nonzero result even when the Safety Net runs advisory. Include concrete integration/removal instructions with the delivered hook interface.
|
||||
- Done when a disposable Git remote accepts/rejects pushes correctly with an existing stdin-reading hook, and removal restores the preceding behavior without touching source or refs.
|
||||
- _Boundary: Result and Hook Adapter, CLI integration_
|
||||
- _Requirements: 1.2, 5.3, 5.4, 7.4_
|
||||
- [ ] 7.3 Keep optional model assistance within existing evidence rules
|
||||
- Reuse explicit endpoint configuration and redaction only when enabled; propagate the remaining deadline to model calls and record model/prompt/cost provenance.
|
||||
- Restrict explanations to existing witnesses and preserve deterministic no-model behavior; do not implement a new learned ranker or prompt optimizer in this task.
|
||||
- Done when a delayed or misleading model cannot extend the push budget, leak unredacted input in the tested path, raise a rung or bypass actionability admission.
|
||||
- _Boundary: Push runner, existing model and policy integration_
|
||||
- _Requirements: 3.5, 4.1, 4.2, 7.1, 7.2, 7.3_
|
||||
|
||||
- [ ] 8. Validate usefulness, runtime and cross-language admission
|
||||
- [ ] 8.1 Exercise complete push failure and regression paths
|
||||
- Combine dirty/non-HEAD/multi-ref/new-branch/force/tag/deletion pushes with unusual paths, external content boundaries, failed census, missing comparisons and interrupted persistence.
|
||||
- Verify no orphan children, no overwritten live state, no vendor code in graphs, exact scope reports and no false clean result from skipped work.
|
||||
- Done when the integration suite covers the critical flows through the real CLI/backend seams and preserves ordinary scan behavior.
|
||||
- _Boundary: Push runner integration tests_
|
||||
- _Requirements: 1.1, 1.2, 1.3, 1.4, 2.2, 2.3, 4.2, 4.4, 5.2, 5.5, 7.1, 7.4, 8.2_
|
||||
- [ ] 8.2 Measure realistic cache and deadline feasibility
|
||||
- Measure identical-tip reuse, a warm one-function edit, dependency/configuration edits, cold starts, repository growth and multi-ref workloads on declared hardware.
|
||||
- Publish p50/p95, timeouts, cancellation allowance, completed/deferred population and stage costs; the denominator is fixed before ranker selection.
|
||||
- Done when the versioned scorecard evaluates the accepted latency/coverage gates without hiding timeouts or calling repeated-identical-tip hits representative warm edits; a failed gate remains NO-GO for rollout.
|
||||
- _Boundary: Evaluation harness, runtime integration_
|
||||
- _Requirements: 4.1, 4.2, 6.2, 6.3, 6.5, 7.3_
|
||||
- [ ] 8.3 Evaluate independent transfer and admit only passing capabilities
|
||||
- Run the frozen candidate on reserved Node changes and benign/fixed controls; preserve PHP/Python regressions, including a verified VAmPI transitive target, and report libpng envelope/unsupported bounds separately.
|
||||
- Score actual actionable alerts, reviewed precision with counts/intervals, supported recall, benign interruptions and fixed-side silence alongside latency and coverage. Record adapter changes separately; a core retune requires a new untouched evaluation.
|
||||
- Generate versioned capability eligibility only when all predeclared gates pass; no positive controls, no reviewed alerts, no untouched workload, wrong family semantics or unsupported properties prevent admission.
|
||||
- Require untouched evaluation for the particular language/framework/family being admitted. Node results cannot qualify PHP/WordPress or Python; capabilities without their own untouched evidence remain experimental with an explicit disposition.
|
||||
- Done when the policy consumes a reproducible PASS/NO-GO capability artifact tied to exact semantics and rejects stale eligibility; a NO-GO result never silently enables advisory alerts.
|
||||
- _Boundary: Evaluation harness, Delta and Actionability Policy integration_
|
||||
- _Depends: 1.2, 1.3, 5.2, 7.3, 8.2_
|
||||
- _Requirements: 3.1, 3.2, 6.1, 6.2, 6.3, 6.4, 6.5, 8.1, 8.2, 8.3, 8.4_
|
||||
- [ ] 8.4 Verify the packaged end-to-end outcome and acceptance coverage
|
||||
- Run the required regression checks and a local packaged smoke over supported security/fixed/benign pushes; verify no-model startup, preservation of hook behavior and truthful partial results.
|
||||
- Reconcile every acceptance criterion with fresh evidence and record unresolved capability/performance blockers separately from completed plumbing.
|
||||
- Done when a reproducible validation artifact states GO only with all mandatory integration and joint quality/latency gates satisfied; otherwise it names the remaining NO-GO conditions without marking the feature complete.
|
||||
- _Boundary: Packaged runtime, full feature validation integration_
|
||||
- _Requirements: 3.1, 3.4, 5.1, 5.2, 5.3, 5.4, 5.5, 6.5, 7.1, 7.3, 7.4, 8.3, 8.4_
|
||||
Reference in New Issue
Block a user