mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-19 21:19:28 +00:00
docs(pre-push-safety-net): record approved design and implementation plan
This commit is contained in:
@@ -0,0 +1,75 @@
|
||||
# Brief: pre-push-safety-net
|
||||
|
||||
Status: requirements, design and tasks approved on 2026-09-12; group 1 implementation started. Language: English.
|
||||
|
||||
## Problem
|
||||
|
||||
AI increases the rate of code changes. Developers need a bounded pre-push check that catches
|
||||
actionable security regressions without a recurring queue of speculative SAST warnings. The
|
||||
maintainer explicitly rejects noisy advisory output as well as noisy blocking output.
|
||||
|
||||
## Current State
|
||||
|
||||
The scan driver and Joern arbiters exist, with demonstrated PHP/Python cases. There is no pre-push
|
||||
snapshot resolver, change context for the ranker, persistent graph cache, or end-to-end deadline.
|
||||
The driver builds a whole-root graph before cutting the region budget. The latest PMPro ranking
|
||||
experiment alone spends about 130 seconds building and 296 seconds extracting evidence. A manually
|
||||
selected two-file slice finding a CVE does not demonstrate automatic incremental analysis.
|
||||
|
||||
## Desired Outcome
|
||||
|
||||
A developer pushes committed code and receives either a concise actionable security finding or a
|
||||
bounded result with explicit coverage. Unrelated working-tree changes, backlog findings, and model
|
||||
speculation do not become push warnings. The same analysis can be invoked explicitly for diagnosis.
|
||||
|
||||
## Approach
|
||||
|
||||
Resolve pushed snapshots, feed changed and affected first-party context to the existing ranker,
|
||||
reuse compatible results, and run admitted detector capabilities within a total deadline. The ranker
|
||||
remains the scope mechanism. Exclude third-party vendor code from targets AND graphs as already decided.
|
||||
Compare relevant base/head evidence
|
||||
before calling a defect new. Apply a separate actionability policy after arbitration. Evaluate on
|
||||
security and benign changes across repositories/frameworks, with cold and warm caches separated.
|
||||
|
||||
## Scope
|
||||
|
||||
- In: resulting pushed tips, affected context, caching, bounded execution, change-related actionable
|
||||
alerts, coverage status, optional enforcement, evaluation and reproducible local artifacts.
|
||||
- Out: every historical commit introduced by a push, whole-repository certification, style rules,
|
||||
dependency-vulnerability inventory, automatic source edits, mandatory cloud/LLM services,
|
||||
prompt optimization, and new memory-safety abstractions.
|
||||
|
||||
## Boundary Candidates
|
||||
|
||||
One product boundary owns the pre-push transaction from Git input to user outcome. It consumes the
|
||||
existing ranker's scope and analysis engine through explicit scope/budget/results contracts. Reusable ranking and
|
||||
family semantics remain with their existing specs rather than being reimplemented in the hook.
|
||||
|
||||
## Upstream / Downstream
|
||||
|
||||
- Upstream: `contributor-scan` engine/packaging/family correctness; existing facts and rungs.
|
||||
- Adjacent: `flow-aware-ranking` schedules work; `zero-with-a-reason` diagnoses missing work;
|
||||
`finding-feedback-loop` owns durable human review and later learning.
|
||||
- Downstream: developer pre-push workflow and explicit local/CI replay of the same snapshot contract.
|
||||
|
||||
## Constraints and Initial Evaluation Targets
|
||||
|
||||
- User direction: actionability and low noise are mandatory in every mode.
|
||||
- Proposed rollout: advisory first; blocking is explicit opt-in after per-capability validation.
|
||||
- Proposed latency experiment: 30-second warm-push p95 on a declared reference workload/hardware,
|
||||
with a configurable deadline and 2-second cancellation/report allowance. This is an unproven
|
||||
design target, not a demonstrated SLA. Cold runs use the same deadline and disclose incompleteness.
|
||||
- No promise of all-language coverage; capabilities are admitted per language/framework/family.
|
||||
- Initial work must establish feasibility and noise before investing in an LLM ranker.
|
||||
|
||||
## Cross-language amendment — 2026-09-12
|
||||
|
||||
Keep PMPro as the primary worked optimization example, and require a JavaScript/Express/Node API
|
||||
transfer experiment before claiming the core improvement generalizes. Abstract evidence, ranking,
|
||||
scope and arbitration contracts remain shared. Language/frontend/framework facts supply concrete
|
||||
semantics; missing semantics remain a stated limit. Do not add PHP-specific core ranking shortcuts.
|
||||
|
||||
See `evaluation.md` for the proposed workload matrix: NodeGoat as an Express wiring/teaching
|
||||
candidate, a separate untouched Node API repository, existing Python regressions, and libpng as
|
||||
a C envelope/limit control. C/C++ detection is measured only for a labeled property the chosen
|
||||
abstraction can decide; bounds reasoning is not inferred from successful taint analysis.
|
||||
Reference in New Issue
Block a user