mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 03:49:26 +00:00
chore(skills): apply the 2026-10-02 review -- current CLI and evidence, merges, removals, steering names
openultrasast-scan: evidence ladder as verification.py defines it, with the levels nothing produces today; --mode prerequisites, --fail-on, recorded degradations, run artifacts, ousast mcp; retired [evidence] threshold language removed. openultrasast-triage: reasons verified against calibration.py; deterministic fusion; adjudications as labels, plane verdicts as features, rule changes through improve or a registered experiment; fix-audit merged as a fix-and-re-scan section. kiro: spec-quick and spec-batch removed; validate-gap and validate-design merged into spec-design (--gap, --review; the review gate stays); steering-custom merged into steering; core steering names overview.md, roadmap.md, safety-net.md; instrument rule in debug and verify-completion; repo gates named in validate-impl. AGENTS.md, README skills paragraph and tests/test_agent_skills.py (.agents/skills, two product skills, drift tests) updated. Co-Authored-By: Claude Fable 5.1 <[email protected]> Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo
This commit is contained in:
co-authored by
Claude Fable 5.1
parent
5010b15ffc
commit
16d9ef7110
@@ -55,6 +55,14 @@ Extract:
|
||||
- The command that produced the failure
|
||||
- Whether the failure is deterministic or intermittent
|
||||
|
||||
### 1b. Verify the Instrument Before You Believe the Measurement
|
||||
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
|
||||
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
|
||||
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
|
||||
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
|
||||
- **When a result contradicts a working baseline, suspect the instrument first.**
|
||||
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
|
||||
|
||||
### 2. Inspect Local Runtime and Repository State
|
||||
Inspect the repository for local evidence:
|
||||
- `package.json`, `pyproject.toml`, `go.mod`, `Makefile`, `README*`
|
||||
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
|
||||
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
|
||||
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
|
||||
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
|
||||
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
|
||||
|
||||
## Output Format
|
||||
|
||||
|
||||
Reference in New Issue
Block a user