marius/Open_Ultra_SAST
1
0
Fork 0
mirror of https://github.com/norandom/OpenUltraSAST.git synced 2026-10-16 03:49:26 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs(agents): verify the instrument before believing the measurement

Browse Source 
Makes the day's lesson durable rather than a note in a transcript. Every wrong
diagnosis this project has recorded has one shape: the input was unreadable and
the tool reported a plausible ZERO. file_exists() false, the parser writes
nothing, the frontend exits 0 -- and "no errors, 0 findings" reads exactly like
a clean result. Four wrong root causes in one day, two in an earlier session,
twice from the same php shim.

Four rules, each earned:

  1. Prove the tool read its input -- open one file, print its size. Afterwards,
     a zero from an unread tree and a zero from clean code are identical.
  2. Distrust a number that is too good. A joern script finishing in 0.5s when
     a JVM takes 15s to start is a failed launch, and a harness must say so with
     the exit code rather than print `rows=?` beside a time.
  3. Prefer the shipped environment; local wrappers that proxy stdio are not
     valid instruments for php2cpg, which depends on stream ordering.
  4. When a result contradicts a working baseline, suspect the instrument.
     Slices that found CVEs an hour ago do not stop working because a repository
     got bigger.

The local shim is now fixed rather than merely condemned: merging stderr into
stdout INSIDE the container gives 0 drops and a 2,872,996-byte graph,
byte-identical to a native interpreter across runs. The reasoning is written
into the shim so it survives the next person who tries to simplify it.

853 tests pass, 8 skipped; gates byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
Marius Ciepluch
2026-09-10 11:38:31 +02:00
co-authored by Claude Opus 5
parent 07df314e12
commit 128950a586
1 changed files with 18 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+18
View File
@@ -61,6 +61,24 @@ Skills are located in `.opencode/skills/kiro-*/SKILL.md`
- Keep steering current and verify alignment with `/kiro-spec-status`
- Follow the user's instructions precisely, and within that scope act autonomously: gather the necessary context and complete the requested work end-to-end in this run, asking questions only when essential information is missing or the instructions are critically ambiguous.
### Verify the instrument before you believe the measurement
Every wrong diagnosis this project has recorded came from the same shape: **the input was unreadable and the
tool reported a plausible zero.** `file_exists()` false, the parser writes nothing, the frontend exits 0 —
and "no errors, 0 findings" reads exactly like a clean result. It has cost four wrong root causes in one day
and two in an earlier session, twice from the same `php` shim.
So, before reporting any measurement:
1. **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree
and a zero from clean code are indistinguishable afterwards. `JoernBackend` now does this itself; a
scratch harness must do it too.
2. **Distrust a number that is too good.** A joern script "finishing" in 0.5s when a JVM takes 15s to start
is a failed launch. A harness must fail loudly with the exit code, never print `rows=?` next to a time.
3. **Prefer the shipped environment.** The Docker image has a native `php`; local wrappers that proxy stdio
are not valid instruments for `php2cpg`, which depends on stream ordering.
4. **When a result contradicts a working baseline, suspect the instrument first.** Slices that found CVEs an
hour ago do not stop working because a repository is larger.
## Steering Configuration
- Load entire `.kiro/steering/` as project memory
- Default files: `product.md`, `tech.md`, `structure.md`