measure(independent): evaluation of population v2 under protocol v2 -- fails every M4 gate

Browse Source

The analyzer was frozen at a75b9f6. 17 cases, 102 scans (vulnerable and fixed pins
twice each), scored exactly as pre-registered.
- Recall: 0/17 (Wilson 95% 0-0.18). No unstable detections.
- Precision: 0/17 sampled (0-0.18). Every sampled finding is false: regex
  match/exec read as sinks, int casts, ORM statements, client-SDK code, the
  process's own argv, a generated database name, a registered-URI check.
- Completion: 13206/108374 questions (12%).
- Fixed-side alerts: 0, trivially, since nothing was detected.
- Benign alerts: 3.
- Stability: 2 of 100 findings differ between runs.
The zero is not an instrument artefact: FUXA, GoPay, cct-studio, ReactPress, User
Registration and Directorist completed questions in the sink file itself and still
found nothing.
Instrument failures, reported as failures: LMDeploy (every taint batch failed on all
six pins) and Froxlor's two benign scans.

This record is committed before any analyzer change informed by v2.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01EmM1j46qgKMh3zqD1QMwEo

...

This commit is contained in:

Marius Ciepluch

2026-09-28 08:52:53 +02:00

co-authored by Claude Opus 5.5

parent a75b9f62f9

commit 21adc408f1

1 changed files with 1205 additions and 0 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

benchmarks/independent/results-v2.json

+1205

File diff suppressed because it is too large Load Diff