Open_Ultra_SAST/benchmarks/measurements/2026-10-13-access-control-screen/NOTES.md

Access-control feature table and separation screen, no model calls (2026-10-13/14)

What was built

table.py checks out the head of every change of the frozen access-control training pool (digest 53f9f364: 5 fix pushes with 48 verified locations, 100 ordinary pushes, 5 repositories), enumerates the candidate rows (functions the push touched, plus the verified fix sites of a positive) and runs the four producers over them. Rows are labelled 1 at a verified fix site, 0 on an ordinary push. The screen scores every engineered feature alone against the label (ROC-AUC) and names the strongest feature per model annotation. Repositories are the clusters, and there are five, so the screen is a direction, not a result.

The final table: 952 rows, 74 positive rows (62 functions; a function with two verified spans counts twice), 878 negative rows, digest 02774edc (152a72ce with folds). Producer coverage: diff 952, source 933, semantic 262 (the semantic producer handles TypeScript and JavaScript only; Java has no obligations support and PHP is unsupported), taxonomy 952 (a constant: it renders the family list, it is not a row feature).

It took three builds, and the first two printed plausible numbers

1. Build 1 (952 rows, source ok 783): the screen printed AUCs. 150 rows, 16%, every positive of trigger.dev among them, had read an empty file although git holds the blobs. The gate source == 0 did not fire because most rows were fine.
2. Build 2, after adding a checkout proof (git status clean after the sparse checkout is undone) and an unread-row gate (every unread row checked against git cat-file -s): the gate refused the table, the same 150 rows. The checkout proof never fired, so the checkout was not the cause.
3. The cause was a producer bug: the entry-point scan read every product file of the tree, one empty file raised empty_source, the exception escaped the cached property and failed the source producer on every row of the change, re-reading the whole tree per row (fcb43fd). Build 3: 0 unread rows, source ok 933; the 19 remaining failures are files the push deleted (FileNotFoundError, 16) and three unparseable tsx files.

A producer that fails on every row of whole changes while the same files read fine by hand is a producer bug, not disk or concurrency. The memory note carries it.

The diff producer was blind to the shape of this class

A positive push here is the fix replayed backwards (base = fix commit, head = its parent). A fix that adds a permission check becomes a push that only removes lines, and the diff producer kept only hunks with added lines at head. In build 1, 38 of 74 positive rows had no hunk in their file at all and 69 of 74 were "not on a changed line", while 60% of ordinary rows were: the hunks annotation separated the labels by construction and showed the model nothing at the fix site. The producer (version 2, 537b61e) now records the head lines after which the push removed code; distance and on_changed_line count them; the model's hunks input renders them as removed_after_line; diff.deletion_count is an engineered feature. In build 3, 66 of 74 positive rows have a deletion anchor in their file and 12 are on a changed line.

Screen (build 3, separation.csv, screen.csv)

Annotation	Strongest feature	AUC	rows
hunks	diff.deletion_count	0.89	952
obligations	semantic.obligations_undischarged	0.76	262
callers	source.caller_count	0.72	933
entry_points	source.entry_point_count	0.71	933
roles	semantic.sinks	0.17	46
function_text	source.span_lines	0.36	933
change_summary	source.is_module	0.36	933
def_use	semantic.binds	0.36	262

Read with care:

• deletion_count (0.89) and the low hunk_lines (0.15) are the construction, not the class. Reversed fixes are deletion-heavy pushes with few added lines; ordinary pushes are not. Any feature that says "this push removed code and added little" separates these labels. separation_deletion_rows.csv repeats the screen on the 235 rows whose file lost lines (66 positive, 169 negative, all 5 repositories): there deletion_count drops to 0.71, hunk_lines and hunk_count stay at 0.12 and 0.14, and entry_point_count rises to 0.83, caller_count stays at 0.73. Those two are the class: the verified access-control sites are web entry points with callers.
• obligations_undischarged (0.76) rests on 8 positive rows from one repository (trigger.dev; the semantic producer covers no Java or PHP). Not evidence yet.
• roles scored 46 rows. The semantic annotation is absent for 80% of this pool.
• leak_audit.csv is the same screen over the folded table (ablation.leak_audit), kept for the ablation record.

Folds and the first ablation manifest (registered, not run)

Folds are repository-grouped (ablation.assign_folds, k=5, seed 0), one repository per fold: 0 p4-plugin (6 positive rows), 1 dependency-track (4), 2 job-import (2), 3 shopper (54), 4 trigger.dev (8). ablation.json registers the first manifest in ~/ousast-results/hunt/ac-ablation-1/ for a decision: arms full, no-hunks, no-entry_points, no-callers, no-semantic; compile folds 0-2 (the Jenkins plugins), holdout 3-4 (shopper and trigger.dev, 62 of the 74 positive rows); metric search_outcome at tolerance 10; one repeat; 8 dollars and 1,500 calls per arm. The baseline cost 6.13 dollars for one arm over this pool, so this manifest is about 35 dollars. With 5 positive pushes analyse will report dry_run; the location-level numbers are what it can say.

Open questions for the maintainer

• Whether the hunks annotation should be ablated at all while positives are reversed fixes, or whether the ordinary draw should be stratified by "push removed lines" so the construction signal cancels.
• Whether to extend the semantic producer to Java (obligations) and PHP before the next class round; 80% of this pool gets no roles, def_use or obligations today.
• The 21 access-control sources dropped by the disk guard and batch 2 (27 changes, 69 locations, awaiting the spot-check) would take positives past 40.

Files

• table.py: the build (worktree per change, full checkout, producers, labels, gates, screen).
• summary.json, separation.csv, screen.csv, separation_deletion_rows.csv, leak_audit.csv, log.json (per-change seconds and rows; the slowest ordinary change of shopper took 2,407 s in the repository-wide caller scan), ablation.json.
• Table: ~/ousast-results/valid/train/work/ac-screen/table.jsonl (94 MB) and table_folds.jsonl; builds 1 and 2 kept beside it as ac-screen-v1 and ac-screen-v2-refused.