Open_Ultra_SAST/.kiro/specs/access-control-prover/research.md

# Research & Design Decisions

## Summary
- **Feature**: `access-control-prover`
- **Discovery Scope**: Extension of `authorization-obligations` (checker, facts, dominance) and the hunter's
  per-change context; gap analysis against the code on 2026-10-14 (commit 9feb8f1).
- **Key Findings**:
  - The checker already takes `paths` and `entries` and already decides reached-ness and along-path dominance
    (`semantic/obligations/check.py::_reached`, `dominance.py::_dominates_along`); nothing supplies paths today,
    so every operation outside an entry handler is invisible to it.
  - The entry-point mapper (`mapping.py`) recognises Flask, Django, Express, Laravel routes, WordPress hooks,
    Next and Remix route modules partially, and nothing for Java: no Stapler `doXxx` web method, no Servlet or
    Jetty handler, no JAX-RS resource. Seventeen of the pool's 23 fixes are Stapler web methods.
  - The hunter's `Context` (hunt/producers.py) caches product files, texts, file IR, a call index and the
    repository's entry points per change; `source.callers_of` gives one hop of callers from the index. A bounded
    upward walk from a function to an entry point is a loop over that, no new index.
  - Enforcement outside the function fits the existing `Discharge` scopes `router` and `decorator`, which
    `OrderDominance.dominates` accepts without a line, through the checker's `extra_witnesses` parameter. No
    schema change to the facts loader is needed for them; the facts files gain operation rows only.
  - `benchmarks/valid/metrics.py` scores replay-shaped records with `benchmarks.unseen.score`; the hunter arm
    (`benchmarks/valid/hunter_arm.py`) writes that shape. A prover arm writing the same shape is scored by the
    same code, so its recall and false alarms per 100 stand beside the hunter arms.
  - The pool's fixes by shape: 17 Jenkins web methods (`doTestConnection`, `doFill...Items`, `doCheck...`,
    `doValidate...`, `doChangeSubmit`, `doNotifyCommit`, `doPython`, `persistValue`, `create`) missing
    `checkPermission`; one Jetty service without authentication (CWE-306); one Tornado terminal websocket open
    without authentication (CWE-306); one Laravel Livewire `store` without authorization (CWE-285); one Symfony
    controller action (CWE-284); one Remix action deactivating a schedule by id without an ownership check
    (CWE-639); one Flarum JavaScript extension (CWE-863, out of reach for this prover: client-side permission
    gating, counted as uncovered).

## Research Log

### Which checker parameters exist and what they expect
- **Context**: avoid a second checker.
- **Sources Consulted**: `src/openultrasast/semantic/obligations/check.py` (`check_obligations`, `_reached`,
  `_relevant`), `dominance.py` (`OrderDominance`, `_dominates_along`, `_hop_index`), the approved design of
  `authorization-obligations` (Dominance, Checker, EntryPoints sections).
- **Findings**: `paths` is a sequence of objects with a `hops` attribute; each hop is `"path::function"` or a
  `(node, line)` pair; a path reaches an operation when its last hop equals `f"{op.path}::{op.function}"`. With
  no paths, only operations inside entry handlers (`entries`) are reached. A witness of scope `hop` on an
  earlier hop dominates along the path. Witness scopes `decorator` and `router` dominate without a line.
- **Implications**: the prover produces `PathRecord(hops=(...))` objects ending at the operation's function and
  passes them as `paths`; it produces router and decorator witnesses as `extra_witnesses`; it needs no change to
  the checker's logic. One gap: the checker builds `operations` from the IRs it is given, so the prover passes the
  IR of every file on a path, not only the touched file.

### Where entry points come from
- **Sources Consulted**: `src/openultrasast/mapping.py` (`analyze_entry_points`, `_entry`, the per-framework
  recognisers around lines 260-930), `EntryPointRecord` fields.
- **Findings**: recognisers are code, one block per framework, each calling `_entry(target, start, end, handler,
  name, kind, access, trust_boundary, evidence, conditions, ...)`. No Java recogniser exists. `frameworks.toml` is
  a lexicon for priors and stratification, not for entry recognition.
- **Implications**: add recognisers in `mapping.py` following the existing block pattern (Stapler, Jetty,
  Symfony attributes, Livewire, Tornado, Remix exports where missing). Each names its evidence.

### How a rule arm is scored
- **Sources Consulted**: `benchmarks/valid/metrics.py` (`summarize`, `scoring_records`), `benchmarks/unseen/score.py`
  (`catches`, `location_match`, `false_alarm`, `coverage`), `benchmarks/valid/hunter_arm.py` (`run`, record
  writing), `src/openultrasast/hunt/metric.py::search_outcome`.
- **Findings**: a record needs `id`, `kind`, `repo`, `family`, the change's `path`/`function_lines` (copied from
  the pool), `findings` with `path`, `line`, `family`, and `coverage` fields the scorer reads; a push is caught
  when a finding of the change's family lands within the tolerance of a verified location.
- **Implications**: the prover arm copies the hunter arm's record envelope and instrument fields (bytes read per
  changed file, head verified), writes per-change files, and calls `benchmarks.valid.metrics.summarize`.

### Budget
- **Findings**: the second screen's feature table took 53 minutes for 483 changes with every producer, 7 seconds
  per change on average and 117 seconds for the slowest seatunnel change; the caller index is the dominant cost
  and is already cached per change. A depth-3 upward walk adds a few index lookups per function.
- **Implications**: the 120-second bound of Requirement 6 is realistic; the runner measures it per change.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|---|---|---|---|---|
| Extend the checker's inputs (chosen) | Supply paths, entries and witnesses to `check_obligations` from the hunter's context | One checker, one dominance rule, facts stay closed data | Order dominance is textual inside a function; nesting-aware dominance (`guard-dominance-regime`) never existed | Honest for "check present before operation"; the Jenkins shape |
| New prover module with its own graph | Build a call graph and CFG per change | Precise dominance | Weeks of work, duplicates the checker, no CFG for Java or PHP in the tree | Rejected for this increment |
| Joern engine | Dataflow and call graph from the engine | Precise | 225 s per batch, pilot produced no findings, no PHP in the budget | Rejected as the gate; usable later as a proof backend |

## Design Decisions

### Decision: paths from the caller index, not a call graph
- **Context**: Requirement 3.
- **Alternatives Considered**: a per-change call graph over all product files; the engine's call graph.
- **Selected Approach**: an upward walk from the function through `source.callers_of` on the cached index,
  bounded by depth and fan-out, stopping at the first entry point of each chain.
- **Rationale**: the index exists, is cached per change, and the pool's shapes are shallow (the operation is in the
  web method or one helper below it).
- **Trade-offs**: name-based caller resolution conflates same-named functions across files; chains are kept per
  file but a caller of `save` in another class may be a false hop. Accepted and visible in the evidence.
- **Follow-up**: measure how many positives are reached at depth 1, 2 and 3 on the pool.

### Decision: framework enforcement as router and decorator witnesses
- **Context**: Requirement 2.
- **Selected Approach**: a small recogniser per framework reads route registrations and attributes and emits
  `Discharge` witnesses with scope `router` or `decorator` for the handlers they cover; passed through
  `extra_witnesses`.
- **Rationale**: the dominance rule already accepts those scopes; the facts files stay about calls.
- **Trade-offs**: coverage is as wide as the recognisers; an unrecognised middleware name means a finding on a
  guarded route. Every recogniser lists its names in the facts header and the coverage statement counts routes
  with unrecognised middleware.

### Decision: an opt-in premise in the checker, not a synthesized policy
- **Context**: Codex's first attempt stopped on a real conflict: with path records and neither a declared clause
  nor a sibling norm the checker returns no finding by design (gating 3), so the prover's core verdict could not
  be emitted without a change.
- **Alternatives Considered**: synthesize a `DeclaredPolicy` in memory marking every recognised entry
  `authenticated` (dishonest: the finding would read "route declared authenticated" when nothing was declared);
  a second checker in the prover (duplicates the rules); an opt-in keyword and a distinct label (chosen).
- **Selected Approach**: `assume_entries_protected=False` keyword, label `unguarded_path`, evidence naming the
  premise. Default behaviour and tests unchanged.
- **Trade-offs**: the label makes the premise visible in every finding and in the static tags; consumers that
  grouped by label need the new value (revalidation trigger for `authorization-obligations`).

### Decision: no model call inside the prover
- **Context**: Requirement 4.4; the maintainer's direction after the engine pilot: the model's job is the trust
  boundary question the static side cannot answer, asked after a static result exists.
- **Selected Approach**: the prover is deterministic; the intent adjudication hook of the checker stays off in
  the arm; a later increment may wire it as a separate, measured arm.

## Risks & Mitigations
- Jenkins web methods whose protected operation is an outbound connection test with stored secrets
  (`doTestConnection`) call no listed operation name — mitigate by adding operation rows for credential
  lookups and HTTP client calls used in those handlers, checked in the clones, and by counting the handlers the
  facts do not cover.
- Textual order dominance inside a function misses a check in an `else` branch — accepted for this increment;
  recorded as a degradation when a witness exists but does not dominate.
- Same-name conflation in callers — chains kept per file; evidence names the hop files.
- Over-wide entry recognition (every `doXxx`) raises false alarms on helper methods — the Stapler recogniser
  requires a public method on an exposed type or a Stapler-typed parameter; the pool's ordinary pushes measure
  the rate.

## References
- Jenkins AccessControlled: https://javadoc.jenkins.io/hudson/security/AccessControlled.html
- Stapler web methods: https://www.jenkins.io/doc/developer/security/form-validation/ and
  https://www.jenkins.io/doc/developer/security/secrets-and-credentials/
- Laravel authorization and route middleware: https://laravel.com/docs/authorization, https://laravel.com/docs/middleware
- Symfony security: https://symfony.com/doc/current/security.html
- Remix route module API: https://remix.run/docs/en/main/route/action
- Tornado authentication: https://www.tornadoweb.org/en/stable/guide/security.html

---

## Design amendment synthesis -- 2026-10-29

Context: gates 1-3 (measurements `2026-10-28-access-control-{prover,judge,ablation}`) found one cross-cutting
lever -- interprocedural resolution -- behind both the 8 recall misses (B: operations never linked; C: coarse
guards discharge) and the kept false alarms (guards present but unmodeled, name-collision resolution, base-class
projection). Requirements amendment `[2026-10-29]` (AC 1.9, 2.9, 2.10, 3.7-3.10, 5.6, 7.3); this section
records the code survey behind the design amendment (same date tag).

### Research Log: resolution seams (verified against current code)

- Caller index keys on bare function name only: `src/openultrasast/source.py:447` `callers_of` (regex over
  `CodeLine` indexes); `CallSite(number, name)` at `source.py:22` carries no receiver. The prover adapter
  `hunt/prover.py:161-172` re-qualifies hits to `file::enclosing_fn` by IR span containment.
- Tree-sitter IR drops receiver text: `semantic/ir.py:21-29` `CallSite` (callee is trailing name only;
  `semantic/cst.py:220` `_callee_name` strips `$obj->method` / `x.y.z`); receiver must be captured in
  `cst.py:203` `_call` and added to `ir.CallSite` (python-ast path `ir.py:100+` likewise).
- Target enumeration is one-hop trailing-name join: `hunt/prover.py:174-185` (`call_names` from touched
  functions' IR calls; any `(path,name)` with that name and operations). Second receiver-blind seam.
- Guards on hops: `Hop.guards_before_call` populated only at `semantic/obligations/trace.py:224-231`
  (serialization); the verdict path uses `semantic/obligations/check.py:312-323` `_on_caller_hop`, which today
  re-judges only `path_guard`/`ownership_check` (router/decorator => True; other kinds => False). Dominance
  order rule `dominance.py:52-60`; CFG deferral `cfg_dominance.py:63-65`.
- Discharge has no resource identity: `semantic/obligations/operations.py:35-44` (`detail` is identifier-only).
  The only resource token today is `Operation.resource` (`operations.py:31`, `_resource_token` at 229-243:
  receiver first segment, SQL table, or `resource_arg`). Suppression loop `check.py:117-122`; relevance
  `check.py:326-327`; scope matching is `(path, function)` equality (`operations.py:124`, `dominance.py:38`).
- Enforcement witnesses: `semantic/obligations/enforcement.py:18` covers Tornado decorators, Symfony
  `IsGranted`/`Security`/`access_control`, Laravel middleware + `authorizeResource`; called per language at
  `hunt/prover.py:223-227`. In-function recognizer `_call_witnesses` (`operations.py:163-183`) matches
  discharger rows by trailing name -- `denyAccessUnlessGranted`/`isGranted` already match existing rows
  (`ruleset/obligations/php.toml:68-78`); `voteOnAttribute` needs no row (interface method, trailing-name
  match). Jenkins guard rows exist (`java.toml:68-76`).
- Mapping: `_jetty_entries` (`mapping.py:562`) anchors at method declarations only; call-site registration
  precedent exists (`_python_registered_routes` `mapping.py:945-1015`, WordPress `_PHP_REGISTRATION`
  `mapping.py:657-680` with `registered_by` field on `EntryPointRecord`, `mapping.py:53-74`); java dispatch at
  `mapping.py:185-198`; framework tag table `entry_framework` `mapping.py:472-488`.
- prover_arm refusal seams: per-file zero-byte loop `benchmarks/valid/prover_arm.py:86-113`; stale-anchor
  validation belongs in `_prove` between lines 114 and 125 (before `record.update(findings=...)`), mirrored in
  `summarize()` (184-187); type-name-only catch at `_prove` 129-133.
- Checker boundary: `authorization-obligations` owns check.py/facts.py/operations.py/paths.py/dominance.py.
  The design's third extension (resource field + `resource_bound` keyword + all-kind `_on_caller_hop`) touches
  `operations.py` (dataclass field) and `check.py` (keyword + relevance compare), default-off. Other
  `check_obligations` callers (cli.py, hunt/producers.py, pairs.py) keep unchanged defaults.

### Decisions

1. New module `semantic/obligations/resolution.py` + additive `CallSite.receiver` (captured in `cst.py`)
   rather than widening the source.py index: the index is shared with hunt search; a per-change class table
   keeps resolution local, cacheable on the Context, and testable in isolation. Bare-name fallback +
   `receiver_unresolved` coverage counter keeps dynamic languages (TS/PHP dynamic receivers) working as today.
2. Resource binding as checker keyword (`resource_bound`) with emission-side tokens rather than prover-side
   filtering: the suppression decision is per (witness, operation) pair inside `check_obligations`; filtering
   outside would need a restructured call. Default-off keeps `authorization-obligations` tests untouched.
3. Programmatic registration follows the existing call-site registration pattern (python routes, WordPress)
   with class-literal resolution through `resolution.class_table` -- one owner for class resolution, reused
   by the mapping recogniser and the callers adapter.

### Risks

- Dynamic receivers (TS `this.handler.get`, PHP `$container->get(...)`) stay name-keyed; the counter makes the
  fallback visible instead of silent. If the pool shows large `receiver_unresolved` rates, the fact-proposal
  lane (deferred injection spec) is the designated place to grow type facts, not this spec.
- `down_depth` widens target enumeration; the fan-out bound and the 120 s budget cap it. Per-repository
  seconds-per-change in the runner summary will show any cost.

---

## Abstract judgment and admission amendment -- 2026-10-29

### Context

The design review after the capability-level lift found that the proposed amendment still allowed two degraded
instruments to participate in binary decisions: unresolved receivers fell back to a bare-name call key, and
unavailable CFGs fell back to source order. It also found that the fact-initialization task used known development
misses and then remeasured the same population, which is valid regression work but cannot qualify an admission arm.

### Synthesis

1. **Generalization**: call certainty, path completeness, resource compatibility, authority compatibility and
   dominance are instances of one evidence-relation problem. They share a finite domain
   (`MUST | MAY | NO | UNKNOWN`) and one incompleteness record instead of separate booleans and counters.
2. **Build vs adopt**: the repository already owns the semantic IR, structural CFG, obligation checker and path
   records. A small abstract-domain layer and monotone function summaries extend those contracts; adopting another
   graph or abstract-interpretation engine would duplicate the current architecture and violate steering.
3. **Simplification**: the prover keeps one checker and one concrete trace. Function summaries accelerate and
   stabilize interprocedural discovery, but final evidence remains a materialised path. No second verdict engine is
   introduced.

### Decisions

#### Decision: three-way judgment
- **Selected approach**: every path and aggregate pair is `unguarded`, `discharged`, or `indeterminate`.
- **Invariant**: less precise evidence may preserve a verdict or move it to `indeterminate`; it may not switch
  directly between the two final verdicts.
- **Algebra**: `MUST` and `NO` are incomparable, both below `MAY`, which is below `UNKNOWN`. Alternative
  evidence uses the least-upper-bound table; dimensions of one witness use conjunction; witnesses for one
  obligation use existential aggregation. These operations are distinct so `MUST + NO` on alternative paths
  becomes `MAY`, while one definite governing witness remains sufficient for the obligation it covers.
- **Consequence**: prover mode does not use textual order as a semantic fallback. Legacy checker callers retain it.

#### Decision: stable symbols and finite candidate sets
- **Selected approach**: identify repository declarations by module, enclosing type, method and available signature.
  Partial resolution returns a finite candidate set plus incompleteness; unbounded resolution returns unresolved.
- **Rejected approach**: global bare-name fallback, because it conflates unrelated methods and can both invent paths
  and borrow unrelated guards.

#### Decision: resource and authority are separate abstract values
- **Selected approach**: match witness kind, scope, resource and authority independently, joining each relation into
  the pair judgment. Request-global guards are valid only when the operation fact explicitly allows that scope.
- **Rejected approach**: `None` as a resource wildcard. It cannot distinguish intentionally global authentication
  from missing resource evidence.

#### Decision: summaries discover; concrete paths prove
- **Selected approach**: monotone summaries over strongly connected call components collect operations, witnesses,
  call target sets and incompleteness. Concrete paths are materialised from exact edges for trace evidence.
- **Trade-off**: summary and path code add one engine boundary, but they remove recursion and depth handling from
  ad-hoc traversal and make every cutoff explicit.

#### Decision: development attribution before admission
- **Selected approach**: compare old/new analyzer crossed with old/new facts on the current development population,
  then freeze analyzer, facts, acquisition record, configuration and a separate untouched population for admission.
- **Consequence**: the 2026-10-28 numbers remain the immutable baseline. A better rerun on the same repositories is
  regression evidence, not a G1 admission claim.

### Risks and mitigations
- **Indeterminate inflation**: strict uncertainty may initially lower apparent coverage. Report the stage funnel and
  reason distribution; improve the dominant reason rather than weakening the lattice.
- **Summary cost**: a repository-wide symbol graph may threaten the pre-push budget. Reuse Context caches, use finite
  summary sets, and fail the budget gate when any measured change exceeds 120 seconds.
- **Authority vocabulary growth**: permission implication can become target-specific code. Keep implications in
  promoted facts and include the fact identifier in every match.
- **Admission scarcity**: the current pool is already development data. Source and review a separate population
  through `valid-detection-benchmark`; do not relabel the current pool as untouched.

## Admission outcome (2026-10-30)

The first admission run on the untouched pilot population returned **NO-GO** (0/12 pairs,
in-family 0/3; 9/12 repositories with zero framework recognition; 0 instrument failures;
determinism 4/4). See `benchmarks/measurements/2026-10-30-access-control-admission/`. The
next lever is entry-recognition and call-resolution breadth for the valid pool's
ecosystems, gated by the same admission workflow. The pilot pool remains unspent: no
acquisition observed its labels; only admission scoring read them. Standing rules:
`steering/detection-quality.md`.

---

## Detection-core fact-authority ownership amendment (2026-11-01)

### Context

Detection-core task 9.4 reached a real boundary gap while migrating the installed entry
pack. The generic core already validates fact packs, structural matches, lineages, custody,
and relation application, but no owner supplied the production access-control contract,
reviewed target authority, structural matcher, or persisted match artifacts. Putting those
objects into core would make a vulnerability-neutral package own access-control and target
vocabulary.

The instrument was verified before diagnosis: the installed entry file was readable at 963
bytes, and the focused legacy/v2 fact suites passed. The blocker was therefore ownership and
missing authority, not an unread input or broken validator.

### Synthesis and decision

1. **Dependency direction:** `access-control-prover` imports detection-core contracts and
   supplies family-owned objects. Detection-core never imports access-control or target code.
2. **Authority:** a checked-in immutable bundle references the production family contract,
   one reviewed fact pack per language, recognition projection, policies, lineages, matcher
   identity, and canonical evidence. A separately configured bootstrap policy authenticates
   the bundle's reviewer and store; the bundle cannot trust itself.
3. **Evidence custody:** the downstream adapter persists canonical structural observations
   and match bytes before calling core. Core receives only typed objects, immutable content
   references, and read-only custody/resolver ports.
4. **Compatibility:** existing regex rows remain a byte-stable recognition projection during
   migration. They are explicitly non-authoritative; exact typed observations are required
   for complete fact evidence.
5. **Task routing:** access-control-prover tasks 14.1–14.8 define the family contract,
   authenticate authority, implement installed codecs/resolution, record the approved review
   ceremony, migrate reviewed data, persist evidence, match structure, and publish the
   conformance handoff. Detection-core task 9.4 then runs the target-neutral handoff-acceptance gate. Access-control-prover task 14.9 subsequently migrates the
   production caller and proves zero bypasses before detection-core task 10.2 cuts over.

### Risks and mitigations

- **Self-authorized trust:** keep bootstrap policy outside the bundle, re-derive every digest,
  require producer-reviewer separation, and enforce forbidden identity pairs.
- **Pattern authority inflation:** preserve recognition behavior separately and require the
  complete typed predicate set before core application.
- **Circular match references:** persist the completed canonical match first, then pass its
  reference to core; the reference is never part of match-digest construction.
- **Admission contamination:** bind fact and matcher lineages to current custody and the
  evaluation population so label access refuses the same run.

### Approved repo-local review ceremony (2026-11-01)

- The maintainer approved a repo-local ceremony after Task 14.3 correctly refused to invent
  its own independent reviewer. Generator, maintainer reviewer, and custodian are separate
  canonical identity manifests; the maintainer-reviewer manifest records the explicit
  session approval as its basis.
- The bootstrap file admits the maintainer reviewer and installed content-addressed store;
  the authenticated authority file cannot modify either admission list.
- Closed canonical decoders, fixed pack filenames, embedded policies/lineages, and a
  digest-addressed artifact directory make installed bytes reproducible. Storage receipts
  are derived from verified bytes and bootstrap-admitted store identity rather than stored
  as independently mutable claims.
- This resolves the external identity decision but adds two prerequisite implementation
  units: wire decoders/resolver first, then the closed identity/bootstrap manifests, before
  the Express data migration.