**Purpose.** Give OpenUltraSAST a second detection regime beside data flow: an operation that carries an obligation (a protected read or write, a privileged action, a security-relevant setting) is reported when an entry point reaches it and nothing on the way discharges the obligation (a path guard, an identity constraint bound from the authenticated context, an ownership check, a non-permissive value, validated input). The application's own policy says which obligations exist: the consistency of sibling handlers, and optionally a declared policy file.
**Users.** AppSec engineers get absence findings that name the missing discharger, the evidence for the obligation, and the known fix. Maintainers get a leave-one-out number per obligation kind and an improve lever that admits obligation shapes under the existing gate. Project owners may declare their policy in a small versioned file.
**Impact.** New `semantic/obligations/` package (facts, shapes, siblings, declared policy, dominance protocol, checker). One more MAP proposer. Obligation rows in the existing mechanism store (second shape family). New harvest mode and one additive label field in the corpus. Reports, manifest and ranking gain obligation fields. No change to dispositions, the evidence ladder, taint, facts for flows, the call graph, or any gate.
### Goals
- Absence pairs in `vibe-py` and `agent-vfc` move from 0 detected to a measured leave-one-out baseline per obligation kind.
- Every obligation finding states obligation, missing discharger, evidence (siblings or policy clause) and known fix.
- Works today in function-local mode; becomes path-aware when reachability lands; becomes nesting-aware when guard dominance lands, without caller changes.
-`PathRecord` shape change upstream (the checker's path consumer must be re-checked).
- Consistency findings that demote or suppress anything.
## Architecture
### Existing Architecture Analysis
MAP already runs three proposers over one parse: inventory findings adjudicated by the overlay, the tool hunter, and variant search. Each produces `StaticFinding`s at or below `static_corroboration`, merges with overlay records where a call site coincides, and is counted in the manifest. The entry-point mapper runs before MAP and yields routes with an access level. The mechanism store is append-only with deterministic ids and a folding loader; the improve loop has a bounded lever contract with byte-for-byte revert. This design adds a fourth proposer and a second shape family, following those patterns exactly; the only new concept is the `Dominance` protocol, introduced because the spec that owns real dominance is not written yet.
LOO --> LEVER[obligations lever: admit/retract shapes, pending policy clauses]
LEVER --> STORE
```
**Architecture Integration**
- Selected pattern: closed facts plus learned shapes plus recovered policy, evaluated by a proposer that never claims proof.
- Domain boundaries: facts/shapes (semantic data), siblings/policy (application intent), checker (evaluation), reports/rank/lever (consumers). No component writes another's data except the lever writing the store through the existing contract.
- Existing patterns preserved: append-only store with deterministic ids; MAP proposer at `suspicion`; lever with validator, snapshot, per-profile clause; leave-one-out as the objective.
- New component rationale: `Dominance` protocol decouples the checker from the two upstream engine specs so obligations are useful in function-local mode today.
- Dependency direction: obligation facts → shapes → siblings/policy → checker → cli/reports/rank → loo → lever. `obligations/` never imports the overlay, the scorer, or `improve`.
### Technology Stack
| Layer | Choice | Role | Notes |
|-------|--------|------|-------|
| Facts | TOML data next to the flow facts | operation and discharger kinds | closed loader, separate file so `facts.py` stays untouched |
| Shapes | stdlib dataclasses over `FileIR` | derive, key, dict round-trip | second family in the mechanism store |
| Policy | TOML file in the scanned project's calibration dir | declared resources and routes | closed schema, version hash in manifest |
| Evaluation | MAP proposer | findings at `suspicion` | function-local without paths |
| Corpus | harvest mode + label field | teach and score absence pairs | additive |
| Improve | existing lever machinery | admit/retract, pending clauses | same gate |
## File Structure Plan
```
src/openultrasast/
├── semantic/obligations/
│ ├── __init__.py # public surface: load_obligation_facts, derive_obligation, check_obligations, ObligationFinding
-`src/openultrasast/cli.py`: `_check_obligations` after `_search_variants` in MAP (standard/deep); manifest `obligations` block; `mechanisms export` writes obligation rows too.
-`src/openultrasast/pairs.py`: `ExpectedFinding.obligation: str | None`; `_detections`-side additive rule: for a row with `obligation`, a finding with tag `obligation:<kind>` inside the labeled function counts.
-`src/openultrasast/semantic/mechanisms.py`: no schema change; `corpus_mechanisms()` unchanged; a helper `obligation_mechanisms(records)` selects `shape["family"] == "obligation"` (added in `obligations/shapes.py`, not in the store module).
Gating: the checker emits a finding only when (1) the operation is reached (from an entry point along a `PathRecord`, or, without paths, when the operation sits in a handler that is itself an entry point) and (2) no discharger of the required kind dominates, and (3) the obligation is justified by a sibling set (`consistency_violation`), a policy clause (`declared_policy_violation`) or, in degraded mode without either, by the operation fact alone (`function_local`). A declared public route suppresses path-guard obligations for that route only.
Y --> Z[store row family=obligation, deterministic id]
Z --> L[leave-one-out: check_obligations on held-out excerpts with the other pairs' shapes]
L --> M[per obligation kind: recall, silence, Youden; found_by]
```
### Lever
Admit/retract of obligation shape rows follows the mechanisms lever exactly (validator dispatches on `family`; same snapshot, tombstone, per-profile clause via `evaluate_mechanism_profiles` extended to run the checker for obligation rows). A policy clause proposal (`PolicyClauseEdit`) is journaled as `pending` with its sibling evidence and is never read by a scan until a human copies it into the policy file; accepting is a human edit of the file, not a loop action.
| 4.1–4.5 | declared policy: load, reject unknown, declared violation, public routes, never written by the tool | DeclaredPolicy, Checker, Lever | evaluation, lever |
defclassify_discharger(vuln:FileIR,fixed:FileIR,*,function:str,facts:ObligationFacts,texts:tuple[str,str])->tuple[str,str]|None:...# (discharger kind, provenance) added on the fixed side
```
Preconditions: both sides parse; the labeled function exists on both. Postconditions: None when no operation of a required kind is in the labeled function or no discharger was added; the shape carries no text. Stored through the existing writer with `shape = to_dict()`; id from `key()`.
### Operations and Siblings
```python
@dataclass(frozen=True)
classOperation:
path:str;line:int;function:str;kind:str;fact_id:str;resource:str|None# resource token, identifier only
@dataclass(frozen=True)
classDischarge:
path:str;line:int|None;function:str;kind:str;fact_id:str;provenance:str;scope:str# scope: decorator | router | statement | hop
defconsistency_anomalies(sets,*,min_siblings:int)->tuple[Anomaly,...]:...# handler, kind, siblings that discharge
```
Invariants: a set with fewer than `min_siblings` members yields no anomaly and is counted as under-populated; anomalies never touch existing findings.
### DeclaredPolicy
```toml
# .openultrasast/obligations.toml (project-owned; the tool never writes it)
version=1
identity_source="request.user"# closed: one of the discharger facts' identity_sources or "token"
[[resource]]
name="book"# resource token as seen in operations
sensitivity="high"# low | medium | high
identity_field="user_id"# constraint param that ties the resource to its owner
[[route]]
path="/health"
access="public"# public | authenticated | role
roles=[]
```
`load_declared_policy(path) -> DeclaredPolicy | None`; unknown field or value → `PolicyError` naming it; the scan continues without a policy and records `{"stage": "obligations", "reason": "policy_invalid", "detail"}`. `DeclaredPolicy.version_hash` goes to the manifest.
Rules: reached = a `PathRecord` ends at the operation, or (no paths) the operation's function is an entry point; label precedence declared > consistency > function_local; a declared public route suppresses `path_guard` obligations for that route; an operation with a matching store shape gains `known_fix`; a discharge that binds `request_input` where `authenticated_context` is required is reported with `provenance = "request_input"`.
`mapping.analyze_entry_points` names a handler from a decorator today. Real projects register handlers by call (`add_url_rule(path, view_func=fn)`, `api.add_resource(Cls, path)`, `router.get(path, requireAuth, handler)`) or by convention (`Resource` verb methods, Next.js `route.ts` exports). Because both the sibling grouping (Req 3.1) and the function-local reach test (Req 5.3) key on a named entry-point handler, an unnamed registration makes every obligation unreachable. The mapper gains those four forms; access classification, evidence, conditions and the record shape are unchanged, and middleware identifiers on a registration line are access evidence exactly as decorators are.
When a hunter client is available (the same resolution as `pairs --hunter`), the checker asks one closed question per finding with a route: "Is this route meant to be public?" with the handler text and siblings as context; the answer is recorded as `intent` and in the rationale. It never changes the label or the evidence level; when the client is absent a degradation `intent_adjudication_unavailable` is recorded once.
Markdown per finding: `- Obligation: <kind> on <resource>`, `- Missing discharger: <kind> (<provenance>)`, `- Evidence: <label> — <siblings | clause>`, `- Known fix: <guard shape> learned from <pairs>`, `- Intent: <adjudication>`; a `## Obligations` section lists sibling sets and the policy version. SARIF properties `obligation_kind`, `obligation_missing`, `obligation_label`, `obligation_evidence`, `obligation_known_fix`, `obligation_intent`. Manifest `obligations = {sibling_sets, under_populated, operations, findings_by_label, policy_version | null, degradations, sets}` (`sets` lists `{module, resource, handlers}` so the report can name them). Rank: `ranking_priority = base(sensitivity) x weight(label)` with `declared > consistency > function_local`, capped strictly below the lowest sandbox-proven finding in the run. Proof rungs live on `CandidateVerdict`, never on `StaticFinding`, so the cap takes the proven finding ids from the run's `TRIGGERABLE` verdicts: `rank_obligations(findings, proven_ids=...)` runs once at MAP and again after REGRESS when the sandbox proved anything.
Harvest mode `handler_context`: the labeled function plus its decorators and the registration statements that name it (router `.get(...)`/`.post(...)` calls, `add_url_rule`, middleware attachment) as one excerpt per side. Catalog row `obligation = "<OPERATION_KIND>"` (additive); vocabulary gains `unconstrained_protected_read`, `unconstrained_protected_write`, `unguarded_privileged_action` (existing `missing_auth_guard`, `identity_from_request_body`, `permissive_default` stay). `loo.score_pair_with_store` runs `check_obligations` on both sides for obligation rows; grouping adds `per_obligation_kind`; `found_by` names the shape record that supplied `known_fix`.
### Lever
`validate_mechanism` dispatches on `shape["family"]`: obligation rows check `operation_kind`, `discharger_kind`, `provenance`, `resource_class` against the closed sets. `PolicyClauseEdit(kind="resource"|"route", fields, evidence, rationale)` is validated against the policy schema, journaled under `pending_policy_clauses`, and never applied by the loop. `propose_mechanism_edits` treats obligation rows like sink rows (admit when a trial store recovers a missed holdout absence pair without leaking; retract leakers); `evaluate_mechanism_profiles` runs the checker for obligation rows.
## Data Models
- Store row: existing `Mechanism` with `shape = {family: "obligation", language, operation_kind, discharger_kind, provenance, resource_class, mechanism}`; `guard` field carries `discharger_kind` for report continuity.
- Facts: `obligations.toml` with `[[operation]]` and `[[discharger]]` tables per language; `version`.
- Policy: as above; `version_hash = sha1(normalized file)`.
| Store row with malformed obligation shape | skipped with `obligations_store_row_invalid` |
## Testing Strategy
- Unit: facts loader rejects an unknown kind (1.4); `derive_obligation` on the vampi pair yields `protected_read` / `identity_constraint` / `authenticated_context` and a text-free key (2.1, 2.3); a pair whose fix adds no discharger is skipped with a reason (2.5); `sibling_sets` groups three handlers on `/books` and `consistency_anomalies` flags the unconstrained one with the two siblings as evidence, and stays silent under `min_siblings` (3.2, 3.3); a public route among authenticated siblings is a missing `path_guard` with the access classification as evidence (3.4); policy loader rejects an unknown field and a declared public route suppresses the guard obligation but not the identity one (4.2, 4.4); `OrderDominance` with a stub `PathRecord` accepts a guard on an earlier hop and rejects one after the operation (5.1, 5.2); no paths → `function_local` plus degradation (5.3); identity bound from `request.args` → `provenance = request_input` (5.4).
- Integration: standard scan on a fixture with three handlers emits one `obligation:` finding at `suspicion` with tags and report lines, manifest block present, absent in quick (5.5, 6.1, 6.2, 6.5); ranking places it below a sandbox-proven finding (6.4).
- Corpus: `handler_context` excerpt keeps decorators and registration (7.1); an obligation-labeled catalog row counts an obligation finding inside the function (7.2); LOO on a three-pair toy slice reports per obligation kind and `found_by` (7.4).
- Lever: admission of an obligation shape that recovers a holdout absence pair is accepted; one that fires on a fixed twin is rejected with byte-identical store; a `PolicyClauseEdit` lands in `pending_policy_clauses` and no scan reads it (8.3, 8.4).
- Gates: detection, map and local pair gates unchanged; extra-free suite green with semantic tests skipping (9.2, 9.3).
## Security Considerations
Shapes and policy carry identifiers only; the policy file is project-owned and never written by the tool; the hunter receives redacted handler text and can only annotate. Obligation findings never execute anything and never reach the sandbox candidate set.
## Performance
Operations and discharges are found in the same parse the overlay uses; sibling grouping is linear in handlers; the checker is O(operations x witnesses) per handler and O(paths) with path records; bounded by the MAP file budget.
## Migration Strategy
1. Facts, shapes, exporter and vocabulary; measure how many absence pairs teach.
2. Operations, discharges, siblings, policy, order dominance, checker in function-local mode; MAP wiring behind `[obligations].enabled`; reports and manifest.
3. Corpus harvest mode and label; leave-one-out per kind; roadmap baseline.
4. Lever and pending clauses.
5. When `reachability-flow-model` lands: pass `PathRecord`s to the checker (no interface change). When `guard-dominance-regime` lands: swap the `Dominance` implementation.
