mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-16 11:49:28 +00:00
Compare commits
278
Commits
@@ -55,6 +55,14 @@ Extract:
|
||||
- The command that produced the failure
|
||||
- Whether the failure is deterministic or intermittent
|
||||
|
||||
### 1b. Verify the Instrument Before You Believe the Measurement
|
||||
Every wrong diagnosis recorded in this project had the same shape: the input was unreadable and the tool reported a plausible zero (`file_exists()` false, the parser wrote nothing, the frontend exited 0), and "no errors, 0 findings" read exactly like a clean result. Before trusting any measurement in the failure report:
|
||||
- **Prove the tool read its input.** Make it open one file and print the size. A zero from an unread tree and a zero from clean code are indistinguishable afterwards.
|
||||
- **Distrust a number that is too good.** A Joern script "finishing" in 0.5 s when a JVM takes 15 s to start is a failed launch. A harness must fail loudly with the exit code.
|
||||
- **Prefer the shipped environment** (the Docker image has a native `php`); local wrappers that proxy stdio are not valid instruments for `php2cpg`.
|
||||
- **When a result contradicts a working baseline, suspect the instrument first.**
|
||||
A plausible zero is a failed instrument until proven otherwise; classify it as `CONFIG_GAP` or `RUNTIME_MISMATCH`, not as `LOGIC_ERROR` in the code under test.
|
||||
|
||||
### 2. Inspect Local Runtime and Repository State
|
||||
Inspect the repository for local evidence:
|
||||
- `package.json`, `pyproject.toml`, `go.mod`, `Makefile`, `README*`
|
||||
@@ -134,6 +142,7 @@ If the issue is fixable by repo changes inside the current task plan, do not esc
|
||||
| “Let’s try a few fixes” | Multi-fix guessing hides root cause. |
|
||||
| “The spec is probably wrong, I’ll adapt it” | Spec conflicts must be surfaced explicitly. |
|
||||
| “The docs search is optional” | For runtime/dependency issues, docs and version issues often contain the shortest path to root cause. |
|
||||
| “It ran clean: no errors, zero findings” | A tool that never read its input reports the same. Prove the read (one file, its size) before believing the zero. |
|
||||
|
||||
## Output Format
|
||||
|
||||
@@ -20,7 +20,7 @@ description: Entry point for new work. Determines the best action path or work d
|
||||
Gather **only metadata** to determine the action path. Do NOT read full file contents yet.
|
||||
|
||||
- **Specs inventory**: Scan `.kiro/specs/*/spec.json` for `name`, `phase` fields and `approvals` status. Note feature names and their current status.
|
||||
- **Steering existence**: Check which files exist in `.kiro/steering/` (product.md, tech.md, structure.md, roadmap.md). Do NOT read their contents yet.
|
||||
- **Steering existence**: Check which files exist in `.kiro/steering/` (core: overview.md, roadmap.md, safety-net.md; custom files may exist too). Do NOT read their contents yet.
|
||||
- **Roadmap check**: If `.kiro/steering/roadmap.md` exists, read it. This contains project-level context (approach, scope, constraints, spec list) from a previous discovery session. Use it to restore project context.
|
||||
- **Top-level structure**: List the project root directory to note key directories and files. Do NOT recurse into subdirectories.
|
||||
|
||||
@@ -60,7 +60,7 @@ For Path A/B, recommend the next action and stop.
|
||||
**Only for Path C, D, and E.** Now load the context needed for discovery.
|
||||
|
||||
**In main context** (essential for dialogue with user):
|
||||
- **Steering documents**: Read product.md and tech.md (if they exist) for project goals, constraints, and tech stack
|
||||
- **Steering documents**: Read overview.md and safety-net.md (if they exist) for project goals, constraints, and the current state of the stack
|
||||
- **Relevant specs**: If the request is adjacent to an existing spec, read that spec's requirements.md to understand boundaries and avoid overlap
|
||||
|
||||
**Delegate to sub-agent** (keeps exploration out of main context):
|
||||
@@ -198,7 +198,7 @@ Use this roadmap structure:
|
||||
- [ ] feature-c -- [one-line description]. Dependencies: feature-a, feature-b
|
||||
```
|
||||
|
||||
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. This enables parallel spec creation via `/kiro-spec-batch`.
|
||||
Then write `.kiro/specs/<feature>/brief.md` for **every** feature listed under `## Specs (dependency order)` using the Path C brief format. Each spec is then created one at a time with `/kiro-spec-init`, in dependency order.
|
||||
|
||||
**For Path E (mixed decomposition)**:
|
||||
|
||||
@@ -219,7 +219,7 @@ Use the same roadmap structure as Path D, plus these additional sections:
|
||||
```
|
||||
|
||||
Path E rules:
|
||||
- Keep `## Specs (dependency order)` reserved for **new specs only** so `/kiro-spec-batch` can still parse it unchanged
|
||||
- Keep `## Specs (dependency order)` reserved for **new specs only** so the list stays a plain, parseable creation order
|
||||
- Record existing-spec extensions under `## Existing Spec Updates`
|
||||
- Record true no-spec work under `## Direct Implementation Candidates`
|
||||
- Write `brief.md` only for the **new specs** listed under `## Specs (dependency order)`
|
||||
@@ -235,15 +235,12 @@ Suggest the next command and stop. Do NOT automatically run downstream spec gene
|
||||
|
||||
- Path A: `/kiro-spec-requirements {feature}` to update the existing spec
|
||||
- Path B: Recommend direct implementation without creating a spec
|
||||
- Path C: Default to `/kiro-spec-init <feature-name>`
|
||||
- Optional fast path: `/kiro-spec-quick <feature-name>` when the user explicitly wants to continue immediately
|
||||
- Path D: Default to `/kiro-spec-batch` (creates all specs in parallel based on roadmap.md dependency order)
|
||||
- Optional cautious path: `/kiro-spec-init <first-feature-name>` when the user wants to validate the first slice before batching the rest
|
||||
- Path C: `/kiro-spec-init <feature-name>`, then the step-by-step phases (requirements, design, tasks), each human-reviewed
|
||||
- Path D: `/kiro-spec-init <first-feature-name>` for the first spec in roadmap.md dependency order; validate that slice before the next one
|
||||
- Path E: Choose the next command based on the new-spec portion of the decomposition
|
||||
- If there is exactly one new spec: `/kiro-spec-init <new-feature-name>`
|
||||
- If there are multiple new specs: `/kiro-spec-batch`
|
||||
- `/kiro-spec-init <new-feature-name>` for the first (or only) new spec, the rest in dependency order afterwards
|
||||
- Also note which existing specs should be revisited with `/kiro-spec-requirements <feature>`
|
||||
- Re-entry: `/kiro-spec-init <next-feature-name>` or `/kiro-spec-batch` if multiple specs remain
|
||||
- Re-entry: `/kiro-spec-init <next-feature-name>` for the next spec that remains
|
||||
|
||||
If the decomposition contains only existing-spec updates plus direct implementation candidates, do NOT use Path E. Prefer Path A when one existing spec is the clear home, or recommend the existing-spec update plus direct implementation work without creating roadmap entries.
|
||||
|
||||
@@ -26,7 +26,7 @@ You operate in two modes:
|
||||
If steering/spec context is already available from conversation, skip redundant file reads.
|
||||
Otherwise, load all necessary context:
|
||||
- `.kiro/specs/{feature}/spec.json`, `requirements.md`, `design.md`, `tasks.md`
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
|
||||
- Additional steering files only when directly relevant to the selected task's boundary, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect implementation or validation
|
||||
- Relevant local agent skills or playbooks only when they clearly match the task's host environment or use case; read the specific artifact(s) you need, not entire directories
|
||||
|
||||
+29
-5
@@ -1,13 +1,18 @@
|
||||
---
|
||||
name: kiro-spec-design
|
||||
description: Create comprehensive technical design for a specification
|
||||
description: Create comprehensive technical design for a specification; optionally run the implementation gap analysis first (--gap) or an interactive design review of an existing design (--review)
|
||||
metadata:
|
||||
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md"
|
||||
shared-rules: "design-principles.md, design-discovery-full.md, design-discovery-light.md, design-synthesis.md, design-review-gate.md, gap-analysis.md, design-review.md"
|
||||
---
|
||||
|
||||
|
||||
# Technical Design Generator
|
||||
|
||||
**Arguments**: `$1` is the feature. `$2` may be `-y` (auto-approve requirements), `--gap` (run the
|
||||
optional gap analysis of Step 1b before designing) or `--review` (run only the optional interactive
|
||||
design review of Step 7 on an existing `design.md`). The design review gate of Step 5 always runs
|
||||
when a design is generated; the optional steps never replace it.
|
||||
|
||||
<background_information>
|
||||
- **Success Criteria**:
|
||||
- All requirements mapped to technical components with clear interfaces
|
||||
@@ -24,8 +29,8 @@ metadata:
|
||||
|
||||
**Read all necessary context**:
|
||||
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md` (if exists)
|
||||
- `.kiro/specs/$1/research.md` (if exists, contains gap analysis from `/kiro-validate-gap`)
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- `.kiro/specs/$1/research.md` (if exists, contains the gap analysis from Step 1b or an earlier run)
|
||||
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (this repository's core steering; custom steering files may also exist in `.kiro/steering/`)
|
||||
- Additional steering files only when directly relevant to requirement coverage, architecture boundaries, integrations, runtime prerequisites, security/performance constraints, or team conventions that affect implementation readiness
|
||||
- `.kiro/settings/templates/specs/design.md` for document structure
|
||||
- Read `rules/design-principles.md` from this skill's directory for design principles
|
||||
@@ -35,6 +40,16 @@ metadata:
|
||||
- If `-y` flag provided ($2 == "-y"): Auto-approve requirements in spec.json
|
||||
- Otherwise: Verify approval status (stop if unapproved, see Safety & Fallback)
|
||||
|
||||
### Step 1b (optional, `--gap`): Implementation Gap Analysis
|
||||
|
||||
Recommended for brownfield features; skip for greenfield. Analyze the gap between the approved requirements and the existing codebase to inform the design, following `rules/gap-analysis.md` from this skill's directory.
|
||||
|
||||
1. Read `rules/gap-analysis.md` for the analysis framework.
|
||||
2. Research in parallel where possible (sub-agents if available, otherwise sequentially): existing implementations, architecture patterns, integration points and extension possibilities; external dependency compatibility and version constraints when needed; the requirements, core steering, task-relevant extra steering and relevant local agent skills or playbooks.
|
||||
3. Execute the gap analysis: Grep and Read the codebase, use WebSearch/WebFetch for external dependencies if needed, evaluate multiple implementation approaches (extend / new / hybrid) with trade-offs, flag areas needing further research. Provide information and options, not final implementation choices.
|
||||
4. Write the analysis to `.kiro/specs/$1/research.md` (append after a `---` rule if the file exists; never overwrite earlier research) and read it back to verify the write. Use the language specified in spec.json.
|
||||
5. Unapproved requirements are a warning here, not a stop: gap analysis can inform requirement revisions. Then continue with Step 2, or stop and report if the user asked for the gap analysis alone.
|
||||
|
||||
### Step 2: Discovery & Analysis
|
||||
|
||||
**Critical: This phase ensures design is based on complete, accurate information.**
|
||||
@@ -137,6 +152,15 @@ After all findings return, synthesize in main context before proceeding.
|
||||
- Set `approvals.requirements.approved: true`
|
||||
- Update `updated_at` timestamp
|
||||
|
||||
### Step 7 (optional, `--review`): Interactive Design Review
|
||||
|
||||
A human-paced quality review of an existing `.kiro/specs/$1/design.md`, in addition to the automatic gate of Step 5. Follow `rules/design-review.md` from this skill's directory.
|
||||
|
||||
1. Gather context: spec.json, requirements.md, design.md, core steering, review-relevant extra steering and local playbooks; survey existing codebase patterns, naming conventions and component structure for reference (parallel sub-agents if available).
|
||||
2. Review: Analysis -> Critical Issues -> Strengths -> GO/NO-GO. Limit to the 3 most important concerns; recognize 1-2 strengths; engage interactively (clarifying questions, alternatives); leverage conversation history for requirements context and user intent. Quality assurance, not perfection seeking; accept acceptable risk; every suggestion must be implementable.
|
||||
3. Decide: a clear GO/NO-GO with rationale. GO: `/kiro-spec-tasks $1` (or `-y`). NO-GO: address the critical issues, re-run `/kiro-spec-design $1`, then `/kiro-spec-design $1 --review` again.
|
||||
4. If design.md does not exist, stop: "Run `/kiro-spec-design $1` first to generate design document". If the design phase is not marked generated in spec.json, warn but proceed.
|
||||
|
||||
## Critical Constraints
|
||||
- **Type Safety**:
|
||||
- Enforce strong typing aligned with the project's technology stack.
|
||||
@@ -199,7 +223,7 @@ Provide brief summary in the language specified in spec.json:
|
||||
|
||||
**If Design Approved**:
|
||||
- Review generated design at `.kiro/specs/$1/design.md`
|
||||
- **Optional**: Run `/kiro-validate-design $1` for interactive quality review
|
||||
- **Optional**: Run `/kiro-spec-design $1 --review` for the interactive quality review (Step 7)
|
||||
- Then `/kiro-spec-tasks $1 -y` to generate implementation tasks
|
||||
|
||||
**If Modifications Needed**:
|
||||
+3
-3
@@ -24,7 +24,7 @@ metadata:
|
||||
- Read `.kiro/specs/$1/spec.json` for language and metadata
|
||||
- Read `.kiro/specs/$1/brief.md` if it exists (discovery context: problem, approach, scope decisions, boundary candidates)
|
||||
- Read `.kiro/specs/$1/requirements.md` for project description
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
|
||||
- Additional steering files only when directly relevant to feature scope, user personas, business/domain rules, compliance/security constraints, operational constraints, or existing product boundaries
|
||||
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and contain domain terminology or workflow rules that shape user-observable requirements
|
||||
|
||||
@@ -40,7 +40,7 @@ The following research areas are independent. Decide the optimal decomposition b
|
||||
**In main context** (essential for requirements generation):
|
||||
- Spec files: spec.json, brief.md, requirements.md (project description)
|
||||
- EARS format rules, requirements review gate, requirements template
|
||||
- Core steering: product.md, tech.md (directly inform scope and constraints)
|
||||
- Core steering: overview.md, safety-net.md (directly inform scope and constraints)
|
||||
|
||||
**Delegate to sub-agent** (keeps exploration out of main context):
|
||||
- **Codebase hints** (brownfield projects): Spawn a sub-agent to explore existing implementations that inform requirement scope. Ask it to summarize: (1) what already exists, (2) relevant interfaces/APIs, (3) patterns that new requirements should align with. Return a summary under 150 lines.
|
||||
@@ -133,7 +133,7 @@ Provide output in the language specified in spec.json with:
|
||||
**If Requirements Approved**:
|
||||
- Review generated requirements at `.kiro/specs/$1/requirements.md`
|
||||
- **Optional Gap Analysis** (for existing codebases):
|
||||
- Run `/kiro-validate-gap $1` to analyze implementation gap with current code
|
||||
- Run `/kiro-spec-design $1 --gap` to analyze the implementation gap with the current code before designing
|
||||
- Identifies existing components, integration points, and implementation strategy
|
||||
- Recommended for brownfield projects; skip for greenfield
|
||||
- Then `/kiro-spec-design $1 -y` to proceed to design phase
|
||||
@@ -25,7 +25,7 @@ metadata:
|
||||
**Read all necessary context**:
|
||||
- `.kiro/specs/$1/spec.json`, `requirements.md`, `design.md`
|
||||
- `.kiro/specs/$1/tasks.md` (if exists, for merge mode)
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
|
||||
- Additional steering files only when directly relevant to requirements coverage, design boundaries, runtime prerequisites, or team conventions that affect task executability
|
||||
|
||||
**Validate approvals**:
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
name: kiro-steering
|
||||
description: Manage .kiro/steering/ as persistent project knowledge
|
||||
description: Manage .kiro/steering/ as persistent project knowledge, including custom steering documents for specialized contexts
|
||||
metadata:
|
||||
shared-rules: "steering-principles.md"
|
||||
---
|
||||
@@ -11,9 +11,12 @@ metadata:
|
||||
<background_information>
|
||||
**Role**: Maintain `.kiro/steering/` as persistent project memory.
|
||||
|
||||
**Core steering in this repository**: `overview.md` (what the project is and what is actually true of it, dated current state), `roadmap.md` (milestones and status), `safety-net.md` (product direction). Custom files may exist beside them (for example a domain or standards file); all `.kiro/steering/*.md` are loaded as project memory. The generic bootstrap templates under `.kiro/settings/templates/steering/` are named `product.md`, `tech.md`, `structure.md`; they are starting points for a fresh project, not this repository's file names.
|
||||
|
||||
**Mission**:
|
||||
- Bootstrap: Generate core steering from codebase (first-time)
|
||||
- Sync: Keep steering and codebase aligned (maintenance)
|
||||
- Custom: Create specialized steering documents beyond the core files (optional, see below)
|
||||
- Preserve: User customizations are sacred, updates are additive
|
||||
|
||||
**Success Criteria**:
|
||||
@@ -27,8 +30,9 @@ metadata:
|
||||
|
||||
Check `.kiro/steering/` status:
|
||||
|
||||
**Bootstrap Mode**: Empty OR missing core files (product.md, tech.md, structure.md)
|
||||
**Sync Mode**: All core files exist
|
||||
**Bootstrap Mode**: Empty OR missing core files (here `overview.md`, `roadmap.md`, `safety-net.md`; in a fresh project the template names `product.md`, `tech.md`, `structure.md`)
|
||||
**Sync Mode**: All core files exist
|
||||
**Custom Mode**: The user asks for a specialized steering document (see Custom Steering below)
|
||||
|
||||
---
|
||||
|
||||
@@ -75,6 +79,20 @@ After all parallel research completes, synthesize patterns for steering files.
|
||||
|
||||
---
|
||||
|
||||
## Custom Steering (optional)
|
||||
|
||||
Create a specialized steering document for one domain beyond the core files.
|
||||
|
||||
1. **Ask the user** for the domain/topic (e.g. "API standards", "testing approach") and the specific patterns to document
|
||||
2. **Check for a template** in `.kiro/settings/templates/steering-custom/`: `api-standards.md`, `testing.md`, `security.md`, `database.md`, `error-handling.md`, `authentication.md`, `deployment.md`. Load the matching one as a starting point; without a template, generate from domain knowledge
|
||||
3. **Analyze the codebase** (JIT) for the domain's patterns with Glob/Grep/Read (template and principles in parallel with the domain analysis when multi-agent is available)
|
||||
4. **Generate** the document: follow the template structure if any, apply `rules/steering-principles.md`, patterns not exhaustive lists, one domain per file, 100-200 lines (a 2-3 minute read), never secrets
|
||||
5. **Create** `.kiro/steering/{name}.md`; ensure it does not duplicate core steering content
|
||||
|
||||
Custom files are loaded as project memory like the core files and carry equal weight.
|
||||
|
||||
---
|
||||
|
||||
## Granularity Principle
|
||||
|
||||
From `rules/steering-principles.md` (in this skill's directory):
|
||||
@@ -106,9 +124,9 @@ Chat summary only (files updated directly).
|
||||
✅ Steering Created
|
||||
|
||||
## Generated:
|
||||
- product.md: [Brief description]
|
||||
- tech.md: [Key stack]
|
||||
- structure.md: [Organization]
|
||||
- overview.md: [What it is, current state]
|
||||
- roadmap.md: [Milestones, status]
|
||||
- safety-net.md: [Product direction]
|
||||
|
||||
Review and approve as Source of Truth.
|
||||
```
|
||||
@@ -118,8 +136,8 @@ Review and approve as Source of Truth.
|
||||
✅ Steering Updated
|
||||
|
||||
## Changes:
|
||||
- tech.md: React 18 → 19
|
||||
- structure.md: Added API pattern
|
||||
- overview.md: Current state dated, retired component removed
|
||||
- roadmap.md: Milestone status updated
|
||||
|
||||
## Code Drift:
|
||||
- Components not following import conventions
|
||||
@@ -128,6 +146,20 @@ Review and approve as Source of Truth.
|
||||
- Consider api-standards.md
|
||||
```
|
||||
|
||||
### Custom:
|
||||
```
|
||||
✅ Custom Steering Created
|
||||
|
||||
## Created:
|
||||
- .kiro/steering/api-standards.md
|
||||
|
||||
## Based On:
|
||||
- Template: api-standards.md
|
||||
- Analyzed: src/api/ directory patterns
|
||||
|
||||
Review and customize as needed.
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Bootstrap
|
||||
@@ -136,7 +168,12 @@ Review and approve as Source of Truth.
|
||||
|
||||
### Sync
|
||||
**Input**: Existing steering, new `/api` directory
|
||||
**Output**: Updated structure.md, flagged non-compliant files, suggested api-standards.md
|
||||
**Output**: Updated overview.md, flagged non-compliant files, suggested api-standards.md
|
||||
|
||||
### Custom
|
||||
**Input**: "Create API standards steering"
|
||||
**Action**: Load template, analyze src/api/, extract patterns
|
||||
**Output**: api-standards.md with project-specific REST conventions
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
+8
-4
@@ -84,7 +84,11 @@ Never include:
|
||||
|
||||
## File-Specific Focus
|
||||
|
||||
- **product.md**: Purpose, value, business context (not exhaustive features)
|
||||
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
|
||||
- **structure.md**: Organization patterns, naming rules (not directory trees)
|
||||
- **Custom files**: Specialized patterns (API, testing, security, etc.)
|
||||
This repository's core steering files:
|
||||
|
||||
- **overview.md**: What the project is and what is actually true of it; a dated current-state section wins over older history
|
||||
- **roadmap.md**: Milestones, their measured status, and the accepted order of work (not a task list)
|
||||
- **safety-net.md**: Product direction and acceptance criteria (not feature catalogs)
|
||||
- **Custom files**: Specialized patterns (API, testing, security, etc.); they may exist beside the core files and carry equal weight
|
||||
|
||||
(The generic bootstrap templates `product.md`, `tech.md`, `structure.md` under `.kiro/settings/templates/steering/` cover purpose, key frameworks and organization patterns for a fresh project.)
|
||||
+8
-2
@@ -68,10 +68,16 @@ For each detected feature:
|
||||
- Read `.kiro/specs/<feature>/requirements.md` for requirements
|
||||
- Read `.kiro/specs/<feature>/design.md` for design structure
|
||||
- Read `.kiro/specs/<feature>/tasks.md` for task list and Implementation Notes
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Core steering context: `overview.md`, `roadmap.md`, `safety-net.md` (custom steering files may also exist in `.kiro/steering/`)
|
||||
- Additional steering files only when directly relevant to the validated boundaries, runtime prerequisites, integrations, domain rules, security/performance constraints, or team conventions that affect the GO/NO-GO call
|
||||
|
||||
**Discover canonical validation commands**:
|
||||
**This repository's gates** (`.github/workflows/ci.yml`; run them as written, judge each by its own exit code):
|
||||
- `TEST_COMMANDS`: `pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"` -- judge ONLY by the printed exit code, never by a summary line; `ruff check src tests`; `ruff format --check src tests`; `mypy src/openultrasast`
|
||||
- Detection gates: `python -m openultrasast.gate` (>=90% recall, <10% FP per language; hard, score-independent), `python -m openultrasast.map_gate`, and the pair gate `python -m openultrasast.pair_gate` (local vuln vs fix). A ruleset change that fails a gate is NO-GO.
|
||||
- Docs: `mkdocs build --strict` when `docs/` or `mkdocs.yml` changed
|
||||
- `SMOKE_COMMANDS`: `ousast --help`, and for a scan change `ousast scan <fixture> --mode quick` whose `manifest.json` names the stages that ran and the files preprocess kept. A plausible zero (0 findings, 0 degradations) from a tool that never read its input is a failed instrument, not a pass: prove the tool read its input first.
|
||||
|
||||
**Discover canonical validation commands** (for other repositories, or anything the list above does not cover):
|
||||
- Inspect repository-local sources of truth in this order: project scripts/manifests (`package.json`, `pyproject.toml`, `go.mod`, `Cargo.toml`, app manifests), task runners (`Makefile`, `justfile`), CI/workflow files, existing e2e/integration configs, then `README*`
|
||||
- Derive a feature-level validation set for this repo: `TEST_COMMANDS`, `BUILD_COMMANDS`, and `SMOKE_COMMANDS`
|
||||
- Prefer commands already used by repo automation over ad hoc shell pipelines
|
||||
+3
-1
@@ -61,7 +61,8 @@ Use the language specified in `spec.json`.
|
||||
4. Check exit code, failure count, skipped scope, and missing coverage.
|
||||
5. Reject claims that are broader than the evidence.
|
||||
6. If mandatory validation cannot be completed, return `MANUAL_VERIFY_REQUIRED`.
|
||||
7. Only then allow the claim.
|
||||
7. Verify the instrument before you believe the measurement: prove the tool read its input (it opened one file and printed the size; the stage list names what ran). A plausible zero -- "0 findings, 0 errors, exit 0" from a tree the tool never read, or a Joern script that "finished" before a JVM could start -- is a failed instrument, not evidence. For pytest, judge only by the printed exit code (`pytest -q -p no:cacheprovider < /dev/null; echo "exit=$?"`), never by a summary line.
|
||||
8. Only then allow the claim.
|
||||
|
||||
## Claim-Specific Rules
|
||||
|
||||
@@ -116,6 +117,7 @@ Return `NOT_VERIFIED` when:
|
||||
| “Build should be fine because lint passed” | Lint does not prove build success. |
|
||||
| “Tests passed and build succeeded, so it must run” | Type erasure, module loading, native ABI, and boot-time config issues can still fail at runtime. |
|
||||
| “The feature is done because all tasks are checked off” | `FEATURE_GO` also requires coverage, integration, and design alignment. |
|
||||
| “The scan found nothing, so the code is clean” | An unread input and clean code both report zero. Prove the tool read its input first. |
|
||||
|
||||
## Output Format
|
||||
|
||||
@@ -0,0 +1,123 @@
|
||||
---
|
||||
name: openultrasast-scan
|
||||
description: OpenUltraSAST scan workflows - `ousast scan` in quick, standard or deep mode, the evidence ladder, `--fail-on` exit codes, recorded degradations, the run directory and `ousast mcp`. Use when running or planning a scan, reading a run directory, or wiring OpenUltraSAST into an agent.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Scan
|
||||
|
||||
Use this skill to run `ousast scan PATH --mode quick|standard|deep` from an agent and to read what
|
||||
it wrote.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Confirm the target is a local checkout (a path, not a URL).
|
||||
2. Run through the project entry point: `uv run ousast ...` inside the repository, or `ousast` from an
|
||||
environment where it is installed. Never activate a virtualenv by hand.
|
||||
3. Start with `--mode quick` unless the task needs the Joern model layer (`standard`) or sandboxed
|
||||
regression snippets (`deep`). Check the prerequisites below first; a mode whose engine is missing
|
||||
still exits 0 and records a degradation.
|
||||
4. Read the printed `scan_id=` and `run_dir=` lines; the run directory is `.openultrasast/runs/<scan-id>/`.
|
||||
5. Report the finding count, how many the verifier accepted, and **every** recorded degradation
|
||||
(`manifest.json`, key `degradations`). No findings plus a degradation is not a clean scan: the scan
|
||||
answered a smaller question than it was asked.
|
||||
6. Never call a finding verified from its title or severity. `verification.json` holds the status.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
uv run ousast scan <repo> --mode quick
|
||||
uv run ousast scan <repo> --mode standard --fail-on verified
|
||||
uv run ousast scan <repo> --mode deep --fail-on worth-fixing
|
||||
uv run ousast scan <repo> --mode quick --config openultrasast.toml
|
||||
uv run ousast mcp # the MCP server over stdio, for agent hosts
|
||||
```
|
||||
|
||||
## Modes and prerequisites
|
||||
|
||||
| Mode | Adds | Needs | Without it |
|
||||
| --- | --- | --- | --- |
|
||||
| `quick` | language-scoped pattern rules, entry-point hints, ranking, verification, score; deterministic | nothing | - |
|
||||
| `standard` | the MAP stage: complexity map, semantic overlay, authorization obligations, the Joern model layer (one code property graph per repository; taint, guard-dominance and configuration questions), deterministic fusion | Joern (shipped in the Docker image, with `php-cli` for PHP); a provider key for the residual LLM question | `cpg_unavailable`; without a key the LLM question is not asked and only the graph's entailed findings are reported |
|
||||
| `deep` | REGRESS: promoted candidates run as snippets in a Docker sandbox (no network, read-only source, non-root) and get a `triggerable` / `not_triggerable` / ... verdict in `verdicts.json` | a working `docker` | `sandbox_unavailable` |
|
||||
|
||||
Model stages read `DEEPSEEK_API_KEY` (every LLM call); with only `OPENROUTER_API_KEY` set, chat
|
||||
calls fall back to OpenRouter. An exported shell variable wins over `.env` silently. `[models]
|
||||
hunter` in the config enables the tool hunter in `standard`; it is off by default.
|
||||
|
||||
## Exit code: `--fail-on`
|
||||
|
||||
| Policy | Exit 1 when |
|
||||
| --- | --- |
|
||||
| `never` (default) | never |
|
||||
| `findings` | any finding was reported |
|
||||
| `verified` | any verification result is `verified` (accepted at `static_corroboration` or above) |
|
||||
| `worth-fixing` | a deep-mode sandbox verdict is marked `worth_fixing` |
|
||||
|
||||
A degradation never changes the exit code. Read `manifest.json` for it.
|
||||
|
||||
## Recorded degradations
|
||||
|
||||
Each entry in `manifest.json` `degradations` is `{"stage": ..., "reason": ...}`. The ones to expect:
|
||||
|
||||
- `cpg_unavailable` (stage `model`): no Joern on the host; the model layer ran nothing.
|
||||
- `sandbox_unavailable` (stage `regress`): no working `docker`; deep mode fell back to standard.
|
||||
- `learning_endpoint_unavailable` (stage `model`): no provider key resolved. The graph's entailed
|
||||
findings are still reported; the residual question is skipped. This is recorded in `manifest.json`
|
||||
but has no line in `report.md`'s "What could not be analysed" section.
|
||||
- `hunter_model_unavailable` (stage `map`): `[models] hunter` is unset, so the tool hunter did not run.
|
||||
Recorded on every standard scan without that setting; it is expected, not an error.
|
||||
- coverage facts from the graph, rendered in `report.md` under "What could not be analysed":
|
||||
`cpg_empty`, `cpg_build_failed`, `cpg_sharded`, `files_unparsed`, `query_failed`,
|
||||
`query_budget_reserved`, `query_too_expensive`, `budget_exhausted`, `regions_truncated`,
|
||||
`region_failed`; plus `max_findings_exceeded` and `facts_unavailable` (overlay, obligations).
|
||||
|
||||
Prove the instrument read its input before believing a zero: `manifest.json` names the stages that
|
||||
ran and the files the preprocess step kept (`preprocess/file_targets.json`).
|
||||
|
||||
## Evidence ladder
|
||||
|
||||
`src/openultrasast/verification.py` defines the levels in this order. A level moves one rung at a
|
||||
time and never backward.
|
||||
|
||||
| Level | Meaning | Produced today by |
|
||||
| --- | --- | --- |
|
||||
| `suspicion` | a rule, model or weak signal raised a hypothesis | the Joern model layer (its rung, `model_entailed` or `suspicion`, travels in the finding's tags), the tool hunter, authorization-obligation checks |
|
||||
| `static_corroboration` | deterministic code evidence at the line | quick rules, the semantic overlay |
|
||||
| `crash_reproduced` | an artifact-backed reproducer | nothing: there is no crash reproducer. Deep-mode sandbox verdicts (`verdicts.json`, `worth_fixing`) do not raise the level |
|
||||
| `root_cause_explained` | independent verifier reasoning | nothing: the LLM judge was retired in v2.0.0 |
|
||||
| `exploit_demonstrated` | a demonstrated exploit | nothing |
|
||||
| `patch_validated` | relevant checks re-run after a patch | nothing: there is no patch stage |
|
||||
|
||||
The verifier's status per finding (`verification.json`): below `static_corroboration` is
|
||||
`needs_evidence`; `static_corroboration` with reachability `unknown` is `rejected` (the tie-breaker
|
||||
asks for call-graph, route, CLI, parser or dynamic evidence); `static_corroboration` with
|
||||
function-level reachability is `accepted`. `verified` means accepted at `static_corroboration` or
|
||||
above, and that is what `--fail-on verified` counts. There is no configurable report threshold: the
|
||||
`[evidence]` config section was retired in v2.0.0 and is ignored.
|
||||
|
||||
## Run artifacts
|
||||
|
||||
Under `.openultrasast/runs/<scan-id>/`:
|
||||
|
||||
- `report.md`, `report.sarif` (SARIF 2.1.0 for code hosts and IDEs), `manifest.json` (stages, degradations, model payload)
|
||||
- `findings.json`, `verification.json`, `score.json`, `shadow_findings.json` (rules shipped as `shadow`)
|
||||
- `harness.json`: the resolved harness configuration the run used
|
||||
- standard: `fusion.json`, `complexity_map.json`, `overlay.json`; deep: `verdicts.json`
|
||||
- `calibration/false_positive_learnings.json`, `rank/ranking.json`, `mapping/`, `preprocess/`, `trace/events.jsonl`
|
||||
|
||||
## Changed code only
|
||||
|
||||
`ousast pre-push --base <rev> --head <rev> --artifact <file> <repo>` replays a revision pair and
|
||||
reports only new or worsened defects (experimental; `--mode advisory` is the default and never
|
||||
blocks). The capability registry is empty, so every candidate is diagnostic.
|
||||
|
||||
## MCP
|
||||
|
||||
`ousast mcp` serves a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
|
||||
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
|
||||
`explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a shell, Docker or a
|
||||
free-form command.
|
||||
|
||||
```jsonc
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
```
|
||||
@@ -0,0 +1,75 @@
|
||||
---
|
||||
name: openultrasast-triage
|
||||
description: OpenUltraSAST finding triage - false-positive adjudication with the fixed reason taxonomy, the evidence ladder, what a verdict may feed (labels, never rule edits), and fixing then re-scanning. Use when reviewing findings, reducing false positives, deciding whether evidence is enough, or auditing a fix for a finding.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Triage
|
||||
|
||||
Use this skill when triaging OpenUltraSAST findings or auditing a fix for one.
|
||||
|
||||
## Triage Loop
|
||||
|
||||
1. Identify the finding: id, path, rule, CWE and policy severity, confidence, evidence level and
|
||||
verification status (`findings.json`, `verification.json` in the run directory).
|
||||
2. Separate deterministic evidence (rule match, graph entailment, attached reachability) from model
|
||||
assumptions.
|
||||
3. Check attacker control, reachability, sanitizer behaviour, duplicates, impact, and contradictory
|
||||
evidence.
|
||||
4. If rejected or unresolved, record the false-positive reason and its scope.
|
||||
5. Prefer scoped demotion over deleting a vulnerability class globally: a rejection in `auth/`
|
||||
demotes `auth/...`, never the class (`calibration.py`).
|
||||
6. Record the adjudication. Do not edit a rule because of one finding (see below).
|
||||
|
||||
## False-Positive Reasons
|
||||
|
||||
The taxonomy is `FalsePositiveReason` in `src/openultrasast/calibration.py`; use these values exactly:
|
||||
|
||||
- `unreachable_path`
|
||||
- `missing_attacker_control`
|
||||
- `sanitizer_disproved`
|
||||
- `static_rule_mismatch`
|
||||
- `incorrect_model_assumption`
|
||||
- `duplicate`
|
||||
- `insufficient_impact`
|
||||
- `unsupported`
|
||||
- `contradicted`
|
||||
- `unverified`
|
||||
|
||||
## Decision Standard
|
||||
|
||||
If evidence is uncertain, mark the finding `needs_evidence` and name the next evidence instead of
|
||||
escalating severity. Fusion (`fusion.json`, standard mode) is deterministic: it adjudicates between
|
||||
the quick-rule and MAP panels by fixed triggers and is not a second opinion from a model. Independent
|
||||
model agreement exists only in the plane's verify passes a and b, with pass c on disputes, and the
|
||||
`agree` step; those verdicts arrive as features of the decision engine, never as a verification
|
||||
status.
|
||||
|
||||
## What an Adjudication Feeds
|
||||
|
||||
- Recorded adjudications (a per-finding verdict with a reason) become decision-engine labels through
|
||||
the fail-closed source list `src/openultrasast/learn/sources.toml` (`[[source]] id =
|
||||
"adjudications"`). Aggregate counts label nothing; only per-finding verdicts do.
|
||||
- Plane verdicts (`agreed`, `rejected`, `disputed`) are features, never labels. "Rejected by the
|
||||
plane" is not a negative.
|
||||
- Rule changes go through the ledger: `ousast improve` (the validator and the detection gate, holdout
|
||||
pairs per round) or a registered experiment (`ousast learn experiment register|units|run|analyse`).
|
||||
Never a hand edit of `rules.toml` per miss or per false positive.
|
||||
|
||||
## Fix and Re-scan
|
||||
|
||||
1. Intake: restate the finding id, evidence level, affected path, and the acceptance criterion.
|
||||
2. Plan: the smallest defensive change that addresses the root cause.
|
||||
3. Implement: modify only the files the finding requires.
|
||||
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits. Use a
|
||||
differential review for changes in auth, crypto, config, parsers, and sandbox behaviour.
|
||||
5. Reconcile: fix each review finding or document why it does not apply.
|
||||
6. Verify: a re-scan of the checkout in the same mode, or a replay of the change with
|
||||
`ousast pre-push --base <before> --head <after> --artifact <file> <repo>`, shows the finding gone
|
||||
and no new findings. That re-scan is the evidence. Nothing in the tool records `patch_validated`;
|
||||
do not claim it.
|
||||
|
||||
Guardrails:
|
||||
|
||||
- Do not rewrite modules when a local fix is enough.
|
||||
- Do not change public APIs unless the vulnerability requires it.
|
||||
- Do not call a finding fixed from model reasoning alone; the re-scan or replay decides.
|
||||
+8
-2
@@ -6,9 +6,8 @@
|
||||
DEEPSEEK_API_KEY=
|
||||
# DEEPSEEK_BASE_URL=https://api.deepseek.com
|
||||
|
||||
# Embeddings only: OpenAI text-embedding-3-small through OpenRouter.
|
||||
# Embeddings only (`ousast learn memory embed`, default openai/text-embedding-3-small through OpenRouter).
|
||||
OPENROUTER_API_KEY=
|
||||
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
|
||||
# Plane memory store. Default is a local FileStore under ~/ousast-results/plane/memory.
|
||||
# For an S3-compatible server (RustFS is the tested one): OUSAST_MEMORY=s3://<bucket>[/<prefix>], or
|
||||
@@ -22,3 +21,10 @@ OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-small
|
||||
# AWS_SESSION_TOKEN=
|
||||
# The bucket must have versioning enabled, a lifecycle rule expiring runs/, S3 Select and readable object tags
|
||||
# (see docs/rustfs.md); the store verifies them at startup and refuses to run otherwise.
|
||||
|
||||
# Operator notifications (Discord). Status of OpenUltraSAST processes and their results for the
|
||||
# system/AI operator -- not developer debugging, and not k3s/ax infra health. See
|
||||
# .kiro/steering/operator-notifications.md. Webhook posts to the operator channel; the operator id
|
||||
# is the account to @mention on a stuck/impaired alert. Leave blank to disable notifications.
|
||||
DISCORD_WEBHOOK_URL=
|
||||
DISCORD_OPERATOR_ID=
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
name: Docs
|
||||
|
||||
# Builds the MkDocs site and publishes it to GitHub Pages on every push to main that touches it.
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths: ["docs/**", "mkdocs.yml", "README.md", ".github/workflows/docs.yml"]
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
concurrency:
|
||||
group: pages
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
enable-cache: true
|
||||
- uses: actions/setup-python@v6
|
||||
with:
|
||||
python-version-file: .python-version
|
||||
- run: uv sync --extra docs
|
||||
- run: uv run mkdocs build --strict
|
||||
- uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: site
|
||||
deploy:
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
@@ -0,0 +1,199 @@
|
||||
name: Engine image
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
tags: ['v*']
|
||||
paths:
|
||||
- 'plane/Dockerfile.*'
|
||||
- 'plane/task-base.digest'
|
||||
- 'Dockerfile'
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
concurrency:
|
||||
group: engine-image-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
pins:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
base_tag: ${{ steps.pins.outputs.base_tag }}
|
||||
base_digest: ${{ steps.pins.outputs.base_digest }}
|
||||
browser_tag: ${{ steps.pins.outputs.browser_tag }}
|
||||
browser_digest: ${{ steps.pins.outputs.browser_digest }}
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
# No login: every pinned base must be anonymously pullable.
|
||||
- id: pins
|
||||
run: |
|
||||
python3 - <<'PYTHON'
|
||||
import json, os, re, subprocess
|
||||
from pathlib import Path
|
||||
pins = json.loads(Path("plane/task-base.digest").read_text())
|
||||
manifests = {}
|
||||
for key, output in (("base", "base"), ("base-browser", "browser")):
|
||||
ref = pins.get(key)
|
||||
pattern = r"ghcr.io/norandom/ousast-task-base:(base-[a-z0-9-]+)@(sha256:[0-9a-f]{64})"
|
||||
match = re.fullmatch(pattern, ref or "")
|
||||
if not match:
|
||||
raise SystemExit(f"{key}: unpublished/invalid pin; run task-base-image and update plane/task-base.digest")
|
||||
raw = subprocess.check_output(["docker", "buildx", "imagetools", "inspect", "--raw", ref])
|
||||
manifests[key] = json.loads(raw)["layers"]
|
||||
with open(os.environ["GITHUB_OUTPUT"], "a") as out:
|
||||
out.write(f"{output}_tag={match[1]}\n{output}_digest={match[2]}\n")
|
||||
base = manifests["base"]
|
||||
if not base or manifests["base-browser"][:len(base)] != base:
|
||||
raise SystemExit("Browser pin does not share the pinned base's layers")
|
||||
PYTHON
|
||||
image:
|
||||
needs: pins
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- package: openultrasast
|
||||
dockerfile: Dockerfile
|
||||
- package: ousast-search-task
|
||||
dockerfile: plane/Dockerfile.search-task
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: names
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
PACKAGE: ${{ matrix.package }}
|
||||
run: |
|
||||
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
|
||||
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
|
||||
- id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.dockerfile }}
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
provenance: false
|
||||
build-args: |
|
||||
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
|
||||
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
|
||||
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
|
||||
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
|
||||
cache-from: |
|
||||
type=gha,scope=${{ matrix.package }}
|
||||
type=gha,scope=ousast-engine-task
|
||||
cache-to: type=gha,mode=max,scope=${{ matrix.package }}
|
||||
tags: |
|
||||
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
|
||||
${{ steps.names.outputs.image }}:main
|
||||
- env:
|
||||
IMAGE: ${{ steps.names.outputs.image }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: |
|
||||
echo "$IMAGE@$DIGEST"
|
||||
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
|
||||
docker logout ghcr.io
|
||||
docker buildx imagetools inspect "$IMAGE@$DIGEST"
|
||||
engine:
|
||||
needs: pins
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
image: ${{ steps.names.outputs.image }}
|
||||
digest: ${{ steps.build.outputs.digest }}
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: names
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
PACKAGE: ousast-engine-task
|
||||
run: |
|
||||
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
|
||||
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
|
||||
- id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: plane/Dockerfile.engine-task
|
||||
target: engine
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
provenance: false
|
||||
build-args: |
|
||||
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
|
||||
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
|
||||
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
|
||||
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
|
||||
cache-from: type=gha,scope=ousast-engine-task
|
||||
cache-to: type=gha,mode=max,scope=ousast-engine-task
|
||||
tags: |
|
||||
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
|
||||
${{ steps.names.outputs.image }}:main
|
||||
- env:
|
||||
IMAGE: ${{ steps.names.outputs.image }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: |
|
||||
echo "$IMAGE@$DIGEST"
|
||||
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
|
||||
docker logout ghcr.io
|
||||
docker buildx imagetools inspect "$IMAGE@$DIGEST"
|
||||
replay:
|
||||
needs: [pins, engine]
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: names
|
||||
env:
|
||||
OWNER: ${{ github.repository_owner }}
|
||||
PACKAGE: ousast-replay-task
|
||||
run: |
|
||||
echo "image=ghcr.io/${OWNER,,}/$PACKAGE" >> "$GITHUB_OUTPUT"
|
||||
echo "sha=sha-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
|
||||
- id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: plane/Dockerfile.engine-task
|
||||
target: replay
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
provenance: false
|
||||
build-args: |
|
||||
ENGINE_REF=${{ needs.engine.outputs.image }}@${{ needs.engine.outputs.digest }}
|
||||
TASK_BASE_TAG=${{ needs.pins.outputs.base_tag }}
|
||||
TASK_BASE_DIGEST=${{ needs.pins.outputs.base_digest }}
|
||||
TASK_BROWSER_TAG=${{ needs.pins.outputs.browser_tag }}
|
||||
TASK_BROWSER_DIGEST=${{ needs.pins.outputs.browser_digest }}
|
||||
cache-from: |
|
||||
type=gha,scope=ousast-replay-task
|
||||
type=gha,scope=ousast-engine-task
|
||||
cache-to: type=gha,mode=max,scope=ousast-replay-task
|
||||
tags: |
|
||||
${{ steps.names.outputs.image }}:${{ steps.names.outputs.sha }}
|
||||
${{ steps.names.outputs.image }}:main
|
||||
- env:
|
||||
IMAGE: ${{ steps.names.outputs.image }}
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
run: |
|
||||
echo "$IMAGE@$DIGEST"
|
||||
echo "$IMAGE@$DIGEST" >> "$GITHUB_STEP_SUMMARY"
|
||||
docker logout ghcr.io
|
||||
docker buildx imagetools inspect "$IMAGE@$DIGEST"
|
||||
@@ -34,3 +34,45 @@ jobs:
|
||||
generate_release_notes: true
|
||||
prerelease: ${{ contains(github.ref_name, '-') }}
|
||||
make_latest: ${{ !contains(github.ref_name, '-') }}
|
||||
|
||||
images:
|
||||
# The scan image (Joern + php-cli, `docker run ... scan`) and the plane's task runner, pushed to GHCR under the
|
||||
# tag's version. Pin them by the digests this job prints, not by tag: Agent Substrate rejects tags.
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
include:
|
||||
- name: openultrasast
|
||||
dockerfile: Dockerfile
|
||||
- name: openultrasast-runner
|
||||
dockerfile: plane/Dockerfile.runner
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}
|
||||
tags: |
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}},enable=${{ !contains(github.ref_name, '-') }}
|
||||
type=raw,value=latest,enable=${{ !contains(github.ref_name, '-') }}
|
||||
- id: build
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.dockerfile }}
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
- run: |
|
||||
echo "ghcr.io/${{ github.repository_owner }}/${{ matrix.name }}@${{ steps.build.outputs.digest }}" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
name: Task base image
|
||||
on:
|
||||
workflow_dispatch:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'plane/Dockerfile.base'
|
||||
- 'pyproject.toml'
|
||||
- 'uv.lock'
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
concurrency:
|
||||
group: task-base-image
|
||||
cancel-in-progress: false
|
||||
jobs:
|
||||
base:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
- uses: docker/setup-buildx-action@v3
|
||||
- uses: docker/login-action@v3
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- id: names
|
||||
run: |
|
||||
echo "tag=base-${GITHUB_SHA:0:7}" >> "$GITHUB_OUTPUT"
|
||||
content=$(cat plane/Dockerfile.base pyproject.toml uv.lock | sha256sum | cut -d' ' -f1)
|
||||
echo "content=base-content-$content" >> "$GITHUB_OUTPUT"
|
||||
- id: base
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: plane/Dockerfile.base
|
||||
target: base
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
provenance: false
|
||||
sbom: false
|
||||
tags: |
|
||||
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}
|
||||
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.content }}
|
||||
cache-from: type=gha,scope=task-base
|
||||
cache-to: type=gha,mode=max,scope=task-base
|
||||
- id: browser
|
||||
uses: docker/build-push-action@v6
|
||||
with:
|
||||
context: .
|
||||
file: plane/Dockerfile.base
|
||||
target: base-browser
|
||||
platforms: linux/amd64
|
||||
push: true
|
||||
provenance: false
|
||||
sbom: false
|
||||
build-args: |
|
||||
TASK_BASE_REF=ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}@${{ steps.base.outputs.digest }}
|
||||
tags: |
|
||||
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}-browser
|
||||
ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.content }}-browser
|
||||
cache-from: type=gha,scope=task-base-browser
|
||||
cache-to: type=gha,mode=max,scope=task-base-browser
|
||||
- name: Print manual pin update and verify public access
|
||||
env:
|
||||
BASE: ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}@${{ steps.base.outputs.digest }}
|
||||
BROWSER: ghcr.io/norandom/ousast-task-base:${{ steps.names.outputs.tag }}-browser@${{ steps.browser.outputs.digest }}
|
||||
run: |
|
||||
python3 - <<'PYTHON' >> "$GITHUB_STEP_SUMMARY"
|
||||
import json, os
|
||||
print("Replace plane/task-base.digest with this JSON after both public checks pass:")
|
||||
print("```json")
|
||||
print(json.dumps({"base": os.environ["BASE"], "base-browser": os.environ["BROWSER"]}, indent=2))
|
||||
print("```")
|
||||
PYTHON
|
||||
docker logout ghcr.io
|
||||
docker buildx imagetools inspect "$BASE"
|
||||
docker buildx imagetools inspect "$BROWSER"
|
||||
@@ -36,3 +36,9 @@ workspace/
|
||||
|
||||
# MkDocs build output (mkdocs build)
|
||||
site/
|
||||
|
||||
# Unseen evaluation pointers for non-permissive repositories.
|
||||
benchmarks/unseen/private/
|
||||
|
||||
# Feasibility pilot identities remain local; only aggregate records are tracked.
|
||||
benchmarks/search/private/
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"feature_name": "ai-service-plane",
|
||||
"created_at": "2026-09-29T15:40:00Z",
|
||||
"updated_at": "2026-09-29T17:55:00Z",
|
||||
"updated_at": "2026-10-02T00:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"phase": "implementation-complete",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document: contributor-scan
|
||||
|
||||
> **Status 2026-10-02:** paused since 2026-09-23 (its last change). 27 of 51 tasks done: the MAP integration was delivered and `v1.2.0-alpha.1` rolled out NO-GO (`census-repair-verification.md`); the remaining tasks wait on `pre-push-safety-net` M2/M3 and nothing here moved after the ax plane landed. Not active.
|
||||
|
||||
## Introduction
|
||||
|
||||
`model-grounded-detection` built an arbiter and measured it: 41.3% of the corpus reaches `model_entailed`
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
# Engine change-scoped analysis (G2 workstream)
|
||||
|
||||
Status: **proposed / in progress** (opened 2026-10-11). Owner: engine. Tracks a concrete lever for the
|
||||
engine-detection gap found while driving the G0 baseline. This is G2 ("improve what moves the unseen-repo
|
||||
numbers"), gated on the pool metric; it is NOT part of the G0 record-07 baseline (that runs on the quick-rule tier).
|
||||
|
||||
## Problem (measured, not assumed)
|
||||
|
||||
On real unseen vulnerability changes the engine produces **no finding** — but the diagnosis (live, 300 s long-deadline
|
||||
replays) shows it is usually **not a detection-capability gap**:
|
||||
|
||||
- **Clean Python injection change** (`aws/bedrock-agentcore-sdk-python`, `install_packages`, introducing, 3-file
|
||||
diff): `wall=303 s`, hit the deadline. The ranker found candidates (`static_rank=1.0`) but **`selected: []`** and
|
||||
every candidate `deferred` with `reason: deadline_exhausted`, `analysis_not_started=deadline_exhausted`,
|
||||
`change_context_incomplete`. **Zero analyses ran.** And the ranked candidates were **repo-wide and unrelated to
|
||||
the push** (`auth.py` access_control/config_secrets), not the changed `install_packages`. The engine is trying to
|
||||
re-audit the whole repo and runs out of budget before analyzing the change.
|
||||
- **Noisy last_touch config_secrets change** (`jenkinsci/metrics-plugin`, 14-file diff): finished in 206 s but
|
||||
`comparison_unknown` + `capability_unavailable` + `eligibility_provenance_mismatch` +
|
||||
`contributor-scan ... projection_unavailable`. Multi-causal; correspondence was one part (fixed separately,
|
||||
commit c39fbb8), capability registry is Task 8.3.
|
||||
|
||||
## Hypothesis
|
||||
|
||||
A pre-push hook should vet **the code this push changed**, not re-scan the whole repository. Seeding/scoping the
|
||||
engine's analysis to the **changed declarations and their dataflow neighborhood** (the `change_sites` are already in
|
||||
the replay spec) should let analysis **actually run within budget** instead of being deferred — which is where
|
||||
detection is being lost, not in Joern's capability.
|
||||
|
||||
## Anchors (confirm in increment 1)
|
||||
|
||||
- `src/openultrasast/rank.py` — the static-evidence ranker (`ranking_mode="evidence"`, `existing-static-evidence-v1`).
|
||||
- `src/openultrasast/mapping.py` — builds the `selected`/`deferred` scope + the budget allocation that defers on
|
||||
`deadline_exhausted`.
|
||||
- `src/openultrasast/push/policy.py`, `push/runner.py` — consume the scope; change_sites arrive via the push input.
|
||||
- `benchmarks/unseen/replay_entry.py` — carries `changed_paths` / `declaration_paths` in the spec.
|
||||
|
||||
## Increments (progress tracking)
|
||||
|
||||
- [x] **0. Cost split confirmed (2026-10-11, timed live replay).** `build_seconds=66.69` (CPG build is cheap),
|
||||
`query_seconds=0.0` (no query ever executed), total 303 s (hit the deadline), **4,095** candidate questions all
|
||||
`analysis_not_started=deadline_exhausted`, `selected: []`, `regions_scanned=0`. So of the 303 s, ~67 s built the
|
||||
CPG and ~236 s went to enumerating/ranking/preparing 4,095 repo-wide candidate questions for a **3-file** change —
|
||||
the budget was spent before a single query ran. The CPG build is NOT the bottleneck; the over-broad candidate set
|
||||
is. Change-scoping is confirmed as the right lever, and because per-candidate prep includes the line-correspondence
|
||||
work, fewer candidates also cuts that cost (the two fixes compound).
|
||||
- [x] **1. Located (2026-10-11).** The over-enumeration is upstream of the scan, not inside it:
|
||||
- `push/runner.py:345-348` — `head_regions = discovery(tip.root, …)` discovers regions across the **whole head
|
||||
tree** (the ~4,095), independent of the change.
|
||||
- `push/runner.py:397-407` — those regions are passed wholesale to `scan_repository(..., change_context=context)`;
|
||||
`change_context` is threaded in but **never scopes the region set**.
|
||||
- `model/scan.py:167` and `:364` — `[_identity(unit, r, f) for r in regions for f in r.families]` enumerates all
|
||||
regions × families; with `order_by_evidence=True` (`runner.py:288/403`) the scan evidence-tiers **every** pair
|
||||
before analysis — that pre-work is the ~236 s measured in increment 0, so the deadline hits at `query_seconds=0`.
|
||||
- Primitive already present: `model/regions.py:279` computes the changed-path set from the context;
|
||||
`compare_targeted_base` (`runner.py:411`) is a precedent for change-targeted work.
|
||||
- **Fix locus (increment 2):** restrict the *analyzed* region set to the change's declarations + dataflow
|
||||
neighbourhood (filter `head_regions` by `context.changed_paths` before the scan, or scope the enumeration inside
|
||||
`scan_repository`). Keep the **CPG whole-repo** so taint queries can still trace sources from unchanged code into
|
||||
the changed regions — only the question set is change-scoped.
|
||||
|
||||
**Correction (2026-10-11), after a reverted first attempt — two things the naive filter got wrong:**
|
||||
- **The engine already reaches into unchanged code by design, and it is tested.** A changed `source` that reaches
|
||||
an unchanged `sink` must still select the unchanged sink's question
|
||||
(`tests/test_model_change_context.py::test_changed_dependency_reaches_unchanged_ranked_question`;
|
||||
`test_change_does_not_replace_ranker_selection` pins that change context must NOT alter the ranker's selection).
|
||||
`_changed_files` feeds `_evidence_pass` only as `contextFilter`/`contextPaths` — which locations to itemise vs
|
||||
summarise — NOT to shrink the tier population. A "changed files only" scope deletes the reach-into-unchanged
|
||||
behaviour. Any scope must keep changed regions **plus the regions they reach** (the affected/`relationships`
|
||||
neighbourhood), not changed files alone.
|
||||
- **The 236 s is now split (2026-10-11, saved artifacts).** `total=291s build=52s query=0s arbitrate=238.75s`,
|
||||
and `query_seconds_by_kind["evidence"]=238.54s` — the evidence-tier pass is **99.9 %** of the arbitrate phase;
|
||||
`affected_context` + `_scope_work` are **0.21 s**. The pass tiered 1,015 pairs (`tier_counts 0:1009, 1:6`) at
|
||||
**~235 ms/pair** (contended; ~120 ms uncontended) and that alone blew the budget — `selected=0`,
|
||||
`deferred=1365` all `deadline_exhausted`. Secondary anomaly: the design assumes ~2.5 ms/pair
|
||||
("17,700 in 44 s"); here it is 50–100× that, i.e. the evidence query is pathologically slow on a cold large
|
||||
CPG — a separate optimisation from cutting the pair count.
|
||||
- **So increment 2's single target is the evidence pass** (`scan.py:338-359`, `_evidence_pass`). The fork is HOW
|
||||
to cut it without deleting reach-into-unchanged:
|
||||
- (i) **Scope + bound:** order the evidence candidates change-first and bound the pass to a fraction of the
|
||||
budget so analysis queries still run; reached unchanged regions are tiered only while budget lasts.
|
||||
- (ii) **Neighbourhood pass:** tier the changed regions, read the reached regions from the context rows the
|
||||
pass already returns, tier those in a second small pass — exact reach-into-unchanged, more code.
|
||||
- (iii) **Per-pair speedup:** attack the ~235 ms/pair (warm server / batching) rather than the pair count —
|
||||
helps every scan but is a deeper engine change and does not by itself bound a huge repo.
|
||||
|
||||
**CORRECTION (2026-10-11, after implementing (i) + four live re-verifies — the cost model was wrong).**
|
||||
Scope + bound was built and tested (change-first ordering, adaptive budget-fitted portioning, `_context_pass`
|
||||
scoped to changed files, honest `evidence_pass_bounded` degradation; branch `engine-change-scoped-evidence-pass`).
|
||||
It correctly cut the tiered pair count — but `query_seconds` stayed **0** every time. The live numbers show why:
|
||||
|
||||
| pairs tiered | run_batch calls | evidence s | build s |
|
||||
|---|---|---|---|
|
||||
| 1015 | 1 | 238 | 52 |
|
||||
| 512 | 1 | 219 | 68 |
|
||||
| 64 | 2 | 220 | 68 |
|
||||
| 4 (probe) | 1 | 160 | 127 |
|
||||
|
||||
Evidence time is **flat regardless of pair count** → the cost is **per-*call* fixed** (~150–220 s: loading the CPG
|
||||
into the query engine + the census the evidenceOnly batch computes), ~85 ms/pair on top. Pair-count scoping
|
||||
therefore cannot free budget, and portioning re-pays the fixed cost per call (so (i) can be counterproductive).
|
||||
Build is also large and swings 52→127 s — two engines thrashing the 7 GB host (the record-07 baseline shares it).
|
||||
The real levers are now (iii)-shaped, not (i): **warm/persistent query engine or a cached census** to kill the
|
||||
~150 s per-call fixed cost, and/or a **bigger engine-arm budget** than 300 s; plus a **clean (uncontended)
|
||||
measurement**, because contention alone may be inflating build + fixed cost 2–3×. Scope+bound is NOT merged.
|
||||
|
||||
**UNCONTENDED measurement (2026-10-11, baseline paused via SIGSTOP + container kill, then resumed).** Shipped
|
||||
engine, one evidence call over 1,015 pairs: `build=48 s`, `evidence=243.79 s`, `query=0`. The evidence pass is
|
||||
**the same ~244 s uncontended as ~238 s contended** → the cost is **INTRINSIC, not contention**; contention only
|
||||
inflated `build` (48 → 127 s) and added run-to-run noise. The model is `evidence ≈ 150 s fixed (CPG load into the
|
||||
query engine + the census the evidenceOnly batch computes) + ~93 ms/pair` (244 = 150 + 1015×0.093). Consequences:
|
||||
- **~200 s of fixed overhead** (`build` + evidence-load) precedes any analysis, so a 300 s budget leaves only
|
||||
~100 s for everything else. This is why `query=0` even on a clean host.
|
||||
- **Change-scoping still helps, but only by cutting the per-pair tail** (~1,015×93 ms ≈ 95 s): one evidence call
|
||||
over ~30 changed pairs is ~150 s + 3 s instead of 244 s, which would free ~90 s — possibly enough for analysis,
|
||||
but marginal.
|
||||
- **It must be ONE call, not portioned.** The ~150 s is a first-call load, so the adaptive probe attributed it to
|
||||
its 4 probe pairs (per-pair≈37 s), concluded the budget fit ~0 more, and tiered only 4 — the implemented
|
||||
scope+bound is doubly wrong for this cost model. The correct shape: change-first candidates, count-capped, ONE
|
||||
evidence call, `deadline=None`.
|
||||
- **Pre-push viability:** the ~200 s fixed floor means the engine arm cannot be a fast (≤30 s) pre-push check on a
|
||||
repo this size without a warm/persistent session or a cached census. A larger budget (e.g. 600 s) makes the
|
||||
benchmark arm run but is not a realistic pre-push deadline. The lever that matters for the product is killing
|
||||
the ~150 s fixed cost, not scoping pairs.
|
||||
- [ ] **2. Seed analysis from the change.** Prioritize (or restrict first-pass to) the changed declarations plus
|
||||
their call/data neighborhood over repo-wide static rank, within budget.
|
||||
- [ ] **3. Soundness.** Keep cross-file dataflow INTO the changed function (don't scope to literal changed lines);
|
||||
add tests; keep existing push tests green.
|
||||
- [ ] **4. Live re-test.** Re-run the bedrock change: does analysis now RUN and produce a finding on
|
||||
`install_packages` (vs `deferred`)? Re-run metrics-plugin.
|
||||
- [ ] **5. Measure the delta.** Engine-arm sample on the pool (catch/false-alarm + engine-coverage rate) — needs the
|
||||
kube lane (blocked on the AX completion path) for scale.
|
||||
|
||||
## Dependencies / notes
|
||||
|
||||
- Coverage `complete_within_scope` also needs the capability registry (Task 8.3) regardless of this work; the win
|
||||
here is findings moving from `comparison_unknown`/deferred to a decided novelty → advisory engine finding.
|
||||
- Implement directly and verify on the LIVE instrument (a fake-client/unit pass is not evidence the live engine
|
||||
changed); the kube-lane and store endpoints are operator-owned infra, not assumptions to make here.
|
||||
</content>
|
||||
@@ -1,5 +1,7 @@
|
||||
# Requirements Document: finding-feedback-loop
|
||||
|
||||
> **Status 2026-10-02:** dormant. Requirements approved and design generated on 2026-09-09; the design was never approved and no tasks exist. It depends on `contributor-scan`, which is paused; reopen it with that spec.
|
||||
|
||||
## Introduction
|
||||
|
||||
Nine engine defects were found in one session against two repositories. **Three were the same bug in three
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Plan: learn fix mechanisms into the engine (draft for review, 2026-10-04)
|
||||
|
||||
Status: approved 2026-10-04 by the maintainer. Increment 1 = discharge facts only; coverage additions (Req 7) come later.
|
||||
Decisions: minimum 2 proposing repositories before a candidate is tried; admission per language (not pooled by callee name).
|
||||
|
||||
## Why
|
||||
|
||||
The Joern trace run over exp-005's 501 units shows the engine cannot tell a vulnerable function from its
|
||||
fix. In the 105 pairs where both sides were analysed, a taint path appears on both sides in 12, on the
|
||||
vulnerable side only in 2, on the fixed side only in 0, and on neither in 91
|
||||
(`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`).
|
||||
|
||||
So the fix is invisible to the engine. Either the flow survives and the engine does not know the fix's
|
||||
mechanism (an `allowed_classes` argument, a path normalisation check, an escaping call), or the engine's
|
||||
sources and sinks miss the function on both sides. A classifier fed this evidence cannot improve on it.
|
||||
|
||||
The corpus now holds 710 complete fix pairs (`2026-10-04-decision-engine-harvest-3/record.json`). Each fix
|
||||
diff shows what was added. That is the material the engine lacks.
|
||||
|
||||
## Goal
|
||||
|
||||
Grow the engine's discharge knowledge (sanitizers, guards, safe-argument forms) and, second, its source and
|
||||
sink coverage, from the fix diffs. Admit a fact only when it improves held-out separation. Spend no model
|
||||
money.
|
||||
|
||||
## Requirements (proposed)
|
||||
|
||||
1. **Mining.** From each fix pair, extract what the fixed function adds on the vulnerable flow: new calls,
|
||||
new conditions guarding the sink, new or changed sink arguments. Record each as a candidate fact with its
|
||||
language, family, form (sanitizer call, guard predicate, safe argument, sink restriction) and the pairs
|
||||
that propose it. Mining is deterministic (tree-sitter diff of the two function bodies). No model calls.
|
||||
2. **Generalisation is explicit.** A candidate is a pattern, not a repository name: a callee name, an
|
||||
argument key, a predicate shape. Candidates naming project-specific identifiers are rejected (Req 8: no
|
||||
application-specific trees).
|
||||
3. **Admission by held-out separation.** Folds are by repository, as in the decision engine. A candidate is
|
||||
admitted for a fold only if, on that fold's held-out pairs, adding it to the fact tables:
|
||||
- increases pairs where the vulnerable side keeps a path and the fixed side loses it or becomes
|
||||
sanitized;
|
||||
- and does not remove any held-out vulnerable-side path (no recall loss).
|
||||
A candidate proposed by fewer than 2 distinct repositories in the training folds is not tried.
|
||||
4. **No train-on-test.** Mining and admission never read held-out pairs; the closed-loop leak recorded in
|
||||
memory (`closed-loop-train-on-test-leak`) is the failure this rule exists to prevent. Population v3 is
|
||||
never read.
|
||||
5. **Measurement by the trace runner.** Each admission round re-runs the engine on the affected pins only
|
||||
(the runner's resume and selection already support this; on k3s the worker Deployment runs them in
|
||||
parallel). The outcome is the pair-asymmetry table per family, before and after, on held-out folds.
|
||||
6. **Gate before production.** An admitted fact enters `src/openultrasast/ruleset/semantic/*.toml` only after
|
||||
the detection gates (`gate`, `map_gate`, `pair_gate`) show no regression, and only as a registered
|
||||
experiment whose decision rule was fixed before the run.
|
||||
7. **Coverage second.** After discharge facts, the same loop proposes source and sink additions from pairs
|
||||
where neither side has a path (91 of 105 today), admitted under the same held-out rule.
|
||||
|
||||
## How it relates to what exists
|
||||
|
||||
- `corpus-seeded-mechanisms` already has a closed loop that admits and retracts mechanisms by leave-one-out.
|
||||
This plan reuses its lever and store, and points it at discharge facts mined from real fix diffs instead
|
||||
of variant search.
|
||||
- `benchmarks/pairs/mechanisms.toml` (23 mechanisms) labels what kind of bug a pair is. The new candidates
|
||||
describe what a fix does. They live in the fact tables, not in the mechanism vocabulary.
|
||||
|
||||
## Measures of success
|
||||
|
||||
- Pair asymmetry on held-out folds: vulnerable-only paths rise from 2 of 105 toward a substantial share,
|
||||
with zero fixed-only paths.
|
||||
- No held-out recall loss on vulnerable sides.
|
||||
- Then the decision engine is re-run with the new engine evidence (exp-005 design), and the pooled block gate
|
||||
(exp-004 design) is re-measured.
|
||||
|
||||
## Cost and time
|
||||
|
||||
- Model cost: $0 (mining and admission are deterministic; the classifier re-run afterwards is a separate,
|
||||
approved spend).
|
||||
- Engine time: one pass over affected pins per admission round; about 2 hours per full pass on k3s plus the
|
||||
VM, about 6 hours on the VM alone.
|
||||
|
||||
## Open questions for the maintainer
|
||||
|
||||
1. Should coverage additions (Req 7) be in the same increment or a later one?
|
||||
2. Minimum proposing repositories: 2 is the proposal. Higher is safer, lower finds more.
|
||||
3. Admission per family and language, or pooled across languages for the same callee name?
|
||||
@@ -119,6 +119,26 @@ rule hits.
|
||||
2. Without a trained model available the tools fall back to today's behaviour and say so in the report.
|
||||
3. Hand-written rules remain as signal producers; changing their enabled/shadow status no longer changes what is
|
||||
reported unless the engine's decision changes.
|
||||
4. *(Amended 2026-10-03, maintainer decision; refines the BLOCK rule of 7.3 and 7.4. Tested by exp-004.)* BLOCK
|
||||
precision is demonstrated pooled across families, not per family:
|
||||
- **Normalisation**, declared before any run: each family's raw score is put on one scale by a per-family Platt
|
||||
map (`learn.calibrate.fit_map` with its floors) fitted on training folds only. A family the training folds
|
||||
cannot map does not block on that fold.
|
||||
- **t_block**: one threshold on the normalised score for all families, chosen on training folds as the lowest
|
||||
one whose Wilson 95% lower bound on precision, over the training flags of all families together, is >= 0.95.
|
||||
It is reported by the Wilson 95% lower bound over all held-out flags of all families, before the floor.
|
||||
- **Family floor**: a family may block only if its own held-out flags have zero errors and number at least F
|
||||
(F = 10, reasoned in the exp-004 registration); otherwise that family stays ADVISORY.
|
||||
- **t_quiet**: per family, the highest threshold keeping >= 90% of the paired positives at or above it (chosen
|
||||
on training folds, its held-out recall reported); findings below it are silent.
|
||||
- No held-out label moves a threshold or a normalisation map; candidates whose verdict is `unsure` never block.
|
||||
5. *(Amended 2026-10-05, maintainer decision, with `search-with-proof`.)* Where a family has an owned demonstration
|
||||
oracle, the engine is **triage, not the gate**: BLOCK comes only from a verified demonstration (`search-with-proof`
|
||||
Req 3, 6.3), and the engine's job is to put the true vulnerable function among the top K candidates of a push.
|
||||
For those families the engine is judged by (a) top-K recall per push (K = 3 to start) on the unseen-repo pool and
|
||||
(b) its advisory false-alarm rate there; the pooled block gate of 6.4 then governs only families without an
|
||||
oracle (access control, deserialisation, configuration and secrets), where the engine remains the only signal.
|
||||
Every arm reports top-K recall next to its catch and false-alarm rates.
|
||||
|
||||
### Requirement 7: Built for unseen repositories -- a repo safety net
|
||||
|
||||
@@ -136,7 +156,8 @@ rule hits.
|
||||
ADVISORY, trading precision for recall; both reported with out-of-repository intervals. The decision unit is a
|
||||
candidate in changed code (`pre-push` delta), with whole-repository scans as a secondary mode.
|
||||
4. Calibration is checked on held-out repositories; where it does not hold, only ADVISORY is offered for that
|
||||
family.
|
||||
family. *(Amended 2026-10-03: once exp-004 adopts the pooled gate of 6.4, BLOCK precision is shown by its held-out
|
||||
bound and family floor, and calibration is reported, not required.)*
|
||||
5. Any adaptation to a user's own repository (facts reuse, their dismissals) is opt-in, separate from the
|
||||
generalisation model, and never contributes to its reported numbers.
|
||||
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"feature_name": "learned-decision-engine",
|
||||
"created_at": "2026-09-30T14:00:00Z",
|
||||
"updated_at": "2026-10-01T10:20:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
|
||||
@@ -8,6 +8,10 @@ memory, compiled DSPy-style in-house); tasks 1-5 are unaffected, tasks 6-10 are
|
||||
against the provider balance read just before it, and none starts with a ceiling above that balance. No test makes
|
||||
a model or embedding call.
|
||||
|
||||
Direction 2026-10-04: `.kiro/steering/roadmap.md` (section of 2026-10-04, phases G0-G5) is the authority. Every
|
||||
change is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. Within-pair AUC stays a development signal. Tasks 11-14 below follow that roadmap.
|
||||
|
||||
- [x] 1. Schema and features
|
||||
- `learn/schema.py` (closed, typed allow-list; no identities; missing != zero), `learn/features.py` (host
|
||||
builder), new memory `KINDS`, `plane/tasks/features.py` and the `remember` input; tests enforcing the
|
||||
@@ -45,6 +49,11 @@ a model or embedding call.
|
||||
- _Requirements: 1.1_
|
||||
|
||||
- [ ] 6. Program, local memory and compilation (design sections 4-5)
|
||||
- Status 2026-10-04: 6.1-6.6 are done. 6.7 is partial: leave-one-source-out, the 0/25/50% memory-size points and
|
||||
the source-count curve are not run. Around this task the experiment machinery grew: rule arms
|
||||
(`precision_bound_block`, `pooled_block_gate` with a family floor) in `learn/experiments.py`, the response-cache
|
||||
fill with a hard ceiling and a dry-run estimate in `learn/fill.py`, and the in-function slice in `learn/slice.py`.
|
||||
These ran exp-003, exp-004 and exp-005 (7.5-7.7). Memory now holds 710 complete fix pairs (7.9).
|
||||
- [x] 6.1 Excerpts and the example store (no model)
|
||||
- `learn/excerpt.py` (function span at the label's pin, bounds 80/4,000 for candidates and 40/2,000 for
|
||||
examples, numbered lines, no path/repository/commit, advisory ids and security wording in comments
|
||||
@@ -116,12 +125,63 @@ a model or embedding call.
|
||||
- _Requirements: 5.1, 5.2, 8.3_
|
||||
|
||||
- [ ] 7. Experiments (exp-001 and program variants need budget go-ahead)
|
||||
- `learn/experiments.py`, `register`, `experiment-outcome`, Run annotation checks, `OUSAST_VERIFY_CALLERS`;
|
||||
exp-001 (known-callers line on/off, ~$7.7 expected, ceiling $15) registered, run and recorded -- an
|
||||
inconclusive result is pre-registered as likely.
|
||||
- Program-variant experiments (paired on candidates, repository cluster, arm A replayed from cache at $0): the
|
||||
machinery and its test; the first variant (`k = 5` vs `k = 1`, or the confidence source) **(budget go-ahead:
|
||||
~$0.9-1.35 per arm-B pass of 300 candidates, ceiling $2 each)**.
|
||||
- [x] 7.1 Machinery (no model in tests): `learn/experiments.py` -- YAML manifests under `plane/experiments/<id>.yaml`
|
||||
(hypothesis, arms as program-spec overrides or plane `task_env`, unit, pairing, repository cluster, metrics with
|
||||
`[label=]` selectors and targets, MDE, two looks with O'Brien-Fleming boundaries, budget ceiling, families, folds,
|
||||
frozen units file), `register` (`experiment` row with both digests; refuses an uncommitted, modified or re-declared
|
||||
manifest), `units` (freeze the families' evaluation candidates, paired by candidate across pins; no model), `run`
|
||||
(paired arms in a seeded repository order, a seeded coin per unit for which arm goes first, `arm_outcome` rows,
|
||||
resumable, arm A replay-only at $0, canary replicates under a fresh cache salt, the first look during the run),
|
||||
`analyse` (paired repository bootstrap, exact McNemar, the looks, adopt / reject / equivalent / inconclusive,
|
||||
`experiment_result` row + `benchmarks/experiments/<id>/result.json`), `ousast learn experiment
|
||||
register|units|run|analyse`; `OUSAST_VERIFY_CALLERS=off` in `plane/tasks/verify.py`; the retrieval-ensemble
|
||||
sampling variant (`ProgramSpec.sampling = retrieval_ensemble`: `k` samples at temperature 0 over `k` disjoint
|
||||
retrieved example sets; default unchanged). Tests: `test_learn_experiments.py` (18, scripted clients, FileStore).
|
||||
- [x] 7.2 Registered, not run: `exp-001-known-callers` (plane arms, `flag_rate[label=1]`, MDE 0.15, ceiling $15;
|
||||
units `pending` -- model-free candidate generation on the 41 development repositories is not built, and plane Run
|
||||
annotation checks / the `experiment-outcome` task are not written) and `exp-002-retrieval-ensemble` (arm A today's
|
||||
k = 5 temperature sampling, arm B the retrieval ensemble; primary canary agreement (target 0.90) and within-pair
|
||||
AUC, secondary usd per candidate, AUC, ADVISORY recall/precision; families injection and access_control on the
|
||||
slice-2 candidates; ceiling $4: A $1, B $3). Both registered in `s3://sast-memory` at fbc3220
|
||||
(`benchmarks/experiments/<id>/registration.json`). exp-002's units are `pending` too: on 2026-10-02 every example
|
||||
row in the store carried null plane-only instruments (`agree`, `model_sinks`, `verify`) and `load_examples` raised
|
||||
on them, so `ousast learn experiment units` could not freeze the candidates.
|
||||
- [ ] 7.3 exp-001 run and recorded (~$7.7 expected, ceiling $15) -- an inconclusive result is pre-registered as
|
||||
likely; needs the frozen units file first.
|
||||
- [x] 7.4 exp-002 run and recorded 2026-10-02 (budget go-ahead ~$2 expected, ceiling $4): **reject B**. 160 paired
|
||||
units (injection 119, access_control 41; 56 repositories; 108 units in 54 candidate pairs), both arms, two
|
||||
replicates each, no early stop. Primary: canary agreement A 0.875 -> B 0.906 (diff +0.03, CI [-0.03, +0.10],
|
||||
spans; B meets the 0.90 target), within-pair AUC A 0.815 -> B 0.741 (diff -0.07, CI [-0.13, -0.02], **against B**).
|
||||
Secondary: AUC -0.02 (CI [-0.04, -0.003], against), ADVISORY recall -0.06 (spans; McNemar 7/2, p = 0.18),
|
||||
precision +0.01 (spans), usd per candidate A $0.0021 -> B $0.0051 (CI [+0.0027, +0.0032], against: the five
|
||||
disjoint example sets share only the prompt prefix in DeepSeek's cache). Priced usage A $0.68 (1206 paid calls,
|
||||
395 replayed), B $1.63 (1350 paid, 250 replayed). Result: `benchmarks/experiments/exp-002-retrieval-ensemble/result.json`,
|
||||
`benchmarks/measurements/2026-10-02-exp-002/record.json`. Three repairs on the way (each its own commit): example
|
||||
rows' null plane-only features (1f525ff), the 788 embedding vectors absent from RustFS (copied from the file
|
||||
store), the result row's column-type collision under S3 Select (780caaa). Further program variants (`k = 5` vs
|
||||
`k = 1`, the confidence source) as new manifests.
|
||||
- [x] 7.5 exp-003 run and recorded 2026-10-03 ($0, replay only): **inconclusive**. Selective blocking (a per-family
|
||||
raw-score threshold chosen on training folds by its Wilson lower bound) offers no BLOCK on held-out paired units
|
||||
in any family; the incumbent stays. 211 of 353 frozen units evaluated (142 replay misses). Proving 0.95 needs at
|
||||
least 73 clean flags. Result: `benchmarks/experiments/exp-003-precision-bound-blocking/result.json`.
|
||||
- [x] 7.6 exp-004 run and recorded 2026-10-03: **inconclusive**. One pooled BLOCK threshold over all families with a
|
||||
family floor (Req 6.4). No outer fold found a threshold, pooled or per family; B offers no BLOCK. 501 units,
|
||||
replay at $0 after a cache fill of $1.24 (ceiling $2; the dry-run estimate over-priced 4.5x). Result and record:
|
||||
`benchmarks/experiments/exp-004-pooled-block-gate/result.json`, `record.json`.
|
||||
- [x] 7.7 exp-005 run and recorded 2026-10-03: **inconclusive**. The in-function def-use slice in the prompt of the
|
||||
six slice-2 programs. Within-pair AUC 0.762 -> 0.746 (CI [-0.063, 0.032], spans 0); unsure rate 0.29 -> 0.24
|
||||
(favours B); usd per candidate slightly higher. A $0 leak audit came first
|
||||
(`benchmarks/measurements/2026-10-03-graph-slice-leak-audit/record.json`). Result:
|
||||
`benchmarks/experiments/exp-005-graph-slice/result.json`.
|
||||
- [x] 7.8 Negatives harvest 2026-10-03 ($0.0014 embeddings, no chat-model call): 98 real fix pairs whose function
|
||||
survives the fix, from reviewed advisories, as pointer-pair corpus `advisory-fixes-2`; memory 937 -> 1133
|
||||
examples. Record: `benchmarks/measurements/2026-10-03-decision-engine-negatives-harvest/record.json`.
|
||||
- [x] 7.9 Harvest 3, 2026-10-04 ($0.003 embeddings, no chat-model call): 251 pairs as `advisory-fixes-3`; complete
|
||||
pairs in memory 459 -> 710. Record: `benchmarks/measurements/2026-10-04-decision-engine-harvest-3/record.json`.
|
||||
- [x] 7.10 Joern trace run over exp-005's 501 units and its leak audit, 2026-10-04 ($0): no leak flagged. In the 105
|
||||
pairs where both sides were asked, a path appears on both sides in 12, on the vulnerable side only in 2, on the
|
||||
fixed side only in 0, and on neither in 91. The engine cannot see what a fix changed. Record:
|
||||
`benchmarks/measurements/2026-10-04-joern-slice-leak-audit/record.json`.
|
||||
- _Requirements: 3.6, 4.1-4.4_
|
||||
|
||||
- [ ] 8. Integration
|
||||
@@ -147,3 +207,60 @@ a model or embedding call.
|
||||
`result-v3-engine.json` is committed. Nothing between the two commits may touch the program, memory snapshot,
|
||||
features, priors or thresholds. This spec's code never opens v3 files.
|
||||
- _Requirements: 5.3_
|
||||
|
||||
- [ ] 11. Fix mechanisms (plan approved 2026-10-04; see `fix-mechanisms-plan.md`)
|
||||
- Roadmap G2 arm (`.kiro/steering/roadmap.md`, section of 2026-10-04): one candidate improvement, not the spine.
|
||||
It is judged on the unseen-repository pool (G0, spec `unseen-repo-evaluation`) by catch rate and false-alarm
|
||||
rate per push. **Paused after 11.1 until the G0 pool exists.**
|
||||
- [ ] 11.1 Mine candidate discharge facts from the 710 fix diffs (sanitizer calls, guarding conditions, safe sink
|
||||
arguments, safe replacements); deterministic, $0. In progress.
|
||||
- [ ] 11.2 Admission by held-out separation per outer fold: a candidate turns "path on both sides" into "path on the
|
||||
vulnerable side only" on folds it was not mined from, removes no held-out vulnerable path, has at least 2
|
||||
proposing repositories, and is admitted per language.
|
||||
- [ ] 11.3 Engine re-run on the affected pins (kube-ax lanes plus the VM lane).
|
||||
- [ ] 11.4 Registered experiment, with the detection gates green, before `ruleset/semantic/*.toml` changes.
|
||||
- [ ] 11.5 Coverage increment: source and sink additions for pairs where neither side has a path (later).
|
||||
- [ ] 11.6 TypeScript admission in the engine (branch `engine-typescript`).
|
||||
- _Requirements: 4.1-4.4, 7.1, 8.1_
|
||||
|
||||
- [ ] 12. exp-006: Joern witness slice in the prompt **(budget go-ahead, about $2-4)**
|
||||
- Planned, roadmap G1/G2. Against the current prompt, after a $0 leak audit, on re-frozen units. Judged on the
|
||||
unseen-repository pool, not on within-pair AUC alone.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 13. exp-007: the combiner ($0 replay)
|
||||
- Planned, roadmap G1/G2. Three or four weights (score, engine entailment, neighbour agreement) over cached
|
||||
responses. Judged on the unseen-repository pool.
|
||||
- _Requirements: 4.1-4.4, 7.1_
|
||||
|
||||
- [ ] 14. Re-run the pooled block gate on re-frozen units
|
||||
- Planned, roadmap G3. The exp-004 design with the family floor (Req 6.4), together with the pool's false-alarm
|
||||
rate per family.
|
||||
- _Requirements: 6.4, 7.3_
|
||||
|
||||
- [ ] 15. Trained classifier arm (maintainer decision 2026-10-04: train a model, not hand-set profiles; judged on the
|
||||
unseen-repo pool, roadmap G1)
|
||||
- [ ] 15.1 Features: code embedding of the excerpt (stored vectors) + every allow-listed signal; no hand-picked
|
||||
profile. Automated screening: a signal that separates pair sides by itself on training folds above the leak
|
||||
threshold is dropped for that fold (the leak audit as a filter, fitted per fold).
|
||||
- [ ] 15.2 Models: regularised logistic regression and gradient-boosted trees, fitted per repository-held-out outer
|
||||
fold with inner-fold hyper-parameter search; per-family and pooled variants.
|
||||
- [ ] 15.3 Scores on held-out folds: within-pair AUC, pooled AUC, top-k precision, Wilson bound at the pooled
|
||||
threshold; compared with the current LLM programs on the same units.
|
||||
- [ ] 15.4 Scored on the unseen-repo pool when G0 exists: catch rate at fixed false-alarm budgets.
|
||||
- _Requirements: 3 (revised 2026-10-04 by maintainer decision), 6, 7_
|
||||
|
||||
- [ ] 16. Jointly optimised LLM program arm (judged on the unseen-repo pool)
|
||||
- Search input fields, retrieval parameters, k, instruction and demonstrations together (DSPy MIPRO-style) on
|
||||
training folds; no hand-set input profile. Budget go-ahead before paid search.
|
||||
- _Requirements: 3, 6, 7_
|
||||
|
||||
- [ ] 17. Triage objective (Req 6.5, amended 2026-10-05 with `search-with-proof`)
|
||||
- [ ] 17.1 Metric: top-K recall per push (K = 1, 3, 5) of the true vulnerable function among changed functions,
|
||||
on held-out folds and on the unseen-repo pool; added to the pool scorer next to catch and false alarm.
|
||||
- [ ] 17.2 Score the existing arms on it ($0 replay): trained logistic (cheapest per candidate, likely default),
|
||||
LLM programs (exp-005 A), engine-evidence ordering, and an averaged ensemble.
|
||||
- [ ] 17.3 Re-scope 14 (pooled block gate) to families without a demonstration oracle; 15.4 and 16 report top-K
|
||||
recall as their primary unseen-pool number for families with an oracle.
|
||||
- _Requirements: 6.5, 7_
|
||||
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Brief: MCP shift-left agent + skills
|
||||
|
||||
Status: brief / not yet specced. Proposed 2026-10-07. Delivery surface for the
|
||||
qualified safety net; builds on the existing `ousast mcp` server and the product
|
||||
skills. Related: `proof-arm-opencode` (the proof model), `pre-push-safety-net`
|
||||
(the gate), `unseen-repo-evaluation` (the qualification).
|
||||
|
||||
## Motivation
|
||||
|
||||
The pre-push hook is the gate. Developers and their coding agents want the same
|
||||
security feedback **earlier**, while the code is still being written, not only at
|
||||
push. And in an agent-driven workflow, a mandatory security check is how a human
|
||||
oversees agents: the agent's output has to clear it before it ships.
|
||||
|
||||
Proposal: expose the safety net through an MCP agent and a small set of skills so
|
||||
an editor or a coding agent can ask for a security read of the changed code on
|
||||
demand, shift-left, and so a coding agent can be required to pass it before it
|
||||
proposes a push.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Transport:** extend the existing narrow MCP server (`ousast mcp`, stdio) that
|
||||
OpenCode and other agents already speak. No new network surface; same
|
||||
no-secret, bounded contract.
|
||||
- **What the agent offers:**
|
||||
1. Scan the current diff with the static arm (fast, no model, no cluster) and
|
||||
return findings scoped to the change, with location and a concrete repair
|
||||
direction, not a confidence label.
|
||||
2. Triage a finding: explain it in plain terms and whether it is reachable.
|
||||
3. Prove on demand: hand the top candidate to the proof arm when a runnable
|
||||
harness exists, and return the demonstrated proof-of-concept.
|
||||
- **Skills:** the existing `openultrasast-scan` and `openultrasast-triage`
|
||||
product skills, plus shift-left additions (scan-the-diff, explain-a-finding,
|
||||
suggest-a-fix-direction) that a coding agent invokes mid-task.
|
||||
- **Oversight mode:** a coding agent can be configured to call the scan as a
|
||||
required step before it finalizes a change, so security review is automatic
|
||||
rather than remembered.
|
||||
|
||||
## Control (same model as the hook)
|
||||
|
||||
Advisory by default; blocking is opt-in. Budget and a push/call deadline bound
|
||||
the work. The inline path runs the fast static arm only; heavy proof is explicit
|
||||
and lane-selected. Models and spend caps stay in `.env`.
|
||||
|
||||
## Open questions for the spec
|
||||
|
||||
1. Inline latency budget: in-editor feedback must return in seconds, so the
|
||||
static arm must be fast on a single diff. What is the time contract, and does
|
||||
the large-repo latency problem block it?
|
||||
2. Return contract: exactly what a coding agent gets back (findings, locations,
|
||||
repair direction, optional proof) and how it decides pass/fail in oversight
|
||||
mode. The return should be ticket-ready evidence a busy developer can paste
|
||||
into the issue tracker to win capacity in a prioritization discussion with a
|
||||
product owner, since a reproducible proof is what moves a security fix out of
|
||||
the backlog.
|
||||
3. Does oversight mode need a signed/attestable result so a reviewer can trust
|
||||
that the agent actually ran the check?
|
||||
4. Reuse: the `ousast mcp` tool set already exists; which tools are enough and
|
||||
what is genuinely new.
|
||||
|
||||
## Scope and sequencing
|
||||
|
||||
This is a delivery and adoption surface, correctly **after** the safety net is
|
||||
qualified on unseen repositories (G0 through G5). It is not a detection change;
|
||||
it is how developers and agents consume the detection early. The proof model
|
||||
improvement lives in `proof-arm-opencode`; this brief is the dev- and
|
||||
agent-facing feedback channel.
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"feature_name": "mcp-shift-left-agent",
|
||||
"created_at": "2026-10-07T15:10:00Z",
|
||||
"updated_at": "2026-10-07T15:10:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"approvals": {"requirements": {"generated": false, "approved": false}, "design": {"generated": false, "approved": false}, "tasks": {"generated": false, "approved": false}},
|
||||
"ready_for_implementation": false,
|
||||
"summary": "MCP agent + skills so developers and coding agents get the safety net's feedback early (shift-left, in-editor) and so a coding agent can be required to pass a security check before it ships. Delivery surface after G0-G5 qualification; builds on the existing ousast mcp server."
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
# Brief: the plane on a Kubernetes cluster
|
||||
|
||||
Maintainer, 2026-10-02: "the plan is to move the whole to a kube system soon. what is in the way?" and earlier
|
||||
"later i will deploy this in a separate kube env."
|
||||
|
||||
## Where we are
|
||||
|
||||
The plane already runs on google/ax over Agent Substrate, on a single-node kind cluster on the maintainer's
|
||||
laptop (`ops/ax/`). Its manifests are ax's own kinds, every task is one digest-pinned runner image, the memory
|
||||
store is external (S3-compatible RustFS), and provider credentials travel only in the start request. What still
|
||||
assumes the laptop is small and listed in `docs/deployment.md` and `ops/ax/README.md`:
|
||||
|
||||
1. the reconciler is a host CLI process (`ousast plane run`);
|
||||
2. the artifact receiver is a thread of that process, reached through a Service + EndpointSlice to the kind
|
||||
gateway; the router is reached through a kubectl port-forward;
|
||||
3. the kube context `kind-ousast` and the registry `localhost:5001` are hard-coded in `doctor.py`, `egress.py`,
|
||||
`router.py`;
|
||||
4. images come from the kind registry;
|
||||
5. ax's deploy manifest points `AX_SNAPSHOTS_BUCKET` at the ax authors' bucket;
|
||||
6. the Joern engine runs as a host Docker container (2.8 GB image, 3 GB memory), not as an ax task;
|
||||
7. the worker pool is two 1 CPU / 1.5 GiB workers with no autoscaling.
|
||||
|
||||
## Direction
|
||||
|
||||
Make the plane deployable to any Kubernetes cluster that runs Agent Substrate and ax, prove every change on the
|
||||
kind cluster first (it already behaves like the target), and leave nothing that only works on this host. The
|
||||
production use (a pre-push safety net fed by CI) needs, in addition, a delta Run command and Workspaces from a
|
||||
URL and commit; those are in scope as the last increment because they define what the cluster is for.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Provisioning the maintainer's cluster, Substrate's and ax's own deployment procedures (followed, not changed),
|
||||
and any detection change.
|
||||
@@ -0,0 +1,426 @@
|
||||
# Design Document
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
## Overview
|
||||
|
||||
The plane keeps its shape: `ousast plane run` is the reconciler, ax is the only executor, every task is one
|
||||
digest-pinned image under the runner contract, memory is the S3 store. What changes is where the reconciler's
|
||||
peers live and how they are named. Three facts from the code and the upstream checkouts decide the design:
|
||||
|
||||
- **Every host assumption is an address or a process in four places.** `kind-` + `KIND_CLUSTER_NAME` is built in
|
||||
`doctor.py:39`, `egress.py:115`/`:148` and `router.py:145`; `localhost:5001` in `doctor.py:45-47` and
|
||||
`up.sh:10`; the receiver is a thread of the CLI (`reconciler.py:350-352`) reached by an EndpointSlice to the kind
|
||||
gateway (`receiver-service.yaml.tmpl`, `up.sh:55-57`); the router and ax-server are reached by `kubectl
|
||||
port-forward` (`router.py:146-148`; ax's own CLI does the same, `ax/internal/tunnel/tunnel.go:180-238`).
|
||||
- **The store is already the one thing every profile shares.** `S3Store` is verified at every open
|
||||
(`memory.py:608-640`), its `runs/` prefix expires after 30 days (`RUNS_RULE_ID`, `memory.py:589-591`), and no
|
||||
task reaches it today. A presigned URL is a credential, and the plane already has a channel for credentials that
|
||||
never touches a manifest or the golden boot: the start request (`router.py`, `start_task`, `runner.py:219-275`).
|
||||
- **ax-server authenticates nobody; Kubernetes does.** `deploy/ax-server.yaml` runs `--addr=:8080` with no auth
|
||||
flag; the `ax` CLI tunnels to `svc/ax-server` through kubectl; `kubectl-ate` port-forwards to `ate-api-server`
|
||||
and sends a Kubernetes ServiceAccount token (`cmd/kubectl-ate/internal/cmd/root.go:59-60`). So the only
|
||||
authenticated door into either control plane is the Kubernetes API with a kubeconfig.
|
||||
|
||||
Hence (maintainer, 2026-10-02): the reconciler stays the CLI and gains an execution mode. `local` is the kind
|
||||
profile as today; `remote` is the same `ousast plane run` (later `ousast plane scan`) from a laptop or a CI job
|
||||
against ax in k3s, with task artifacts delivered to the S3 store through presigned URLs minted by the CLI and
|
||||
carried in the start request, and the CLI polling the store for completion. No in-cluster reconciler is needed
|
||||
for the move; the receiver thread goes in both profiles once the store path passes its kind proof.
|
||||
|
||||
### Goals
|
||||
|
||||
- Two profiles of one code base, `kind` and `k3s`; every change proven on kind first (Req 2.4, 7.1).
|
||||
- No module names a host (`kind-ousast`, `localhost:5001`, `172.19.0.1`); a test enforces it (Req 1.3).
|
||||
- One delivery path (the store) in both profiles; one credential channel (the start request).
|
||||
- The Joern engine as a plane task on its own pool (Req 4); a delta Run for a push (Req 6).
|
||||
|
||||
### Non-Goals
|
||||
|
||||
Provisioning k3s; changing Substrate or ax (their manifests are applied with our values, not edited in place);
|
||||
any detection, scoring or population change; an unattended in-cluster reconciler (deferred, section 2.4).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph laptop ["kind profile: laptop"]
|
||||
cli1["ousast plane run --exec local"]
|
||||
env1[".env: provider + S3 keys"]
|
||||
end
|
||||
subgraph kind ["kind cluster (same host)"]
|
||||
ax1["ax-server (ax-system)"]
|
||||
sub1["Substrate: atenet-router, atenet-egress, ate-api"]
|
||||
pool1["ousast-pool: 2 gVisor workers"]
|
||||
epool1["engine-pool: 1 worker, 4 GiB"]
|
||||
reg1["kind registry: images by digest"]
|
||||
end
|
||||
s3[("S3 store (RustFS, external): runs/, repos/, facts/")]
|
||||
cli1 -- "ax CLI tunnel; kubectl-ate; router port-forward" --> ax1
|
||||
cli1 --> sub1
|
||||
sub1 --> pool1
|
||||
sub1 --> epool1
|
||||
pool1 -- "presigned PUT/GET via egress (TLS 443)" --> s3
|
||||
epool1 --> s3
|
||||
cli1 -- "mint URLs, poll, ingest" --> s3
|
||||
reg1 -.-> pool1
|
||||
```
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph ci ["k3s profile: laptop or CI job"]
|
||||
cli2["ousast plane run|scan --exec remote"]
|
||||
kc["kubeconfig: SA ousast-ci token"]
|
||||
sec["CI secrets: provider + S3 keys"]
|
||||
end
|
||||
subgraph k3s ["k3s cluster"]
|
||||
api["Kubernetes API (RBAC: portforward in ax-system, ate-system)"]
|
||||
ax2["ax-server"]
|
||||
sub2["Substrate router / egress / ate-api"]
|
||||
pool2["ousast-pool + HPA"]
|
||||
epool2["engine-pool + HPA"]
|
||||
end
|
||||
ghcr["GHCR ghcr.io/norandom: runner + engine images by digest"]
|
||||
s3b[("S3 store (RustFS)")]
|
||||
cli2 -- "port-forward tunnels only" --> api
|
||||
api --> ax2
|
||||
api --> sub2
|
||||
sub2 --> pool2
|
||||
sub2 --> epool2
|
||||
pool2 -- "presigned PUT/GET" --> s3b
|
||||
epool2 --> s3b
|
||||
cli2 --> s3b
|
||||
ghcr -. "imagePullSecret if private" .-> pool2
|
||||
```
|
||||
|
||||
The two diagrams differ in the kubeconfig principal, the registry, the pool sizes and the HPA. Nothing else.
|
||||
|
||||
New code: `plane/profile.py` (the profile), `plane/delivery.py` (presign, poll, extract), `plane/k8s.py`
|
||||
(manifest rendering per profile), `plane/tasks/engine.py`, `plane/scan.py` (Req 6). `reconciler.py` loses the
|
||||
receiver wiring and gains a `Delivery` dependency; it stays under the 500 lines `test_plane_reconciler.py:803`
|
||||
enforces because every new concern is a sibling module, as `router.py` and `egress.py` already are.
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Configuration surface: `PlaneProfile` (Req 1)
|
||||
|
||||
**Decision.** One frozen dataclass `PlaneProfile` in `plane/profile.py`, loaded by `load_profile()` from a TOML
|
||||
file named by `OUSAST_PLANE_PROFILE` (a name resolved under `ops/k8s/profiles/<name>.toml`, or a path; default
|
||||
`kind`), with every field overridable by an environment variable of the same upper-cased name
|
||||
(`OUSAST_KUBE_CONTEXT`, `OUSAST_REGISTRY`, ...). Profiles hold no secret: keys stay in the environment or
|
||||
`.env`, as today.
|
||||
|
||||
| Field | `kind.toml` | `k3s.toml` | Replaces |
|
||||
| --- | --- | --- | --- |
|
||||
| `exec` | `local` | `remote` | the implicit host mode |
|
||||
| `kube_context` | `kind-ousast` | `ousast-k3s` (the maintainer's context name) | `"kind-" + KIND_CLUSTER_NAME` in three modules |
|
||||
| `registry` | `localhost:5001` | `ghcr.io/norandom` | `doctor.py:45`, `up.sh:10` |
|
||||
| `images` | `kind-images.json` | `images.json` (from the release, section 8) | the `image:` literals in `plane/tasks/*.yaml` |
|
||||
| `router_url` | `""` (port-forward) | `""` (port-forward) | `OUSAST_ROUTER_URL` stays as the test override |
|
||||
| `ax_server` | `""` (ax CLI tunnel) | `""` | `$AX_SERVER` stays as ax's own override |
|
||||
| `memory` | `s3://sast-memory` | `s3://sast-memory` | `OUSAST_MEMORY` default `file://` is removed for the plane (no fallback) |
|
||||
| `atespace` | `default` | `default` | `generate.ATESPACE` |
|
||||
| `snapshots_bucket` | `s3://ax-snapshots` | `s3://ax-snapshots` | ax's `gs://dberkov-gke-dev3/ate-env/` (rendered into our ax-server patch) |
|
||||
| `pools.default`, `pools.engine` | `{replicas: 2, cpu: 1, memory: 1536Mi}`, `{1, 2, 4Gi}` | documented production values | `workerpool.yaml.tmpl` |
|
||||
| `image_pull_secret` | `""` | `ghcr-pull` | none today |
|
||||
|
||||
**Why a file plus env, not env only.** Eleven values change together per cluster; env-only has no reviewable
|
||||
record of what "k3s" is (the `ops/ax/README.md` table at lines 193-204 already struggles to list them), and a
|
||||
CI job would carry eleven exports. The env override keeps the existing tests (`OUSAST_ROUTER_URL`,
|
||||
`OUSAST_ARTIFACT_HOST`) and one-off experiments working. **Rejected:** a `[plane]` section in the scan config
|
||||
TOML (the scan config is a user's tool configuration; the plane is a maintainer surface and must not leak into
|
||||
`ousast scan`); a Kubernetes ConfigMap (there is no in-cluster reconciler to read it).
|
||||
|
||||
`doctor()` takes the profile: it checks `kube_context` reachability, the two namespaces, that the `images` file's
|
||||
runner reference resolves (`docker manifest inspect` for GHCR, the `/v2/` catalog for a local registry), that
|
||||
`memory` opens (`open_store` verifies the bucket), and prints each configured address (Req 1.2). The literal scan
|
||||
(section 9) covers `src/`, `ops/`, `plane/` and `docs/` except the profile files themselves and
|
||||
`ops/ax/up.sh`, which may name kind because it creates it.
|
||||
|
||||
### 2. Execution mode: the CLI with `--exec local|remote` (Req 2)
|
||||
|
||||
**Decision.** `ousast plane run RUN.yaml [--exec local|remote]` (default from the profile). Both modes run the
|
||||
same `reconciler.run`: render, `ax apply`, egress policy, resume, start, await, delete. What differs is
|
||||
addressing and principal, and none of it is new client code:
|
||||
|
||||
| Call | Through | Principal in `remote` |
|
||||
| --- | --- | --- |
|
||||
| `ax apply/resume/get/delete` (`reconciler.Ax`) | ax's own tunnel: `kubectl port-forward -n ax-system svc/ax-server` (`tunnel.go:203-238`), context from the profile | kubeconfig for SA `ousast-ci` |
|
||||
| `kubectl ate create/get/update egress-policy` (`egress.Egress`) | kubectl-ate's own port-forward to `ate-api-server`, bearer = the SA token (`root.go:59-60`) | same SA; Substrate's authorization of that token for `egress-policy` is an open question (below) |
|
||||
| `POST /ousast/v1/start` (`router.open_router`) | `kubectl port-forward svc/atenet-router` in `ate-system`, as today | same SA |
|
||||
| artifacts, inputs, state | the S3 store (section 3) | the store's agent key from the CI secrets |
|
||||
|
||||
**Auth: kubeconfig with a ServiceAccount token, not an ingress.** ax-server has no authentication
|
||||
(`deploy/ax-server.yaml:71-79`: `--addr=:8080`, nothing else) and Substrate's router routes on a header. Exposing
|
||||
either through Traefik would publish an unauthenticated control plane behind whatever middleware we bolt on;
|
||||
tunnelling through the Kubernetes API keeps one authenticated, audited door and uses the clients as they ship.
|
||||
**Rejected:** Traefik IngressRoute for ax-server and the router (unauthenticated upstreams, a second auth layer to
|
||||
design, and ax's CLI would still tunnel); a bespoke ax HTTP client (Req boundary: no replacement of the `ax` CLI
|
||||
beyond what addressing needs, and the tunnel already handles addressing).
|
||||
|
||||
**RBAC (`ops/k8s/base/rbac-ci.yaml`).** ServiceAccount `ousast-ci` in `ax-system`; Role in `ax-system` and in
|
||||
`ate-system`: `pods` get/list, `services` get, `pods/portforward` create; nothing cluster-wide, no verbs on
|
||||
Substrate's or ax's CRDs (both are reached through their servers, not through the Kubernetes API). The token is a
|
||||
bound TokenRequest (`kubectl create token ousast-ci --duration=...`) stored as the CI secret `KUBECONFIG`. Secrets
|
||||
for the provider and the store stay where the reconciler runs: `.env` on a laptop, repository secrets in CI,
|
||||
Kubernetes Secrets only on the day an in-cluster reconciler exists (2.4). The start-request path is unchanged
|
||||
(Req 2.3).
|
||||
|
||||
**Run state of record moves to the store.** `state.json` is written locally under `$OUSAST_RESULTS/plane/<run>/`
|
||||
as today and mirrored to `runs/<run>/state.json` on the store after each change; a rerun from another machine
|
||||
seeds its local copy from the store. This is what lets CI job N+1 resume CI job N's Run (Runs are long,
|
||||
resumable and artifact-driven: `reconciler.py:272-312`).
|
||||
|
||||
**Kind keeps `local` (Req 2.4).** Same command, same manifests, local registry, laptop as reconciler; the only
|
||||
differences are the profile's values. The kind proof of each increment is run in `local`, and increment 3 runs
|
||||
`remote` against kind from a container that holds only the kubeconfig and the secrets (section 7).
|
||||
|
||||
#### 2.4 Deferred: an in-cluster reconciler
|
||||
|
||||
Not needed for the move. It would add: survival of a laptop sleeping or a CI job timing out mid-Run, scheduled
|
||||
Runs, and Secrets injected by Kubernetes. It would cost: a workload (Job per Run is the natural shape, keeping
|
||||
the CLI path identical), a PVC or store-only state, a submit path, and RBAC for ax's and Substrate's Services from
|
||||
inside the cluster. Revisit when unattended Runs are wanted; the store-backed state above is its prerequisite.
|
||||
|
||||
### 3. Delivery: presigned URLs in the start request, both profiles (Req 3)
|
||||
|
||||
**Decision.** The runner PUTs `output.tar` to a presigned S3 URL and GETs each declared input from a presigned
|
||||
URL; the CLI mints them and polls the store. The URLs travel in the start body, never in env:
|
||||
|
||||
```json
|
||||
{"run": "...", "task": "...", "credentials": {"DEEPSEEK_API_KEY": "..."},
|
||||
"delivery": {"put": "<presigned PUT runs/<run>/<task>/output.tar>",
|
||||
"inputs": {"CANDIDATES": "<presigned GET runs/<run>/<producer>/<artifact>>"}}}
|
||||
```
|
||||
|
||||
- **Why the start body.** A presigned URL is a bearer capability, so the credentials-only-in-the-start-request
|
||||
rule applies; env would bake it into the ActorTemplate and hand it to the golden boot; and the body has no
|
||||
32 KB ceiling (Substrate rejects env values over 32768 characters, `reconciler.py:141`). `StartGate.offer`
|
||||
(`runner.py:250-275`) validates `delivery` the way it validates `credentials`: `https` URLs only, hostnames,
|
||||
no echo. Expiry = `OUSAST_TASK_TIMEOUT` (7200 s) + 600 s; a re-start after a re-resume re-mints.
|
||||
- **Why presigned rather than a scoped delivery credential.** A per-task S3 key needs a per-task policy on RustFS,
|
||||
gives the sandbox `ListBucket` and reads beyond its inputs, and would travel in the same body. Presigned URLs
|
||||
name exactly the objects a task may write and read, which is what the receiver enforces today
|
||||
(`Receiver.input_file`, `egress.py:230-247`).
|
||||
- **Why not an in-cluster receiver Service.** It is a server to run, holds state in a process, and is reachable
|
||||
only from a pod; that is the gap remote mode has. **Rejected.**
|
||||
- **Objects.** `runs/<run>/<task>/output.tar` (the runner's PUT); after the CLI extracts it, the declared
|
||||
`outputs[]` are re-put as `runs/<run>/<task>/<artifact>` so a consumer's presigned GET names one artifact, and
|
||||
`runs/<run>/<task>/done.json` carries the summary's status. The 30-day `runs/` expiry already applies; memory
|
||||
rows are ingested from the local extraction by `remember` as today (f: annotations and `memory_key` seeding
|
||||
unchanged, `generate.py:_chain_inputs`, `memory.seed`).
|
||||
- **Completion.** `_await`'s `delivered()` becomes `Delivery.delivered(task)`: a HEAD on `output.tar`, polled at
|
||||
`OUSAST_POLL_SECONDS` (3 s). The runner's completion marker logic (`runner.py`, `_run_once`) is unchanged.
|
||||
- **Inputs no task produces** (`candidates.json`, `functions.json`, `case.json`) may also be store objects
|
||||
declared as inputs, which lifts the ~20 KB inline-files limit (`ops/ax/README.md:117`); Workspace `files`
|
||||
stay for the smoke Task and small cases.
|
||||
|
||||
**One path for both profiles (d).** The store is required and verified in both; keeping the receiver for kind
|
||||
would keep `receiver_address`, `OUSAST_ARTIFACT_DIAL`, the `Host`-dial trick and the EndpointSlice alive for one
|
||||
profile and leave the kind proof proving a path k3s never runs. The receiver thread, `receiver-service.yaml.tmpl`,
|
||||
`receiver_cluster_ip` and the `up.sh` step go after increment 2's proof (the `harnessx-removal` sequencing rule:
|
||||
nothing deleted before its replacement passes). Tests that used `OUSAST_ARTIFACT_HOST` move to a local HTTP PUT
|
||||
target that accepts presigned-shaped URLs.
|
||||
|
||||
**Egress per task (c).** `policy_for` drops the `http` rule and emits one `tls_passthrough` rule on 443 for the
|
||||
sorted union of the store endpoint's hostname (`urlsplit(S3_ENDPOINT).hostname`, IPs rejected as today), the bound
|
||||
Workspaces' Git hosts (`git_hosts`) and the bound Model's `openultrasast.io/egress-hosts`. A model-free task's
|
||||
policy names the store and its Git hosts only. The gateway already passes TLS to the Model host by hostname, so
|
||||
the store host is the same mechanism (`egress.py:7-16`). Open question: the store endpoint must be a DNS name
|
||||
the gateway resolves; `files.because-security.com` is; an in-cluster RustFS would need a resolvable Service name.
|
||||
|
||||
### 4. The engine as a plane task (Req 4)
|
||||
|
||||
**Decision.** `plane/tasks/engine.py` runs in the Joern image under the runner contract, on a second WorkerPool
|
||||
`engine-pool`. The root `Dockerfile` gains the two lines `Dockerfile.runner:25-29` has (the `ax-task-runner`
|
||||
shim and `/workspace`), so one *contract* in two images: `ousast-runner` (slim, PyYAML) and `ousast-engine`
|
||||
(JRE + Joern + php-cli, ~2.8 GB). **Rejected:** Joern inside the single runner image (every task would pull
|
||||
2.8 GB and every golden snapshot carry it; the engine runs for a minority of cases); Docker-in-pod (no socket on
|
||||
containerd; `Dockerfile:1-7` already argues against it).
|
||||
|
||||
- **Task.** Template `plane/tasks/engine.yaml`: `command: ["engine"]`, env `OUSAST_WORKSPACE_DIR`,
|
||||
`OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins and `OUSAST_FAMILIES`, bound to the same two Workspaces the
|
||||
`alerts` task binds (`generate.py:342-353`). The module lifts `benchmarks/push/finding_dump.py`'s scan into
|
||||
`openultrasast.cpg.dump` (the image ships `src/`, not `benchmarks/`) and reuses `engine_alerts.engine_rows`,
|
||||
`read_record` and the `alerts` helpers (`scan`, `coverage`, `write_rows`, `write_summary`). It emits the same
|
||||
`alerts.jsonl` rows (`rule_id: engine:<family>`, `source: "engine"`) and `summary.json` (coverage with the
|
||||
engine's languages, files and bytes read, questions, seconds, degradations per pin) as `alerts-engine`, with
|
||||
`host: false`. The instrument rule stays: no `read N files` line or `questions == 0` fails the task loudly.
|
||||
- **Sizing from the record** (`benchmarks/measurements/2026-09-30-php-engine-alerts/record.json`): container
|
||||
peak 2.129 GiB under a 3 GiB limit, 789 s for both pins of froxlor, deadline 1800 s per pin. Task resources:
|
||||
requests 3 Gi / 1 CPU, limits 4 Gi / 2 CPU; `engine-pool` workers 4 Gi / 2 CPU, replicas 1 on kind (the 7 GB
|
||||
host holds it only with `ousast-pool` at one worker), production replicas in `k3s.toml`. Task timeout: two pins
|
||||
times 1800 s plus grace fits `OUSAST_TASK_TIMEOUT` 7200.
|
||||
- **Placement is the open question of this increment.** ax's templates set no worker selector
|
||||
(`workerpool.yaml.tmpl:1-3`), so Substrate picks any free worker of the sandbox class. The kind proof must show
|
||||
a 3 Gi actor lands only on `engine-pool`. If Substrate does not place by resource fit, the fallback is a
|
||||
separate atespace with its own pool if Substrate scopes pools that way; failing that, the engine stays
|
||||
`alerts-engine` on a Docker host and the k3s engine becomes a plain Kubernetes Job outside ax, recorded as such.
|
||||
**Probe outcome (task 1.5, 2026-10-02, `benchmarks/measurements/2026-10-02-k8s-01-profile/`): `placement: any`.**
|
||||
With `ousast-pool` at one 1536Mi worker and a `probe-pool` of one 3Gi worker both ACTIVE, a `repo-facts` actor
|
||||
requesting 2560Mi (limit 3Gi) was placed on the 1536Mi `ousast-pool` worker (`kubectl ate get workers`: ACTORS
|
||||
1/1000 there, 0/1000 on `probe-pool`), and so was the 256Mi actor. Substrate does not place by resource fit, so
|
||||
group 5 takes the fallback: the engine pool must be the only pool its actors can land on. Before 5.2, check whether
|
||||
Substrate scopes pools by atespace (an `engine` atespace with `engine-pool` as its only pool); if it does not, the
|
||||
k3s engine runs as a plain Kubernetes Job outside ax and `alerts-engine` stays the kind path.
|
||||
- **`alerts-engine` stays** for development on a host with Docker (kind profile), its help text and
|
||||
`docs/plane.md` row marked "development only; the plane task is `engine`" (Req 4.3). Generated Runs choose
|
||||
`engine` when the profile's `images` file has an engine image, else leave `alerts` for `alerts-engine`.
|
||||
|
||||
### 5. `ops/k8s/` layout (Req 5)
|
||||
|
||||
```
|
||||
ops/k8s/
|
||||
profiles/kind.toml, k3s.toml # section 1; k3s values documented, not secrets
|
||||
base/
|
||||
workerpool-default.yaml.tmpl # replicas, limits, nodeSelector from the profile (render: plane/k8s.py)
|
||||
workerpool-engine.yaml.tmpl # engine-pool, same template, engine values
|
||||
ax-server-snapshots.yaml # a strategic-merge patch setting AX_SNAPSHOTS_BUCKET (applied over ax's deploy)
|
||||
rbac-ci.yaml # SA ousast-ci; Roles + RoleBindings in ax-system and ate-system (section 2)
|
||||
secrets.example.env # the variable names only: DEEPSEEK_API_KEY, OPENROUTER_API_KEY, S3_*, AWS_*
|
||||
k3s/
|
||||
imagepullsecret.yaml.tmpl # ghcr-pull from a GitHub PAT, referenced by the pools' pod template
|
||||
hpa-default.yaml, hpa-engine.yaml # External metric ate_workerpool_workers{state=at_capacity}, averageValue 0.7
|
||||
prometheus-adapter.yaml # from substrate/demos/autoscaled-workerpool, namespace ax-system
|
||||
render.sh # ousast plane manifests --profile <p> --out DIR; kubectl apply -f DIR
|
||||
```
|
||||
|
||||
`ops/ax/up.sh` keeps creating kind and calls `render.sh kind` for the pools; its receiver step is deleted in
|
||||
increment 2. **gVisor on k3s is a correction to the requirements text:** Substrate does not use a Kubernetes
|
||||
`RuntimeClass`. The `ateom-gvisor` worker (`workerImage: ko://.../cmd/ateom-gvisor`) fetches the gvisor release
|
||||
tarball named by `manifests/ate-install/sandboxconfig-gvisor.yaml` (`gs://gvisor/releases/nightly/...`) and runs
|
||||
`runsc` itself inside the worker pod (`cmd/ateom-gvisor/actorpath.go:35-45`). The node checklist is therefore:
|
||||
PodSecurity in `ax-system` admits the worker pod's security context (recorded from kind in increment 6), the
|
||||
node reaches the tarball URL or a mirror named in a copied SandboxConfig, and `/var/lib/ate` sits on local-path
|
||||
storage with room for golden actors (24 MB each, `ops/ax/README.md:112`). Autoscaling is k3s-only: the HPA
|
||||
needs Prometheus and the adapter (the Substrate demo README says kind-only for the demo; we take its manifests),
|
||||
and the 7 GB host cannot scale anyway. The queueing consequence of one actor per worker is recorded in the
|
||||
runbook: a Run with `--workers N` beyond the pool's replicas queues in ax as Suspended until a worker frees.
|
||||
|
||||
### 6. `ousast plane scan <url> --base <c> --head <c>` (Req 6)
|
||||
|
||||
`plane/scan.py`, reusing `generate.py`'s render path with a new case source:
|
||||
|
||||
1. **Workspace generation** (`generate.scan_cases(url, base, head, cache)`): a shallow clone into the case cache
|
||||
(`runner.find_cached_checkout` layout), two pinned Workspaces `<repo_dir>-<head7>` and `<repo_dir>-<base7>`
|
||||
(`workspaces.py` shape, pins in `openultrasast.io/git-commits`), repository key by `memory.repo_key`.
|
||||
2. **Candidates from changed functions**: `generate.fix_ranges(repo, base, head, side="new")` gives the head-side
|
||||
hunks (the `push/` delta primitive `_diff_records` is the same `git diff -U0`), `repo_facts.enclosing` names
|
||||
the function per hunk line, filtered to `repo_facts.product_files`. Quick rules on the changed files
|
||||
(`tasks.alerts.scan`) add alerted functions; `roles` runs once per repository when memory has no `roles` rows
|
||||
for it; `repo-facts` supplies callers. `candidates.json` and `functions.json` are store inputs (section 3).
|
||||
3. **Run composition**: per changed-function set `facts -> va, vb -> agree -> vc -> final -> features ->
|
||||
remember`, plus `engine` when a changed file's language is engine-covered and not quick-covered, or PHP
|
||||
(`ENGINE_ALONGSIDE_QUICK`). Budgets from `generate.verify_budget` (0.022 USD per candidate, headroom 1.5);
|
||||
`--ceiling` caps the sum. `--all` (whole repository, secondary) takes every product function.
|
||||
4. **Results** land in memory under `repos/<host>__<owner>__<name>/<head>.jsonl` through `remember` and
|
||||
`ousast plane remember` (rows already carry `repo` and `pin`, `memory.py:77`); `runs/<run>/` holds the raw
|
||||
artifacts for 30 days. The report prints the attribution table and the plane's agreement (`final`); the
|
||||
decision engine's verdict only when a model is adopted (learned-decision-engine Req 6 unchanged).
|
||||
5. **Latency split (e).** `ousast pre-push` stays the blocking hook: quick rules and the delta engine inside
|
||||
`OUSAST_PUSH_DEADLINE` (30 s, `ops/README.md:322`). `ousast plane scan --exec remote --no-wait` submits and
|
||||
prints the Run name in the hook's report; `--wait SECONDS` blocks for CI. `ousast plane status <run>` reports
|
||||
later. The wrapper `ops/pre-push` gains an opt-in `OUSAST_PLANE_SCAN=1`; it never blocks on the plane.
|
||||
|
||||
### 7. Increment order, kind proof and measurement
|
||||
|
||||
| # | Increment | Kind proof (`local` unless stated) | Recorded under `benchmarks/measurements/` |
|
||||
| --- | --- | --- | --- |
|
||||
| 1 | `PlaneProfile`, literals replaced, `doctor` by profile | `ousast plane doctor` passes with `kind.toml`; literal scan 0 | `<date>-k8s-01-profile/record.json`: doctor lines per profile, scan count |
|
||||
| 2 | Store delivery + receiver-less egress; receiver removed after | `ops/ax/smoke-run.sh` delivers `facts.json` via the store; policy has one TLS rule | `-02-store-delivery/`: bytes, PUT-to-poll seconds, rendered policies |
|
||||
| 3 | `--exec remote` from a CI-shaped container (kubeconfig + secrets only) against kind | smoke Run completes; audit log lists only the Role's verbs | `-03-remote-exec/`: kube audit verbs vs Role, run status |
|
||||
| 4 | Images on GHCR, digests, repin (section 8) | kind pulls `ghcr.io/norandom/ousast-runner@sha256` (pull secret if private); smoke Run | `-04-ghcr-images/`: sizes, pull seconds per image |
|
||||
| 5 | `engine` task + `engine-pool` | froxlor case as `engine`: rows equal the 2026-09-30 host record (6/6), placement on engine-pool shown | `-05-engine-task/`: rows diff, seconds, peak memory, placement |
|
||||
| 6 | Pools per profile, HPA manifests, node checklist | kind pools rendered and applied; worker securityContext recorded; HPA renders (k3s-only live) | `-06-pools/`: footprint table, securityContext |
|
||||
| 7 | `plane scan` on a real pair (a PHP CVE pair from the dev corpus) | Run completes; rows under `repos/<repo>/<head>` | `-07-plane-scan/`: candidates, cost, agreement, rows |
|
||||
| 8 | Runbook (`docs/deployment.md`), checklist mapped to R1-R7 | n/a (documentation; every "planned" label removed only with a recorded run) | the checklist links the records above |
|
||||
|
||||
### 8. GitHub Actions and re-pinning
|
||||
|
||||
`.github/workflows/images.yml` on tags `v*` (separate from `release.yml`, which builds the wheel): build
|
||||
`plane/Dockerfile.runner` as `ghcr.io/norandom/ousast-runner` and the root `Dockerfile` as
|
||||
`ghcr.io/norandom/ousast-engine` with `docker/build-push-action`, tag by version, and write the two digests to a
|
||||
release asset `images.json` (`{"runner": "ghcr.io/...@sha256:...", "engine": "..."}`) and to job outputs.
|
||||
`generate.repin_templates` (`generate.py:484-497`, one image for all templates) becomes `repin(plane, images)`:
|
||||
`engine.yaml` gets `images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`
|
||||
exposes it and `--runner-image FILE` keeps working for a single-image file. Private packages: the pools' pod
|
||||
template names `imagePullSecret` from the profile.
|
||||
|
||||
### 9. Tests
|
||||
|
||||
- `tests/test_plane_literals.py`: scans `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh`, `ops/k8s/profiles/`) and
|
||||
`docs/` for `kind-ousast`, `localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py`
|
||||
and the ax tunnel note; built at runtime like `test_removed_plane_references.py`.
|
||||
- `tests/test_plane_profile.py`: defaults, file + env precedence, unknown key rejected, no secret field, both
|
||||
committed profiles load.
|
||||
- `tests/test_plane_k8s.py`: render both profiles; kind has no HPA and no pull secret; k3s has both; engine pool
|
||||
limits >= the engine task's limits; every image reference is digest-pinned.
|
||||
- `tests/test_plane_rbac.py`: the rendered Role's verbs equal the set of Kubernetes API calls the CLI makes
|
||||
(fake `kubectl`/`ax`/`kubectl-ate` record their argv; the test maps them to `pods/portforward`, `pods`,
|
||||
`services`) and nothing more.
|
||||
- `tests/test_plane_egress.py`: for every committed Task template, `policy_for` yields one TLS rule whose hosts are
|
||||
exactly the store host, the bound Git hosts and the Model hosts; no `http` rule; an IP endpoint is refused.
|
||||
- `tests/test_plane_delivery.py`: start body with `delivery` accepted; URLs never logged; expiry; PUT and GET
|
||||
against a local server; `delivered()` polling; declared-only inputs.
|
||||
- `tests/test_plane_engine_task.py`: fixture record -> rows and summary equal `engine_alerts.produce`'s shape;
|
||||
the instrument failures.
|
||||
- `tests/test_plane_scan.py`: candidates from a two-commit fixture repository; Run composition; budgets.
|
||||
- **Only live on kind:** placement on `engine-pool`; Substrate accepting the SA token for `egress-policy`;
|
||||
the gateway passing TLS to the store host; GHCR pulls; the HPA (k3s only). These are the kind proofs above.
|
||||
|
||||
## Data Models
|
||||
|
||||
Store layout additions: `runs/<run>/state.json`, `runs/<run>/<task>/output.tar`, `runs/<run>/<task>/<artifact>`,
|
||||
`runs/<run>/<task>/done.json`, `runs/<run>/inputs/<name>`; all under the existing 30-day `runs/` rule. Start body:
|
||||
`credentials` as today plus optional `delivery {put, inputs{name: url}}`. `PlaneProfile` as in section 1.
|
||||
`images.json` as in section 8.
|
||||
|
||||
## Error Handling
|
||||
|
||||
A profile naming an unreadable context, a bucket that fails `verify_bucket`, or an images file without a digest
|
||||
fails before `ax apply` with the field named. A presigned PUT that fails three times ends the task `failed` with
|
||||
the HTTP status (`runner.deliver` semantics kept); an expired URL after a re-resume is re-minted by the next
|
||||
start. A store HEAD error during polling is retried with the resume backoff and reported after ten consecutive
|
||||
failures, like `ax get` (`reconciler.py`, `_await`). A `delivery` URL that is not `https` to a hostname is a 400
|
||||
from `StartGate.offer`. The engine task fails loudly on an unread input, as `read_record` does.
|
||||
|
||||
## Requirement Traceability
|
||||
|
||||
| Req | Components | Tests | Kind proof / record |
|
||||
| --- | --- | --- | --- |
|
||||
| R1 nothing assumes this host | `profile.py`, `doctor.py`, `egress.py`, `router.py`, `ops/k8s/profiles/` | literals, profile | increment 1 |
|
||||
| R2 reconciler in the cluster -> CLI with `--exec remote`; kind first-class | `reconciler.py` (`Delivery`), `Ax` with context, `rbac-ci.yaml`, store-backed state | rbac, profile | increments 3, 1 |
|
||||
| R3 receiver-less artifacts and egress | `delivery.py`, `runner.py` start body and PUT/GET, `egress.policy_for` | delivery, egress | increment 2 |
|
||||
| R4 engine as a task | `tasks/engine.py`, `cpg/dump.py`, `Dockerfile` shim, `engine.yaml`, `engine-pool` | engine task, k8s | increment 5 |
|
||||
| R5 sizing and autoscaling | `k8s.py`, `ops/k8s/base`, `ops/k8s/k3s` | k8s | increment 6 (HPA live only on k3s) |
|
||||
| R6 delta Run | `scan.py`, `generate.scan_cases`, `ops/pre-push` opt-in | scan | increment 7 |
|
||||
| R7 proven on kind, documented | every increment's record; `docs/deployment.md` | n/a | increment 8 |
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Substrate authorization for the CI token.** `kubectl-ate` sends a ServiceAccount token; which audience and
|
||||
which Substrate-side grant let `ousast-ci` create `egress-policy` in `default` is not in the checkouts read
|
||||
here. Proven or refuted in increment 3.
|
||||
2. **Engine placement.** Whether Substrate places a 3 Gi actor only on workers that fit it (section 4). The
|
||||
fallback is named.
|
||||
3. **gVisor tarball reachability on k3s** (`gs://gvisor/...` in the SandboxConfig): mirror or egress needed.
|
||||
4. **Store endpoint hostname** for any future in-cluster RustFS; today's external endpoint is fine.
|
||||
5. **Two images under one contract** versus the maintainer's "one runner image for all tasks": the design
|
||||
reads it as one contract; confirm.
|
||||
6. **GHCR visibility** (public packages need no pull secret) and whether Substrate's and ax's `ko` images go to
|
||||
the same registry namespace.
|
||||
7. **Reconciler size**: remote mode and `Delivery` must stay under the 500-line test; if they do not, the test's
|
||||
bound moves with a reason, not silently.
|
||||
@@ -0,0 +1,135 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The plane moves from this laptop's kind cluster to a Kubernetes cluster of the maintainer's choosing without
|
||||
changing what runs: the same ax manifests, runner image, memory store and credential path. Every host-bound
|
||||
piece becomes a cluster resource or a configuration value, each change is proven on kind first, and the
|
||||
production entry point (a delta Run for a push) is added. Context: `brief.md`, `docs/deployment.md`.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: two profiles of one code base, `kind` (local, kept) and `k3s` (production); a remote execution mode of the CLI; artifacts delivered to the store; configurable
|
||||
cluster context, registry and image pins; our own snapshot and memory buckets; the Joern engine as a plane task
|
||||
on a larger worker pool; worker pool sizing and autoscaling; a delta Run command and Workspaces from URL +
|
||||
commit; a deployment runbook with a proof on kind and a checklist for the target cluster.
|
||||
- **Out of scope**: provisioning or operating the target cluster; modifying Agent Substrate or ax; any change to
|
||||
detection, scoring, populations or the decision engine; replacing the `ax` CLI with a bespoke client beyond what
|
||||
remote submission needs.
|
||||
- **Adjacent expectations**: `ai-service-plane` requirements (one Model + budget per task, attribution,
|
||||
credentials only in the start request, egress policies, verify-at-startup memory store) keep holding;
|
||||
`harnessx-removal` Req 6 (memory on the S3 store) keeps holding; nothing is deleted before its replacement
|
||||
passes its test (the `harnessx-removal` sequencing rule).
|
||||
|
||||
## Target Environment (maintainer, 2026-10-02)
|
||||
|
||||
- **Cluster: k3s.** Consequences: containerd (no Docker socket; the engine's host-Docker path does not exist there),
|
||||
gVisor needs `runsc` and a `RuntimeClass` installed on the nodes Substrate's workers run on, k3s ships Traefik
|
||||
and local-path storage, and single-node k3s is close enough to kind that the kind proof transfers.
|
||||
- **Images and source: GitHub.** Images are published to GitHub Container Registry (`ghcr.io/norandom/...`),
|
||||
built by GitHub Actions from tagged commits, digest-pinned in the manifests; private packages need an
|
||||
`imagePullSecret` on the worker ServiceAccounts and the CI principal. Substrate's and ax's own images are built with
|
||||
`ko` from their checkouts and pushed to the same registry.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Nothing assumes this host
|
||||
|
||||
**User Story:** As the maintainer, I want every host-bound assumption replaced by a cluster resource or a
|
||||
configuration value, so the plane runs the same on kind and on the target cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The kube context, the registry (GHCR in production), the ax-server address, the router address, the receiver address and the
|
||||
snapshot and memory buckets are configuration (environment or one config file), with the kind values as the
|
||||
documented defaults for development; no module names `kind-ousast`, `localhost:5001` or `172.19.0.1`.
|
||||
2. `ousast plane doctor` checks the configured cluster, not kind by name, and reports each configured address.
|
||||
3. A test enumerates the source tree for host-only literals and fails on any.
|
||||
|
||||
### Requirement 2: The reconciler stays the CLI, with a remote execution mode
|
||||
|
||||
**User Story:** As the maintainer, I want to trigger the checks in the cluster from the CLI and get the results
|
||||
fed back, so that remote execution is a mode of the same tool, not a separate service.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. `ousast plane run` (and the later `ousast plane scan`) takes an execution mode, `local` or `remote`, from the
|
||||
profile or a flag; `remote` submits the Run to ax-server in the target cluster (over the cluster's ingress or a
|
||||
kubeconfig), `local` is today's kind path. The manifests, the memory-key seeding, the budgets and the
|
||||
attribution table are identical in both modes.
|
||||
2. In remote mode the CLI needs no inbound path: it polls the memory store for task completion and artifacts and
|
||||
prints the same status, attribution table and verdicts; `ousast plane status <run>` reports a Run started
|
||||
earlier from any machine with the store credentials.
|
||||
3. Provider and store credentials come from the caller's environment (a laptop `.env`, a CI secret), travel only
|
||||
in the start request, and never appear in a manifest, an ActorTemplate or a log.
|
||||
4. The principal the CLI uses against the cluster has the least privilege the ax API needs and nothing against
|
||||
Agent Substrate directly; a test compares the rendered role with the calls the CLI makes.
|
||||
5. kind stays a first-class local execution mode, not a stepping stone: `ousast plane run` from a laptop
|
||||
against the kind cluster keeps working with the development profile (same code, same manifests, local
|
||||
registry), and every release is proven on it before anything is applied to k3s.
|
||||
6. An unattended in-cluster reconciler (a Deployment or Job that runs without a caller) is a later option, not
|
||||
part of this spec.
|
||||
|
||||
### Requirement 3: Artifacts land in the store, not on a host
|
||||
|
||||
**User Story:** As the maintainer, I want task artifacts delivered to the memory store so that no host has to
|
||||
be reachable from the cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Tasks deliver their artifacts to the S3 memory store (presigned URLs minted by the CLI and carried in the
|
||||
start request, or a scoped delivery credential; the design chooses one and states why) and the CLI reads
|
||||
completion from the store. The receiver thread and the kind gateway EndpointSlice remain only if the design
|
||||
shows the local profile needs them; one delivery path for both profiles is preferred.
|
||||
2. Egress policies grant each task exactly the hosts it needs (the store's endpoint, git hosts, the bound
|
||||
Model's host) and nothing else; a test renders the policy for every committed Task.
|
||||
|
||||
### Requirement 4: The engine is a plane task
|
||||
|
||||
**User Story:** As the maintainer, I want Joern to run where the other tasks run, on a worker sized for it.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A second WorkerPool sized for the engine (memory and CPU from a measured run) exists alongside the default
|
||||
pool; an `engine` task module wraps today's host engine run and emits the same `alerts.jsonl` rows and
|
||||
coverage as `alerts-engine`.
|
||||
2. The engine image is digest-pinned in the registry like the runner image; its size and pull time are recorded.
|
||||
3. `alerts-engine` on the host remains available for development and is marked as such.
|
||||
|
||||
### Requirement 5: Sizing and autoscaling
|
||||
|
||||
**User Story:** As the maintainer, I want the pool to follow the load of pushes without manual resizing.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Worker pool replicas, limits and the autoscaling rule (Substrate's assigned-worker signal) are manifests under
|
||||
`ops/k8s/` with the kind values as a small profile and a documented production profile.
|
||||
2. The runbook records the measured footprint per task type and the queueing consequence of one actor per
|
||||
worker.
|
||||
|
||||
### Requirement 6: A delta Run for a push
|
||||
|
||||
**User Story:** As a user with a repository the project has never seen, I want a push checked by the plane.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. `ousast plane scan <repo-url> --base <commit> --head <commit>` generates Workspaces from the URL and commits,
|
||||
derives candidates from the changed functions (quick rules, inferred roles, repository facts), and submits a
|
||||
Run of facts, verify a/b, tie-break and agree with per-task budgets; a whole-repository mode is secondary.
|
||||
2. Results are written to the memory store under the repository and head commit and reported with the
|
||||
attribution table; the decision engine's verdict is reported only when a model is adopted, else the plane's
|
||||
agreement (ai-service-plane Req 6 and learned-decision-engine Req 6 unchanged).
|
||||
3. The latency split is explicit: a blocking pre-push carries quick rules and the delta engine within its
|
||||
deadline; plane verdicts arrive asynchronously.
|
||||
|
||||
### Requirement 7: Proven on kind, then documented for the target
|
||||
|
||||
**User Story:** As the maintainer, I want each step shown working on kind before I touch the real cluster.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Every requirement above is exercised end to end on the kind cluster (remote mode included, with the kind cluster as the remote target), with
|
||||
a recorded run under `benchmarks/measurements/`.
|
||||
2. `docs/deployment.md` becomes the runbook: prerequisites for the target cluster (Substrate, ax, registry,
|
||||
buckets, Secrets, node requirements for gVisor workers), the profile switch, and a checklist whose every item
|
||||
maps to a requirement here; planned items are labelled planned until their run is recorded.
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "plane-on-kubernetes",
|
||||
"created_at": "2026-10-02T20:00:00Z",
|
||||
"updated_at": "2026-10-04T18:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,295 @@
|
||||
# Implementation Plan
|
||||
|
||||
> **Status 2026-10-04.** Groups 1-2 are done on kind (records `benchmarks/measurements/2026-10-02-k8s-01-profile/`
|
||||
> and `2026-10-02-k8s-02-store-delivery/`). Task 2.6 is paused. The production target changed. The operator runs a
|
||||
> kube-ax cluster (k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes). There we may only submit AX Tasks: no k8s
|
||||
> object creation, no secrets, no access to `ate-system`. The engine now runs there as AX Tasks through
|
||||
> `benchmarks/learn/engine_trace_ax.py`. The engine-task image (`plane/Dockerfile.engine-task`) is built FROM
|
||||
> `ghcr.io/norandom/ax-task-runner:v0.3.1`. Tasks read and write through presigned HTTPS links, are named
|
||||
> `ousast-engine-*`, and run a 1.7 GB heap with `ActiveProcessorCount=2` and one resident Joern session per pin. A
|
||||
> pin whose sandbox dies is retried once under a new name, then handed to the VM lane (2.5 GB heap). Egress is
|
||||
> operator-managed: HTTPS 443 to public hosts is open; LAN and cluster-internal addresses are blocked. Archive
|
||||
> names (`.tar`, `.gz`, `.zip`, `.bin`) failed through the store host's Cloudflare cache until the operator added a
|
||||
> cache-bypass rule; presigned objects are staged as `.dat` as a safeguard. Groups 3-8 are **to re-plan** against
|
||||
> that cluster under roadmap G4 (`.kiro/steering/roadmap.md`, section of 2026-10-04). Their text below is kept
|
||||
> as written. G4 (detection on push for any repository: `ousast plane scan --base --head`, engine pins on kube-ax,
|
||||
> model calls on the VM or CI) is the new target for group 7, sized by what G1-G3 keep.
|
||||
|
||||
One task group per increment of `design.md` section 7, in that order; a later group never starts before the
|
||||
earlier group's kind proof is recorded. Every task ships with its tests; a task is done when its tests pass in
|
||||
the host suite, ruff and mypy are clean, and nothing in `src/`, `plane/`, `ops/` (minus `ops/ax/up.sh` and
|
||||
`ops/k8s/profiles/`) or `docs/` names `kind-ousast`, `localhost:5001`, `172.19.` or a port-forward address
|
||||
(task 1.3 makes that a test). Nothing is deleted before its replacement passes its proof. Each group ends with
|
||||
its kind proof and the record it writes under `benchmarks/measurements/<date>-k8s-NN-<slug>/record.json`
|
||||
(commit, images, the instrument checks, the numbers). Steps that call a model are marked **paid: needs a budget
|
||||
go-ahead** with a ceiling; steps that need a hand on a cluster that does not exist yet are marked **manual**.
|
||||
|
||||
- [x] 1. Profile and literals (design section 1; open-question checks 2, 3, 7)
|
||||
- [x] 1.1 `PlaneProfile`
|
||||
- `src/openultrasast/plane/profile.py`: frozen dataclass with the fields of design section 1; `load_profile()`
|
||||
reads `ops/k8s/profiles/<OUSAST_PLANE_PROFILE>.toml` (a name or a path; default `kind`), then applies
|
||||
`OUSAST_<FIELD>` overrides; rejects an unknown key, a secret-looking key (`*_KEY`, `*_SECRET`), an `exec`
|
||||
outside `local|remote`, an `images` file without digest pins. `ops/k8s/profiles/kind.toml` and `k3s.toml`
|
||||
committed with the design's values; `ops/k8s/profiles/kind-images.json` from `~/.cache/ousast/ax-src/runner-image`.
|
||||
- `tests/test_plane_profile.py`: defaults; file then env precedence; each rejection names the field; both
|
||||
committed profiles load; no field holds a value matching a credential pattern.
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.2 Literals out of `doctor.py`, `egress.py`, `router.py`, `generate.py`
|
||||
- `doctor(profile)`: context from `profile.kube_context`; the registry check resolves the `images` file's runner
|
||||
reference (`docker manifest inspect` for a remote registry, `/v2/_catalog` for a local one); `open_store`
|
||||
verifies `profile.memory`; prints every configured address (Req 1.2). `Egress(context=...)`,
|
||||
`open_router(context=...)` and `receiver_cluster_ip(context)` take the profile's context; `generate.ATESPACE`
|
||||
becomes `profile.atespace`. `cli.py`: `ousast plane doctor|run|workspaces|harvest` accept `--profile`.
|
||||
- `tests/test_plane_reconciler.py`, `test_plane_router.py`, `test_plane_egress.py`: the fakes receive the
|
||||
profile's context, never a built one.
|
||||
- Evidence: `grep -rn "kind-" src/openultrasast/plane` matches only comments citing the kind profile.
|
||||
- _Requirements: 1.1, 1.2_
|
||||
- [x] 1.3 Host-literal scan test
|
||||
- `tests/test_plane_literals.py`: pattern built at runtime (as `tests/test_removed_plane_references.py` does)
|
||||
over `src/`, `plane/`, `ops/` minus `ops/ax/up.sh` and `ops/k8s/profiles/`, and `docs/`, for `kind-ousast`,
|
||||
`localhost:5001`, `172.19.`, `127.0.0.1:` and `port-forward` outside `router.py` and lines tagged
|
||||
`ax-tunnel`; fails listing `path:line`.
|
||||
- Evidence: the test passes on this commit and fails when a literal is reinserted (shown once in the record).
|
||||
- _Requirements: 1.3_
|
||||
- [x] 1.4 `up.sh` by profile
|
||||
- `ops/ax/up.sh`: `KO_DOCKER_REPO` and the context from `kind.toml` (read with `python -m
|
||||
openultrasast.plane.profile --print registry|kube_context`); writes `kind-images.json` instead of
|
||||
`runner-image`; `generate.repin_templates` accepts that file (runner key only until task 4.3).
|
||||
- Evidence: `ops/ax/up.sh` reruns idempotently on kind; `ops/ax/smoke-run.sh` passes.
|
||||
- _Requirements: 1.1, 2.4_
|
||||
- [x] 1.5 Placement probe (open question 2, decided before the engine work) -- result: `placement: any`
|
||||
- `ops/ax/probe-placement.sh`: applies a second WorkerPool `probe-pool` (1 replica, 3 Gi) from the design's
|
||||
engine template, then one model-free `repo-facts` Task with `resources.requests.memory: 2560Mi`, and records
|
||||
with `kubectl ate get workers` which worker took the actor; then the same Task with 256Mi. Removed after.
|
||||
- Evidence: `record.json` says whether Substrate places by resource fit (`placement: fit|any`). `any` selects
|
||||
the design's fallback for group 5 and is written into `design.md` section 4 before group 5 starts.
|
||||
- _Requirements: 4.1_
|
||||
- [x] 1.6 Reconciler bound (open question 7) -- 489 lines after 1.2; projected 514 during group 2: the bound moves to 550 in 2.4
|
||||
- Record `wc -l src/openultrasast/plane/reconciler.py` (477 at 17ac7ad) and the lines task 2.4 may add
|
||||
(receiver wiring removed: `-12`, `Delivery` calls: `+8` estimated). If the estimate crosses 500, the bound
|
||||
in `tests/test_plane_reconciler.py:803` moves to 550 in task 2.4 with the reason in the assertion message.
|
||||
- Evidence: the numbers in `record.json`.
|
||||
- _Requirements: 2.1_
|
||||
- [x] 1.7 gvisor tarball reachability (open question 3) **manual** -- documented as planned in `docs/deployment.md` B.1; the live check waits for k3s
|
||||
- `docs/deployment.md` (B.1, planned): the k3s node must fetch
|
||||
`gs://gvisor/releases/nightly/2026-09-02/<arch>/gvisor.tar.zstd` (`substrate/manifests/ate-install/sandboxconfig-gvisor.yaml`)
|
||||
or a mirrored copy named in a copied SandboxConfig; the check is `kubectl -n ate-system logs` of the worker
|
||||
pod's atelet fetch line, run by hand on the day k3s exists. Labelled planned until then.
|
||||
- _Requirements: 7.2_
|
||||
- [x] 1.8 Kind proof and record 01
|
||||
- `ousast plane doctor --profile kind` passes every line; `ousast plane doctor --profile k3s` fails only on the
|
||||
unreachable context (the expected line); `test_plane_literals` 0 offenders; the probe of 1.5; the bound of 1.6.
|
||||
- `benchmarks/measurements/<date>-k8s-01-profile/record.json`.
|
||||
- _Requirements: 1.1, 1.2, 1.3, 7.1_
|
||||
|
||||
- [ ] 2. Store delivery, receiver-less egress (design section 3)
|
||||
- [x] 2.1 Presigned URLs and the store run directory
|
||||
- `src/openultrasast/plane/delivery.py`: `Delivery(store, run)` mints presigned PUT for
|
||||
`runs/<run>/<task>/output.tar` and GET for `runs/<run>/<producer>/<artifact>` (boto3 `generate_presigned_url`
|
||||
on `S3Client.sdk`, expiry `OUSAST_TASK_TIMEOUT + 600`); `delivered(task)` HEADs the tar; `collect(task)`
|
||||
downloads, extracts under the run dir with the member checks `Receiver.accept` has, re-puts declared
|
||||
`outputs[]` as single objects, writes `done.json`; `put_input(name, path)` for inputs no task produces;
|
||||
`state.json` mirrored to `runs/<run>/state.json` and seeded from it on a rerun.
|
||||
- `tests/test_plane_delivery.py`: against a local S3-shaped fake (PUT/GET/HEAD by key); URLs never appear in
|
||||
logs; expiry; member rejection; declared-only re-put; state seeding.
|
||||
- _Requirements: 3.1, 2.1_
|
||||
- [x] 2.2 Runner: `delivery` in the start body
|
||||
- `src/openultrasast/plane/runner.py`: `StartGate.offer` validates an optional `delivery {put, inputs}` (https,
|
||||
hostname, no echo); `_deliver_output` PUTs the tar to `delivery.put` (same retry and `DeliveryError`
|
||||
semantics); `fetch_inputs` GETs from `delivery.inputs[name]`; `OUSAST_ARTIFACT_URL` and `OUSAST_ARTIFACT_DIAL`
|
||||
stay honoured until task 2.6 so both paths run in the same build.
|
||||
- `tests/test_plane_runner.py`: accepted body with `delivery`; 400 on an `http` or IP URL; PUT to a local
|
||||
server; inputs fetched from presigned-shaped URLs; the marker and retry-only boot unchanged.
|
||||
- _Requirements: 3.1_
|
||||
- [x] 2.3 Egress without the receiver
|
||||
- `src/openultrasast/plane/egress.py`: `policy_for(task, workspaces, model, store_host)` emits one
|
||||
`tlsPassthrough` rule on 443 over the sorted union of store host, Git hosts, Model hosts; no `http` rule; an
|
||||
IP store host raises naming `S3_ENDPOINT`.
|
||||
- `tests/test_plane_egress.py`: for every committed template under `plane/tasks/` the rendered policy is the
|
||||
expected set; the IP refusal.
|
||||
- _Requirements: 3.2_
|
||||
- [x] 2.4 Reconciler on `Delivery`
|
||||
- `src/openultrasast/plane/reconciler.py`: `run()` opens the store first (no fallback), builds `Delivery`,
|
||||
passes `delivery` to `router.starter`/`start_task` (body extended in `router.py`), `_await` on
|
||||
`Delivery.delivered`, outcome from the collected `summary.json`; the receiver thread is kept behind
|
||||
`OUSAST_DELIVERY=receiver` for this group only. Line bound per task 1.6.
|
||||
- `tests/test_plane_reconciler.py`: the ax fake run completes through the delivery fake; a rerun from an empty
|
||||
results root resumes from the store's `state.json`.
|
||||
- _Requirements: 2.1, 3.1_
|
||||
- [x] 2.5 Kind proof and record 02
|
||||
- `ops/ax/smoke-run.sh` with `OUSAST_DELIVERY=store`: `facts.json` and `summary.json` arrive through the store;
|
||||
the actor's policy read back has one TLS rule naming the store host; `ousast plane status ax-e2e-smoke`
|
||||
prints the table. Instrument: the tar's byte size from the store equals the extracted files' total.
|
||||
- `benchmarks/measurements/<date>-k8s-02-store-delivery/record.json`: bytes, PUT-to-poll seconds, rules.
|
||||
- _Requirements: 3.1, 3.2, 7.1_
|
||||
- [ ] 2.6 Remove the receiver (last; after 2.5 is recorded) -- paused 2026-10-04
|
||||
- Delete `Receiver`, `_Handler`, `receiver_address`, `receiver_cluster_ip`, `RECEIVER_HOST`, `RECEIVER_PORT`
|
||||
from `egress.py`; `OUSAST_ARTIFACT_URL/DIAL/HOST/PORT` from `runner.py` and `reconciler.py`; the
|
||||
`OUSAST_DELIVERY` switch; `ops/ax/receiver-service.yaml.tmpl` and the `up.sh` receiver step; the EndpointSlice
|
||||
from kind (`kubectl delete`). Docs: `docs/plane.md` sequence diagram, `docs/deployment.md`, `ops/ax/README.md`.
|
||||
- Tests using `OUSAST_ARTIFACT_HOST` move to the delivery fake; `test_plane_literals` covers `172.19.`.
|
||||
- Evidence: `ops/ax/smoke-run.sh` passes again on the build without the receiver; recorded in 02's record as
|
||||
`receiver_removed: <commit>`.
|
||||
- _Requirements: 3.1, 1.3_
|
||||
|
||||
- [ ] 3. `--exec remote` from a CI-shaped principal (design section 2; open-question check 1) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 3.1 Execution mode and RBAC manifests
|
||||
- `cli.py`: `ousast plane run --exec local|remote` (default `profile.exec`); `reconciler.Ax` passes the
|
||||
profile's context to `ax` (its context flag, read from `ax --help` and pinned in a test); `remote` refuses a
|
||||
`file://` store. `ops/k8s/base/rbac-ci.yaml`: SA `ousast-ci` in `ax-system`; Roles and RoleBindings in
|
||||
`ax-system` and `ate-system` with `pods` get/list, `services` get, `pods/portforward` create, nothing else.
|
||||
- `tests/test_plane_rbac.py`: fake `kubectl`, `ax`, `kubectl-ate` record argv over a full fake Run; the
|
||||
verbs implied (`port-forward` -> `pods/portforward` create + `pods` get + `services` get) equal the rendered
|
||||
Role's rules exactly.
|
||||
- _Requirements: 2.1, 2.2_
|
||||
- [ ] 3.2 Substrate's authorization of the SA token (open question 1)
|
||||
- `ops/ax/probe-ci-token.sh`: `kubectl create token ousast-ci --duration=2h` into a kubeconfig; `kubectl ate
|
||||
--token-file - get workers` and `create egress-policy` on a throwaway actor with that kubeconfig. If refused,
|
||||
read `substrate/docs/api-guide.md` for the grant (atespace role or token audience) and add the manifest to
|
||||
`ops/k8s/base/rbac-ci.yaml`; if no grant exists for a non-admin token, record it and the design's remote mode
|
||||
uses the maintainer's admin kubeconfig for the proof, with the gap listed in the runbook.
|
||||
- Evidence: `record.json` names the grant used (`grant: <kind/name>` or `admin-kubeconfig`).
|
||||
- _Requirements: 2.1_
|
||||
- [ ] 3.3 CI-shaped container
|
||||
- `ops/k8s/ci-shell.Dockerfile`: `python:3.12-slim` + this package + `kubectl`, `kubectl-ate`, `ax` (built from
|
||||
the checkouts); no `.env`; runs with `-e KUBECONFIG=/run/kc -e DEEPSEEK_API_KEY -e S3_*` only.
|
||||
`.github/workflows/plane-remote.yml` (dispatch only, no schedule): the same image running `ousast plane run
|
||||
--exec remote --profile k3s` with repository secrets; disabled until k3s exists (labelled in the file).
|
||||
- _Requirements: 2.2, 2.3_
|
||||
- [ ] 3.4 Kind proof and record 03 **paid: needs a budget go-ahead, ceiling 0.50 USD**
|
||||
- From the container of 3.3 against kind (`kind.toml` with `exec = "remote"`): the smoke Run (model-free), then
|
||||
one `verify` pass over one case of `plane/runs/validation-46.yaml` (budget `{usd: 0.5, calls: 60}`) so the
|
||||
credential path is exercised remotely. Kube audit log (kind apiserver `--audit-policy` of the SA's verbs)
|
||||
lists only the Role's verbs.
|
||||
- `benchmarks/measurements/<date>-k8s-03-remote-exec/record.json`: audit verbs vs Role, grant of 3.2, run
|
||||
status, usd spent.
|
||||
- _Requirements: 2.1, 2.2, 2.3, 2.4, 7.1_
|
||||
|
||||
- [ ] 4. Images on GHCR (design section 8) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 4.1 `images.yml`
|
||||
- `.github/workflows/images.yml` on tags `v*`: `docker/build-push-action` for `plane/Dockerfile.runner` ->
|
||||
`ghcr.io/norandom/ousast-runner:<tag>` and the root `Dockerfile` -> `ghcr.io/norandom/ousast-engine:<tag>`;
|
||||
digests as job outputs and as the release asset `images.json` (`{"runner": "...@sha256:...", "engine": ...}`);
|
||||
`GITHUB_TOKEN` with `packages: write`.
|
||||
- Evidence: a dry run on a `v*-rc` tag publishes both and the asset (the engine build is ~2.8 GB: record the
|
||||
build minutes).
|
||||
- _Requirements: 1.1, 4.2_
|
||||
- [ ] 4.2 Engine image under the runner contract
|
||||
- Root `Dockerfile`: the `ax-task-runner` shim and `/workspace` as `plane/Dockerfile.runner:25-29`; the `s3`
|
||||
extra is not needed in either image (the runner PUTs by URL).
|
||||
- `tests/test_plane_images.py`: both Dockerfiles declare the shim and entrypoint (text check).
|
||||
- _Requirements: 4.2_
|
||||
- [ ] 4.3 `repin` for two images
|
||||
- `generate.repin(plane, images)`: `engine.yaml` (added in 5.1; until then absent is fine) takes
|
||||
`images["engine"]`, every other template `images["runner"]`; `ousast plane repin images.json`;
|
||||
`--runner-image FILE` keeps the single-image file working. `ops/k8s/profiles/k3s.toml` names `images.json`
|
||||
downloaded from the release.
|
||||
- `tests/test_plane_increment.py`: repin with a two-key file; a template with two image lines is refused.
|
||||
- _Requirements: 1.1, 4.2_
|
||||
- [ ] 4.4 Kind proof and record 04
|
||||
- Pools' pod template takes `imagePullSecret` from the profile (`ops/k8s/base/workerpool-*.tmpl`, task 6.1
|
||||
brings the full template; here the field only); kind pulls `ghcr.io/norandom/ousast-runner@sha256:...` with
|
||||
`ghcr-pull` if the package is private; the smoke Run passes on the GHCR image.
|
||||
- `benchmarks/measurements/<date>-k8s-04-ghcr-images/record.json`: image sizes, pull seconds, visibility.
|
||||
- _Requirements: 1.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 5. The engine as a plane task (design section 4) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 5.1 `cpg/dump.py` and `tasks/engine.py`
|
||||
- Lift `benchmarks/push/finding_dump.py`'s scan into `src/openultrasast/cpg/dump.py` (the script becomes a
|
||||
thin caller; `ops/README.md:78` command unchanged). `src/openultrasast/plane/tasks/engine.py`: env
|
||||
`OUSAST_WORKSPACE_DIR`, `OUSAST_FIXED_DIR`, `OUSAST_INPUT_CASE`, pins, `OUSAST_FAMILIES`, `OUSAST_DEADLINE`;
|
||||
per pin the quick scan (`tasks.alerts.scan`) and the engine in-process; rows via `engine_alerts.engine_rows`,
|
||||
the instrument checks via `read_record`'s rules; `summary.json` as `engine_alerts.produce` writes it with
|
||||
`host: false`. `plane/tasks/engine.yaml`: `command: ["engine"]`, requests 3Gi/1, limits 4Gi/2, image from
|
||||
`images["engine"]`. `generate.TEMPLATES` gains `engine`; `--loop` emits `engine` instead of `alerts` for a
|
||||
case needing the engine when the images file has an engine key.
|
||||
- `tests/test_plane_engine_task.py`: a fixture record -> rows and summary equal to `produce`'s shape; the
|
||||
unread-input and zero-question failures; the generator's choice.
|
||||
- _Requirements: 4.1_
|
||||
- [ ] 5.2 `engine-pool`
|
||||
- `ops/k8s/base/workerpool-engine.yaml.tmpl` rendered by `plane/k8s.py` (task 6.1 generalises; here the engine
|
||||
values): kind replicas 1, 4Gi/2; if 1.5 recorded `any`, apply the design's fallback and write it here.
|
||||
- Evidence: `kubectl -n ax-system get workerpool engine-pool` ready on kind with `ousast-pool` at 1 replica.
|
||||
- _Requirements: 4.1, 5.1_
|
||||
- [ ] 5.3 `alerts-engine` marked development-only
|
||||
- `cli.py` help, `docs/plane.md` task table, `ops/README.md`: "development only (needs Docker on the host);
|
||||
the plane task is `engine`". No code removed.
|
||||
- _Requirements: 4.3_
|
||||
- [ ] 5.4 Kind proof and record 05
|
||||
- The froxlor case of `validation-46` as an `engine` task on kind (model-free, deadline 1800 s per pin):
|
||||
rows equal `benchmarks/measurements/2026-09-30-php-engine-alerts/` (6/6 per pin), placement on `engine-pool`
|
||||
shown by `kubectl ate get workers`, peak memory from `kubectl top`.
|
||||
- `benchmarks/measurements/<date>-k8s-05-engine-task/record.json`: rows diff, seconds per pin, peak memory,
|
||||
placement, image size and pull seconds.
|
||||
- _Requirements: 4.1, 4.2, 7.1_
|
||||
|
||||
- [ ] 6. Pools, autoscaling, node checklist (design section 5) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 6.1 `plane/k8s.py` and the manifests
|
||||
- `src/openultrasast/plane/k8s.py`: `render(profile, out)` writes `workerpool-default.yaml`,
|
||||
`workerpool-engine.yaml`, `ax-server-snapshots.yaml` (patch with `profile.snapshots_bucket`), `rbac-ci.yaml`;
|
||||
for `k3s` also `imagepullsecret.yaml`, `hpa-default.yaml`, `hpa-engine.yaml`, `prometheus-adapter.yaml`
|
||||
(copied from `substrate/demos/autoscaled-workerpool`, namespace `ax-system`, metric
|
||||
`ate_workerpool_workers{state=at_capacity}`, `averageValue 0.7`). `ousast plane manifests --profile P --out
|
||||
DIR`; `ops/k8s/render.sh`. `ops/ax/workerpool.yaml.tmpl` deleted once `up.sh` uses the renderer.
|
||||
- `tests/test_plane_k8s.py`: both profiles render; kind has no HPA and no pull secret; k3s has both; engine
|
||||
pool limits >= `engine.yaml` limits; every image digest-pinned; no host literal in the output.
|
||||
- _Requirements: 5.1_
|
||||
- [ ] 6.2 Node checklist and footprint
|
||||
- `docs/deployment.md`: the gVisor correction (no RuntimeClass; `ateom-gvisor` runs `runsc` from the
|
||||
SandboxConfig tarball), PodSecurity for the worker pod's securityContext (recorded from kind), local-path for
|
||||
`/var/lib/ate`, the measured footprint per task type (`verify`, `repo-facts`, `engine` from 05) and the
|
||||
queueing consequence of one actor per worker with `--workers`.
|
||||
- _Requirements: 5.2, 7.2_
|
||||
- [ ] 6.3 Kind proof and record 06 (HPA: render only; live is k3s-only, **manual** later)
|
||||
- `render.sh kind` applied on kind replaces the hand template with identical pools (`kubectl diff` empty but
|
||||
for labels); worker pod `securityContext` captured; `render.sh k3s` validates with `kubectl apply
|
||||
--dry-run=client` against kind's API (CRDs present).
|
||||
- `benchmarks/measurements/<date>-k8s-06-pools/record.json`: footprint table, securityContext, dry-run result.
|
||||
- _Requirements: 5.1, 5.2, 7.1_
|
||||
|
||||
- [ ] 7. `ousast plane scan` (design section 6) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 7.1 Workspaces and candidates from URL + commits
|
||||
- `generate.scan_cases(url, base, head, cache, *, all=False)`: shallow clone into the case cache, two pinned
|
||||
Workspaces, head-side hunks via `fix_ranges(side="new")`, functions via `repo_facts.enclosing` over
|
||||
`product_files`, quick alerts via `tasks.alerts.scan` on the changed files, `candidates.json` and
|
||||
`functions.json` as store inputs (`Delivery.put_input`); `--all` takes every product function.
|
||||
- `tests/test_plane_scan.py`: a two-commit fixture repository yields exactly the changed functions; a rename
|
||||
and a deleted function are handled; `--all` count.
|
||||
- _Requirements: 6.1_
|
||||
- [ ] 7.2 Run composition and the command
|
||||
- `src/openultrasast/plane/scan.py` + `cli.py`: `ousast plane scan URL --base C --head C [--all] [--ceiling USD]
|
||||
[--exec] [--wait S | --no-wait]`: per set `facts -> va, vb -> agree -> vc -> final -> features -> remember`,
|
||||
`engine` when a changed file's language needs it, `roles` once per repository when memory has no `roles`
|
||||
rows for `repo_key(url)`; budgets from `verify_budget`; Run annotated `openultrasast.io/population: scan`,
|
||||
`split: <repo_dir>`; after the Run, `memory.ingest`, then the attribution table and the plane's agreement
|
||||
(`final`), the decision engine's verdict only when adopted. `--no-wait` prints the Run name and exits 0.
|
||||
- `tests/test_plane_scan.py`: the rendered Run for the fixture; budgets sum under the ceiling; `roles`
|
||||
included only when the store lacks rows; the report's verdict line.
|
||||
- _Requirements: 6.1, 6.2_
|
||||
- [ ] 7.3 Hook opt-in
|
||||
- `ops/pre-push`: with `OUSAST_PLANE_SCAN=1`, after the blocking check, `ousast plane scan <remote url>
|
||||
--base <old> --head <new> --exec remote --no-wait` and the Run name in the report; the hook's exit code never
|
||||
depends on it. `ops/README.md` and `docs/deployment.md`: the latency split.
|
||||
- `tests/test_push_hook.py`: the wrapper calls the scan only with the variable set and ignores its failure.
|
||||
- _Requirements: 6.3_
|
||||
- [ ] 7.4 Kind proof and record 07 **paid: needs a budget go-ahead, ceiling 3.00 USD**
|
||||
- A PHP pair from the development corpus (pmpro's CVE commit and its parent, never a v2 or v3 case) scanned on
|
||||
kind: candidates, the Run through `final`, `engine` for the PHP files, rows under
|
||||
`repos/<host>__<owner>__<name>/<head>.jsonl` on the store, the report; then `ousast plane status`.
|
||||
- `benchmarks/measurements/<date>-k8s-07-plane-scan/record.json`: candidates, usd, calls, agreement, rows,
|
||||
seconds from submit to verdict, whether the known CVE site is among the agreed.
|
||||
- _Requirements: 6.1, 6.2, 6.3, 7.1_
|
||||
|
||||
- [ ] 8. Runbook (design section 7, Req 7) -- to re-plan (kube-ax, roadmap G4)
|
||||
- [ ] 8.1 `docs/deployment.md` as the runbook
|
||||
- Sections: prerequisites for k3s (Substrate, ax with the snapshots patch, GHCR and the pull secret, buckets,
|
||||
the CI SA and its grant from 3.2, the node checklist of 6.2), the profile switch, `--exec remote` from a laptop
|
||||
and from CI, and the checklist whose every item names its requirement and its record 01-07; items without a
|
||||
recorded run (HPA live, 1.7's fetch) labelled planned. `ops/ax/README.md` and `ops/README.md` point to it;
|
||||
the "moving to a separate cluster" tables are replaced by the checklist.
|
||||
- _Requirements: 7.2_
|
||||
- [ ] 8.2 Spec closure
|
||||
- `design.md` open questions updated with the outcomes of 1.5, 1.6, 3.2 (1.7 stays open until k3s); the
|
||||
`kind-tooling` memory and `.kiro/steering/overview.md` current-state line updated; `/kiro-spec-status`.
|
||||
- Evidence: every record 01-07 linked from the runbook's checklist.
|
||||
- _Requirements: 7.1, 7.2_
|
||||
@@ -67,7 +67,7 @@ real capability; task 8.3 still owns actual eligibility. Task 4.2 brings progres
|
||||
## Task 4.3 verification
|
||||
|
||||
The agent service rejected the task 4.3 dispatch with `agent thread limit reached`.
|
||||
The [kiro-impl skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly provides:
|
||||
The [kiro-impl skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly provides:
|
||||
“If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
Implementation, review and completion checks therefore ran in the main context.
|
||||
This is a manual review record, not an independent-review claim.
|
||||
|
||||
@@ -7,7 +7,7 @@ Real capability eligibility and rollout readiness remain unproven.
|
||||
## Execution and review mode
|
||||
|
||||
Fresh implementer dispatch for 5.1 returned `agent thread limit reached`.
|
||||
The [Kiro implementation skill](../../../.opencode/skills/kiro-impl/SKILL.md) explicitly
|
||||
The [Kiro implementation skill](../../../.agents/skills/kiro-impl/SKILL.md) explicitly
|
||||
states: “If multi-agent is not available, fall back to manual mode execution for all tasks.”
|
||||
Its manual review rule permits review in the main context when a fresh reviewer is unavailable.
|
||||
This run therefore uses manual implementation and adversarial review, not independent review.
|
||||
|
||||
@@ -147,6 +147,43 @@ with Express-style routes and middleware, including vulnerable, fixed and benign
|
||||
frontend/framework adaptations and new abstract capabilities, and shall report C/C++ envelope or
|
||||
unsupported-property results separately from demonstrated vulnerability detection.
|
||||
|
||||
## Requirement 9: Operate as an advisory safety net on any repository
|
||||
|
||||
Added 2026-10-02, maintainer-approved, from the operational-readiness audit. The maintainer's
|
||||
definition: a user installs the hook on ANY repository and gets honest, advisory-only results;
|
||||
nothing ever blocks a push unless blocking was explicitly chosen.
|
||||
|
||||
9.1 (added 2026-10-02, maintainer-approved) When the installed hook cannot run the check, because the
|
||||
`ousast` executable is missing or an argument or setting is invalid, the hook shall allow the push and
|
||||
print one line saying the check did not run and how to fix it; only an explicit `--mode blocking`
|
||||
opt-in may turn an analysis result into a rejected push.
|
||||
|
||||
9.2 (added 2026-10-02, maintainer-approved) When a check is skipped or incomplete, the terminal output
|
||||
shall name each distinct reason in plain words, one line per reason, with repository paths decoded and
|
||||
no opaque hash tokens; a language with no engine frontend and no quick rules shall be named with its
|
||||
file count; a first push of a new branch shall say there was no base to compare against and what was
|
||||
checked instead. The JSON artifact keeps the raw reason values.
|
||||
|
||||
9.3 (added 2026-10-02, maintainer-approved) Before the engine, within the same deadline, the Safety Net
|
||||
shall run the shipped quick pattern rules over the head side of the changed lines and show their
|
||||
findings as advisory pattern matches, labeled separately from admitted alerts. This amends 3.2 for this
|
||||
tier only: quick-rule findings are shown, never admitted, and never block.
|
||||
|
||||
9.4 (added 2026-10-02, maintainer-approved) When a finding's operation lies in a file that is new in the
|
||||
head revision, the comparison shall classify it as new rather than unknown; an engine finding whose
|
||||
novelty is established but whose capability is not yet qualified shall be shown as an advisory engine
|
||||
finding, not an admitted alert. Genuinely unknown comparisons keep their vetoes.
|
||||
|
||||
9.5 (added 2026-10-02, maintainer-approved) When the engine cannot finish within the deadline, one line
|
||||
shall say so; on explicit request (`--engine background`) the Safety Net shall start the engine
|
||||
detached after the push, record its result in the artifact directory, and show it on the next run as
|
||||
the engine result for that head from the previous push.
|
||||
|
||||
9.6 (added 2026-10-02, maintainer-approved) The hook installer and remover shall ship in the wheel
|
||||
(`ousast pre-push install|uninstall`), refuse to overwrite an existing hook unless forced, and a
|
||||
hook-callable container wrapper shall pass the push input and artifact path through; the documentation
|
||||
shall give a first-run guide, a glossary of skip reasons, a CI recipe and help text for every flag.
|
||||
|
||||
## Accepted experimental acceptance profile
|
||||
|
||||
These values are the initial implementation/evaluation profile accepted for task generation, not
|
||||
|
||||
@@ -475,7 +475,7 @@ Preserve all completed task evidence below.
|
||||
- _Depends: 13.1_
|
||||
- _Requirements: 6.2, 6.3, 8.1, 8.2_
|
||||
|
||||
- [ ] 14. Ask every declared family whether it establishes anything (precondition for M4)
|
||||
- [x] 14. Ask every declared family whether it establishes anything (precondition for M4)
|
||||
- Seven families are declared and two were proven end to end. A declared family that has never been asked whether it finds anything is advertised coverage, and M4 cannot qualify a capability that establishes nothing.
|
||||
- [x] 14.1 Census the seven declared families on NodeGoat and VAmPI
|
||||
- Run production discovery and scan over each whole checkout and report, per family, the regions that offered it, the questions asked, the questions completed, and what was found with the witness and rung. No comparison, no novelty, no admission: detection only.
|
||||
@@ -575,7 +575,7 @@ Preserve all completed task evidence below.
|
||||
- _Boundary: Evaluation harness, full-run integration_
|
||||
- _Depends: 13.2, contributor-scan 7.3, contributor-scan 7.8_
|
||||
- _Requirements: 6.2, 6.3, 8.1, 8.2_
|
||||
- [ ] 13.4 Let a question inside one partition complete, whatever else the repository holds
|
||||
- [x] 13.4 Let a question inside one partition complete, whatever else the repository holds
|
||||
- **Diagnosed 2026-09-21, and PHP was never the variable.** A repository holding TWO languages can never complete a question, so it can never admit a finding. `partitions.py` raises `cross_partition_semantics_unresolved` whenever more than one language partition exists; that reason is a member of `GRAPH_INTEGRITY_GAPS`; and a graph-integrity gap demotes every completed outcome in the scan.
|
||||
- The evidence is one column wide: NodeGoat is JavaScript alone and completes 299 of 299, VAmPI is Python alone and completes 35 of 35, PMPro is PHP with 36 regions of bundled JavaScript and completes 0 of 127 and 0 of 520 while establishing 19 and 43 findings. NodeGoat's own `vendor_semantics_unresolved` is a DECLARED exclusion rather than an integrity gap, which is why it does not demote.
|
||||
- **This is larger than the PHP runtime limit and it lands on the declared v0.1 target.** A WordPress plugin is PHP plus its admin JavaScript, so every one of them is multi-language, so on the target this tool was built for it can detect defects and never admit one. Neither subject that currently passes could reveal it.
|
||||
@@ -587,6 +587,7 @@ Preserve all completed task evidence below.
|
||||
- _Three causes, not one, and the third was hidden by the record rather than by the code._ The same census gap is raised in TWO passes, arbitration and evidence, and carrying the partition in one but not the other left an unscoped copy that kept demoting everything, so the fix looked applied and was not. It was found by making the census print each outcome's reason instead of only its status, after which three separate causes stopped looking identical.
|
||||
- _Admission is scoped too._ `_blocking_degradations` already exempted declared exclusions; it now also ignores a gap another language partition raised, so a JavaScript build artefact cannot block a PHP finding. A gap naming no partition still reaches everything, and so does every gap when the claim's own language is unknown, because unknown scope is not evidence of independence. Evidence: `benchmarks/measurements/2026-09-21-partition-gap-blast-radius.json`.
|
||||
- _Boundary: contributor-scan query/context evidence, Ranker Scope Contract_
|
||||
- _Checked 2026-10-02 per the operational-readiness audit:_ implemented and verified on PMPro (120 of 127, above). 13.3 stays open on its own blocker; this box was left unchecked only because it waits on it.
|
||||
- _Depends: 13.3_
|
||||
- _Requirements: 2.1, 2.2, 6.3, 8.1_
|
||||
|
||||
@@ -619,6 +620,55 @@ Preserve all completed task evidence below.
|
||||
- _Requirements: 8.1, 8.2_
|
||||
|
||||
|
||||
- [x] 17. Operational readiness (2026-10-02)
|
||||
- **Named by the 2026-10-02 operational-readiness audit**, maintainer-approved as the first increment. The maintainer's definition: a user installs the hook on ANY repository and gets honest, advisory-only results; nothing ever blocks a push. The audit found the plumbing sound (snapshots, resolution, deadline accounting, advisory exit 0, the JSON artifact) and five things in the way: the hook can block a push, quick rules are not part of pre-push, the engine cannot finish a 26-file repository inside 30 s, the terminal hides every skip reason, and there is no supported install path without the source checkout. Before numbers for every item are the audit's live runs on records (Python), serve-static (JavaScript) and unrolled/secure (Go), each with one planted commit.
|
||||
- This group is plumbing and honesty. It enables no capability, edits no rule for a planted sample, and leaves graph reuse (the engine speed work decided in 11.6) as a later task.
|
||||
- Record: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (see 17.7).
|
||||
- _Done 2026-10-02:_ all seven subtasks done. The engine still does not fit the 30 s hook deadline (graph reuse, the 11.6 direction, stays a later task), and plane verdicts after the push were not in this increment.
|
||||
- _Requirements: 9.1–9.6_
|
||||
- [x] 17.1 Never block a push the user did not choose to block
|
||||
- The installed hook exits 0 with one line when `ousast` is missing (the check did not run, and how to install). In hook (stdin) mode, an argument or usage error (an invalid `OUSAST_PUSH_DEADLINE`, a cache directory inside the repository) is advisory: one line and exit 0. `--mode blocking` stays the explicit opt-in that can reject a push, and keeps its usage errors. The prior-hook input-retention failure keeps exit 1, because it is the chained hook's verdict that cannot be obtained, not ours.
|
||||
- Files: `ops/pre-push`, `src/openultrasast/cli.py`. Tests: `tests/test_push_hook.py` (a push with `ousast` absent from PATH succeeds and prints the line; a bad deadline in hook mode exits 0; blocking mode keeps exit 2).
|
||||
- Evidence: the audit's `hook.sh` push 1 (rejected before) and `fail.sh` (rc=2 before) rerun with exit 0.
|
||||
- _Done 2026-10-02:_ the hook prints one line and exits 0 (chaining a prior hook if present) when `ousast` (or `OUSAST_COMMAND`) is missing; in hook mode without `--mode blocking`, argparse/usage errors and unexpected exceptions print one line and exit 0. Tests: `test_missing_ousast_never_blocks_the_push`, `test_hook_mode_usage_errors_are_advisory_unless_blocking` (blocking keeps exit 2), and `test_cli_refuses_the_flag_outside_explicit_replay` updated (the experimental flag is still refused in hook mode, now advisory). Gates: 1987 passed, 42 skipped; ruff, format, mypy clean.
|
||||
- _Requirements: 9.1, 5.3, 5.4_
|
||||
- [x] 17.2 Name every skipped check in plain words
|
||||
- One terminal line per distinct coverage reason: hex-encoded paths decoded, sha256-suffixed reasons grouped with a count, JSON unchanged. "language not covered: <lang> (<n> files)" for a language with no engine frontend and no quick rules, in both `pre-push` and `scan`. A first push of a new branch says there is no base and what was checked instead. `reports.py` gains lines for the eight scan degradations `cli.py` records and the report did not render (`learning_endpoint_unavailable`, `cpg_unavailable`, `facts_unavailable`, `policy_invalid`, `model_withheld_execution_budget`, `max_findings_exceeded`, `hunter_model_unavailable`, `sandbox_unavailable`).
|
||||
- Files: `src/openultrasast/push/report.py`, `src/openultrasast/push/explain.py` (new), `src/openultrasast/reports.py`, `src/openultrasast/cli.py`. Tests: `tests/test_push_report.py`, `tests/test_reports.py`.
|
||||
- Evidence: the three repositories' terminal output names engine-missing, Go not covered and the new branch.
|
||||
- _Done 2026-10-02:_ new `push/explain.py` turns every raw coverage reason into one plain `Skipped:` line per distinct kind (hex paths decoded, sha256 tokens dropped and counted, unknown reasons still printed); the existing one-line `Notice:` keeps its next action and gains an engine-missing variant. `language_not_covered:<lang>:<changed>:<total>` is recorded for a changed language with no engine frontend and no quick rules; `ousast scan` prints `notice: language not covered: go (N files)` and its report says so. `reports.py` renders the eight previously dropped degradations and prints any reason it has no sentence for. The hook passes `OUSAST_COMPARISON_BASE`. The new-branch line names the base it lacks; what was checked instead lands with the quick tier (17.3). Tests: `tests/test_push_explain.py` (6), two in `tests/test_reports.py`; module-audit manifest regenerated for the new module. Gates: 1995 passed, 42 skipped; ruff, format, mypy clean.
|
||||
- _Requirements: 9.2, 5.2, 8.2_
|
||||
- [x] 17.3 Run the quick rules as the fast first tier
|
||||
- Inside the deadline and before the engine, the shipped quick rules run over the head side of the changed files and keep the findings on changed lines or in changed functions. They print as advisory pattern matches, labeled apart from admitted alerts, recorded in the artifact's `quick_tier`, and never block.
|
||||
- Files: `src/openultrasast/push/quick.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_quick.py` (new).
|
||||
- Evidence: serve-static's planted `eval` in the new `debug.js` is shown; whether records' planted `%`-formatted SQL is shown is recorded as is, with no rule edits for it.
|
||||
- _Done 2026-10-02:_ new `push/quick.py` runs the shipped quick rules (no shadow rules, no model) over the changed head files inside `_prepare` under the shared deadline, before the engine, keeping matches on a changed line or inside a changed function (entry-point ranges); it records `files`/`bytes_read` as proof of input. The terminal shows them under "Quick rules: N pattern match(es) on changed code ... Advisory: not verified by the engine."; the artifact keeps `quick_tier`; admission and push disposition are untouched. A new branch runs the quick tier over its last commit's changes and says so. `_analyze` gained `engine="off"`. On the audit's serve-static planted commit the `eval` in the new `debug.js` is now shown (js-eval, js-xss-response at debug.js:2, ~1.1 s on the host); the planted `%`-formatted SQL in records is NOT matched by any quick rule (coverage gap, no rule edited). Tests: `tests/test_push_quick.py` (3). Gates: 1998 passed, 42 skipped; ruff, format, mypy clean.
|
||||
- _Requirements: 9.3_
|
||||
- [x] 17.4 Classify a file new in head as new, not unknown
|
||||
- `ChangeContext` records the paths Git reports as added. An operation in an added file, when no add/delete pairing is unresolved and its head evidence is complete, is `new` with reason `file_added_in_head`, and admission accepts that reason as change evidence. An engine finding with established novelty whose only vetoes are capability vetoes renders as an advisory engine finding, not an alert. Every other unknown comparison keeps its veto.
|
||||
- Files: `src/openultrasast/model/contracts.py`, `src/openultrasast/push/snapshot.py`, `src/openultrasast/push/policy.py`, `src/openultrasast/push/report.py`. Tests: `tests/test_push_policy.py`, `tests/test_push_snapshot.py`, `tests/test_push_report.py`.
|
||||
- Evidence: unit tests on the comparison and admission; a live engine run if the host allows one (free memory >= 3 GB).
|
||||
- _Done 2026-10-02:_ `ChangeContext.added_paths` records Git status-A paths (rename targets excluded). In `_compare_evidence`, after every existing context/head/deadline check, an operation in an added file is `new`/`file_added_in_head` when the claim is entailed, undischarged and has change evidence, and no base answer holds the same operation (else `operation_moved_within_change`); an unpaired add/delete still blocks as `change_context_unresolved`. Admission accepts `(new, file_added_in_head)`; with the empty registry such a candidate's only veto is `capability_unavailable`, and the terminal now shows it under "Engine: N finding(s) this push introduced. Advisory ... not an alert." Any comparison, evidence or location veto keeps a candidate out of that line. Tests: `tests/test_push_new_file.py` (9). Live engine evidence depends on host memory (17.7). Gates: 2007 passed, 42 skipped; ruff, format, mypy clean.
|
||||
- _Requirements: 9.4, 3.4_
|
||||
- [x] 17.5 Say when the engine cannot finish, and offer it in the background
|
||||
- One line when the engine did not finish (deadline, engine not installed). `--engine background` starts the engine detached after the quick tier, writes its result under the artifact directory and, on the next run, prints "engine result for <head> from the previous push". One background engine at a time; `--engine off` skips it with a stated line. No speed work.
|
||||
- Files: `src/openultrasast/push/background.py` (new), `src/openultrasast/push/runner.py`, `src/openultrasast/cli.py`, `ops/pre-push`. Tests: `tests/test_push_background.py` (new).
|
||||
- Evidence: a hook push with `OUSAST_PUSH_ENGINE=background` followed by a second push showing the previous result.
|
||||
- _Done 2026-10-02:_ `--engine inline|background|off` (hook: `OUSAST_PUSH_ENGINE`) and `--background-deadline` (default 900 s). `off` and `background` run the quick tier and record `engine_off`/`engine_background` with a plain skip line; the deadline line names the deadline and the background option. New `push/background.py` starts one detached explicit replay (`start_new_session`, stdin closed, output to a log) per artifact directory and repository (lock file with a live-pid check: a second start reports busy), writes its artifact under `<artifact dir>/engine/<repo key>/<head>.json`, and the next run prints "Engine result for <head> from the previous push: ..." once, then marks it shown. Not usable through the Docker wrapper (the container exits with the hook); documented in 17.6. Tests: `tests/test_push_background.py` (4, one launching a real detached run). Gates: 2011 passed, 42 skipped; ruff, format, mypy clean.
|
||||
- _Requirements: 9.5, 4.1, 4.2_
|
||||
- [x] 17.6 Install, Docker and documentation for a first-time user
|
||||
- `ousast pre-push install|uninstall [--force] [--docker] [REPO]` ships the hook as package data, honours `core.hooksPath`, refuses an existing hook unless `--force` (which chains it as `pre-push.before-ousast`), and works from a pip install. `ops/ousast-docker` is a hook-callable script (not a shell function) that passes stdin and the artifact path through. The `openultrasast:dev` image is rebuilt only if `df -h /` shows at least 4 GB free; otherwise the rebuild command is documented and the record says it was not rebuilt. `docs/scanning.md` gains a first-run guide, a glossary of skip reasons, a GitHub Actions recipe and full `--help` text; README tiers 1 and 2 updated, pulling `ghcr.io/norandom/openultrasast:2.0.1` where the wrapper allows.
|
||||
- Files: `src/openultrasast/push/hook/pre-push`, `src/openultrasast/push/install.py` (new), `src/openultrasast/cli.py`, `ops/ousast-docker`, `docs/scanning.md`, `README.md`, `ops/README.md`. Tests: `tests/test_push_install.py` (new), `mkdocs build --strict`.
|
||||
- Evidence: install from the package into a scratch clone, push through it, uninstall restores the prior hook.
|
||||
- _Done 2026-10-02:_ `ousast pre-push install|uninstall [REPO] [--force] [--docker]` (`push/install.py`) installs the hook shipped as package data (`openultrasast/push/hook/pre-push`, byte-identical to `ops/pre-push`, verified in the built wheel), honours `core.hooksPath` without editing it, refuses an existing hook unless `--force` (ours is replaced; a foreign hook moves to `pre-push.before-ousast` and is chained), and uninstall removes only a hook it wrote and restores the chained one. `ops/ousast-docker` (also package data; `--docker` installs it beside the hook, which then uses it) is a script that mounts the repository read-only and the artifact directory at their own absolute paths, passes stdin with `-i`, runs as the user with no network and `--memory 3g`, defaults to `ghcr.io/norandom/openultrasast:2.0.1`, never pulls inside a hook, and exits 0 unless `OUSAST_PUSH_MODE=blocking`. The hook passes `--engine`/`--comparison-base` only when set, so an older ousast or image still runs. Every pre-push flag has help text; `docs/scanning.md` has the first-run guide, settings table, new-branch and background notes, the artifact keys, a skip-reason glossary (a test keeps it in sync with `push/explain.py`), a GitHub Actions recipe and the full `--help` text; README tiers 1 and 2 and `ops/README.md` updated (pull and tag the GHCR image instead of building). **Image not rebuilt:** `df -h /` showed 1.1 GB free (< 4 GB); rebuild with `docker build -t openultrasast:dev .`. Tests: `tests/test_push_install.py` (11) and the glossary test. Gates: 2022 passed, 42 skipped; ruff, format, mypy clean; `mkdocs build --strict` exit 0.
|
||||
- _Requirements: 9.6, 7.4_
|
||||
- [x] 17.7 Prove it on the audit's three repositories
|
||||
- Rerun records, serve-static and unrolled/secure (same planted commits) with the new build in quick mode and with the hook installed, plus one Docker engine run on serve-static if free memory >= 3 GB and free disk >= 1 GB. Prove each run read its input (files, bytes).
|
||||
- Files: `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json`. Evidence: before (the audit) and after: exit codes, wall time against the deadline, findings shown, every skip line printed. A planted sample still missed is a coverage gap, recorded, never a rule edit.
|
||||
- _Done 2026-10-02:_ `benchmarks/measurements/2026-10-02-pre-push-operational-1/record.json` (before = the audit, after = this build installed from its wheel into a scratch venv). 10 hook pushes and 6 replays, all exit 0, 0.69-1.21 s against 30 s on the host; the quick tier read exactly the changed file and every head snapshot read the full tree (records 43,577 B, serve-static 61,525 B, secure 103,023 B). The planted JS `eval` in the new `debug.js` surfaces: quick tier on the host, and as two advisory engine findings classified `new`/`file_added_in_head` in one Docker engine run (190.9 s against a 300 s deadline; MemAvailable 3.28 GB, 1.16 GB disk). **The planted Python `%`-formatted SQL is still missed**: no quick rule matches it and the host has no engine. Recorded as a coverage gap, no rule edited. Go is named as not covered. The proof caught a wrong install hint (`pip install openultrasast`, not on PyPI), corrected in the hook in this commit.
|
||||
- _Requirements: 9.1–9.6, 6.3_
|
||||
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Task5.2 measured PMPro timing out after both snapshots were materialized but before change context/graph analysis. Group6 must account for compatible snapshot/discovery reuse as well as graph reuse; identical-tip hits cannot substitute for the later representative changed-code latency gate. No new detector, scope heuristic or incremental CPG mutation is implied.
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# Brief: OpenCode-MCP proof arm (follow-up to search-with-proof)
|
||||
|
||||
Status: brief / not yet specced. Proposed 2026-10-07 by the maintainer. Builds on
|
||||
`search-with-proof` (see `../search-with-proof/verification-architecture.md`).
|
||||
|
||||
## Motivation
|
||||
|
||||
The search arm (directed, Cairn-inspired) works. The **proof-authoring** last mile is the drag:
|
||||
|
||||
1. **Safety-classifier refusals.** Authoring exploit demos repeatedly tripped a safety classifier
|
||||
this session when exploit payloads or step-by-step attack prose passed through the model or a
|
||||
tool prompt. The mitigation so far (keep payloads on disk, treat demos as opaque) is a workaround,
|
||||
not a fix, and it limits how directly the proof model can reason about the exploit.
|
||||
2. **Nondeterministic wiring.** The agent authors a demo but intermittently mis-wires the app to
|
||||
the oracle fixture (served directory for path, database URL for SQL), so a demonstrable pair
|
||||
comes back inconclusive until a retry. Each miss has cost a paid run.
|
||||
|
||||
Proposal: drive the proof with an **agentic coding tool (OpenCode MCP)** running **inside gVisor**,
|
||||
using a model chosen for exploit authoring rather than one that refuses it, and **learn** the
|
||||
proof-authoring policy with DSPy from **traces logged in OpenObserve**.
|
||||
|
||||
## Shape
|
||||
|
||||
- **Search model:** DeepSeek (current). Good at the directed fact/intent search.
|
||||
- **Proof model:** GLM 5.3 (Anthropic-recommended) and/or an abliterated model via
|
||||
`ABLITERATION_BASE_URL` (`abliterated large v2`). The abliterated model authors exploit **demos**
|
||||
without the refusals that block a safety-trained model on this defensive task.
|
||||
- **Proof driver:** OpenCode MCP (github.com/AlaeddineMessadi/opencode-mcp) as the agentic
|
||||
demo-author, running inside the gVisor verification task so the exploit-authoring is sandboxed.
|
||||
- **Learning:** DSPy compiles/optimizes the proof-authoring program (demo schema, wiring, steps)
|
||||
from past traces, instead of the per-family hand-guidance we keep adding (which is now hitting
|
||||
the prompt character budget — a signal the hand-rule approach is near its limit). Matches the
|
||||
maintainer's "learned decisions, not hand-tuning" direction.
|
||||
- **Tracing:** proof runs (reason/explore/author/verify, timings, outcomes, error classes) log to
|
||||
**OpenObserve AI** at `OPENOBSERVE_URL` (org `OPENOBSERVE_ORG`, stream `OPENOBSERVE_TRACES_STREAM`).
|
||||
DSPy consumes these traces as its training signal.
|
||||
- **Credentials:** stored in `.env` like the existing keys, referenced by name only:
|
||||
`ABLITERATION_API_KEY`, `ABLITERATION_BASE_URL`, `ABLITERATION_MODEL`, `OPENOBSERVE_URL`,
|
||||
`OPENOBSERVE_ORG`, `OPENOBSERVE_TRACES_STREAM`, `OPENOBSERVE_TOKEN`. No secret value ever enters a
|
||||
tracked file, a prompt, a log stream, or a commit.
|
||||
|
||||
## The trust-domain decision (must be resolved first)
|
||||
|
||||
The current design keeps the **model key on the VM only**; cluster tasks carry no secrets and have
|
||||
restricted egress (verify tasks reach the files host only). Running the proof **model** inside gVisor
|
||||
(as OpenCode MCP would) breaks both: the task would hold a model key and need egress to the model
|
||||
API host. Two options:
|
||||
|
||||
- **A. Scoped key + egress in the cluster (operator-managed).** Inject a spend-limited proof-model
|
||||
key into the proof task (like the existing `gemini-api-secret` slot) and add a per-host egress
|
||||
allowlist for the proof-model API. Keeps the fully-sandboxed agentic authoring the maintainer
|
||||
wants; requires an operator change and widens the cluster's secret/egress surface.
|
||||
- **B. Brain on the VM, execution in gVisor (current split preserved).** OpenCode MCP and the proof
|
||||
model run on the VM; only the DAST execution stays in gVisor. No new cluster secret or egress;
|
||||
but the agentic authoring loop is not itself sandboxed.
|
||||
|
||||
Recommendation: start with **B** (no trust-domain change, fastest to try the abliterated proof model
|
||||
and DSPy loop), and move to **A** only if sandboxing the authoring loop itself proves necessary.
|
||||
|
||||
## Security scope (why an abliterated model is in-scope here)
|
||||
|
||||
This is authorized **defensive** work: authoring proof-of-concept demos for **known CVE fix-pairs**
|
||||
in a **gVisor-isolated** corpus, to demonstrate a regression before push. Guardrails to keep:
|
||||
|
||||
- Demos are **data**, not agent code; they stay in the gitignored private area, never echoed.
|
||||
- No real-world or third-party live targets; only the frozen fix-pair corpus and owned oracles.
|
||||
- gVisor is the execution boundary; the abliterated model never runs code itself, it only writes a
|
||||
declarative demo the trusted verifier executes.
|
||||
- The abliterated model is confined to proof-authoring; the search and all reporting stay on the
|
||||
normal models.
|
||||
|
||||
## Open questions for the spec
|
||||
|
||||
1. A or B for the trust domain (above).
|
||||
2. Does the abliterated proof model actually raise the demonstrated rate and remove the wiring
|
||||
misses, measured on the pairs already shown demonstrable (xiaomusic, glance)? Qualify before scaling.
|
||||
3. DSPy program shape: what is the proof-authoring signature (inputs: candidate + oracle contract;
|
||||
output: a valid, wired demo), and what metric does it optimize (demonstrated rate at zero false
|
||||
proofs) from OpenObserve traces?
|
||||
4. Does OpenCode MCP fit the existing executor/verifier task contract (no-secret, presigned I/O,
|
||||
scratch/egress limits), or does it need a new task kind?
|
||||
5. Cost envelope per proof vs the current DeepSeek authoring.
|
||||
|
||||
## OpenCode MCP as the endpoint for both PoC and fix
|
||||
|
||||
Maintainer note (2026-10-07): OpenCode MCP is a good server endpoint for the PoC **and** the fix,
|
||||
not only proof authoring. The same agentic endpoint can:
|
||||
|
||||
1. Author the PoC (the demonstrated proof) for a caught finding.
|
||||
2. Write a candidate fix against the finding's repair direction and the failing PoC.
|
||||
3. Hand the fix back to the differential verifier, which re-runs the PoC and confirms absence
|
||||
(the proof flips from failing to passing).
|
||||
|
||||
This turns the "no auto-patch" gap in `../../../docs/use-case-fix-and-reverify.md` into "the proof
|
||||
endpoint also proposes the fix, and the fix is accepted only when the proof it generated goes green."
|
||||
The PoC is the acceptance test for the fix, so a wrong fix is caught, not trusted. The fix stays a
|
||||
proposal for the developer to review; the verifier's green proof is the objective signal, not the
|
||||
model's say-so. This fix capability is specced alongside the proof authoring when this arm is built.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Changing the search arm, the oracle differential, or the abstract-interpretation triage. This brief
|
||||
is about the **proof-authoring and fix** driver and its learning loop, planned as a full spec after
|
||||
the safety net is qualified.
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"feature_name": "proof-arm-opencode",
|
||||
"created_at": "2026-10-07T14:40:00Z",
|
||||
"updated_at": "2026-10-07T14:40:00Z",
|
||||
"language": "en",
|
||||
"phase": "brief",
|
||||
"approvals": {"requirements": {"generated": false, "approved": false}, "design": {"generated": false, "approved": false}, "tasks": {"generated": false, "approved": false}},
|
||||
"ready_for_implementation": false,
|
||||
"summary": "OpenCode-MCP proof-authoring arm for search-with-proof: abliterated/GLM proof model, DSPy-optimized from OpenObserve traces; trust-domain decision (A scoped-key-in-cluster vs B brain-on-VM) must be resolved first. Creds in .env by name."
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
# Search-with-proof — handoff (2026-10-06)
|
||||
|
||||
State at handoff, for resuming on the search problem and abstract interpretation.
|
||||
|
||||
## RESOLVED 2026-10-07 — the one open question is CLOSED: first cluster demonstration achieved
|
||||
|
||||
The "first `demonstrated`" question below is answered. search-with-proof produced its first autonomous
|
||||
`demonstrated` finding end to end on kube-ax, and a blind negative confirmed no false proof.
|
||||
|
||||
- **Pair:** hanxi/xiaomusic (pilot pair 13), family `path`, vulnerability class **prefix-check bypass**
|
||||
(`startswith(root)` without a trailing separator; the fix adds `os.sep`). Image `e2c8b460`.
|
||||
- **Vulnerable side:** `demonstrated` — owned effect in 3/3 affected runs, 0/3 safe runs. $0.37, 55 calls.
|
||||
- **Fixed side (blind negative, Req 6.3):** `inconclusive`, **0 false proofs**. $0.56, 85 calls.
|
||||
- **Pair total:** $1.22. Record: `benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
|
||||
|
||||
**The original pair-6 both-sides-false was a TWO-part diagnosis, both now fixed:** the demo did not reach the
|
||||
observable (demo/agent), and — the decisive one — the path oracle only modelled **classic parent traversal** while
|
||||
many real path CVEs (xiaomusic included) are **prefix-bypass**, which reaches a sibling that string-extends the served
|
||||
dir, not the parent. The oracle now also plants a prefix-sibling canary.
|
||||
|
||||
**Fix chain that enabled the demonstration (each merged to main and validated by a paid run):**
|
||||
1. Parity — agent is told the observable's location (path oracle reached parity with command/ssrf).
|
||||
2. Verify-time-only — the observable is injected at verify time and absent during exploration; stop the explore-time
|
||||
reachability trap that burned budget without authoring a demo.
|
||||
3. Fair-share pilot budget — each search gets a fair share of `--ceiling-usd` with rollover, so one expensive search
|
||||
no longer starves the rest.
|
||||
4. Served-root guidance — the demo MUST set the app's served directory to the `${served_root}` placeholder.
|
||||
5. Oracle prefix-bypass coverage — plant a sibling canary for the `startswith`-without-separator class.
|
||||
6. Control-signal feedback — the keystone control (does the app serve `${served_root}`?) is fed back to the agent as a
|
||||
`served-root` hint so it self-corrects its wiring. This was the decisive last step.
|
||||
|
||||
**Keystone (honest measurement), merged earlier:** the verifier runs a benign positive control; a broken or mis-wired
|
||||
app yields `could_not_run`/annotation, never a silent clean-looking `inconclusive`.
|
||||
|
||||
**Next toward the pilot exit (Req 6.2):** fan out the path family to ≥5 demonstrations with 0 false proofs, then the
|
||||
≥73-negative BLOCK qualification (Req 6.3). The glance pair (classic traversal) also demonstrates once its npm
|
||||
dependency build-break (`filed`→`mime@4`) is pinned.
|
||||
|
||||
---
|
||||
|
||||
## The one open question (HISTORICAL — resolved above)
|
||||
|
||||
The full chain runs end to end on kube-ax. Every execution layer is cleared. The **first `demonstrated` has not
|
||||
happened yet**, and the remaining gap is narrow and reproducible at **$0 model cost** via `demo_replay`.
|
||||
|
||||
No-model replay of pilot pair 6 (path / Node / npm build), both sides, image `aa54880`:
|
||||
- Both sides **build, start, and run** (ready ~18–20 s, "fresh task run completed", no `could_not_run`).
|
||||
- Oracle observed **no effect on either side** → `inconclusive`, "no consistent differential".
|
||||
|
||||
Both-sides-false means it is **not the harness** any more. It is one of:
|
||||
1. The demo's attack step as written does not trigger the effect even on the vulnerable revision, or
|
||||
2. The oracle does not capture the app's output where the effect would show.
|
||||
|
||||
## How to close it (no model spend)
|
||||
|
||||
- Control: the **probe path app** (`src/openultrasast/search/probe_apps/`) is known-good and *does* observe through the
|
||||
same verifier code path. Diff its behaviour against the real app: how the served directory is wired
|
||||
(`${served_root}` substitution in `start.arguments`), and where the oracle reads the nonce from (app stdout/response
|
||||
capture in `oracles.py` `PathOracle.observe`).
|
||||
- Re-run with `python -m benchmarks.search.demo_replay --both --demo <stored demo.json> --family path --lane ax
|
||||
--image <digest> ...`. Stored demos live under `benchmarks/search/private/runs/*/<pair>/demo/demo.json` (gitignored).
|
||||
- **Classifier note:** keep exploit payloads on disk. Hand Codex file-path tasks that treat demo contents as opaque;
|
||||
print outcomes / phase timings / error *classes*, never request strings, env values, or step bodies. See memory
|
||||
`search-with-proof-direction` and the classifier false-positive note. Code changes never tripped it; echoing
|
||||
payloads or step-by-step attack prose did.
|
||||
|
||||
## What is done and merged (main)
|
||||
|
||||
- Trust-domain split: model key on the VM only; executor and verifier tasks carry no secrets; exchange via presigned
|
||||
objects. Declarative demos (data, not agent code). gVisor is the cluster boundary.
|
||||
- Verifier + 5 oracles pass inside a gVisor task (probe: 5/5 demonstrated, gaming demos inconclusive).
|
||||
- Shared task base image, code-last layers; `plane/task-base.digest` pins it; `engine-image.yml` builds only on
|
||||
dispatch / tags / Dockerfile / pin changes. Measured unpacked: search 2.07 GB, engine 3.61 GB.
|
||||
- Pilot driver (`benchmarks/search/pilot_run.py`), budgets with reservation, named exhaustion limits, `demo_replay`.
|
||||
- Per-host egress live: exec → any 443; verify/probe/pass → files host; replay → files + github. Task names carry
|
||||
the right prefixes.
|
||||
|
||||
## Plumbing defects fixed along the way (each cost one run)
|
||||
|
||||
step allowance → family-vs-oracle mapping → build memory (256 MB SIGTRAP) → manifest path appended positionally →
|
||||
startup env + data files → dead-end intent dedupe → archive size (edge ~100 MB, now split+gzip) → two-interpreter
|
||||
image (pip 3.13 vs venv 3.12 → use `sys.executable`) → checkout root on sys.path → `node_modules` path.
|
||||
|
||||
## Next, once a demo demonstrates
|
||||
|
||||
1. Fan out the pilot across all 20 pairs, both sides (feasibility: demonstrated rate per family, zero false proofs on
|
||||
fixed sides). Exit: ≥1 family with ≥5 demonstrations and zero false proofs (Req 6.2).
|
||||
2. BLOCK qualification per family: 0 false proofs over ≥73 negatives from ≥73 repos (Req 6.3).
|
||||
3. G1 arm on the unseen pool vs the triage/decision arms at equal false-alarm budgets (Req 6.5).
|
||||
|
||||
## Cluster / spend facts
|
||||
|
||||
- kube-ax: **2 tasks in flight, one per worker** (operator `--max-actors=1`). Dispatcher caps AX concurrency at 2;
|
||||
raise it only when the operator raises worker count. "no free workers available" = retry, not failure.
|
||||
- `/workspace` ~3 GiB free per node until the volume move; scratch guard 2 GiB/task.
|
||||
- Pilot spend to date ~$11 across 9 paid runs; the decisive replay was $0.
|
||||
|
||||
## Decision model vs search — current read
|
||||
|
||||
- Decision/classification: measured, **promising as triage** (within-pair AUC 0.762; trained logistic ties at 0.721),
|
||||
**capped as a blocker** (cannot prove 95% precision). Reframed as triage (`learned-decision-engine` Req 6.5).
|
||||
- Search-with-proof: **not yet demonstrated**, but no observation so far argues against the idea — every failure was
|
||||
our execution, now cleared. The pair-6 demo/oracle question is the first verdict on the method itself.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Brief: search with proof
|
||||
|
||||
Maintainer, 2026-10-05, after reviewing Cairn (github.com/oritera/Cairn): "we are still not progressing through the
|
||||
state space search / not finding what we need and the progress is sort of slow ... i think something is missing."
|
||||
And: "I'd move to primarily executing in the cluster."
|
||||
|
||||
## What was missing
|
||||
|
||||
Every detection step so far classifies: it scores one function excerpt and then tries to make the score precise
|
||||
enough to block. Calibration, Wilson bounds, pooled block gates, prompt variants and trained models all hit the same
|
||||
wall: ranking is decent (within-pair AUC about 0.76), but precision at the top cannot be proven to 95% on our data.
|
||||
|
||||
Cairn treats a security problem as a directed search with an objectively checkable goal. A shared board holds
|
||||
confirmed facts, open intents and human hints. A reasoner proposes independent intents; explorers with real tools
|
||||
each confirm one fact; the loop ends when the goal is demonstrably met.
|
||||
|
||||
Three things follow for OpenUltraSAST:
|
||||
|
||||
1. **An objective success condition.** A finding is demonstrated when a test or proof of concept fails on the
|
||||
vulnerable code and passes on the fixed code (or, for a push, fails on head and passes on base where applicable).
|
||||
A demonstrated finding is right by construction; it needs no calibration and no training data, so it generalises
|
||||
to unseen repositories by construction.
|
||||
2. **Exploration with tools.** Agents that read other files, follow callers, build and run the code, instead of a
|
||||
single-shot judgement of one excerpt.
|
||||
3. **Search state per repository and push.** Facts and intents accumulate on a board instead of every question
|
||||
starting from zero.
|
||||
|
||||
## Shape
|
||||
|
||||
- The cheap layers (quick rules, engine, classifier) become triage: they rank candidates.
|
||||
- Agentic search runs only on the top-ranked candidates of a push, under a per-push budget.
|
||||
- BLOCK only on a demonstrated finding. Advisory for plausible, unproven ones. "Could not demonstrate" is recorded as
|
||||
such, never as "not vulnerable".
|
||||
- Execution is primarily in the kube-ax cluster: reason and explore steps are AX Tasks; the VM coordinates.
|
||||
- Model access: a scoped, budget-limited key in the cluster, managed by the operator.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Copying Cairn code (AGPL-3.0; this repository is Apache-2.0). Offensive use against systems we do not own: all
|
||||
execution is against code under analysis, inside gVisor sandboxes.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Design: search with proof (v2, after two Codex reviews; the Claude Doc is the readable version)
|
||||
|
||||
## Why: from scoring excerpts to proving findings
|
||||
BLOCK will mean a vulnerability was demonstrated and re-run by the system, not that a score passed a threshold. Every detection step so far classified one function excerpt, then tried to make the score precise enough to block. Same wall in every experiment: LLM programs (exp-005 A) within-pair AUC 0.762, pooled gate needs >=73 clean flags; trained logistic 0.721 (tie, CI spans zero); gradient boosting 0.597 (worse). Cairn (AGPL, design only) treats a security problem as directed search with a checkable goal. Three things follow: (1) objective goal: test/PoC fails on vulnerable, passes on fixed; needs no calibration or training data, so it carries over to unseen repos; (2) exploration with tools (read other files, follow callers, build, run); (3) search state: facts and intents accumulate on a board per search.
|
||||
|
||||
## Architecture
|
||||
VM coordinates and is the only writer to the board (memory store); every model call and execution runs as an AX Task on kube-ax. Triage (quick rules, engine, classifier) -> top K candidates -> first facts on the board. Reason task (reads snapshot, returns complete or <=3 intents) <-> board; Explore tasks (one per intent; read, grep, run, write demo/; return one fact) <-> board; Verify task (fresh sandbox, planted canary, vulnerable vs fixed / head vs base) -> outcome: BLOCK only if demonstrated, else advisory or silent. Scheduler and budgets on the VM check caps before every dispatch. Tasks read/write via presigned links; store credentials stay on the VM. VM is fallback lane, kind local fallback, same image.
|
||||
|
||||
## The demonstration contract (v2)
|
||||
The verifier, not the agent, observes the effect, on one side and not the other, consistently, in fresh sandboxes. Artefact: demo/ with build.sh and attack.sh (or a request file) touching only the documented external interface; its exit code and output are ignored. Oracles owned by the verifier per family: injection SQL = canary row seen in verifier's query log/response capture; path = canary file outside root read, seen by file-access watch; XSS = script executes in verifier's headless browser; SSRF = request arrives at verifier's in-sandbox listener; injection (command) = canary side effect seen by process watch. Deserialisation, access control, config/secrets: no owned oracle yet -> plausible_unproven at most. Attacker boundary excludes source tree, dependencies, runtime, fixtures; checkout and dependencies read-only during run; untracked files, dependency swaps, runtime patching refused. Differential: fix pair vulnerable-vs-fixed; push head-vs-base. Both sides must build and pass readiness independently; each side 3 runs from fresh state; inconsistent runs rejected; a side that fails to build or start => inconclusive, never a pass. Outcomes: demonstrated (BLOCK once the family is qualified), plausible_unproven (advisory, a visible finding), could_not_build/could_not_run/inconclusive (advisory if facts support, else silent; never "not vulnerable"), no_oracle (as plausible_unproven), no_evidence (silent). REGRESS's nonzero-exit-is-triggerable rule (regress/verdict.py) is not reused for proof; its sandbox plumbing is.
|
||||
|
||||
## Agent worker and sandbox tools
|
||||
Own small tool loop in src/openultrasast/search/ via the existing OpenAI-compatible client (provider swappable, spend metered, Apache-2.0). Image ousast-search-task from the AX task runner, toolchains for Python, Node and TypeScript, PHP, Java; repo fetched over HTTPS. Tools: list_files, read_file (400 lines/call), grep (200 matches), run (per-call timeout, truncated output), write_demo (only write, under demo/), finish (one fact or recorded failure). Engine evidence (taint path, sinks, missing guards) is in the explore prompt. Reason step uses no tools.
|
||||
|
||||
## Triage, budgets and model access (v2)
|
||||
Top K per push (K=3); candidate carries family hint + engine evidence as first facts. Caps (ceilings, not estimates): reason rounds 4/search, intents 3/round, explore 15 min/intent, $0.50/search, $1.50/push, 3 demos x 2 sides x 3 runs. Spend reserved before each model call at max cost and settled after (plane/budget.py gets a reservation step); per-task memory/disk/cleanup limits sized to workers (~3 vCPU, 5-5.75 GiB, two heavy lanes); total run wall-time and retry cap. Worst case 120 task-hours for 40 searches (~60 h on two lanes); a 1,300-change slice at K=3 could reach $1,950. First 3 pilot searches measure real cost; pilot ceiling set from that and asked for. Model key: operator scoped secret (AX injects only GEMINI_API_KEY; operator decides mechanism); never in manifest/log/board/record; repository code cannot read it: model client runs in a separate process as its own user.
|
||||
|
||||
## Pilot, qualification, then the unseen pool (v2)
|
||||
1. Feasibility pilot: 20 dev-corpus pairs (never v3/pool), the four pool families with an owned oracle (injection, path, output encoding, untrusted destination), across the pool's five languages (Python, JS, TS, PHP, Java). Candidate = labelled function (measures proof, not triage); advantages recorded; fixed-side search never sees the vulnerable revision or its demo. Reported: demonstrated rate per family/language, false proofs, could_not_build/run/inconclusive/no_oracle rates, spend/tasks/wall. Exit: >=1 family with >=5 verified demonstrations and zero false proofs, else stop.
|
||||
2. BLOCK qualification per family: 0/20 still allows 16.1% (Wilson 95%); a family BLOCKs only after >=73 negative-side searches (fixed revisions + ordinary changes) with zero demonstrated (upper bound 5%, same bar as the pooled gate). Until then demonstrations show as advisories.
|
||||
3. G1 arm on the unseen pool: triage top K on every replayed change in a fresh slice; scored with the pool's rules on every visible finding incl. advisories, family + location match; 1/5/10% FA budgets beside classifier arms; adopted only under the no-regression rule. No evaluation label, fixing revision or evaluation board is ever an input to unseen search.
|
||||
Every run reports the coverage ceiling (true candidate x owned oracle x both sides runnable x genuine differential, build failures in the denominator) next to catch. Delayed results go via push/background.py and are scored separately from push-time results.
|
||||
|
||||
## Decisions, risks and open questions
|
||||
Largest risk: environment failure caps the demonstrated rate. Risks: builds fail; agent games oracle (mitigated by re-run, canary, differential, tracked-file check); repo code misuses egress (gVisor; ask operator for no-egress verify class); cost (hard caps); minutes per search too slow for synchronous hook (decide product path after pilot). Open: operator key mechanism + no-egress class; own tool loop vs CLI agent; pilot go-ahead $20; how a minutes-long search reaches the pre-push user.
|
||||
@@ -0,0 +1,80 @@
|
||||
# Declarative verification and separate execution domains
|
||||
|
||||
Implemented on `search-with-proof`, 2026-10-05, under the maintainer's explicit instructions.
|
||||
`git merge -q main` failed because the shared Git metadata is read-only. The amended requirements
|
||||
were read from the main checkout, including Requirements 3.1 and 5.2. No commits or network operations.
|
||||
|
||||
## Changes
|
||||
|
||||
- Strict, bounded JSON demonstration schema, also supplied to the model's `write_demo` tool.
|
||||
Verifier-owned recipes cover pip, npm, composer, Maven, Gradle and no build. Runtime entrypoints,
|
||||
CLI/HTTP sequences and earlier-response captures are validated; private fixtures, environment
|
||||
substitutions, canary captures and script artefacts are rejected. Injection/traversal strings remain
|
||||
legitimate request payload data, never executable verifier instructions.
|
||||
- The verifier owns build, readiness and request execution. Local runs retain Bubblewrap (`userns`
|
||||
on this host). HTTP requests require an acknowledged readiness gate; incomplete observations fail
|
||||
closed. Build-generated internal symlinks are supported without accepting external or cyclic links.
|
||||
- A supplied AX side dispatcher selects `task-boundary` and creates three separate fresh tasks per
|
||||
side. Failure of local isolation without a dispatcher fails closed.
|
||||
- The model-holding worker delegates list/read/grep/run to an executor interface. Production uses
|
||||
presigned command/result objects; the in-process adapter is for tests. Commands have sequence
|
||||
numbers, identical-retry caching, deadlines, stop and bounded results. Uncertain outcomes prevent
|
||||
later commands from reusing that executor session.
|
||||
- AX executor tasks use the `ousast-engine-search-exec-` prefix. Task entrypoints reject inherited
|
||||
credentials before reading checkout input, then use an explicit environment allowlist.
|
||||
|
||||
## Changed files
|
||||
|
||||
Production:
|
||||
- `src/openultrasast/search/demo.py`
|
||||
- `src/openultrasast/search/executor.py`
|
||||
- `src/openultrasast/search/worker.py`
|
||||
- `src/openultrasast/search/verify.py`
|
||||
- `src/openultrasast/search/verify_task.py`
|
||||
- `src/openultrasast/search/_sandbox.py`
|
||||
- `src/openultrasast/search/oracles.py`
|
||||
- `src/openultrasast/search/probe.py`
|
||||
- `src/openultrasast/search/probe_apps/app.py.txt`
|
||||
- `benchmarks/ax/batch.py`
|
||||
- `benchmarks/ax/search_verify.py`
|
||||
- `plane/Dockerfile.search-task`
|
||||
|
||||
Tests and audit:
|
||||
- `tests/test_search_demo.py`
|
||||
- `tests/test_search_executor.py`
|
||||
- `tests/test_search_verify_transport.py`
|
||||
- `tests/test_search_verify.py`
|
||||
- `tests/test_search_worker.py`
|
||||
- `tests/test_search_isolation.py`
|
||||
- `tests/test_search_probe.py`
|
||||
- `tests/test_ax_batch.py`
|
||||
- `benchmarks/measurements/2026-09-08-module-audit.json`
|
||||
|
||||
This report is the additional documentation file.
|
||||
|
||||
## Fresh validation
|
||||
|
||||
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py tests/test_module_audit.py -rs`
|
||||
|
||||
Result: **171 passed, 3 skipped, exit 0**, in 35.46 seconds.
|
||||
The three skips are Chromium isolation/network setup, SSRF localhost sockets and HTTP localhost sockets,
|
||||
all prohibited by this host's policy. Host verification mode was explicitly checked: `userns`.
|
||||
|
||||
Ruff passes for search sources, AX changes and affected tests. Targeted strict mypy passes for the seven
|
||||
changed Python search implementation modules checked. Independent review approved the implementation,
|
||||
including follow-up timeout, readiness, capture, credential and generated-link regressions.
|
||||
|
||||
Module audit: **171 modules**, **165 load-bearing**, **6 standalone capabilities**, **0 orphaned**.
|
||||
The audit read the worktree source explicitly with `PYTHONPATH=src`.
|
||||
|
||||
## Remaining validation gaps
|
||||
|
||||
- No live AX/gVisor/kind dispatch, image build/pull/publish, or paid model calls were performed.
|
||||
- Actual ecosystem dependency installations remain unqualified. Python dependency loading was tested
|
||||
with a controlled offline build output and real isolated execution.
|
||||
- The three socket/browser integrations need a host or task where those operations are permitted.
|
||||
- Hard aggregate disk isolation in `task-boundary` mode requires an outer task storage quota. The
|
||||
executor records this requirement and additionally bounds individual files and monitors checkout
|
||||
growth; it cannot enforce a quota across every writable task path without that outer resource limit.
|
||||
- AX tasks configured to inject a model credential will fail closed; executor and verifier task classes
|
||||
must be provisioned without that injection.
|
||||
@@ -0,0 +1,101 @@
|
||||
# Host launcher repair and kube-ax operator constraints
|
||||
|
||||
2026-10-05, branch `search-with-proof`. Implemented under the maintainer's explicit
|
||||
instructions. No commits or network operations. Existing uncommitted work was preserved.
|
||||
This supersedes the build/transport and disk-bound statements in
|
||||
`implementation-trust-domains.md` where they differ.
|
||||
|
||||
## Root cause and repair
|
||||
|
||||
The HTTP driver calls `subprocess.Popen(..., stdin=DEVNULL, stdout=DEVNULL,
|
||||
stderr=DEVNULL)` inside bubblewrap. Its filesystem had no `/dev/null`; Python failed
|
||||
in `_get_handles` before starting the app, binding port 18765, or applying captures.
|
||||
Other probe apps do not use this nested HTTP launcher path.
|
||||
|
||||
A socket-free before/after reproduction read a 19-byte instrument input. Removing
|
||||
`--dev /dev` reproduced exit 1 and the reported `subprocess.py:992 → _get_handles`
|
||||
stack, ending with `FileNotFoundError: [Errno 2] No such file or directory: '/dev/null'`.
|
||||
With the private device mount, the identical launcher exited 0 with empty stderr.
|
||||
The original HTTP capture test's fixed port, token exchange, and differential
|
||||
assertion remain unchanged. Reasons now retain the last 2,000 stderr characters;
|
||||
executor result truncation retains stderr tails too.
|
||||
|
||||
## Operator contract
|
||||
|
||||
- Shared two-slot AX reservations cover workload lane instances and long-lived
|
||||
search executors. Capacity rejection retries the identical manifest with 15–30
|
||||
seconds of jitter until the deadline. Exhaustion is `capacity_timeout`, never
|
||||
`sandbox_failure`, and does not trigger another execution attempt or VM fallback.
|
||||
Uncertain task cleanup retains the reservation and halts dispatch.
|
||||
- Task entrypoints configure temporary files and the specified package caches under
|
||||
`/workspace`. The aggregate guard defaults to 2 GiB, configurable through
|
||||
`OUSAST_SCRATCH_BYTES`. It checks workspace files, caches, and unlinked output logs
|
||||
during child execution and after exit. Exceeding it returns `could_not_build` with
|
||||
`scratch limit`; unreadable scratch is not interpreted as zero usage.
|
||||
- The default AX side dispatcher first builds/packages each side in an executor,
|
||||
retrieves its digest-checked archive, and deletes the executor before verification.
|
||||
One archive contains the checkout, products, and spec. Three fresh verifier tasks
|
||||
consume that archive through `VERIFY_INPUT_URL`; results use `RESULT_URL`.
|
||||
Verifiers reject dependency builds. Git metadata is omitted from source archives;
|
||||
safe internal generated links are materialized before export.
|
||||
- Search tasks receive source archives. Executor tool guidance and project/operator
|
||||
documentation require HTTPS GitHub git clone or codeload, never `api.github.com`.
|
||||
First-download retries tolerate delayed egress activation for up to 60 seconds.
|
||||
|
||||
## Files changed in this follow-up
|
||||
|
||||
Production:
|
||||
- `src/openultrasast/search/_sandbox.py`
|
||||
- `src/openultrasast/search/verify.py`
|
||||
- `src/openultrasast/search/executor.py`
|
||||
- `src/openultrasast/search/verify_task.py`
|
||||
- `src/openultrasast/search/task_storage.py` (new)
|
||||
- `src/openultrasast/search/worker.py`
|
||||
- `benchmarks/ax/batch.py`
|
||||
- `benchmarks/ax/search_verify.py`
|
||||
|
||||
Tests:
|
||||
- `tests/test_search_isolation.py`
|
||||
- `tests/test_search_verify.py`
|
||||
- `tests/test_search_verify_transport.py`
|
||||
- `tests/test_search_task_storage.py` (new)
|
||||
- `tests/test_ax_batch.py`
|
||||
|
||||
Documentation/evidence:
|
||||
- `.kiro/steering/kube-ax.md` (new)
|
||||
- `ops/ax/README.md`
|
||||
- `benchmarks/measurements/2026-09-08-module-audit.json`
|
||||
- This report (new).
|
||||
|
||||
Other files already modified/untracked at session start remain uncommitted.
|
||||
|
||||
## Fresh verification
|
||||
|
||||
- Requested suite, with `-rs` for skip reasons:
|
||||
`/home/mc/Source/OpenUltraSAST/.venv/bin/pytest -q tests/test_search_*.py tests/test_ax_batch.py tests/test_plane_literals.py -rs`
|
||||
— **181 passed, 3 skipped, exit 0**, 24.94 seconds.
|
||||
- Final executor/storage regression and module audit tests after tightening stderr
|
||||
truncation: **34 passed, exit 0**, including all **6 module audit tests**.
|
||||
- Shared dispatcher consumer `tests/test_unseen_calibrate.py`: **10 passed, exit 0**.
|
||||
- Module audit regenerated from the worktree using `PYTHONPATH=src`, with an explicit
|
||||
source read/byte count: **172 modules, 166 load-bearing, 6 standalone capabilities,
|
||||
0 orphaned**.
|
||||
- Ruff passes across search, AX, and changed tests; strict mypy passes for the six
|
||||
changed typed implementation modules. `git diff --check` passes.
|
||||
|
||||
## Gaps
|
||||
|
||||
- Host policy prohibits Chromium network/isolation setup, SSRF localhost sockets,
|
||||
and the full HTTP capture test. These are the three skips. The exact missing-device
|
||||
failure is reproduced and repaired without sockets; full HTTP host confirmation
|
||||
remains necessary.
|
||||
- No live AX tasks, image builds/pulls, registry installs, or external downloads were
|
||||
run. Mocked transport plus local real child processes establish code contracts,
|
||||
not deployed cluster readiness. Verifier egress to
|
||||
`files.because-security.com:443` is requested from the operator and still pending.
|
||||
- Scratch enforcement is a polling guard, not an instantaneous filesystem quota.
|
||||
Programs that hardcode RAM-backed paths can still consume the worker memory limit.
|
||||
Archive bounds remain 64 MiB compressed / 256 MiB file contents / 20,000 entries;
|
||||
larger built projects fail explicitly and require a separately qualified envelope.
|
||||
- Multi-ecosystem dependency build/runtime qualification remains open. The offline
|
||||
preparation tests cover no-build projects and controlled Python build products.
|
||||
@@ -0,0 +1,146 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
Add a search-with-proof stage: for the top-ranked candidates of a push or a labelled pair, sandboxed tool-using agents
|
||||
search for a demonstration of the vulnerability, coordinated through a fact/intent board, executing primarily in the
|
||||
kube-ax cluster. BLOCK becomes "demonstrated", not "scored above a threshold". Context: `brief.md`, roadmap G1-G2,
|
||||
`unseen-repo-evaluation` (the pool and metric this arm is judged by).
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the board (facts, intents, hints) in the memory store; reason and explore steps as AX Tasks; the
|
||||
demonstration contract and its verification; triage-to-search budgeting; a pilot on known fix pairs; a G1 arm on the
|
||||
unseen pool; model access through an operator-managed scoped key.
|
||||
- **Out of scope**: copying Cairn code (AGPL-3.0); network actions against anything but the code under analysis;
|
||||
changing the existing quick rules, engine or classifier (they are inputs here).
|
||||
- **Adjacent expectations**: `unseen-repo-evaluation` scores this arm; the AX batch dispatcher (`benchmarks/ax/batch.py`)
|
||||
and the kube-ax constraints (2 heavy lanes, `ousast-engine-` names, egress to public HTTPS) apply;
|
||||
`learned-decision-engine` Req 6.4 BLOCK rule is superseded for this arm by "demonstrated".
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: The board
|
||||
|
||||
**User Story:** As the maintainer, I want each search to keep its state, so exploration builds on what is already
|
||||
known instead of restarting.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Per search (one candidate or one push), the board holds facts (confirmed, objective, with the evidence that
|
||||
confirmed them), intents (declared directions, open or concluded, with the facts they start from) and hints (human
|
||||
input, absorbed on the next read).
|
||||
2. The board lives in the memory store, versioned, and every fact and intent carries the step and the task that wrote
|
||||
it.
|
||||
3. A search ends when a fact meets the goal (Requirement 3), when its budget is spent, or when the reasoner proposes no
|
||||
new intent and none is open; the end reason is recorded.
|
||||
|
||||
### Requirement 2: Reason and explore steps
|
||||
|
||||
**User Story:** As the maintainer, I want the search driven by the board's state, not by fixed roles.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A reason step reads the board and returns: the goal is met (citing facts), or at most N new, independent,
|
||||
non-overlapping intents, or no new intent.
|
||||
2. An explore step claims one intent, works it with tools inside the sandbox (read files, search the repository,
|
||||
follow callers, build, run tests and scripts against the code under analysis), and returns exactly one new fact or
|
||||
a recorded failure.
|
||||
3. Both run as AX Tasks in the kube-ax cluster by default; the VM coordinates (board, dispatch, budgets) and is a
|
||||
fallback lane only.
|
||||
4. Parallel explores of one search never race on the board (claims as in the batch dispatcher).
|
||||
|
||||
### Requirement 3: The demonstration contract
|
||||
|
||||
**User Story:** As the maintainer, I want BLOCK to mean "shown", so precision does not depend on calibration.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. *(Amended 2026-10-05, maintainer decision after the kube-ax probe: AX tasks have no capabilities, so gVisor and one
|
||||
fresh task per side are the only isolation.)* A demonstration is **declarative data, never agent-written code**: a
|
||||
build recipe chosen from the verifier's fixed per-ecosystem recipes, a start command of the form runtime + path +
|
||||
arguments (no shell), and a request sequence (HTTP requests or CLI invocations, with values captured from earlier
|
||||
responses) that the verifier itself sends. The system, not the agent, runs it in fresh sandboxes. The **verifier observes the effect itself** through a per-family oracle it owns (for
|
||||
example a request log, a planted resource it watches, a browser execution check for XSS); the artefact's own exit
|
||||
code or output is never the oracle.
|
||||
2. Each family's oracle states the attacker boundary (what the artefact may touch: only the documented external
|
||||
inputs of the code under analysis, never the verifier's fixtures, the source tree, dependencies or the runtime).
|
||||
Inputs are immutable during a run; untracked files, dependency replacement and runtime patching are refused.
|
||||
3. Differential: for a labelled pair the effect is observed on the vulnerable revision and not on the fixed one; for a
|
||||
push, on head and not on base. Both sides must build and pass a readiness check independently; a side that fails
|
||||
to build or start makes the result **inconclusive**, never a pass. Each side is run at least three times from fresh
|
||||
state; inconsistent runs are rejected.
|
||||
4. Only a verified demonstration counts as `demonstrated`. Outcomes: `demonstrated`, `plausible_unproven`,
|
||||
`could_not_build`, `could_not_run`, `inconclusive`, `no_oracle` (family without an owned oracle), `no_evidence`;
|
||||
none of the failure outcomes is ever reported as "not vulnerable".
|
||||
5. Families without an owned oracle (for now: deserialisation, access control, configuration and secrets) can reach
|
||||
`plausible_unproven` at most until an oracle is specified and reviewed.
|
||||
|
||||
### Requirement 4: Triage and budget
|
||||
|
||||
**User Story:** As the maintainer, I want search spent where it matters, at a known cost.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Candidates come from the existing layers (quick rules, engine, classifier scores); only the top K per push or per
|
||||
pair side are searched; K and the per-search and per-push budgets (model spend, task count, wall time) are
|
||||
configuration, enforced by the coordinator.
|
||||
2. Every search records its model spend, task count and wall time. Spend is **reserved before each model call** (the
|
||||
call's maximum cost) and released after; wall time, memory, disk and retries are capped per task and in total; a
|
||||
budget is never exceeded, including by calls in flight.
|
||||
3. Paid runs need the maintainer's budget go-ahead with a ceiling, as all paid steps do.
|
||||
|
||||
### Requirement 5: Model access in the cluster
|
||||
|
||||
**User Story:** As the maintainer, I want agents in the cluster to reach a model without spreading keys.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The agents' model key is a scoped, budget-limited key provided by the operator inside the cluster (AX v0.3.1
|
||||
injects only `gemini-api-secret` as `GEMINI_API_KEY`; the design names the exact mechanism agreed with the
|
||||
operator). Store credentials stay out of the cluster (presigned links, as today). For this arm, model calls inside
|
||||
the cluster supersede `unseen-repo-evaluation` Req 4.4 (model calls on the VM); that boundary is unchanged for all
|
||||
other arms. *(maintainer decision 2026-10-05: scoped key in the cluster)*
|
||||
2. The key never appears in a manifest, a log, a board entry or a record, and never shares a sandbox with repository
|
||||
code. *(Amended 2026-10-05, maintainer decision: trust domains are separated instead of isolated inside one task.)*
|
||||
The agent loop (the "brain", holding the key) runs on the VM or in its own AX task and executes no repository code;
|
||||
every tool call that touches the repository runs in a separate executor AX task without the key, exchanging
|
||||
commands and results through presigned store objects. A Firecracker executor runtime is considered only if the
|
||||
pilot's `could_not_build` rate shows gVisor cannot build the projects.
|
||||
|
||||
### Requirement 6: Feasibility pilot, false-proof qualification, then the unseen pool
|
||||
|
||||
**User Story:** As the maintainer, I want to know quickly whether search with proof works, and to admit BLOCK only on
|
||||
evidence that meets the 95% precision bar.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. **Feasibility pilot** (amended 2026-10-05 after review): about 20 known fix pairs from development corpora. It
|
||||
measures feasibility only and is reported as such: demonstrated rate on vulnerable sides, false proofs on fixed
|
||||
sides, `could_not_build` / `could_not_run` / `inconclusive` / `no_oracle` rates, spend, tasks and wall time. Its
|
||||
advantages (known location, labelled family, runnable-project bias) are stated in the record. Fixed-side searches
|
||||
never receive the vulnerable revision or its demo.
|
||||
2. Before any paid search, an infrastructure probe without model calls proves each owned oracle works inside a gVisor
|
||||
AX Task (database fixture for SQL, in-sandbox listener for SSRF, headless browser for XSS, file and process
|
||||
watches), that the artefact cannot reach the observers directly, and records image size, RAM and the full
|
||||
verification cost (up to 3 demos x 2 sides x 3 runs per search). Pilot exit: at least one family with at least 5 verified demonstrations on vulnerable sides and zero false proofs;
|
||||
otherwise the result is recorded and the arm is not scaled.
|
||||
3. **BLOCK admission per family** (deferred until the pilot passes; not part of the first tasks) requires a false-proof
|
||||
qualification. A **negative** is a search on code that should be safe: a fixed revision searched on its own (the
|
||||
search never sees the vulnerable revision), or an ordinary change. A **false proof** is the verifier observing the
|
||||
effect on that safe code, confirmed by adjudication (an ordinary-change alarm may be a real new bug and is
|
||||
adjudicated before it counts either way). Qualification needs zero false proofs over at least 73 negatives drawn
|
||||
from at least 73 distinct repositories, so the Wilson 95% upper bound on the false-proof rate per negative is at
|
||||
most 5%. This bounds the false-proof rate, not precision among flags; precision is reported separately from the
|
||||
pool. Repeated runs and repeated searches of one repository are one negative.
|
||||
4. **Coverage ceilings**: every run reports, separately, (a) the demonstrated ceiling: the share of vulnerable changes
|
||||
with a true candidate selected, an owned oracle, both sides runnable and a genuine differential, build and run
|
||||
failures in the denominator; and (b) the advisory ceiling: the share with a true candidate selected. Families
|
||||
without an owned oracle (access control, deserialisation, configuration and secrets) have a demonstrated and BLOCK
|
||||
ceiling of zero today; their advisory ceiling need not be zero.
|
||||
5. **G1 arm on the unseen pool**: scored with the pool's rules on **every visible finding** (BLOCK and advisory), with
|
||||
its family and location match; catch at the 1%, 5% and 10% false-alarm budgets; compared with the classifier arms;
|
||||
adopted only under the pool's no-regression rule on a fresh slice. No evaluation label, fixing revision or board
|
||||
from evaluation searches is ever an input to unseen search.
|
||||
6. Delayed results (a search finishing after the push) reach the user through the existing background delivery and
|
||||
are scored separately from results available at push time.
|
||||
@@ -0,0 +1,29 @@
|
||||
{
|
||||
"feature_name": "search-with-proof",
|
||||
"created_at": "2026-10-05T12:00:00Z",
|
||||
"updated_at": "2026-10-07T13:30:00Z",
|
||||
"language": "en",
|
||||
"phase": "tasks-generated",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false,
|
||||
"pilot": {
|
||||
"status": "in_progress",
|
||||
"first_cluster_demonstration": "2026-10-07",
|
||||
"pair": "xiaomusic (path, prefix-bypass)",
|
||||
"false_proofs": 0,
|
||||
"exit_remaining": "path family >=5 demonstrations / 0 false proofs (Req 6.2), then >=73-negative BLOCK qualification (Req 6.3)"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,130 @@
|
||||
# Implementation Plan
|
||||
|
||||
Scope: infrastructure probe and feasibility pilot only (Req 6.1-6.2). BLOCK qualification (6.3), the unseen-pool arm
|
||||
(6.5) and delivery (6.6) are deferred until the pilot passes. Implementation via Codex in a worktree; Claude reviews,
|
||||
gates, runs every networked or paid step.
|
||||
|
||||
- [ ] 1. Operator prerequisites (Claude, no code)
|
||||
- [ ] 1.1 Ask the operator: scoped spend-limited model key mechanism (reuse `gemini-api-secret` slot or a new
|
||||
injection), and whether a no-egress task class exists for verify runs. _Requirements: 5.1, 3.2_
|
||||
- [ ] 1.2 Record the answers in `kube-ax-cluster` memory and design.md; no key value anywhere. _Requirements: 5.2_
|
||||
|
||||
- [x] 2. Verifier and oracles (no model calls)
|
||||
- [x] 2.1 `src/openultrasast/search/verify.py`: fresh-sandbox runner per side; build, readiness check, read-only
|
||||
checkout and dependencies, untracked-file and runtime-patch refusal, 3 runs per side, consistency check,
|
||||
outcomes `demonstrated|inconclusive|could_not_build|could_not_run|no_oracle`. Artefact exit code ignored.
|
||||
_Requirements: 3.1-3.4_
|
||||
- [x] 2.2 Oracles owned by the verifier: SQL (database fixture + query log), path (file-access watch on a planted
|
||||
canary), XSS (headless browser execution check), SSRF (in-sandbox listener), command injection (process watch).
|
||||
Observers unreachable from the artefact. _Requirements: 3.1, 3.2, 3.5_
|
||||
- [x] 2.3 Unit tests with hand-written vulnerable/fixed toy apps per oracle, including gaming attempts (reading the
|
||||
canary, revision sniffing, patching a dependency, contacting the listener directly): each must not yield
|
||||
`demonstrated`. _Requirements: 3.2, 3.3_
|
||||
|
||||
- [ ] 3. Image and infrastructure probe (no model calls)
|
||||
- [ ] 3.1 `ousast-search-task` image from the AX task runner: Python, Node/TypeScript, PHP, Java toolchains, the
|
||||
oracle fixtures, headless browser; built by CI to GHCR, pinned by digest. _Requirements: 2.3_
|
||||
- [ ] 3.2 Probe: the toy apps from 2.3 verified as AX Tasks on kube-ax (gVisor) and on kind. Record image size,
|
||||
RAM peak, disk, wall time per verification and per 18-run search, oracle success per family, and that observers
|
||||
are unreachable. Record `benchmarks/measurements/2026-10-xx-search-probe/record.json`. _Requirements: 6.2_
|
||||
- [ ] 3.3 Gate: every oracle works under gVisor, or the family is marked `no_oracle` for the pilot.
|
||||
|
||||
- [x] 4. Board, coordinator, budgets
|
||||
- [x] 4.1 Board in the memory store as immutable versioned blobs (`plane/memory.py`); facts, intents, hints with
|
||||
writer step and task; coordinator is the only writer; tasks read and write through presigned links.
|
||||
_Requirements: 1.1-1.3_
|
||||
- [x] 4.2 Spend reservation before each model call in `plane/budget.py`, settled after; per-task memory, disk,
|
||||
cleanup caps; total wall-time and retry caps; tests that in-flight calls cannot overrun. _Requirements: 4.1, 4.2_
|
||||
- [x] 4.3 Dispatch reason, explore and verify as AX Tasks through `benchmarks/ax/batch.py` (extend its environment
|
||||
contract beyond numeric deadlines), VM lane as fallback, resume from checkpoints. _Requirements: 2.3, 2.4_
|
||||
|
||||
- [ ] 5. Agent worker
|
||||
- [x] 5.1 Reason step (no tools): board snapshot in, `complete` / up to 3 intents / nothing out. _Requirements: 2.1_
|
||||
- [x] 5.2 Explore step: tool loop (`list_files`, `read_file`, `grep`, `run`, `write_demo`, `finish`) via the
|
||||
existing OpenAI-compatible client; model client in a separate process and user from executed code; one fact
|
||||
or a recorded failure. _Requirements: 2.2, 5.2_
|
||||
- [ ] 5.3 Smoke test with a stub model (no spend) end to end on one toy pair on kube-ax.
|
||||
|
||||
- [ ] 6. Feasibility pilot (paid, after go-ahead)
|
||||
- [x] 6.1 Select 20 development-corpus pairs (never v3 or the unseen pool) in the four pool families with an owned
|
||||
oracle, across the five languages; record selection advantages. _Requirements: 6.1_
|
||||
- [ ] 6.2 Run 3 searches, measure real cost per search, ask the maintainer for the pilot ceiling. _Requirements: 4.3_
|
||||
- [ ] 6.3 Run the pilot: both sides per pair, fixed side blind to the vulnerable revision; record demonstrated
|
||||
rate, false proofs, failure-outcome rates, both coverage ceilings, spend, tasks, wall time.
|
||||
_Requirements: 6.1, 6.4_
|
||||
- [ ] 6.4 Exit decision against 6.2 and report to the maintainer; on pass, spec tasks for qualification and the
|
||||
unseen-pool arm. _Requirements: 6.2_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- Group 2 (2026-10-05): independently reviewed; `tests/test_search_verify.py` has 32 passing tests and one
|
||||
skipped SSRF integration test because this host forbids even localhost sockets. The namespace handoff and
|
||||
listener request parsing are tested separately; the full SSRF effect still needs the task 3.2 probe before
|
||||
claiming that oracle operational. The module audit and neighbouring sandbox/regression tests pass.
|
||||
- The local lane requires Linux Bubblewrap and util-linux, reuses `SandboxJob`/`SandboxResult`, and fails closed
|
||||
without isolation. Artefacts receive a batch CLI `TARGET` bridge and private scratch; HTTP apps need a trusted
|
||||
CLI adapter. Only trusted `Side.build_command` outputs can enter the app runtime (read-only `/build`);
|
||||
artefact `build.sh` writes scratch only. XSS requires a trusted browser executor, otherwise `no_oracle`.
|
||||
|
||||
- Tasks 3.1 (files only) and group 4 (2026-10-05): independent review approved; final offline validation:
|
||||
107 passed, 2 skipped (Chromium namespace/socket restrictions and the existing SSRF localhost restriction).
|
||||
Ruff and changed-source mypy pass; module audit: 160 load-bearing, 6 standalone, zero orphaned. No network
|
||||
actions, model calls, image builds, pushes or commits were performed. Task 3.1 remains unchecked until the
|
||||
image is built/published and a real digest is available; CI now includes `plane/Dockerfile.search-task`.
|
||||
- The coordinator accepts an injected executor. `BatchExecutor` uses the repository's `benchmarks.ax.batch`
|
||||
transport with one attempt per admission; coordinator owns the global retry budget and VM fallback. It requires
|
||||
a repository checkout import path and a trusted worker command; reason/explore workers remain group 5.
|
||||
Per-task memory, disk and cleanup ceilings are passed in the worker contract; worker enforcement and gVisor
|
||||
compatibility still require group 5 and the 3.2 probe. Only trusted verification results can meet the goal.
|
||||
- Board heads are immutable memory-store hashes; retain the returned head with the host run to resume. Unknown
|
||||
interrupted model calls conservatively consume their full reservation; settled failures with zero spend release
|
||||
it. Cleanup exceptions stop dispatch. Chromium uses a fresh profile in a private network namespace; injected
|
||||
script must call `alert("ousast-xss")`, while escaped/reflected payloads cannot produce the verifier DOM marker.
|
||||
|
||||
- Tasks 3.2 and 5.1–5.3 offline implementation (2026-10-05, explicitly authorized): packaged five-family
|
||||
toy pairs and real/two-gaming demos in `search/probe_apps/`; `python -m openultrasast.search.probe` reports
|
||||
per-side stage times, image versions, child/self peak RSS, retained scratch usage and sampled verification
|
||||
scratch peak. Generic isolation and later oracle namespace failures are instrument failures, never misses.
|
||||
Optional RESULT_URL uses an object-scoped urllib PUT. `benchmarks.ax.search_probe` reuses AX/Docker lanes
|
||||
(kind via its kubeconfig), with `ousast-engine-search-probe-` task names.
|
||||
- Worker reason/explore uses the existing OpenAI-compatible client with retries disabled and SpendBudget
|
||||
reservation before each call. Repository commands run only in the scrubbed sandbox with bounded tmpfs;
|
||||
the key-absence, timeout, output limit and disk limit tests execute real isolated processes. The scripted
|
||||
model/fake coordinator executor smoke reaches `demonstrated` through the real path oracle verifier.
|
||||
- Independent review APPROVED the requested offline scope. Fresh requested search/AX/plane tests plus module
|
||||
audit: 87 passed, 2 skipped, exit 0; focused new tests: 15 passed; Ruff and changed-source mypy pass.
|
||||
Audit: 162 load-bearing, 6 standalone, zero orphaned. Tasks 3.2 and 5.3 remain unchecked for their original
|
||||
live-cluster criteria; their code/local smoke is implemented. No network, paid calls, image builds, pushes
|
||||
or commits. Offline wheel build could not resolve the uncached uv-build backend, so installed-wheel and
|
||||
real AX/gVisor/kind/Docker image validation remain gaps.
|
||||
|
||||
- Task 6.1 and executor smoke (2026-10-05, explicitly authorized): added
|
||||
`benchmarks/search/executor_smoke.py` and `pilot_select.py`, with offline tests and usage documentation.
|
||||
The smoke drives one managed, key-free executor mailbox and records per-command timing/output measurements
|
||||
and sampled workspace peak; AX and Docker lifecycle cleanup is covered by fake-lane tests. No live task ran.
|
||||
Pilot selection read 10 development catalogs (873 rows, 904,050 bytes), retained 192 eligible pairs and selected
|
||||
20 distinct repositories: four per language and five per family. Identities live only in the gitignored local
|
||||
manifest; the aggregate record is `benchmarks/measurements/2026-10-05-search-pilot-selection/record.json`.
|
||||
All 20 selections have unknown manifest/test-suite evidence; no matching local cache or frozen unseen-pool
|
||||
metadata was present. These are feasibility inputs, not proof of buildability or qualification.
|
||||
Both task-local independent reviews approved after a reserved-cache-symlink regression was fixed.
|
||||
Fresh requested search/AX/plane tests, module audit, independent/reserved and unseen guards, and label guards:
|
||||
287 passed, four skipped, exit 0. Skips: Chromium isolation, two localhost-socket cases, and no frozen pool.
|
||||
Ruff and changed-source mypy pass; audit: 166 load-bearing, six standalone, zero orphaned.
|
||||
No network or commits. The requested main merge could not update read-only Git metadata; HEAD and main
|
||||
had identical file trees. Live AX/Docker executor smoke and oracle probes remain unmeasured.
|
||||
|
||||
- **FIRST CLUSTER DEMONSTRATION (2026-10-07) — pilot is live (6.2/6.3 in progress).** search-with-proof produced its
|
||||
first autonomous `demonstrated` finding end to end on kube-ax: pair 13 (hanxi/xiaomusic, path, prefix-check-bypass
|
||||
class), vulnerable side `demonstrated` (3/3 affected, 0/3 safe, $0.37), fixed side blind negative `inconclusive`
|
||||
with 0 false proofs ($0.56); pair total $1.22, image e2c8b460. Record:
|
||||
`benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`. This closes the handoff's open question.
|
||||
The infra probe (3.2) passed earlier; the pilot (6.x) has run many times. Enabling fix chain, each merged + paid-run
|
||||
validated: (1) parity (name the observable to the agent), (2) verify-time-only (observable injected at verify time,
|
||||
absent in exploration — ends the explore reachability trap), (3) fair-share pilot budget (no starvation across
|
||||
searches), (4) served-root guidance (demo MUST serve `${served_root}`), (5) oracle prefix-bypass coverage (sibling
|
||||
canary for `startswith`-without-separator path traversal), (6) control-signal feedback (served-root hint so the
|
||||
agent self-corrects wiring). Plus the keystone (positive control → `could_not_run`, never a silent clean-looking
|
||||
`inconclusive`). NEXT: fan out the path family to ≥5 demonstrations / 0 false proofs (6.2 exit), then the
|
||||
≥73-negative BLOCK qualification (6.3). glance (classic traversal) also demonstrates once its npm `filed`→`mime@4`
|
||||
build-break is pinned.
|
||||
@@ -0,0 +1,52 @@
|
||||
# Search-with-proof: where each stage runs (verification architecture)
|
||||
|
||||
Recorded 2026-10-07. search-with-proof has two stages across a trust boundary: a Cairn-inspired
|
||||
directed state-space **search**, then a DAST-style **proof (verification)**.
|
||||
|
||||
## The two stages
|
||||
|
||||
1. **Search (Cairn-inspired).** A blackboard of facts, intents and hints. A reasoner proposes
|
||||
independent, non-overlapping intents; tool-using explorers each confirm one fact against the
|
||||
code. The goal is a declarative demo (a candidate PoC), never proof by itself.
|
||||
2. **Proof (DAST verification).** The verifier builds and starts the app, drives the demo's steps
|
||||
through the app's real interface, and watches a verifier-owned canary for a **differential**
|
||||
between the vulnerable and fixed revisions: the owned effect appears on the vulnerable side and
|
||||
not on the fixed side (3/3 vs 0/3). Only this differential earns `demonstrated`.
|
||||
|
||||
## Trust boundary — what runs where
|
||||
|
||||
The brain runs on the VM; the agent's tool executions and the DAST proof run as isolated tasks in
|
||||
the kube-ax cluster. Secrets never enter the cluster.
|
||||
|
||||
| Component | Location | Notes |
|
||||
|-----------|----------|-------|
|
||||
| Coordinator, board, budgets | VM (host) | Orchestrates the search; owns the global budget. |
|
||||
| Reasoner + explorer **model calls** | VM (host) | The model key lives only on the VM. This is the brain. |
|
||||
| Explorer **tool runs** (`list_files`, `read_file`, `grep`, `run`, build) | kube-ax AX Task (gVisor) | Carry no secrets; inputs/results via presigned object links. |
|
||||
| **DAST proof / verification** | kube-ax AX Task (gVisor) | Builds, starts, attacks and observes the app. One task per revision. |
|
||||
|
||||
- **gVisor is the isolation boundary.** The proof runs untrusted exploit code against a running
|
||||
app, so it is sandboxed per-task in the cluster.
|
||||
- **Secrets stay on the VM.** Executor and verifier tasks carry no credentials; they exchange
|
||||
checkout archives, demos and results through presigned object-store links. Per-host egress is
|
||||
restricted (verify tasks reach the files host only).
|
||||
|
||||
## Verification lanes (the `--verify-lane` choice)
|
||||
|
||||
- **`ax` (default for paid pilot runs):** verification runs **inside kube-ax** as gVisor AX Tasks,
|
||||
one per side. This is how every paid pilot run this session executed the proof.
|
||||
- **`vm` (local):** verification runs in **Bubblewrap on the host**, no cluster. Used for the
|
||||
zero-cost ceiling proofs (a hand-authored demo verified locally with no model and no cluster),
|
||||
which is how glance and xiaomusic were confirmed demonstrable before any spend.
|
||||
- **`docker` (replay):** `benchmarks.search.demo_replay` runs a stored demo with no model calls in
|
||||
a local Docker lane.
|
||||
|
||||
## Why the proof is dynamic, not abstract interpretation
|
||||
|
||||
Abstract interpretation (the static engine and the decision model) is the cheap, generalizable
|
||||
**triage/ranker**: it selects and orders candidate regions across any repo without executing. It
|
||||
plateaued as a **blocking** gate because it could not prove the required precision. search-with-proof
|
||||
is the **dynamic** confirmation: a demonstrated finding is ground truth by execution, so it earns a
|
||||
block without calibration. The per-app harness cost (building and running the app, wiring its served
|
||||
directory or database to the oracle fixture) is the price of the dynamic proof and is why proof pays
|
||||
off where a runnable harness exists, with abstract interpretation remaining the static backbone.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Brief: does the safety net work on repositories it has never seen?
|
||||
|
||||
Maintainer, 2026-10-04: "we want to spread the safety net across repos in general, unseen code. that was the purpose
|
||||
of the AB tests to see if our approach is generalizeable." And: "we should run our tests / runs in kube. now we can
|
||||
speed it up and design the workloads for ax directly. kind is only a fallback."
|
||||
|
||||
## Why this spec exists
|
||||
|
||||
The A/B experiments so far (exp-003, exp-004, exp-005) score the decision engine on repository-held-out folds of known
|
||||
fix pairs. That measures whether the vulnerable side of a pair scores above its fixed side. It does not measure what a
|
||||
safety net on an arbitrary repository lives on:
|
||||
|
||||
- on an ordinary push, does it stay quiet? (false-alarm rate)
|
||||
- on a push that introduces a real vulnerability, does it say so? (catch rate)
|
||||
|
||||
Populations v1 and v2 measured the engine scan on fresh cases once (1 of 11, 0 of 17) and are spent. Nothing since has
|
||||
measured push-level behaviour on unseen repositories.
|
||||
|
||||
## What this spec delivers
|
||||
|
||||
1. A frozen pool of repositories never used anywhere in development, with two kinds of change each: real
|
||||
vulnerability-introducing commits and ordinary commits.
|
||||
2. A push-level metric: each change replayed through the hook exactly as a user would run it, and through other
|
||||
arms (decision engine, engine evidence) where an experiment asks.
|
||||
3. Execution designed for the kube-ax cluster from the start: one push replay is one AX Task; kind and the VM are
|
||||
fallbacks.
|
||||
4. A baseline for today's hook, then a table of arms scored on the same pool.
|
||||
|
||||
## Not in scope
|
||||
|
||||
Changing detection. This spec only builds the measurement. Improvements are separate, registered experiments judged
|
||||
by it.
|
||||
@@ -0,0 +1,518 @@
|
||||
# Design Document
|
||||
|
||||
## Overview
|
||||
|
||||
This spec builds the instrument for roadmap G0: a frozen pool of repositories that no development step has seen,
|
||||
and a push-level score on it. The score is what a user of the hook sees. On a vulnerability-introducing push, did
|
||||
the hook say so (catch rate)? On an ordinary push, did it say anything (false-alarm rate)? Replays run as AX Tasks
|
||||
on kube-ax. kind and the VM run the same tasks as fallbacks.
|
||||
|
||||
Four facts from the code shape the design:
|
||||
|
||||
- **Populations already have a freeze-and-guard pattern.** `benchmarks/independent/freeze.py` verifies cases by
|
||||
reading git only, `evaluate.py` scores with fixed rules and Wilson intervals, and
|
||||
`tests/test_independent_population.py` fails when a reserved repository is named anywhere in `benchmarks/`,
|
||||
`src/` or `tests/` outside its directory. Harvest batch 3 used that guard as a black box: candidates written,
|
||||
guard run with output discarded, subsets bisected on red. That is how v3 stays unread.
|
||||
- **The hook can already replay a change.** `ousast pre-push PATH --base B --head H --artifact F` resolves two
|
||||
local commits (`snapshot.py`, `resolve_replay`: no fetch, no merge-base), runs quick rules and the engine inside
|
||||
one deadline (default 30 s, engine inline) and writes a JSON artifact. The terminal text has one `Skipped:` line
|
||||
per skipped check. The artifact holds three finding streams: admitted defects, advisory engine findings
|
||||
(new or worsened, vetoed only by capability) and quick-rule matches with their CWE.
|
||||
- **The AX pattern works but lives in one script.** `engine_trace_ax.py` (manifest, egress checks, presigned
|
||||
objects, retry once under a new name, cleanup) and the lane scheduler in `engine_trace.py` (`run_plan`: fixed
|
||||
lanes, fallback queue, `progress.json`, `STOP`, resume by input digest) are engine-specific.
|
||||
- **Placement on kube-ax is random.** One worker per node, shared by all actors. Two heavy tasks fit per worker,
|
||||
but only two concurrent tasks in all guarantee at most two on one worker.
|
||||
|
||||
### Goals
|
||||
|
||||
- A pool frozen by code against every used source (Req 1), with two kinds of change per repository (Req 2).
|
||||
- A scorer whose numbers mean what the hook shows, with silent zeros visible (Req 3).
|
||||
- One replay is one AX Task via a dispatcher shared with the engine (Req 4); a baseline and fresh slices (Req 5).
|
||||
|
||||
### Non-Goals
|
||||
|
||||
Detection changes, population v3, model calls in the cluster, languages the hook does not analyse.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
M[pool manifest slice] --> D[batch dispatcher on VM]
|
||||
D --> P[presign spec GET and record PUT]
|
||||
P --> T[AX Task ousast-engine-replay]
|
||||
T --> G[shallow fetch base and head from GitHub]
|
||||
G --> H[ousast pre-push --base --head]
|
||||
H --> R[record tar: artifact, text, instrument]
|
||||
R --> S[(store via presigned PUT)]
|
||||
D -->|sandbox died twice or OOM| V[VM docker lane]
|
||||
D -->|cluster down| K[kind lane, same tasks]
|
||||
V --> S
|
||||
K --> S
|
||||
S --> C[scorer: catch, false alarm, coverage]
|
||||
S --> A[model arms on VM, budgeted]
|
||||
A --> C
|
||||
C --> O[record: counts and digests]
|
||||
```
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
### 1. Pool sourcing and eligibility (Req 1)
|
||||
|
||||
**Decision 1.1: sources and ecosystems.** Candidates come from GitHub reviewed advisories (`gh api /advisories`,
|
||||
the enumeration harvest-3 used) for pip, npm, maven and composer, published 2019 or later, whose CWE maps to one of
|
||||
the seven families through the project's family CWE table. Each needs a linked fix commit with one parent in one
|
||||
repository. OSV records for the same advisory give the first affected version. That version is a cross-check for
|
||||
the introducing commit (section 2). Reason: these four ecosystems are the five languages the hook analyses
|
||||
(Python, JavaScript, TypeScript, PHP, Java). GHSA links commits and CWEs; OSV adds version brackets. Rejected:
|
||||
drawing from every ecosystem (Go, Ruby, C#). That would mostly measure missing frontends, which is already known.
|
||||
Coverage still reports uncovered files inside the drawn repositories. Rejected: NVD and published fix datasets.
|
||||
NVD links commits weakly, and the datasets are mined corpora that development or model training has likely seen.
|
||||
|
||||
**Decision 1.2: eligibility by code against every used source.** `benchmarks/unseen/eligibility.py` builds the
|
||||
used set before the draw. It never prints names; it records counts per source and a digest of the set.
|
||||
|
||||
| Source (Req 1.1) | Where it is read |
|
||||
| --- | --- |
|
||||
| Pair corpora | every catalog, recipe and dataset file under `benchmarks/pairs/`, plus the local pointer-pair manifests |
|
||||
| Populations v1, v2 | `population-v1.toml`, `population-v2.toml`, read directly |
|
||||
| Population v3 | never opened; the existing guard test only (below) |
|
||||
| Harvest records | the batch catalogs under `benchmarks/pairs/advisory-fixes*` (the records hold counts only) |
|
||||
| Memory examples | `example` rows of the local FileStore and the S3 store (`repo` field) |
|
||||
| Experiment units | `plane/experiments/*.units.jsonl` and registered units files |
|
||||
| Measurement records | everything under `benchmarks/measurements/` and `benchmarks/experiments/` |
|
||||
| Anything else | a text sweep for `github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/`, `.kiro/` |
|
||||
|
||||
Names are normalised (lower case, no `.git`, no trailing slash). Each candidate is resolved through the GitHub API
|
||||
to its canonical `full_name`, its former names (redirects) and its fork network (`parent`, `source`). A candidate
|
||||
is out if any of these is in the used set. A fork shares code with what it was forked from.
|
||||
|
||||
v3 is checked like harvest-3 did. The draft manifest is written into `benchmarks/unseen/`. Then
|
||||
`pytest -q tests/test_independent_population.py -k referenced` runs with its output discarded. The guard already
|
||||
sweeps `benchmarks/` outside the population directory. On red, the candidate list is bisected by rewriting the
|
||||
draft until green, and the dropped count is recorded. No v3 name is ever printed or read by a person.
|
||||
|
||||
**Decision 1.3: post-cutoff preference.** An advisory published after 2026-06-01 (harvest-3's cutoff for the
|
||||
decision engine's model) is flagged `post_cutoff`. Within each ecosystem stratum post-cutoff advisories are drawn
|
||||
first. Results are reported for both groups. Reason: a post-cutoff advisory cannot be in the model's training
|
||||
data as a known vulnerability. Rejected: post-cutoff only. Harvest-3 kept 12 post-cutoff pairs out of 251, and
|
||||
three slices of 100 repositories cannot be filled from that window.
|
||||
|
||||
**Decision 1.4: licences.** The licence is read at the pinned head commit (LICENSE file, then the API's SPDX id).
|
||||
Only permissive licences are published by name: MIT, Apache-2.0, BSD-2-Clause, BSD-3-Clause, ISC, 0BSD, Unlicense,
|
||||
Zlib, PostgreSQL, BSL-1.0. Everything else (copyleft, source-available, none) goes to the gitignored
|
||||
`benchmarks/unseen/private/pool-p1.toml`. The committed manifest then holds an opaque id and the sha256 of
|
||||
(URL, commits) for that entry, so the freeze digest still covers it. Reason: Req 1.4 read literally. Rejected:
|
||||
"OSI-approved" as the bar, as population v2 used. It is wider than the requirement says (open question 1,
|
||||
resolved 2026-10-04: copyleft stays private).
|
||||
|
||||
Also required: anonymous HTTPS, at most 500 MB, at least 40 eligible ordinary commits, one per fork network.
|
||||
|
||||
### 2. Change extraction (Req 2.1, 2.2)
|
||||
|
||||
Extraction runs on the VM over a full `--filter=blob:none` clone in a scratch cache. It uses git only, like
|
||||
`freeze.py`. No scanner, engine or model touches a pool repository before the freeze.
|
||||
|
||||
**Decision 2.1: the vulnerability-introducing range.** Let F be the fix and P its parent. The vulnerable lines are
|
||||
the old-side lines of F's hunks in the vulnerable function. For a pure-addition fix (an absence bug such as a
|
||||
missing guard), they are the vulnerable function's own lines.
|
||||
|
||||
1. **`introducing` (preferred).** `git blame -w -M -C --porcelain P -L <lines> -- <file>`. The candidate V is the
|
||||
commit that last wrote the sink line (for absence bugs, the function's declaration). V is accepted if all hold:
|
||||
at V's first parent the sink line's normalised text is absent from the file's vulnerable function; V is not a
|
||||
merge or root commit; V changes at most 50 files and 2,000 lines; when OSV names a first affected version, V is
|
||||
reachable from that tag and not from the previous release tag. If the text is present at V's parent, blame
|
||||
walks back from there, at most five steps. The replay is `base = V^`, `head = V`.
|
||||
2. **`last_touch` (fallback).** When no V passes, `git log -L :<function>:<file> P` gives the newest commit L
|
||||
before F that changed the vulnerable function. The replay is `base = L^`, `head = P`. The vulnerable code is in
|
||||
head, and the range contains a change to the vulnerable function.
|
||||
|
||||
Each change records its method, family, vulnerable function (resolved in head), head-side changed lines,
|
||||
`post_cutoff` and the advisory id.
|
||||
|
||||
Bias of each, stated in the record. `introducing` inherits SZZ noise: blame finds the last writer, not the author
|
||||
of the flaw. A refactor, a move the heuristics miss or a reformat can stand in for the real introduction. The
|
||||
absence check, the walk-back and the version bracket cut this down, but do not remove it. The bias runs both
|
||||
ways: a wrong V may introduce nothing (a miss is not the hook's fault) or may move a flaw (the delta-based hook is
|
||||
right to call it existing). `last_touch` is pessimistic for a delta hook. The flaw is often already in base, so
|
||||
the hook's novelty rule correctly stays quiet and the change counts as a miss. Catch is therefore reported per
|
||||
method. The headline is `introducing`; `last_touch` is a lower bound. Rejected: fix-parent against a fixed old
|
||||
base (for example 50 commits back). The range often misses the vulnerable function entirely and grows without
|
||||
bound on busy repositories.
|
||||
|
||||
Label check: every third `introducing` change is read by an agent before the freeze, with no model call, as
|
||||
harvest-3 reviewed diffs. It is marked `confirmed`, `refactor` or `unclear`. The confirmed share is the label
|
||||
noise estimate. The scorer also reports catch on the confirmed subset.
|
||||
|
||||
**Decision 2.2: ordinary commits.** The candidates are non-merge commits on the first-parent line of the default
|
||||
branch that change at least one source file of an analysed language. Excluded:
|
||||
|
||||
- explicit security signals in the message: `secur|cve|ghsa|vuln|inject|xss|csrf|ssrf|exploit|sanitiz|escap`;
|
||||
- reverts (`^Revert`, `This reverts commit`);
|
||||
- dependency bumps (bot authors such as dependabot and renovate, `bump|upgrade .* from`, or only manifest and lock
|
||||
files changed);
|
||||
- commits whose source changes are all generated or vendored (`vendor/`, `node_modules/`, `dist/`, `*.min.js`,
|
||||
lock files, `linguist-generated` in `.gitattributes`, "generated" headers);
|
||||
- any fix commit of any advisory of the repository, and any commit touching a known vulnerable function of it;
|
||||
- commits over 1,000 changed source lines (counted, reported).
|
||||
|
||||
Twelve are drawn per repository with a seeded random draw. Strata are three time thirds of the repository's
|
||||
history crossed with four size buckets of changed source lines (1-10, 11-50, 51-200, 201-1,000). Allocation is
|
||||
proportional to the repository's own commits per cell, with at least one per time third. Each change records base
|
||||
(its first parent), head, changed files and line ranges. Reason: proportional allocation keeps the false-alarm
|
||||
rate a per-push rate as a user meets it. Rejected: `freeze.py`'s broad `SECURITY_WORDS`. It also drops commits
|
||||
saying "query", "token", "session" or "valid", which removes the ordinary commits where false alarms are most
|
||||
likely, and the rate would look better than it is. Rejected: docs-only commits. They are trivially quiet and
|
||||
would dilute the rate.
|
||||
|
||||
Label noise: an ordinary commit may itself introduce an unknown vulnerability, so a flag counted as a false alarm
|
||||
may be true. The false-alarm rate stays as Req 3.2 defines it. Flagged ordinary changes are adjudicated beside it
|
||||
(all if 30 or fewer, else every third) against protocol-v2's standard, and the adjudicated share is reported.
|
||||
|
||||
### 3. Sizing and slices (Req 2.3, 5.2)
|
||||
|
||||
Wilson 95% interval, z = 1.96: centre (p + z²/2n)/(1 + z²/n), half width z·sqrt(p(1-p)/n + z²/4n²)/(1 + z²/n).
|
||||
Taking the larger distance from p as the width, the smallest n that meets the bar, computed exactly:
|
||||
|
||||
| Rate | Assumed p | Bar | n needed |
|
||||
| --- | --- | --- | --- |
|
||||
| false alarm | 0.05 | +/- 3 points | 315 |
|
||||
| false alarm | 0.10 | +/- 3 points | 483 |
|
||||
| false alarm | 0.20 | +/- 3 points | 756 |
|
||||
| catch | 0.10 | +/- 10 points | 62 |
|
||||
| catch | 0.25 | +/- 10 points | 88 |
|
||||
| catch | 0.40 | +/- 10 points | 97 |
|
||||
|
||||
Catch: 100 vulnerability-introducing changes per slice, one per repository. At p = 0.4, Wilson gives
|
||||
[0.309, 0.498], within 10 points. One per repository keeps them independent.
|
||||
|
||||
False alarm: ordinary changes of one repository are correlated (noisy code flags often). Wilson assumes
|
||||
independence, so the count is inflated by a design effect 1 + (m - 1)·ICC. With m = 12 per repository and an
|
||||
assumed ICC of 0.05, that is 1.55. 100 repositories × 12 = 1,200 changes, effective n 774 ≥ 756. The plain Wilson
|
||||
interval at p = 0.2, n = 1,200 is about +/- 2.3 points. A repository-cluster bootstrap interval is reported beside
|
||||
it, and the ICC measured on slice 1 is recorded. If it exceeds 0.05, the next pool sizes up. A frozen slice is
|
||||
never changed.
|
||||
|
||||
**Decision 3.1: slices.** Pool p1 has three slices of 100 repositories each (100 + 1,200 changes per slice; 300
|
||||
repositories and 3,900 changes in all). Repositories are assigned by a seeded shuffle, stratified by ecosystem and
|
||||
`post_cutoff`. Slice 1 serves the baseline and the G1 arms table. Slices 2 and 3 serve the first two adoption
|
||||
decisions. When they are spent, pool p2 is drawn by the same code, and it excludes p1. Reason: each slice alone
|
||||
meets the Req 2.3 widths, so any decision can stand on one slice. Rejected: one large pool scored whole. Every
|
||||
decision would then see every repository, which Req 5.2 forbids. Rejected: ten small slices. None would meet the
|
||||
widths. Per family, 100 changes split over seven families give intervals of about +/- 20 to 25 points. They are
|
||||
reported, not sized (open question 3).
|
||||
|
||||
Use ledger: `benchmarks/unseen/usage-p1.jsonl` is append-only, one row per use (slice, decision or experiment id,
|
||||
date, result digest). A slice with a row whose purpose is `informed` cannot appear again as `qualifies` for that
|
||||
change. A test checks this.
|
||||
|
||||
### 4. Scoring rules (Req 3.1, 3.2, 3.3)
|
||||
|
||||
Each replay uses the hook's defaults: `--deadline 30`, `--engine inline`, `--max-regions 500`, `--mode advisory`.
|
||||
That is what a user's push gets. The text output is kept with every `Skipped:` line, and so is the artifact.
|
||||
|
||||
**Decision 4.0: a registered long-deadline arm** (maintainer decision 2026-10-04, open question 5). Beside the
|
||||
default replay, slice 1 is replayed once more with `--deadline 300` and every other flag unchanged. The arm is
|
||||
registered (manifest committed, `unseen: {pool: p1, slice: 1}`) before its first replay. G1 then reports both
|
||||
what the user sees (30 s) and what the engine could catch given time (300 s), so a miss caused by the deadline is
|
||||
told apart from a miss by the engine. It is a measurement arm, not a proposed default: adopting a longer hook
|
||||
deadline would be a separate decision under 7.2.
|
||||
|
||||
**Decision 4.0a: calibrated cluster deadline and native check** (maintainer, 2026-10-05).
|
||||
The gVisor proof exhausted 30 seconds in snapshot preparation on a 1,725-file repository.
|
||||
Cluster replays therefore scale the 30/300-second arms by the median paired cluster/native
|
||||
time through the quick tier, rounded to integer seconds. Native runc replays retain 30/300 seconds.
|
||||
A deterministic sample (default 20, or all of a smaller proof manifest) runs both lanes at
|
||||
600 seconds for calibration. Report per-change stage times and median slowdown with bootstrap
|
||||
95% intervals for time through quick, engine build and engine queries separately. Incomplete
|
||||
timings or degraded scans invalidate calibration. Records contain counts, times and opaque digests.
|
||||
|
||||
A deterministic native sample accompanies the cluster slice. Compare whether each push flags
|
||||
anything, with a Wilson 95% interval. If its lower bound is below 95%, or any sample pair is
|
||||
missing/failed, flag the run and exclude cluster results from the baseline. Disagreements name
|
||||
change IDs only. The scorer reports applied deadlines and factors. Docker pulls explicitly after
|
||||
checking its data-root free space against an unpacked image size upper bound plus 1 GiB; failures
|
||||
retain sanitized stderr tails and exit codes. See the unseen README for coordinator commands.
|
||||
|
||||
**Decision 4.1: catch, per family.** A finding counts if its family is the change's family. Engine streams carry a
|
||||
family; a quick-rule match maps its CWE through the family CWE table. Its head location must also be in the
|
||||
vulnerable function or within 10 lines of the change's head-side hunks (protocol v1's window).
|
||||
|
||||
| Family | Location rule |
|
||||
| --- | --- |
|
||||
| injection, path, deserialization, output_encoding, untrusted_destination | sink line in the vulnerable function, or the changed lines +/- 10 |
|
||||
| access_control | the vulnerable function itself (the handler that lacks the check), or the changed lines +/- 10 |
|
||||
| config_secrets | the changed file and lines +/- 10 (file scope; function `<global>`) |
|
||||
|
||||
All three streams count: admitted defects, advisory engine findings and quick-rule matches. Each catch also names
|
||||
the stream that made it. Reason: the hook shows all three to the user. Rejected: admitted defects only. That
|
||||
scores the blocking path, which G3 decides separately, and would hide what users read.
|
||||
|
||||
**Decision 4.2: false alarm.** An ordinary change is a false alarm when any of the three streams holds at least one
|
||||
finding, of any family, anywhere. The alerts-only rate (admitted defects) is reported beside it.
|
||||
|
||||
**Decision 4.3: coverage, so zeros cannot hide.** Every change gets exactly one state:
|
||||
|
||||
- `instrument_failure`: no record, a failed fetch, the head commit not verified, or zero bytes of the changed files
|
||||
read. It is re-run once, then reported. It never counts as quiet or missed.
|
||||
- `unanalysable`: no changed file is in a language the hook covers (the hook's `language_not_covered` notice).
|
||||
- `quick_only`: the quick tier read the changed files, but no engine question completed on them.
|
||||
- `engine_covered`: at least one engine question completed on a changed file.
|
||||
|
||||
The task entry proves its input as AGENTS.md requires. It records files and bytes in the checkout, bytes of each
|
||||
changed file, the JVM's wall time and the hook's exit code. A replay faster than a JVM can start is flagged
|
||||
`suspect_fast` and goes to the failure path. Catch and false alarm are reported over all changes and over each
|
||||
coverage state.
|
||||
|
||||
### 5. The AX workload (Req 4)
|
||||
|
||||
**Decision 5.1: the image.** `plane/Dockerfile.engine-task` gains a second stage, `FROM engine AS replay`, built
|
||||
with `--target replay` and pinned by digest. It adds `git`, the package with its runtime dependencies and
|
||||
`benchmarks/unseen/replay_entry.py`. Joern, php-cli, Java 21 and the frontend-retention patch come from the engine
|
||||
stage, so both lanes run the same Joern. The import smoke runs `ousast pre-push --help`. The root image stays
|
||||
`ghcr.io/norandom/ax-task-runner:v0.3.1`. Rejected: a separate Dockerfile. It would duplicate the Joern install,
|
||||
and two Joern copies are two instruments that drift. Rejected: one stage for both. The engine image's isolated
|
||||
venv without yaml and boto3 is proven, and replay dependencies would change its digest on every bump.
|
||||
|
||||
**Decision 5.2: the task entry.** `replay_entry.py` is standard library only, like `engine_task_entry.py`, and
|
||||
reuses its `transfer` (egress readiness wait, three attempts, no URL text in errors).
|
||||
|
||||
1. GET `SPEC_URL` (presigned, `.json`): repository URL, base, head, change id, hook flags.
|
||||
2. `git init`, then `git fetch --depth 1 origin <head>` and `git fetch --depth 1 origin <base>` over HTTPS
|
||||
(GitHub serves commits by SHA). Verify both with `git cat-file -e`. The fetch is bounded at 180 s; the 500 MB
|
||||
repository cap bounds its size.
|
||||
3. Run `ousast pre-push <checkout> --base <base> --head <head> --artifact /tmp/out/push.json` with defaults, the
|
||||
backend heap at 1700 MB and the cluster JVM flags of `engine_trace_worker.py`. Capture stdout.
|
||||
4. Pack `push.json`, `push.txt`, `engine.json` (the head and base scan records and witness rows from the artifact,
|
||||
keyed by repository and revision, for engine-evidence arms) and `instrument.json` (section 4.3, image digest,
|
||||
step timings). PUT to `RESULT_URL` as `.dat`. The PUT is completion.
|
||||
|
||||
On any error it PUTs a failure record naming the step and HTTP status, as the engine entry does. Task names are
|
||||
`ousast-engine-replay-<slug>-<digest>`, the prefix the operator's egress watcher matches. No key or store
|
||||
credential enters the manifest. Only the two presigned URLs do, and the spec object keeps repository names out of
|
||||
AX status and cluster telemetry.
|
||||
|
||||
**Decision 5.3: the generic dispatcher.** `benchmarks/ax/batch.py` takes over from `engine_trace_ax.AX` and
|
||||
`engine_trace.run_plan`:
|
||||
|
||||
- `Item`: id, input objects (name to bytes), the result object name, command, extra env, input digest.
|
||||
- `Workload`: image digest, allowed URL env names, deadline, how to validate a received result.
|
||||
- Lanes: `ax` (manifest, apply, poll the result object and AX phase, delete, clean objects), `kind` (the same
|
||||
manifest against the kind AX), `docker` (the VM image, one container at a time).
|
||||
- Scheduler: two AX lanes plus one VM lane by default. A sandbox death is retried once under a new name; a second
|
||||
death or a JVM OOM goes to the VM fallback queue, which beats overflow on the VM lane. `progress.json` is written
|
||||
only by the scheduler. A `STOP` file halts new launches. Resume skips items whose stored record has the same
|
||||
input digest. Cleanup failure stops dispatching.
|
||||
|
||||
The engine pass becomes a workload over it, with the same tests. Reason: the egress, secrecy and cleanup logic took
|
||||
several fixes and must exist once. Rejected: a replay-specific copy of `engine_trace_ax.py`. Rejected: putting it in
|
||||
`src/` now. G4's `ousast plane scan` may need it there, and it moves when G4 is planned.
|
||||
|
||||
Concurrency is two AX tasks. Placement is random, and only two guarantees two per worker at most. Four would need
|
||||
the operator to add placement (open question 2, resolved 2026-10-04: stays at two, no placement change
|
||||
requested). Estimated wall time per slice: about 75 s per replay (fetch, 30 s hook, setup), 1,300 replays over
|
||||
three lanes, about 9 hours. The long-deadline arm (4.0) is bounded at about 345 s per replay; at that bound 1,300
|
||||
replays over three lanes take about 42 hours, less where the hook finishes early. It runs with `STOP` and resume
|
||||
across sessions.
|
||||
|
||||
**Decision 5.4: model arms on the VM.** Arms that need DeepSeek (the decision engine, the combiner) read the stored
|
||||
records on the VM. They build candidates from the head side's changed functions, score them, and take the change
|
||||
score as the maximum. Each runs under its experiment manifest's `budget_usd` through the metered client, dry run
|
||||
first, with responses cached. Nothing in the cluster calls a model.
|
||||
|
||||
**kind (Req 4.5)** runs the identical manifest and image through the `kind` lane. Every new task is proven there
|
||||
before kube-ax, as `plane-on-kubernetes` requires.
|
||||
|
||||
### 6. Records and the guard (Req 1.2, 1.3, 5.1)
|
||||
|
||||
`benchmarks/unseen/` holds `pool-p1.toml` (repositories, licences, slices, changes with base, head, method,
|
||||
family, function, lines), `freeze-p1.json` (used-set digests and counts per source, guard result, label-check
|
||||
counts, freeze digest = sha256 of the canonical manifest), `protocol-p1.md` (the scoring rules of section 4,
|
||||
committed before the first replay) and `usage-p1.jsonl`. The `private/` directory is gitignored.
|
||||
|
||||
The guard grows by one parametrised reservation. The existing sweep becomes a list of (reservation directory,
|
||||
names): `benchmarks/independent/` for populations and `benchmarks/unseen/` for pools, including the private
|
||||
manifest when present. It also sweeps `plane/`. A second test asserts that pool and population names are disjoint,
|
||||
and prints only a count on failure. A third asserts that a frozen pool's digest matches `freeze-p1.json`, so the
|
||||
pool is never edited. Raw per-change records stay outside the repository (`~/ousast-results/unseen/` and the
|
||||
store). Committed result records under `benchmarks/measurements/` carry counts, rates, intervals and digests,
|
||||
never names.
|
||||
|
||||
### 7. Metrics and reporting (Req 3.3, 3.4, 5.3)
|
||||
|
||||
`benchmarks/unseen/score.py` reports per family and overall: catch and false-alarm rate with Wilson 95%
|
||||
intervals, the cluster bootstrap interval for false alarms, coverage counts per state, catch per method,
|
||||
post-cutoff and per stream, and the alerts-only rate.
|
||||
|
||||
**Decision 7.1: arms at fixed budgets.** An arm with a score is thresholded per budget (1%, 5%, 10%). The
|
||||
threshold is the lowest score whose false-alarm rate on the slice's ordinary changes stays within the budget.
|
||||
Catch at that threshold is reported with its Wilson interval and the realised false-alarm rate. That in-slice
|
||||
threshold is labelled exploratory. For a qualification, the threshold is fixed beforehand on an earlier slice or
|
||||
on held-out folds, and it is applied unchanged. A binary arm (today's hook) has one operating point, reported with
|
||||
the budgets it meets. Rejected: AUC over changes (it averages operating points no user runs).
|
||||
|
||||
**Decision 7.2: next to held-out folds.** Experiment manifests registered after this spec lands carry
|
||||
`unseen: {pool, slice}`. `analyse` writes an `unseen` block into `result.json` next to the fold estimates: catch at
|
||||
each budget for each arm, and the paired difference B - A with an exact McNemar test on the 100 changes. The
|
||||
fold rule decides as today. The slice is recorded in the usage ledger.
|
||||
|
||||
**Decision 7.3: adoption gate on the unseen slice** (amended 2026-10-04, maintainer decision; Req 5.4; open
|
||||
question 4 resolved). Reporting is not enough. An arm B is adopted over the production arm A only if, on the
|
||||
untouched slice that qualifies it, both hold:
|
||||
|
||||
- catch at the 5% budget, with B's threshold fixed beforehand (7.1), is not significantly lower than A's: exact
|
||||
two-sided McNemar on the paired vulnerability-introducing changes, alpha 0.05;
|
||||
- for an arm with one operating point (a binary arm such as a hook setting), its false-alarm rate is not
|
||||
significantly higher than A's: exact two-sided McNemar on the paired ordinary changes, alpha 0.05, with the
|
||||
repository-cluster bootstrap interval of the difference reported beside it, since those changes are clustered.
|
||||
|
||||
`analyse` writes `unseen.gate: pass | regression | not_run` into `result.json`. `not_run` (no slice, slice
|
||||
already `informed` for this change, or more than 5% `instrument_failure`) blocks adoption like `regression`. The
|
||||
fold rule and this gate must both pass.
|
||||
|
||||
### 8. Increments
|
||||
|
||||
1. **Used set and guard.** `eligibility.py`, the extended guard and the usage-ledger test. Exit: guard green on an
|
||||
empty pool; a planted used repository, a renamed one and a fork are each rejected in tests.
|
||||
2. **Sourcing and extraction.** Candidates, both change kinds and the label check, on the VM. Exit: a draft for 300
|
||||
repositories with method counts, the confirmed share and exclusion counts.
|
||||
3. **Freeze p1.** Guard bisection for v3, then manifest, freeze record and protocol committed. Exit: freeze digest
|
||||
committed and guard green.
|
||||
4. **Dispatcher.** `benchmarks/ax/batch.py`, with the engine pass moved onto it. Exit: existing tests green and a
|
||||
two-pin engine smoke on kube-ax matching its stored records.
|
||||
5. **Replay task.** Image stage and entry, proven on kind, then kube-ax on five changes from this repository's own
|
||||
history (never pool changes). Exit: instrument fields show bytes read and realistic JVM time, and quick-tier
|
||||
output matches the VM lane.
|
||||
6. **Scorer.** `score.py` on synthetic records, including every coverage state and an instrument failure. Exit:
|
||||
unit tests green.
|
||||
7. **Baseline.** Today's hook, from a frozen export of HEAD, on slice 1 (1,300 replays). Exit: the record committed
|
||||
(counts and digests only) and a `baseline` row in the usage ledger. That is G0's exit.
|
||||
8. **Arms on slice 1 (G1)** (added 2026-10-04 with decisions 4.0 and 7.3). The registered long-deadline arm,
|
||||
the engine-evidence arm from the stored records, the model arms on the VM (5.4) under their budgets, and the
|
||||
`unseen` block with the adoption gate in `analyse`. Exit: one committed table of arms on slice 1 at the 1/5/10%
|
||||
budgets (counts, rates, intervals, digests), and `exploratory` rows in the usage ledger.
|
||||
|
||||
## Error Handling
|
||||
|
||||
A failed fetch, a missing commit or a hook crash gives a failure record, never an empty success. A dead sandbox is
|
||||
retried once, then the VM runs it. A store or AX cleanup error stops the dispatcher. A slice with more than 5%
|
||||
`instrument_failure` after re-runs is not scored; the cause is fixed first.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
Unit tests: used-set normalisation (case, `.git`, renames, forks), blame acceptance on a synthetic repository,
|
||||
ordinary-commit exclusions, Wilson values against section 3, budget thresholds, coverage states, the task manifest
|
||||
and dispatcher resume and fallback with a fake lane. Guard tests run in CI; cluster runs stay manual.
|
||||
|
||||
## Requirement Traceability
|
||||
|
||||
| Requirement | Design |
|
||||
| --- | --- |
|
||||
| 1.1 eligibility against every source, v3 via the guard | 1.2 |
|
||||
| 1.2 frozen manifest, never edited | 6, increment 3 |
|
||||
| 1.3 pool directory, guard extended | 6 |
|
||||
| 1.4 licences, private manifest | 1.4 |
|
||||
| 2.1 introducing changes, family, function, post-cutoff | 1.1, 1.3, 2.1 |
|
||||
| 2.2 ordinary changes across time and size | 2.2 |
|
||||
| 2.3 sizes for interval widths | 3 |
|
||||
| 3.1 default replay, skipped checks kept | 4, 5.2 |
|
||||
| 3.2 catch and false-alarm rules | 4.1, 4.2 |
|
||||
| 3.3 per family, Wilson, coverage | 4.3, 7 |
|
||||
| 3.4 arms at 1/5/10% budgets | 4.0, 7.1, increment 8 |
|
||||
| 4.1 one change one Task, prefix, image by digest | 5.1, 5.2 |
|
||||
| 4.2 HTTPS fetch, presigned record, no keys | 5.2 |
|
||||
| 4.3 generic dispatcher, concurrency, retry, fallback, resume | 5.3 |
|
||||
| 4.4 model arms on the VM within budgets | 5.4 |
|
||||
| 4.5 kind runs the same tasks | 5.3, 5 (kind paragraph) |
|
||||
| 5.1 baseline on slice 1, counts and digests | 6, increment 7 |
|
||||
| 5.2 fresh slice per decision | 3.1, usage ledger |
|
||||
| 5.3 experiments report unseen next to folds | 7.2 |
|
||||
| 5.4 adoption needs no significant unseen regression | 7.3 |
|
||||
|
||||
## Amendments (2026-10-11)
|
||||
|
||||
Folded in after the G0 pool was frozen; the original sections above stay as the design of record, these refine it
|
||||
where reality differed.
|
||||
|
||||
- **Persistence is bucket-first, not local disk.** The draw streams every stage to the `ousast-corpus` RustFS
|
||||
bucket and resumes from it, so a restart (even after a code change) is cheap. Layout: `unseen/<seed>/decisions/`
|
||||
(accept/reject per repo, key `sha256(repo+fix)`, idempotent), `unseen/<seed>/resolved/` (per-advisory
|
||||
resolution), `unseen/<seed>/advisories/list.json`, `unseen/repos/` (get_repo cache, **not** seed-scoped, shared
|
||||
across seeds), and `unseen/<seed>/pool/` (the frozen pool + provenance — the "don't redo sourcing" archive). The
|
||||
local journal is a within-process cursor only. Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`). The
|
||||
VM's local disk is the scarce resource; run artifacts and replay outputs belong on RustFS (see steering
|
||||
`storage-streaming.md`). Credentials: `.env` `CORPUS_S3_*`; boto3 needs path-style + SigV4.
|
||||
- **Harvest cost is GitHub-rate-limit bound, measured yield ~2.2%.** Through the strict change criteria (1 vuln +
|
||||
12 stratified ordinary, >=40 eligible commits) ~2.2% of candidates are accepted, so "100 repos" is ~4,500
|
||||
candidate attempts at ~4 GitHub calls each — the bottleneck is the 5,000/hr API limit, not CPU or cores. Pure
|
||||
GitHub-API harvesting cannot replace the clone: the positive is the SZZ introducing commit and the negatives are
|
||||
stratified over full history, both needing repo history. The convergence fix was a full-objects clone (dropping
|
||||
`--filter=blob:none`, which caused ~600 per-blob lazy fetches) plus the bucket caches above.
|
||||
- **Record 07 is a machine-label baseline.** The pool's 100 vulnerability labels are SZZ-derived and unchecked
|
||||
(`freeze-p1.json` `label_check_counts`: 100 unchecked). The G0 baseline is reported as a machine-label number
|
||||
with that caveat; a human label pass is a later quality refinement, required before any published precision
|
||||
claim, not before record 07.
|
||||
- **Replay image provenance is pinned.** CI `engine-image.yml` builds `ghcr.io/norandom/ousast-replay-task` (the
|
||||
replay harness, which `COPY`s `/app/benchmarks/unseen/replay_entry.py`) and `ousast-engine-task` (the Joern
|
||||
hook), publicly pullable by digest. `ousast-runner` (the 248 MB ax-task-runner, no `/app/benchmarks`, no Joern)
|
||||
is **not** the replay image. The AX lane's store check (`benchmarks/ax/batch.py`) requires a public-HTTPS DNS
|
||||
presign-capable endpoint; how that and the cluster's store access are provided is operator-owned infra, not
|
||||
assumed here.
|
||||
|
||||
- **AX lane blocker (2026-10-11) — kube is the target lane, so this is tracked, not dismissed.** The replay
|
||||
*dispatches* correctly but tasks do not *complete*: `pins_done` stayed 0 for both the 5-change proof and the
|
||||
1,300-change slice 1. Sub-issues resolved along the way: (a) the replay image is `ousast-replay-task` (public);
|
||||
(b) the ax CLI must be `~/go/bin/ax` — the `~/.local/ax-v0.3.1/ax` build calls an `UpdateTask` RPC the deployed
|
||||
ax-server answers `Unimplemented` (client/server version mismatch); (c) the store endpoint
|
||||
`files.because-security.com` / `sast-memory` passes the egress check. **Remaining blocker (AX/cluster-internal,
|
||||
operator-owned):** dispatched tasks reach the `ousast-pool` workers and the replay actors execute and checkpoint,
|
||||
but the result never returns to the coordinator and no completion is recorded. `ousast-pool` logs show a gVisor
|
||||
cleanup failure: `runsc delete ... connecting to control server at PID NNN: connection refused` and `failed to
|
||||
terminate workload after checkpoint ... runsc delete: exit status 128`. Until the completion / checkpoint-cleanup
|
||||
path is fixed, the ax lane cannot produce results; the G0 baseline (7.2) was taken on the **docker lane**
|
||||
instead. The ax lane must be restored before it is the baseline/G1 lane (the stated target).
|
||||
|
||||
- **Engine coverage gap is comparison, not deadline (2026-10-11, preliminary n=1).** The cold-cache default arm
|
||||
(30s) times out the Joern CPG build → `quick_only`. But a long-deadline sample (`--arm long_deadline`, 300s) on a
|
||||
vulnerability change ran **206s with no timeout and still did not cover** (`coverage_status: incomplete`,
|
||||
`quick_only`): the reasons were `ambiguous_line_correspondence`, `unmatched_added_deleted_path_correspondence`,
|
||||
`eligibility_provenance_mismatch`, `witness_unresolved`, `comparison_unknown` — i.e. the engine could not
|
||||
correspond the replayed diff's lines between base and head. So **calibration / a longer deadline alone will not
|
||||
give engine coverage**; the blocker is the engine's base↔head comparison on real replayed diffs (consistent with
|
||||
the earlier "engine near-blind to fixes" slice-leak audit). This is G2 engine-fix territory (comparison /
|
||||
line-correspondence), not a G0 deadline knob. To confirm beyond n=1, the long-deadline arm must run at scale
|
||||
(needs the kube lane). Implication: the record-07 baseline is measured on the **quick-rule tier** (the default
|
||||
arm), and the engine arm's value is gated on fixing the comparison path, not on calibration.
|
||||
|
||||
- **Line-correspondence fix made, but it is necessary-not-sufficient (2026-10-11).** Root cause confirmed in
|
||||
`src/openultrasast/push/snapshot.py`: equal-region base↔head line pairs were mapped only when the line content
|
||||
was *globally unique in the file*, so common lines (`}`, blanks, imports) tripped `ambiguous_line_correspondence`
|
||||
and the finding's anchor was discarded → `comparison_unknown` → finding suppressed. Fixed to **run-aware
|
||||
positional anchoring**: a pair maps when its content occupies exactly one contiguous equal run per side (intra-run
|
||||
duplicates are forced/safe; cross-run duplicates stay withheld — soundness preserved, `test_push_snapshot.py`
|
||||
moved-operation guard still passes). 163 push tests pass. BUT live-testing the fix (mounted over the installed
|
||||
package, 300s) on a real messy change (a 14-file Java diff) did **not** flip coverage: it stayed `incomplete`
|
||||
because the binding constraints are elsewhere — `capability_unavailable` (empty capability registry always blocks
|
||||
`coverage_status=complete` until Task 8.3), the engine produced **no actionable defect at all**
|
||||
(`actionable_defect_ids: []`, a detection gap, not correspondence), plus `eligibility_provenance_mismatch` and
|
||||
`contributor-scan ... projection_unavailable`. So "near-blind to fixes" is **multi-causal**; the correspondence
|
||||
fix removes one real cause but engine coverage on unseen diffs also needs the capability registry (Task 8.3), the
|
||||
engine's actual vuln detection, and the eligibility/projection paths. Validating the correspondence fix's benefit
|
||||
needs a change where correspondence is the sole blocker (a small single-file diff), or the engine arm at scale.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Maintainer decisions of 2026-10-04 resolve questions 1, 2, 4 and 5; question 3 stays open for G3.
|
||||
|
||||
1. **Resolved 2026-10-04.** Licences: only permissively licensed repositories are published by name. Copyleft and
|
||||
all other licences go to the gitignored private manifest, covered in the committed manifest by an opaque id and
|
||||
a digest (Req 1.4 as written; decision 1.4). Population v2's OSI-approved bar is not used.
|
||||
2. **Resolved 2026-10-04.** Concurrency stays at two AX lanes plus the VM lane. No operator placement change is
|
||||
requested (5.3).
|
||||
3. Per-family BLOCK (G3) needs about 100 changes per family. Pool slices together, or draw a family-enriched pool
|
||||
for G3?
|
||||
4. **Resolved 2026-10-04.** An adoption requires no significant regression on the unseen slice, not only a report
|
||||
(Req 5.4, amended; decision 7.3).
|
||||
5. **Resolved 2026-10-04.** G1 adds a registered long-deadline arm (`--deadline 300`) next to the default 30 s, so
|
||||
the table shows both what the user sees and what the engine could catch (decision 4.0, increment 8).
|
||||
@@ -0,0 +1,107 @@
|
||||
# Requirements Document
|
||||
|
||||
## Introduction
|
||||
|
||||
The safety net is meant for repositories it has never seen. This spec builds the instrument that measures that: a
|
||||
frozen pool of unseen repositories, a push-level metric (catch rate and false-alarm rate as the hook reports them),
|
||||
and an execution path that runs the replays as AX Tasks on the kube-ax cluster. Context: `brief.md`, the roadmap's
|
||||
G0-G1 (`.kiro/steering/roadmap.md`).
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: pool selection and freezing; change extraction (vulnerability-introducing and ordinary commits);
|
||||
replay of each change through `ousast pre-push` and through registered experiment arms; the metric and its
|
||||
confidence intervals; AX Task execution with kind/VM fallback; the baseline for today's hook; rotation of pool
|
||||
slices.
|
||||
- **Out of scope**: any detection change; population v3 (stays reserved and unread); model calls inside the cluster.
|
||||
- **Adjacent expectations**: `learned-decision-engine` experiments use this pool as their primary unseen-repo score;
|
||||
`pre-push-safety-net` owns the hook being replayed; `plane-on-kubernetes` and the AX executor
|
||||
(`benchmarks/learn/engine_trace_ax.py`) supply the execution pattern; the reserved-repository guard covers every
|
||||
population and this pool.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: A pool no development step has touched
|
||||
|
||||
**User Story:** As the maintainer, I want the evaluation repositories to be ones nothing in development has seen, so a
|
||||
good number means the approach generalises.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. A repository is eligible only if it appears in no pair corpus, no population (v1, v2, v3), no harvest record, no
|
||||
memory example, no experiment unit and no measurement record. Eligibility is checked by code against every source
|
||||
before the pool is frozen; v3 is checked through the existing guard without being opened.
|
||||
2. The pool is frozen (a committed manifest with repository URLs, commit SHAs, licences and the freeze digest) before
|
||||
any change in it is replayed. A frozen pool is never edited; a correction is a new pool.
|
||||
3. Pool repositories are referenced only inside the pool's own directory (like `benchmarks/independent/`), and the
|
||||
reserved-repository guard is extended to them.
|
||||
4. Only permissively licensed repositories are published by name; others may be used with their names kept out of
|
||||
the repository (pointers in a local, gitignored manifest), recorded as such.
|
||||
|
||||
### Requirement 2: Two kinds of change per repository
|
||||
|
||||
**User Story:** As the maintainer, I want both vulnerable and ordinary pushes from the same repositories, so the
|
||||
false-alarm rate is measured on the code the catch rate is measured on.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Vulnerability-introducing changes: from public advisories with a linked fix, the commit range that introduced the
|
||||
vulnerable code where it can be determined, else the fix's parent compared with an earlier base; each labelled
|
||||
with family and the vulnerable function. Post-model-cutoff advisories are preferred and flagged.
|
||||
2. Ordinary changes: commits from the same repositories' histories with no security fix, revert or advisory link,
|
||||
sampled across time and size; each recorded with its changed files and lines.
|
||||
3. Per repository, at least one vulnerability-introducing change and several ordinary changes; the pool's totals are
|
||||
sized so the false-alarm rate's 95% interval is no wider than +/- 3 points and the catch rate's no wider than
|
||||
+/- 10 points (sizes computed in the design).
|
||||
|
||||
### Requirement 3: The push-level metric
|
||||
|
||||
**User Story:** As the maintainer, I want the score to be what a user would see, so the number means something.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Each change is replayed as `ousast pre-push --base <base> --head <head>` with the hook's default settings, in a
|
||||
clean checkout, with the record of every skipped check kept.
|
||||
2. Catch: a vulnerability-introducing change counts as caught when the hook (or the arm) reports a finding of the
|
||||
right family on the vulnerable function or its changed lines. False alarm: an ordinary change counts as a false
|
||||
alarm when the hook reports any advisory finding.
|
||||
3. Reported per family and overall: catch rate, false-alarm rate, and both with 95% Wilson intervals; plus coverage
|
||||
(changes where the hook could analyse the changed language) so a silent zero is visible.
|
||||
4. Arms (decision engine, combiner, engine evidence, fix mechanisms) are scored on the same replays: catch rate at
|
||||
fixed false-alarm budgets (1%, 5%, 10%), so arms are compared at equal noise.
|
||||
|
||||
### Requirement 4: Execution on kube-ax, designed for AX
|
||||
|
||||
**User Story:** As the maintainer, I want the replays to run on the cluster, so measurements are fast and the VM is
|
||||
free.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. One replayed change is one AX Task, named with the `ousast-engine-` prefix the operator's egress watcher matches,
|
||||
using an image built `FROM ghcr.io/norandom/ax-task-runner:v0.3.1` with this package, Joern and php-cli, pinned by
|
||||
digest.
|
||||
2. The task fetches the repository at the needed commits itself over HTTPS (public hosts are reachable from actors)
|
||||
and writes its record through a presigned link; no API keys or store credentials enter the cluster.
|
||||
3. A generic AX batch dispatcher serves both this replay and the existing engine passes: items to Tasks, at most the
|
||||
cluster's safe concurrency (two heavy tasks per worker), retry once on sandbox death, then a VM or kind fallback
|
||||
lane; progress and resume as in the engine runner.
|
||||
4. Arms that need model calls take the cluster's records and make their calls on the VM, within registered budgets.
|
||||
5. kind runs the same tasks unchanged as the local fallback.
|
||||
|
||||
### Requirement 5: Baseline, rotation and use
|
||||
|
||||
**User Story:** As the maintainer, I want a baseline now and a fresh slice for each decision, so the pool is not
|
||||
worn out by tuning.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. The first use is a baseline of today's hook on the first slice; its record is committed (counts and digests only,
|
||||
no repository names outside the pool directory).
|
||||
2. A slice that informed a change can no longer qualify that change; each adoption decision draws a new, untouched
|
||||
slice, recorded with its freeze digest.
|
||||
3. Every learned-decision-engine experiment from now on reports its unseen-repo score from this pool next to its
|
||||
held-out-fold score.
|
||||
4. An arm is adopted only if, on the untouched slice that qualifies it, it shows no significant regression against
|
||||
the current production arm: catch at the 5% false-alarm budget not significantly lower, and for an arm with one
|
||||
operating point the false-alarm rate not significantly higher (exact paired tests, two-sided, alpha 0.05).
|
||||
Reporting the unseen score alone is not enough to adopt. *(amended 2026-10-04, maintainer decision)*
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"feature_name": "unseen-repo-evaluation",
|
||||
"created_at": "2026-10-04T18:00:00Z",
|
||||
"updated_at": "2026-10-04T20:00:00Z",
|
||||
"language": "en",
|
||||
"phase": "implementation",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"design": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
},
|
||||
"tasks": {
|
||||
"generated": true,
|
||||
"approved": true
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": true
|
||||
}
|
||||
@@ -0,0 +1,402 @@
|
||||
# Implementation Plan
|
||||
|
||||
One task group per increment of `design.md` section 8, in that order. A group starts only after the previous
|
||||
group's exit is recorded. Maintainer decisions of 2026-10-04 are folded in: a registered long-deadline arm
|
||||
(`--deadline 300`) beside the hook's default 30 s (design 4.0, group 8); adoption needs no significant regression
|
||||
on the unseen slice (Req 5.4 as amended, design 7.3, tasks 6.4 and 8.2); only permissively licensed repositories
|
||||
are published by name, everything else goes to the gitignored private manifest under a digest (Req 1.4, design
|
||||
1.4); concurrency stays at two AX lanes plus the VM lane (design 5.3).
|
||||
|
||||
**2026-10-11 amendments folded in** (see design.md "Amendments (2026-10-11)"):
|
||||
|
||||
- **Pool p1 is frozen.** `benchmarks/unseen/pool-p1.toml` = 100 repos, slice 1; `freeze-p1.json` guard=0,
|
||||
dropped=0, deterministic (manifest `6bd2fa5d`); archived to `ousast-corpus/unseen/20260601/pool/`. Groups 1-3 are
|
||||
effectively done (the draw, draft and freeze ran end to end); formal records 02/03 were not written as separate
|
||||
`record.json` files — the freeze was run directly (`bisect_guard` then `freeze`). Freeze recipe: `bisect_guard
|
||||
--root $PWD --draft $CACHE/journal/draft-p1.toml` then `freeze --root $PWD --used-report <FILE>` where the file
|
||||
is the draw report's `used_set` object (not the whole report).
|
||||
- **Persistence is bucket-first** (`ousast-corpus`, `benchmarks/unseen/bank.py`); the local journal is a
|
||||
within-process cursor only. Raw per-change replay records go to **RustFS**, not `~/ousast-results/` on local disk
|
||||
— amend tasks 5.3/7.2's `--out` accordingly.
|
||||
- **Group 5 image note:** the replay image is the CI-built `ghcr.io/norandom/ousast-replay-task` (+ engine-task),
|
||||
publicly pullable by digest — not a local export-HEAD build and not `ousast-runner`. Task 7.1's "build and push
|
||||
the replay image" is satisfied by the CI `engine-image.yml` run; use that digest.
|
||||
- **Record 07 is a machine-label baseline** (labels SZZ-derived, 100 unchecked); state the caveat in the record.
|
||||
- **Baseline lane (7.2):** run on kube (`--lane ax`). The `ax` binary and the AX-lane store endpoint are
|
||||
operator-owned infra to stand up before 7.2; do not assume the cluster/network state.
|
||||
|
||||
Execution rules:
|
||||
|
||||
- **Who does what.** Codex implements every coding task offline: no network, no model, no store, no cluster. The
|
||||
coordinator runs every networked step (GitHub and OSV API calls, git fetches and clones, the S3 store, image
|
||||
pushes, kind and kube-ax). Those tasks are marked **(coordinator, network)**. Steps that need the coordinator but
|
||||
no network (agent label reads, adjudication, the v3 guard bisection, commits) are marked **(coordinator)**.
|
||||
Steps that spend model money are marked **paid: needs a budget go-ahead**. Each starts only after the maintainer
|
||||
confirms a ceiling against the provider balance read just before it.
|
||||
- **Done means.** A coding task is done when its tests pass in the host suite, `ruff check`, `ruff format --check`
|
||||
and `mypy` are clean, and the reserved-repository guard is green. No test makes a network, model or cluster call.
|
||||
- **Names stay in the pool directory.** Pool repository names and URLs appear only under `benchmarks/unseen/`
|
||||
(copyleft and other non-permissive ones only under the gitignored `benchmarks/unseen/private/`). Raw per-change
|
||||
records live in `~/ousast-results/unseen/` and the store, never in the repository.
|
||||
- **Population v3 is never opened.** It is checked only through the guard test, run as a black box with its
|
||||
output discarded.
|
||||
- **Every group ends with a measured exit** and a record at
|
||||
`benchmarks/measurements/<date>-unseen-NN-<slug>/record.json`: commit, image digests where used, the instrument
|
||||
checks (bytes read, JVM time, exit codes), counts and digests. No repository names. The guard sweeps these
|
||||
records.
|
||||
- **Instrument first** (AGENTS.md). A zero, an empty draft or a fast run is treated as an instrument failure until
|
||||
bytes read and wall time prove otherwise.
|
||||
|
||||
- [x] 1. Used set and guard (design 1.2, 6; increment 1)
|
||||
- [x] 1.1 Used-set builder
|
||||
- `benchmarks/unseen/eligibility.py`: normalises names (lower case, no `.git`, no trailing slash) and builds the
|
||||
used set from every source of design table 1.2: pair catalogs, recipes and datasets under `benchmarks/pairs/`
|
||||
plus the local pointer-pair manifests as `benchmarks/pairs/catalog_gen.py` reads them; `population-v1.toml`
|
||||
and `population-v2.toml`; the advisory-fix batch catalogs; memory `example` rows (`repo` field) through an
|
||||
injected `MemoryStore` (`FileStore` locally, the S3 store when the coordinator passes one); experiment units
|
||||
under `plane/experiments/`; `benchmarks/measurements/` and `benchmarks/experiments/`; a text sweep for
|
||||
`github.com/<owner>/<name>` over `benchmarks/`, `src/`, `tests/`, `plane/` and `.kiro/`. It never reads
|
||||
population v3 and never prints a name. Output: counts per source and the sha256 of the sorted set.
|
||||
- `tests/test_unseen_eligibility.py`: a synthetic tree with one repository per source, each found; case, `.git`
|
||||
and trailing-slash variants collapse; stdout and the output JSON contain no name; a v3-shaped file planted in
|
||||
the synthetic tree is not opened (open is patched to fail on it).
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.2 Canonical resolution, renames and forks
|
||||
- `eligibility.py`: `resolve(candidate, api)` with an injected GitHub client returns the canonical `full_name`,
|
||||
former names (redirects) and the fork network (`parent`, `source`). A candidate is ineligible when any of them
|
||||
is in the used set; one candidate per fork network is kept.
|
||||
- Tests with a fake client: a planted used repository, a renamed one and a fork of a used one are each
|
||||
rejected; two forks of one network keep one; the rejection reason is a source label, not a name.
|
||||
- _Requirements: 1.1_
|
||||
- [x] 1.3 Guard extended to pools
|
||||
- `tests/test_independent_population.py`: the sweep becomes a list of (reservation directory, names):
|
||||
`benchmarks/independent/` for populations, `benchmarks/unseen/` for pools (public manifests, plus
|
||||
`benchmarks/unseen/private/*.toml` when present). The sweep also covers `plane/`. New tests: pool and
|
||||
population names are disjoint (failure prints a count only); a frozen pool's manifest digest equals its
|
||||
`freeze-pN.json` (skipped while no pool is frozen). The existing v3 handling is unchanged.
|
||||
- Each reservation is exempt only inside its own directory: population names (v3 included) are still swept in
|
||||
`benchmarks/unseen/`, the gitignored `private/` files included. Task 3.1 depends on this.
|
||||
- `.gitignore`: `benchmarks/unseen/private/`.
|
||||
- Evidence: the guard is green with an empty `benchmarks/unseen/`, and red when a test fixture names a pool
|
||||
repository in `benchmarks/measurements/` (shown once in the record).
|
||||
- _Requirements: 1.2, 1.3, 1.4_
|
||||
- [x] 1.4 Usage ledger
|
||||
- `benchmarks/unseen/ledger.py`: append-only `usage-pN.jsonl`, one row per use (slice, decision or experiment
|
||||
id, purpose `baseline | exploratory | informed | qualifies`, date, result digest, freeze digest). It refuses a
|
||||
`qualifies` row for a change whose slice already has an `informed` row for it, and refuses to rewrite rows.
|
||||
- `tests/test_unseen_ledger.py`: append, the refusal, a rewritten row detected by a digest chain.
|
||||
- _Requirements: 5.2_
|
||||
- [x] 1.5 Used-set measurement and record 01 **(coordinator, network)**
|
||||
- Run `eligibility.py` against the real sources, the S3 memory store included (read only).
|
||||
- Instrument: each source reports files read and bytes; a source with zero rows fails the run unless it is
|
||||
known empty.
|
||||
- `benchmarks/measurements/<date>-unseen-01-used-set/record.json`: counts per source, used-set digest, guard
|
||||
result (green on the empty pool), test counts of 1.1-1.4.
|
||||
- _Requirements: 1.1, 1.3_
|
||||
|
||||
- [ ] 2. Sourcing and extraction (design 1.1, 1.3, 1.4, 2, 3; increment 2)
|
||||
- [x] 2.1 Advisory candidates and repository filters
|
||||
- `benchmarks/unseen/source.py`: from GHSA advisory JSON (pip, npm, maven, composer; published 2019 or later)
|
||||
keep those whose CWE maps to a family through `src/openultrasast/model/taxonomy.py` and that link one fix
|
||||
commit with one parent in one repository; attach the OSV first affected version; flag `post_cutoff` when
|
||||
published after 2026-06-01. Repository filters: anonymous HTTPS, at most 500 MB, licence at the pinned head
|
||||
(LICENSE file, then the API's SPDX id) classed `permissive` (the design 1.4 list) or `private`.
|
||||
- `tests/test_unseen_source.py` on recorded fixture JSON (no network): CWE mapping, multi-parent and
|
||||
multi-repository fixes dropped, cutoff flag, licence classing (GPL, AGPL, LGPL, none and source-available go
|
||||
private).
|
||||
- _Requirements: 1.4, 2.1_
|
||||
- [x] 2.2 Vulnerability-introducing range
|
||||
- `benchmarks/unseen/extract.py`: the `introducing` method of design 2.1 (blame of the sink line, or the
|
||||
declaration for an absence bug; absence at V's first parent; no merge or root; at most 50 files and 2,000
|
||||
lines; OSV version bracket when present; walk-back of at most five steps) and the `last_touch` fallback. Each
|
||||
change records method, family, vulnerable function resolved in head, head-side lines, `post_cutoff`, advisory
|
||||
id. Git only; no scanner or engine.
|
||||
- `tests/test_unseen_extract.py` on a synthetic repository built in a temp dir: accepted V; a reformat walked
|
||||
back; a merge rejected; size cap; bracket rejection; fallback chosen when no V passes.
|
||||
- _Requirements: 2.1_
|
||||
- [x] 2.3 Ordinary-commit draw
|
||||
- `extract.py`: first-parent, non-merge commits touching an analysed language; the exclusions of design 2.2
|
||||
(security words of the narrow pattern, reverts, bot and bump commits, manifest- or lock-only, generated or
|
||||
vendored, advisory fix commits, commits touching a known vulnerable function, more than 1,000 source lines,
|
||||
counted). Seeded draw of 12 per repository over three time thirds by four size buckets, proportional, at least
|
||||
one per third; a repository with fewer than 40 eligible commits is dropped.
|
||||
- Tests: each exclusion; the narrow pattern keeps "query", "token", "session"; allocation and the seed
|
||||
reproduce the same draw; the 40-commit floor.
|
||||
- _Requirements: 2.2, 2.3_
|
||||
- [x] 2.4 Draft manifest and slices
|
||||
- `benchmarks/unseen/draft.py`: writes `benchmarks/unseen/draft-p1.toml` (permissive entries by name) and
|
||||
`benchmarks/unseen/private/draft-p1.toml` (the rest); the public file holds an opaque id and
|
||||
sha256(URL, commits) per private entry. Seeded shuffle into three slices of 100 repositories, stratified by
|
||||
ecosystem and `post_cutoff`; each repository has one introducing change and 12 ordinary changes.
|
||||
- Tests: the digest covers private entries; no private URL in the public file; the same seed gives the same
|
||||
slices; per-slice strata within one repository of proportional.
|
||||
- _Requirements: 1.2, 1.4, 2.3, 5.2_
|
||||
- [ ] 2.5 Sourcing run **(coordinator, network)**
|
||||
- `gh api /advisories` enumeration, OSV lookups, canonical resolution against the used set (1.2), blobless
|
||||
clones into a scratch cache, extraction (2.2, 2.3), draft (2.4).
|
||||
- Instrument: clones counted with bytes on disk; a repository whose clone reads zero source files is an
|
||||
instrument failure, not a drop. Fewer than 300 eligible repositories stops the run with counts; the
|
||||
eligibility bars are not relaxed to fill it.
|
||||
- Observable: a draft of 300 repositories (3,900 changes) on disk.
|
||||
- _Requirements: 1.1, 1.4, 2.1, 2.2, 2.3_
|
||||
- [ ] 2.6 Label check and record 02 **(coordinator)**
|
||||
- Every third `introducing` change read by an agent, no model call, marked `confirmed | refactor | unclear`
|
||||
in the draft.
|
||||
- `benchmarks/measurements/<date>-unseen-02-draft/record.json`: candidates per ecosystem, rejections per used
|
||||
source, licence split (public, private), method counts, post-cutoff counts, confirmed share, ordinary-commit
|
||||
exclusion counts. No names.
|
||||
- _Requirements: 2.1, 2.2_
|
||||
|
||||
- [ ] 3. Freeze p1 (design 1.2, 6; increment 3)
|
||||
- [x] 3.1 Guard bisection tool
|
||||
- `benchmarks/unseen/bisect_guard.py`: rewrites the draft with a candidate subset, runs
|
||||
`.venv/bin/pytest -q -p no:cacheprovider tests/test_independent_population.py -k referenced` with stdout and
|
||||
stderr discarded, reads only the exit code, and bisects until green. Reports the dropped count only.
|
||||
- `tests/test_unseen_bisect.py` with an injected fake guard (a hidden set): finds and drops exactly the hidden
|
||||
members; never logs a candidate; an exit code other than 0 or 1 aborts (instrument failure).
|
||||
- _Requirements: 1.1_
|
||||
- [x] 3.2 Freeze writer
|
||||
- `benchmarks/unseen/freeze.py`: canonical `pool-p1.toml` (and the private counterpart) from the bisected draft;
|
||||
`freeze-p1.json` with used-set digest and per-source counts, guard result, dropped count, label-check counts
|
||||
and the freeze digest (sha256 of the canonical manifest). Refuses to write over a frozen pool; a correction
|
||||
is `p2`.
|
||||
- Tests: digest stable under key order; overwrite refused; the guard's digest test (1.3) passes on a fixture
|
||||
pool.
|
||||
- _Requirements: 1.2_
|
||||
- [x] 3.3 Protocol p1
|
||||
- `benchmarks/unseen/protocol-p1.md`: the scoring rules of design section 4 (default flags, decision 4.0's
|
||||
long-deadline arm, catch and location rules, false alarm, coverage states), the statistics of section 3 and
|
||||
7, and the adoption gate of 7.3, stated before any replay.
|
||||
- Observable: the file exists and its sha256 is listed in `freeze-p1.json`.
|
||||
- _Requirements: 3.1, 3.2, 3.3, 3.4, 5.4_
|
||||
- [ ] 3.4 Freeze run and record 03 **(coordinator)**
|
||||
- Run the bisection (3.1), then the freeze (3.2). Guard green, digest test active.
|
||||
- `benchmarks/measurements/<date>-unseen-03-freeze/record.json`: freeze digest, protocol digest, dropped
|
||||
count, final counts per slice and kind.
|
||||
- _Requirements: 1.1, 1.2, 1.3_
|
||||
|
||||
- [ ] 4. Dispatcher (design 5.3; increment 4)
|
||||
- [x] 4.1 Items, workloads and the AX lane
|
||||
- `benchmarks/ax/batch.py`: `Item` (id, input objects, result object name, command, extra env, input digest),
|
||||
`Workload` (image digest, allowed URL env names, deadline, result validator) and the `ax` lane, moved from
|
||||
`benchmarks/learn/engine_trace_ax.py`: manifest render, egress readiness, presigned GET and PUT as `.dat`,
|
||||
apply, poll result object and AX phase, delete, object cleanup. The `kind` lane is the same lane with the
|
||||
kind context.
|
||||
- `tests/test_ax_batch.py` with a fake `ax` CLI and a fake store: manifest holds only the allowed URL env
|
||||
names and no credential; names carry the `ousast-engine-` prefix; cleanup on success and failure.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
- [x] 4.2 Scheduler, fallback and resume
|
||||
- `batch.py`: two AX lanes and one VM `docker` lane by default (one container at a time); sandbox death retried
|
||||
once under a new name, a second death or a JVM OOM goes to the VM fallback queue ahead of overflow;
|
||||
`progress.json` written by the scheduler only; a `STOP` file halts launches; resume skips items whose stored
|
||||
record has the same input digest; a cleanup failure stops dispatching.
|
||||
- Tests with fake lanes: retry then fallback; OOM straight to fallback; resume skips; `STOP`; cleanup failure
|
||||
halts; never more than two AX tasks in flight.
|
||||
- _Requirements: 4.3_
|
||||
- [x] 4.3 Engine pass on the dispatcher
|
||||
- `benchmarks/learn/engine_trace_ax.py` and `engine_trace.run_plan` become an engine `Workload` over `batch.py`;
|
||||
no behaviour change.
|
||||
- Observable: `tests/test_ax_executor.py` and `tests/test_engine_trace.py` pass with their assertions unchanged
|
||||
(moved imports only).
|
||||
- _Requirements: 4.3_
|
||||
- [ ] 4.4 Engine smoke on kube-ax and record 04 **(coordinator, network)**
|
||||
- Two pins through the new dispatcher on kube-ax, compared with their stored records.
|
||||
- `benchmarks/measurements/<date>-unseen-04-dispatcher/record.json`: image digest, per-pin wall time and JVM
|
||||
time, record digests equal to the stored ones (or the diff counted), cleanup result.
|
||||
- _Requirements: 4.3_
|
||||
|
||||
- [ ] 5. Replay task (design 5.1, 5.2; increment 5)
|
||||
- [x] 5.1 Image stage
|
||||
- `plane/Dockerfile.engine-task`: second stage `FROM engine AS replay` adding `git`, the package with its
|
||||
runtime dependencies and `benchmarks/unseen/replay_entry.py`; smoke `ousast pre-push --help`. The root image
|
||||
stays `ghcr.io/norandom/ax-task-runner:v0.3.1`; the engine stage is unchanged.
|
||||
- `tests/test_unseen_image.py`: parses the Dockerfile; the `replay` stage builds from `engine`; the root image
|
||||
pin; the engine stage's text is byte-identical to before.
|
||||
- _Requirements: 4.1_
|
||||
- [x] 5.2 Task entry
|
||||
- `benchmarks/unseen/replay_entry.py`, standard library only, reusing `engine_task_entry.transfer`: GET the spec;
|
||||
`git init` and depth-1 fetches of head and base over HTTPS (180 s bound), verified with `git cat-file -e`; run
|
||||
`ousast pre-push` with the spec's hook flags (defaults, or `--deadline 300` for the long-deadline arm), heap
|
||||
1700 MB and the cluster JVM flags; pack `push.json`, `push.txt`, `engine.json`, `instrument.json` (files and
|
||||
bytes in the checkout, bytes of each changed file, JVM wall time, hook exit code, step timings, image
|
||||
digest); PUT as `.dat`. Any error PUTs a failure record naming the step and status, never URL text.
|
||||
- `tests/test_unseen_replay_entry.py`: a local bare repository and a localhost presign stub; success record
|
||||
fields; a missing commit, a failed fetch and a hook crash each give a failure record; zero bytes read and a
|
||||
replay faster than JVM start are flagged `suspect_fast`.
|
||||
- _Requirements: 3.1, 4.1, 4.2_
|
||||
- [x] 5.3 Replay workload
|
||||
- `benchmarks/unseen/replay.py`: builds `Item`s from a frozen slice and an arm (`default` or `long_deadline`);
|
||||
the spec object carries the repository URL so names stay out of AX status and telemetry; task names
|
||||
`ousast-engine-replay-<slug>-<digest>` with an opaque slug; the validator rejects a record without
|
||||
`instrument.json`; records go to `~/ousast-results/unseen/<pool>/<slice>/<arm>/` and the store.
|
||||
- Tests: no repository name in any task name or manifest; only the two presigned URLs in the env; input
|
||||
digest changes with the arm's flags; `instrument_failure` items queued for one re-run.
|
||||
- _Requirements: 3.1, 4.1, 4.2, 4.3_
|
||||
- [ ] 5.4 Proof on kind, then kube-ax, and record 05 **(coordinator, network)**
|
||||
- Build and push the `replay` target, pinned by digest. Five changes from this repository's own history (never
|
||||
pool changes) on kind, then the same five on kube-ax, then on the VM lane.
|
||||
- Exit: instrument fields show bytes read and realistic JVM time on every lane; quick-tier findings equal
|
||||
across lanes.
|
||||
- `benchmarks/measurements/<date>-unseen-05-replay-task/record.json`: image digest, per-lane timings, bytes,
|
||||
exit codes, the quick-tier comparison.
|
||||
- _Requirements: 4.1, 4.2, 4.5_
|
||||
|
||||
- [x] 6. Scorer (design 3, 4, 7.1, 7.3; increment 6)
|
||||
- [x] 6.1 Coverage states
|
||||
- `benchmarks/unseen/score.py`: each record gets one state (`instrument_failure`, `unanalysable`, `quick_only`,
|
||||
`engine_covered`) from `instrument.json` and the artifact; `suspect_fast` goes to failure; a slice above 5%
|
||||
`instrument_failure` after re-runs is refused.
|
||||
- `tests/test_unseen_score.py` on synthetic records: every state; the refusal; a failure never counts as quiet
|
||||
or missed.
|
||||
- _Requirements: 3.3_
|
||||
- [x] 6.2 Catch and false-alarm rules
|
||||
- `score.py`: family match (engine family, quick-rule CWE through `taxonomy.py`), the location table of design
|
||||
4.1, all three streams with the stream named per catch; false alarm on any finding of any stream; the
|
||||
alerts-only rate.
|
||||
- Tests: right family wrong place misses; `access_control` on the handler; `config_secrets` file scope; a
|
||||
quick-rule-only catch; one advisory finding on an ordinary change is a false alarm.
|
||||
- _Requirements: 3.2_
|
||||
- [x] 6.3 Rates, intervals and breakdowns
|
||||
- `score.py`: Wilson 95% intervals; repository-cluster bootstrap for false alarms; ICC; breakdowns per family,
|
||||
method, `post_cutoff`, stream, coverage state and the confirmed subset.
|
||||
- Tests: the n values of design section 3 (315, 483, 756, 62, 88, 97) reproduced; bootstrap seeded; ICC on a
|
||||
synthetic clustered set.
|
||||
- _Requirements: 2.3, 3.3_
|
||||
- [x] 6.4 Budgets, paired tests, the gate and record 06
|
||||
- `score.py`: per-budget thresholds (1%, 5%, 10%), labelled `exploratory` in-slice and `fixed` when supplied;
|
||||
a binary arm's single operating point with the budgets it meets; exact two-sided McNemar for catch and for
|
||||
false alarm; `gate(A, B)` returning `pass | regression | not_run` per design 7.3.
|
||||
- Tests: thresholds on a synthetic score set; McNemar exact values; a binary arm with a significant
|
||||
false-alarm increase gives `regression`; missing slice or excess failures give `not_run`.
|
||||
- `benchmarks/measurements/<date>-unseen-06-scorer/record.json`: test counts and the digest of the scorer's
|
||||
output on the committed synthetic fixture.
|
||||
- _Requirements: 3.4, 5.4_
|
||||
|
||||
- [ ] 7. Baseline (design 4, 6; increment 7; G0 exit)
|
||||
- [ ] 7.1 Frozen source and dry run **(coordinator, network)**
|
||||
- Export HEAD to a frozen tree; build and push the `replay` image from it, pinned by digest.
|
||||
- `benchmarks/push/python_overhead.py` run with the stub engine to size per-replay Python cost; dispatcher dry
|
||||
run on slice 1 lists 1,300 items without launching.
|
||||
- _Requirements: 5.1, 4.1_
|
||||
- [ ] 7.2 Baseline replay of slice 1 **(coordinator, network)**
|
||||
- Default arm, 1,300 replays over two AX lanes and the VM lane (about 9 hours); `instrument_failure` items
|
||||
re-run once; the run stops for a cause above 5% failures.
|
||||
- Observable: 1,300 records in `~/ousast-results/unseen/p1/1/default/` and the store, each with
|
||||
`instrument.json`.
|
||||
- _Requirements: 3.1, 4.3, 5.1_
|
||||
- [ ] 7.3 Adjudication **(coordinator)**
|
||||
- Flagged ordinary changes adjudicated against protocol-v2's standard (all if 30 or fewer, else every third);
|
||||
the adjudicated share kept beside the false-alarm rate. ICC measured; above 0.05 noted as "next pool sizes
|
||||
up".
|
||||
- _Requirements: 3.2, 2.3_
|
||||
- [ ] 7.4 Record 07 and the ledger **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-07-baseline/record.json`: `score.py` output (counts, rates, Wilson and
|
||||
bootstrap intervals, coverage per state, per family, method, stream), freeze digest, image digest, commit.
|
||||
`usage-p1.jsonl` row `baseline`.
|
||||
- Observable: guard green over the record; G0's exit met.
|
||||
- _Requirements: 5.1, 5.2, 3.3_
|
||||
|
||||
- [ ] 8. Arms on slice 1 (design 4.0, 5.4, 7.1-7.3; increment 8; G1)
|
||||
- [ ] 8.1 Registered long-deadline arm
|
||||
- `plane/experiments/` manifest for the long-deadline arm: hook flags with `--deadline 300`, budget 0,
|
||||
`unseen: {pool: p1, slice: 1}`, committed before its first replay. `src/openultrasast/learn/experiments.py`
|
||||
accepts `unseen` and a hook-flag arm.
|
||||
- `tests/test_learn_experiments.py`: the manifest loads; an `unseen` block without pool or slice is refused.
|
||||
- _Requirements: 3.4, 5.3_
|
||||
- [ ] 8.2 `unseen` block and adoption gate in `analyse`
|
||||
- `experiments.py` `analyse`: writes `unseen` into `result.json` (catch at each budget per arm, paired B - A,
|
||||
McNemar, `gate`) from `score.py`; checks the usage ledger; adoption requires the fold rule and
|
||||
`gate == pass`.
|
||||
- Tests: `regression` and `not_run` each block adoption; an `informed` slice cannot qualify; result JSON shape.
|
||||
- _Requirements: 5.2, 5.3, 5.4_
|
||||
- [ ] 8.3 Engine-evidence arm
|
||||
- `benchmarks/unseen/arms.py`: a score per change from `engine.json` (witness rows on head-side changed
|
||||
functions), no model; plugs into the budget thresholds of 6.4.
|
||||
- Tests on synthetic records: score monotone in witness strength; a change without engine coverage gets no
|
||||
score and is reported, not counted quiet.
|
||||
- _Requirements: 3.4_
|
||||
- [ ] 8.4 Model-arm adapter
|
||||
- `arms.py`: candidates from head-side changed functions of stored records, scored by the current decision
|
||||
engine program (and the combiner when its program exists; else its row reads `not_built`); change score is the
|
||||
maximum; metered client, dry run first, response cache, the manifest's `budget_usd` as the hard ceiling.
|
||||
- Tests with a scripted client: dry-run estimate, ceiling stop, cache replay; no model call.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.5 Long-deadline replay of slice 1 **(coordinator, network)**
|
||||
- 1,300 replays of the `long_deadline` arm, bounded at about 345 s each (up to about 42 hours over three
|
||||
lanes), with `STOP` and resume across sessions; failures re-run once.
|
||||
- Observable: 1,300 records under `~/ousast-results/unseen/p1/1/long_deadline/`, JVM time in each.
|
||||
- _Requirements: 3.1, 3.4, 4.3_
|
||||
- [ ] 8.6 Model arms on slice 1 **(coordinator, network; paid: needs a budget go-ahead)**
|
||||
- Dry run gives the candidate count; estimate at the injection slice's measured cost (about $0.008 per
|
||||
candidate at k = 5); the maintainer sets the ceiling against the balance read just before. Then the decision
|
||||
engine arm (and the combiner if built) on the VM.
|
||||
- _Requirements: 4.4, 3.4_
|
||||
- [ ] 8.7 Record 08: the arms table **(coordinator)**
|
||||
- `benchmarks/measurements/<date>-unseen-08-arms/record.json`: per arm (hook 30 s, hook 300 s, engine evidence,
|
||||
decision engine, combiner or `not_built`) catch at 1/5/10% budgets with Wilson intervals and realised
|
||||
false-alarm rates, coverage per state, spend; `exploratory` rows in `usage-p1.jsonl`.
|
||||
- Observable: one committed table of arms on slice 1; G1's table exists.
|
||||
- _Requirements: 3.4, 5.2, 5.3_
|
||||
|
||||
## Implementation Notes
|
||||
|
||||
- 2026-10-04: Tasks 1.1–1.4 implemented offline and independently reviewed. Task 1.5 remains for the
|
||||
coordinator; no networked measurement or commit was performed. Command and API contracts:
|
||||
`benchmarks/unseen/README.md`. Offline evidence:
|
||||
`benchmarks/measurements/2026-10-04-unseen-01-offline/record.json`.
|
||||
- Population guard scope is preserved under the maintainer's hard rule: benchmarks/src/tests for
|
||||
populations, plus plane for pools. Each directory exemption is reservation-specific, including private
|
||||
pool manifests. Extending population scope to plane exposed 30 existing references; no historical
|
||||
records were changed or exempted inside the existing population scope.
|
||||
- S3 inventory opens through `open_store(read_only=True)` because ordinary store construction writes a
|
||||
Select capability probe. Default store behavior is unchanged; inventory uses LIST/GET only.
|
||||
- The freeze writer must match the canonical TOML digest encoding documented in the pool README and
|
||||
exercised by `test_independent_population.py`; final pool discovery does not trust mutable status.
|
||||
|
||||
### Group 2 implementation evidence (2026-10-04)
|
||||
|
||||
Tasks 2.1–2.4 are implemented offline and independently reviewed. Task 2.5 is
|
||||
`python -m benchmarks.unseen.draft --root "$PWD" --s3-store s3:// --seed 20260601`
|
||||
with the documented PATH/PYTHONPATH and GH_TOKEN environment. No networked draw,
|
||||
label check, group-2 measurement record, guard bisection, or commit was performed.
|
||||
The group header remains pending its coordinator exits 2.5/2.6.
|
||||
|
||||
The command rebuilds the group-1 used set, resolves renamed/fork repositories,
|
||||
uses disposable external blobless clones, re-reads licences at extracted heads,
|
||||
and emits counts/digests only. The README documents injected clients, method bias,
|
||||
the hard 300-repository floor, and an explicitly unmeasured resource estimate.
|
||||
Independent review caught and fixed ambiguous parent-function absence and stale
|
||||
candidate-404 handling; regression tests cover both.
|
||||
|
||||
Validation: requested pytest selection 103 passed, 1 skipped, exit 0 (population-test
|
||||
output discarded); `ruff check .` exit 0; `ruff format --check .` exit 0 (411 files);
|
||||
`mypy src/openultrasast` exit 0 (160 source files). The CLI `--help` smoke exited 0.
|
||||
Source-read instrumentation printed 20,101 bytes for the extraction implementation.
|
||||
|
||||
### Groups 4–6 offline implementation (2026-10-04)
|
||||
|
||||
Tasks 4.1–4.3, 5.1–5.3 and 6.1–6.4 are implemented and independently reviewed.
|
||||
Tasks 4.4 and 5.4 remain coordinator-owned network proofs; group headers 4 and 5 stay
|
||||
pending those exits. No commit, network request, Docker, AX or kubectl command was run.
|
||||
The sourcing job's draft.py, source.py, extract.py and journal.py are unchanged.
|
||||
|
||||
The shared dispatcher retains the engine adapter's historical questions.json object
|
||||
name to preserve its unchanged assertions; replay uses spec.dat and result.dat.
|
||||
The engine Dockerfile body is unchanged except its required AS engine alias; CI builds
|
||||
engine and replay separately. S3Store's read_only class default preserves existing
|
||||
constructor-bypassing transport fixtures; initialized read-only stores still refuse writes.
|
||||
|
||||
Exact coordinator commands and input contracts are in benchmarks/unseen/README.md,
|
||||
including frozen TOML/private pointer verification, scoring joins, own-history proof
|
||||
preparation and anonymous measurement-record generation. JVM wall time is the hook's
|
||||
aggregate scan build/query timing and is labelled as such. Unsupported-only partitions
|
||||
with explicit language_not_covered evidence do not pretend to have launched a JVM.
|
||||
|
||||
Record 06 contains the committed synthetic fixture and scorer-output digests. The
|
||||
reserved-repository guard was run as a black box with both output streams discarded.
|
||||
Independent review: APPROVED; all reported integration and coverage defects resolved.
|
||||
|
||||
Final verification: requested pytest command 279 passed, 8 skipped, exit 0; separate
|
||||
`tests/test_ax_batch.py` 5 passed, exit 0; `ruff check .`, `ruff format --check .`,
|
||||
`mypy src/openultrasast`, and the output-discarded reserved-repository guard all exit 0.
|
||||
The own-history proof preparer and replay dry run read five items and exit 0.
|
||||
@@ -0,0 +1,32 @@
|
||||
# kube-ax operator constraints
|
||||
|
||||
Verified by the operator on 2026-10-05. These constraints apply across all workloads.
|
||||
|
||||
- Two workers, one task per worker, at most two tasks in flight cluster-wide. AX has no
|
||||
queue: a third apply returns `no free workers available`. Treat this as back-pressure,
|
||||
retry the same task/lane with 15–30 seconds of jitter until its deadline, and never
|
||||
count it as a sandbox failure or trigger VM fallback. The host dispatcher shares a
|
||||
two-slot reservation across workload instances and long-lived search executors.
|
||||
- `/` and `/tmp` are RAM-backed and consume the worker's 3 GiB. `/workspace` is the only
|
||||
disk-backed path, with about 2 GiB usable per task. Executor and verifier entrypoints
|
||||
use `/workspace/tmp`, `/workspace/.npm`, `/workspace/.m2`, `/workspace/.composer`,
|
||||
`/workspace/.pip`, and `/workspace/.gradle` through the respective tool environment
|
||||
variables. Builds and extracted checkouts live under `/workspace` too. The default
|
||||
aggregate workspace scratch guard is 2 GiB; `OUSAST_SCRATCH_BYTES` configures it.
|
||||
Exceeding it returns `could_not_build`, reason `scratch limit`. The guard polls
|
||||
during child execution and checks after exit; it is not an instantaneous filesystem
|
||||
quota or a replacement for the worker memory limit.
|
||||
- `ousast-engine-search-verify-*` tasks have no egress. The requested exception for
|
||||
`files.because-security.com:443` is **pending**, not operationally verified. They
|
||||
receive a presigned manifest for a gzip level-6 archive containing `checkout/`,
|
||||
`products/`, and `spec.json`, built/exported by an executor. Compressed archives
|
||||
retain the 1 GiB cap and travel in parts of at most 90 MB, each digest-checked
|
||||
before assembly and extraction under `/workspace`. Only `VERIFY_INPUT_URL` and `RESULT_URL` cross the
|
||||
verifier boundary. Verifiers reject build recipes and do not install dependencies
|
||||
or fetch repositories. Executors finish and release their slots before verifier
|
||||
repetitions start. Both task types must be provisioned without model credentials.
|
||||
- All tasks share one public IP. Repository acquisition uses HTTPS git clone from
|
||||
`github.com` or archives from `codeload.github.com`, never `api.github.com`. Host-side
|
||||
metadata queries are a separate concern. Search tasks receive checkout archives;
|
||||
executor tool guidance carries the same acquisition rule. Retain the 60-second
|
||||
first-download retry for delayed task egress policy activation.
|
||||
@@ -0,0 +1,60 @@
|
||||
# Operator notifications (Discord)
|
||||
|
||||
Recorded 2026-10-09 from the maintainer's direction. OpenUltraSAST notifies the operator on
|
||||
Discord about the status and results of its own processes, and especially when a process is
|
||||
**stuck or otherwise impaired**. This is an operator channel, not a developer or infrastructure
|
||||
one.
|
||||
|
||||
## Audience
|
||||
|
||||
The **system operator** and the **AI operator** who run and oversee the tool. Not developers
|
||||
debugging the code, and not an infrastructure on-call. The messages answer "is the work moving,
|
||||
did it finish, what did it produce, is it stuck" for someone supervising a run, not "why does this
|
||||
function throw".
|
||||
|
||||
## Scope: OpenUltraSAST processes and their results
|
||||
|
||||
Notify on the actual OpenUltraSAST workloads and what they produce:
|
||||
|
||||
- scans and pre-push checks (a completed check and its verdict/degradations),
|
||||
- unseen-repo sourcing / draft, freeze, and replay / baseline runs,
|
||||
- learning and evaluation runs (compile, evaluate, experiments) and their recorded results,
|
||||
- fusion / adjudication and any long-running job.
|
||||
|
||||
**Not** k3s or ax cluster infrastructure. Cluster, node, and ax-task health is a separate
|
||||
infrastructure concern monitored elsewhere; this channel does not carry it. The line is: a message
|
||||
here is about an OpenUltraSAST process or result, never about the plane's plumbing.
|
||||
|
||||
## When to notify
|
||||
|
||||
- **Completion:** a process finished, with a one-line result summary (counts, status, digest).
|
||||
- **Stuck or impaired (the important case):** a stall, a clean-but-short result
|
||||
(`insufficient_*`), an abort or `instrument_failure`, a timeout or deadline hit, budget
|
||||
exhaustion, or a resource limit (disk, memory). These @mention the operator so a supervised run
|
||||
does not sit silently broken.
|
||||
|
||||
Advisory and low-volume by design: a start/finish and the impaired cases, not per-step chatter.
|
||||
|
||||
## Transport and secrets
|
||||
|
||||
A Discord webhook, with credentials referenced by name only and stored in gitignored `.env`, like
|
||||
every other credential (`.env.example` lists the names, blank):
|
||||
|
||||
- `DISCORD_WEBHOOK_URL` — the operator channel webhook.
|
||||
- `DISCORD_OPERATOR_ID` — the account to @mention on a stuck/impaired alert.
|
||||
|
||||
The webhook URL is a secret (anyone holding it can post to the channel): it never enters a tracked
|
||||
file, a prompt, a log, or a commit, and it is rotated if exposed. Notifications are **off** when the
|
||||
webhook is unset, so the tool runs the same with or without it.
|
||||
|
||||
## Content discipline
|
||||
|
||||
Messages carry status, counts, and digests only, under the same redaction as the hardening and
|
||||
measurement-record conventions: no exploit payloads or request strings, no secrets, no customer
|
||||
content, no repository contents. A notification is safe to read in a shared channel.
|
||||
|
||||
## Status
|
||||
|
||||
Policy recorded; the notifier (reading the env-named webhook, quiet when unset, honoring the
|
||||
redaction above) is to be wired into the process entry points and may be specced separately. Until
|
||||
then this steering states the intended behavior for any process that adds notifications.
|
||||
@@ -1,8 +1,23 @@
|
||||
# Overview — what this is, and what is actually true of it
|
||||
|
||||
## Current state, 2026-10-02
|
||||
## Current state, 2026-10-04
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins. The
|
||||
direction lives in `roadmap.md`, section of 2026-10-04 (phases G0-G5).
|
||||
|
||||
- **Goal.** A pre-push safety net that works on repositories it has never seen. Two numbers per push decide it:
|
||||
catch rate on real vulnerability-introducing changes, and false-alarm rate on ordinary changes. Neither has been
|
||||
measured yet; the unseen-repository pool (G0, spec `unseen-repo-evaluation`) comes first.
|
||||
- **Correction, 2026-10-04.** Infrastructure and engine work had drifted from that goal. It is now done only when
|
||||
it removes a bottleneck for those measurements.
|
||||
- **Engine.** Joern runs on the operator's kube-ax cluster as AX Tasks, with a VM fallback lane; kind stays the
|
||||
development profile. It is almost blind to fixes: paths differ between sides in 2 of 105 both-asked pairs.
|
||||
- **Decision engine.** Ranks but cannot block (exp-003 to exp-005 inconclusive; 710 fix pairs in memory). Fix
|
||||
mechanisms is one G2 arm, paused after mining until the pool exists.
|
||||
- **Hook.** `ousast pre-push` is operational and advisory on any repository; rollout stays NO-GO.
|
||||
|
||||
## State, 2026-10-02 (superseded by the section above)
|
||||
|
||||
Everything below this section is dated history; where it disagrees with this section, this section wins.
|
||||
User-facing documentation: `README.md`, `docs/architecture.md`, `docs/evaluation.md`, `docs/decision-engine.md`,
|
||||
`ops/ax/README.md`.
|
||||
|
||||
@@ -200,10 +215,10 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
|
||||
|---|---|---|
|
||||
| `learned-decision-engine` | implementation | 11 of 17 tasks done |
|
||||
| `pre-push-safety-net` | implementation | 67 of 79 tasks done |
|
||||
| `contributor-scan` | implementation | 27 of 51 tasks done |
|
||||
| `ai-service-plane` | implementation | 11 of 11 tasks done; the ax plane is the agentic path |
|
||||
| `contributor-scan` | paused | 27 of 51 tasks done; paused since 2026-09-23 (`v1.2.0-alpha.1` NO-GO), waits on pre-push M2/M3 |
|
||||
| `ai-service-plane` | complete | 11 of 11; the ax plane is the agentic path |
|
||||
| `harnessx-removal` | complete | 8 of 8; HarnessX removed 2026-09-30 |
|
||||
| `finding-feedback-loop` | design | requirements approved, design awaiting approval |
|
||||
| `finding-feedback-loop` | dormant | requirements approved, design unapproved since 2026-09-09; depends on `contributor-scan` |
|
||||
| `model-grounded-detection` | complete | 16 of 16; built the arbiter this all rests on |
|
||||
| `authorization-obligations` | dormant | tasks approved, 0 of 24 started; superseded in practice by the dominance arbiter |
|
||||
| `reachability-flow-model` | dormant | 23 tasks generated, nothing approved; largely delivered by adopting Joern |
|
||||
@@ -214,6 +229,5 @@ Counted 2026-10-02 from each spec's `tasks.md` checkboxes and `spec.json`.
|
||||
> **Retired 2026-09-30:** HarnessX was removed by `harnessx-removal`; the agentic plane is ax (`ai-service-plane`),
|
||||
> and evolutionary proposals come from the plane memory. The HarnessX specs in the table above are history.
|
||||
|
||||
As of 2026-09-09: 97 source modules, 90 load-bearing and 7 standalone capabilities; 814 tests.
|
||||
150 source modules, 145 load-bearing and 5 standalone capabilities, 0 orphaned
|
||||
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 1,964 tests collected.
|
||||
151 source modules, 146 load-bearing and 5 standalone capabilities, 0 orphaned
|
||||
(`benchmarks/measurements/2026-09-08-module-audit.json`, regenerated 2026-10-02); 2,023 tests collected.
|
||||
|
||||
@@ -1,5 +1,143 @@
|
||||
# Roadmap
|
||||
|
||||
## Roadmap to detection with the decision engine on kube-ax (2026-10-04)
|
||||
|
||||
This section supersedes the ordering below it. The older sections stay as history.
|
||||
|
||||
### Destination
|
||||
|
||||
On every push, from a hook or CI, OpenUltraSAST answers for the changed code. The answer is advisory for every
|
||||
family, and BLOCK only for families whose precision has been proven on repositories the system never saw. The work
|
||||
is split by what each place is good at:
|
||||
|
||||
- **kube-ax cluster** (operator k3s, Agent Substrate v0.3.0, AX v0.3.1, 2 nodes): the Joern engine, as AX Tasks.
|
||||
No API keys and no store credentials in the cluster; tasks read and write through short-lived presigned links.
|
||||
- **The VM or CI runner**: candidate selection, the decision engine's model calls (DeepSeek), the memory store,
|
||||
and the fallback engine lane (2.5 GB heap) for pins that do not fit the cluster's 1.7 GB heap.
|
||||
- **kind** stays the local development profile, with the same code and manifests.
|
||||
|
||||
### The question everything answers
|
||||
|
||||
Does the safety net work on repositories it has never seen? That is the purpose of the A/B experiments, and every
|
||||
phase below is judged by it. On an unseen repository almost every push is an ordinary change, so two numbers decide
|
||||
whether the safety net is useful, measured on the hook's advisory output exactly as a user would see it:
|
||||
|
||||
1. **Catch rate:** on real vulnerability-introducing changes in unseen repositories, how often it flags the change.
|
||||
2. **False-alarm rate:** on ordinary changes in the same unseen repositories, how often it flags anything.
|
||||
|
||||
Within-pair AUC on known fix pairs answers neither. It stays a development signal, not the goal.
|
||||
|
||||
### Where we are (2026-10-04)
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Generalisation measured so far | The classifier ranks above chance on repository-held-out folds (within-pair AUC about 0.76) but cannot reach the block bar; the engine-based scan failed fresh populations (v1 1/11, v2 0/17); push-level catch and false-alarm rates on unseen repositories have never been measured | exp-003/004/005, `results-v1.json`, `results-v2.json` |
|
||||
| Pre-push hook | Operational and advisory on any repository | `2026-10-02-pre-push-operational-1/record.json` |
|
||||
| Labelled data | 710 complete real fix pairs | `2026-10-04-decision-engine-harvest-3/record.json` |
|
||||
| Engine | Almost blind to fixes (paths differ between sides in 2 of 105 both-asked pairs) | `2026-10-04-joern-slice-leak-audit/record.json` |
|
||||
| Infrastructure | Engine runs on kube-ax as AX Tasks with a VM fallback; kind for development | memory `kube-ax-cluster` |
|
||||
|
||||
Correction recorded 2026-10-04: the previous two days went mostly into infrastructure and engine internals that did
|
||||
not move a generalisation number. From here, infrastructure and engine work is done only when it removes a
|
||||
bottleneck for the measurements below.
|
||||
|
||||
### Where we are (2026-10-11)
|
||||
|
||||
The G0 measurement is now runnable: the unseen pool is **frozen** and the whole pipeline is hardened and
|
||||
**bucket-first**. The baseline replay is the one remaining step, to run on kube.
|
||||
|
||||
| Area | State | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Unseen pool (G0.1) | **Frozen.** `pool-p1.toml` = 100 repos (slice 1); `freeze-p1.json` guard=0, dropped=0; deterministic (manifest `6bd2fa5d`). License 49 permissive / 51 private; methods 81 introducing / 19 last_touch. | `benchmarks/unseen/pool-p1.toml`; archived `ousast-corpus/unseen/20260601/pool/` |
|
||||
| Labels | 100 machine-derived (SZZ), **unchecked**. Record 07 is a **machine-label baseline** (caveat stated); a human label pass is a later refinement, before any published claim. | `freeze-p1.json` label_check_counts |
|
||||
| Harvest | Rebuilt to converge. Pure GitHub-API harvesting is a dead end for this benchmark (it needs the SZZ introducing commit + 12 stratified history negatives, not one commit). The fix: a full-objects clone (drop `--filter=blob:none`, which caused ~600 per-blob lazy fetches) + bucket-first durability + a get_repo cache that paid the 4,499-name used-set canonicalization once. Yield ~2.2%; **GitHub-rate-limit bound, not CPU**. | 95bed70, 857263d, 21aebe8, 7b711a6 |
|
||||
| Persistence | **Bucket-first, not local disk.** `ousast-corpus` is the source of truth: enumeration, resolution, get_repo, and accept/reject decisions all stream to and resume from it. The host disk (~9-13 GB free) is the scarce resource; RustFS/S3 has the room, so run artifacts and **replay outputs go to RustFS too**, not `~/ousast-results` on local disk. | `storage-streaming.md`, `bank.py` |
|
||||
| Replay image | Provenance pinned. CI `engine-image.yml` builds `ousast-replay-task` (harness with `/app/benchmarks/unseen/replay_entry.py`) + `ousast-engine-task` (Joern hook), publicly pullable by digest. The old `ousast-runner:dev` (248 MB ax-runner, no `/app/benchmarks`, no Joern) is the **wrong** image for replay. | CI run 38116281676 |
|
||||
| Baseline (G0.2) | **Not yet.** Replay reads the frozen pool cleanly (dry-run = 1,300 changes, `freeze_digest` verified). Remaining: mirror CI's passing `replay` job wiring, and stand up the kube lane (`ax` binary + a public-HTTPS presign-capable store endpoint; GHCR pull already public). | CI `replay` job (passed) |
|
||||
|
||||
Lesson, recorded so it is not re-paid: sourcing and engine-image work cost real time. The durable takeaway now lives
|
||||
in the pipeline and these docs — bucket-first persistence, pinned image provenance, a measured yield — so the next
|
||||
pool or arm is cheap, not a rediscovery.
|
||||
|
||||
### Readiness plan (to G0 exit = record 07)
|
||||
|
||||
1. **Kube replay lane (infra, operator-owned):** install the `ax` binary; provide the store endpoint the AX lane
|
||||
needs. The only thing settled in-repo is a code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS
|
||||
DNS, presign-capable S3 endpoint. How that endpoint and the cluster's access to the store are provided is an
|
||||
operator/AX decision, not an assumption to make here. GHCR pull of the images is verified public.
|
||||
2. **Unblock the replay** by mirroring CI's passing `replay` job (replay-task + engine-task wired, store + network).
|
||||
3. **Run slice-1 baseline on kube:** `--lane ax --arm default`, 1,300 changes, no model cost, resumable; outputs to
|
||||
RustFS.
|
||||
4. **Adjudicate (7.3) -> record 07 (7.4)** = the measured machine-label unseen baseline = **G0 exit**.
|
||||
5. Then **G1 arms** (DeepSeek decision engine) on the frozen pool.
|
||||
|
||||
### Phases
|
||||
|
||||
**G0. The unseen-repository pool and the push-level metric** (new spec `unseen-repo-evaluation`)
|
||||
|
||||
1. A frozen pool of repositories never used for training, tuning, mining or harvesting, and not in any population.
|
||||
Two kinds of change per repository: real vulnerability-introducing commits (the parent of a fix, post-cutoff where
|
||||
possible) and ordinary commits from the same history.
|
||||
2. The metric: replay each change through `ousast pre-push` (and the decision engine where an arm uses it) and
|
||||
record catch rate and false-alarm rate, with confidence intervals, per family and overall.
|
||||
3. Rotation: the pool is spent per decision, like the populations; a new slice is drawn for the next one.
|
||||
|
||||
Exit: the pool is frozen and the current production hook has a measured baseline on it.
|
||||
|
||||
**G1. Re-score the arms we have on the pool**
|
||||
|
||||
Current hook (quick rules + engine), decision-engine advisory (current programs), combiner, engine evidence. Each is
|
||||
an arm with a registered decision rule, scored by catch rate at a fixed false-alarm budget.
|
||||
|
||||
Exit: one measured table of arms on unseen code; keep only what generalises.
|
||||
|
||||
**G2. Improve what moves the unseen-repo numbers**
|
||||
|
||||
Candidates, each admitted only if it improves the pool metric on a fresh slice: fix mechanisms in the engine
|
||||
(approved plan, now one arm, not the spine), engine coverage and TypeScript, decision-engine prompt and retrieval
|
||||
changes, more labelled pairs from more distinct repositories.
|
||||
|
||||
Active G2 workstreams (2026-10-11, opened while driving G0; progress tracked in their briefs):
|
||||
|
||||
- **Change-scoped engine analysis** (`.kiro/specs/engine-change-scoped-analysis/brief.md`). Live diagnosis on the
|
||||
frozen pool showed the engine's "no finding" on unseen changes is usually *analysis-never-runs*: the engine ranks
|
||||
repo-wide candidates and `deferred: deadline_exhausted` before analyzing anything (even at 300 s), and the ranked
|
||||
sites are unrelated to the pushed change. Lever: seed analysis from the change's declarations + dataflow
|
||||
neighbourhood (`rank.py`/`mapping.py`) so it runs within budget. Increment 0 (confirm CPG-build-vs-analysis cost
|
||||
split) gates the rest. Scale measurement needs the kube lane.
|
||||
- **Line-correspondence (done, necessary-not-sufficient).** `snapshot.py` discarded base↔head anchors on
|
||||
non-globally-unique lines → `comparison_unknown` → suppressed findings; fixed to run-aware anchoring
|
||||
(commit c39fbb8, 163 tests). Removes one cause; coverage still needs change-scoped analysis + the capability
|
||||
registry (Task 8.3).
|
||||
|
||||
**G3. Decide BLOCK per family** on the pool's false-alarm rate and the pooled precision bound (Req 6.4).
|
||||
|
||||
**G4. Detection on push for any repository** (`plane-on-kubernetes` P4 work: `ousast plane scan --base --head`,
|
||||
engine pins on kube-ax, model calls on the VM or CI), sized by what G1-G3 kept.
|
||||
|
||||
**G5. Qualification and rollout:** adopt, commit `prediction-v3.json`, spend population v3 once, advisory rollout,
|
||||
BLOCK only for families that passed.
|
||||
|
||||
### Cross-cutting rules
|
||||
|
||||
- No hand-tuning per miss. Every detection change is a registered experiment or an admission by held-out
|
||||
separation.
|
||||
- Records carry counts, digests and pattern names, never repository names: the reserved-repository guard covers
|
||||
every population.
|
||||
- Instrument first: a zero from an unread input is a failed run (AGENTS.md).
|
||||
- Cluster stability: sandbox deaths are retried once, then sent to the VM lane; the operator owns workers, egress
|
||||
and Substrate.
|
||||
|
||||
### Spec map
|
||||
|
||||
| Spec | Role in this roadmap | State |
|
||||
| --- | --- | --- |
|
||||
| `unseen-repo-evaluation` (new) | G0, G1: the pool and the push-level metric | To specify |
|
||||
| `learned-decision-engine` | G1-G3, G5; fix-mechanisms plan is one G2 arm | Implementation; experiments 3-5 recorded |
|
||||
| `plane-on-kubernetes` | G4; the engine executors for kube-ax | Groups 1-2 done (kind); production path re-scoped to the operator's kube-ax (AX Tasks), groups 3-8 to re-plan |
|
||||
| `pre-push-safety-net` | the hook replayed in G0-G1; M4-M5 in G5 | Operational increment 1 done; M4-M5 open |
|
||||
| `ai-service-plane` | The plane the above runs on | Complete |
|
||||
|
||||
## Status, 2026-10-02
|
||||
|
||||
The milestone table below dates from 2026-09-14. Since then (`.kiro/specs/pre-push-safety-net/release-milestones.md`):
|
||||
|
||||
@@ -142,3 +142,19 @@ promising but does not qualify security queries or hook latency. The contributor
|
||||
vendor/correspondence completeness vetoes, and query-to-operation identity as the next
|
||||
shared-contract work. Resolving a boundary requires evidence of its relevance and scope;
|
||||
do not simply drop completeness checks to increase admissions. Rollout remains NO-GO.
|
||||
|
||||
On 2026-10-07, the **search-with-proof** arm reached its first milestone: an autonomous,
|
||||
execution-backed `demonstrated` finding end to end on the kube-ax cluster. On a real CVE
|
||||
fix-pair (hanxi/xiaomusic, path traversal of the prefix-check-bypass class), the agent
|
||||
found the vulnerability, authored a declarative demo, and the differential verifier
|
||||
confirmed the owned effect on the vulnerable revision and its absence on the fixed one;
|
||||
a blind search of the fixed revision produced zero false proofs. This is the proof-based
|
||||
BLOCK mechanism the maintainer directed after classification hit a precision wall: a BLOCK
|
||||
rests on a demonstrated finding with a direct execution trace, not a confidence score.
|
||||
Scope note: this is feasibility-pilot evidence on a development-corpus pair (known location,
|
||||
labelled family, runnable-project selection bias), not release readiness. The pilot exit
|
||||
(≥5 demonstrations in one family with 0 false proofs, Req 6.2) and the ≥73-negative BLOCK
|
||||
qualification (Req 6.3) remain open, and no capability is enabled. The decision/classification
|
||||
engine is retained as a triage ranker (it could not prove 95% precision as a gatekeeper);
|
||||
proof is the BLOCK gate. See `search-with-proof/HANDOFF.md` and
|
||||
`benchmarks/measurements/2026-10-07-search-first-cluster-demonstration/`.
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# Stream run artifacts to RustFS, one bucket per purpose
|
||||
|
||||
Recorded 2026-10-09 from the maintainer's direction. The host's local disk is the scarce
|
||||
resource (it runs near full, and the draw stops itself at under 2 GB free). Runs should stream
|
||||
their durable artifacts to RustFS instead of accumulating them on local disk, and should spread
|
||||
them across **purpose-specific buckets** rather than one shared bucket. This relieves the disk and
|
||||
simplifies runs: local state becomes mostly a scratch cache that can be rebuilt or resumed from the
|
||||
object store.
|
||||
|
||||
## Why per-purpose buckets
|
||||
|
||||
One shared bucket (`sast-memory`) already bit us: an append-rewritten key on a versioned bucket
|
||||
with no noncurrent-version expiry ballooned to >10 GB (see the `engine-queue` command-upload fix and
|
||||
the lifecycle rules added 2026-10-09). Separate buckets let each kind of artifact carry the
|
||||
lifecycle it actually needs, and keep a transient firehose from polluting durable data.
|
||||
|
||||
Proposed split (names illustrative; the admin creates them and grants the store key):
|
||||
|
||||
- `sast-memory` — the plane memory store (existing): facts, excerpts, embeddings, index. Durable,
|
||||
versioned, with the noncurrent-version lifecycle now in place.
|
||||
- `sast-pools` — frozen unseen pools and their provenance: `pool-pN.toml`, `freeze-pN.json`, and the
|
||||
draw journal needed to resume/reproduce. Durable, versioned. This is also the "don't redo the
|
||||
sourcing" archive.
|
||||
- `sast-runs` (or `sast-journals`) — ephemeral run and queue state (engine-queue, replay results,
|
||||
per-run journals). Aggressive lifecycle: short current expiry + 1-day noncurrent expiry, since it
|
||||
is a transient firehose.
|
||||
- optionally `sast-traces` — engine/replay traces, if kept.
|
||||
|
||||
## Principles
|
||||
|
||||
- **Stream, don't hoard.** Write a run's durable outputs to its bucket as they are produced; keep
|
||||
only a working scratch (e.g. the live git clone, which cannot be streamed) on local disk, and let
|
||||
the disk guard stay as the backstop.
|
||||
- **Lifecycle per bucket**, tuned to the artifact: durable+versioned for pools/memory;
|
||||
short+noncurrent-expiry for transient run/queue state. Never a versioned bucket without a
|
||||
noncurrent-version rule.
|
||||
- **Resume from the object store**, so a wiped or small local disk does not lose a long run.
|
||||
- **Least privilege per bucket.** A run gets a key scoped to the bucket it writes; the store never
|
||||
creates or configures a bucket (admin does that once), consistent with the current S3Store.
|
||||
- Content discipline unchanged: no secrets, no exploit payloads; pool manifests keep the existing
|
||||
public/private (licence) split.
|
||||
|
||||
## Foundation and dependency
|
||||
|
||||
`src/openultrasast/plane/memory.py` already speaks boto3 to RustFS (S3Store) with startup
|
||||
verification of versioning/lifecycle/Select/tags; this direction extends that pattern to more
|
||||
buckets rather than inventing a new transport. New buckets and their grants are an **operator
|
||||
admin** action — the store credentials cannot create buckets or set lifecycle (confirmed by the
|
||||
`AccessDenied` on the 2026-10-09 lifecycle apply).
|
||||
|
||||
## Default: RustFS over local disk
|
||||
|
||||
Local disk on the VM is the scarce resource (it runs near full; the draw stops itself under 2 GB free).
|
||||
RustFS/S3 has far more room. So the default for any run's durable state and large artifacts is the bucket,
|
||||
not local disk. Local files are a rebuildable scratch cache only (the live git clone, a within-process
|
||||
journal). A long run must be resumable purely from the bucket, and its outputs — including replay result
|
||||
records — are written to RustFS, not `~/ousast-results` on local disk.
|
||||
|
||||
## Status (implemented 2026-10-10/11)
|
||||
|
||||
Bucket-first persistence is live, not a future spec. The unseen draw streams every stage to the
|
||||
`ousast-corpus` bucket (`http://10.0.0.54:9001`, creds in `.env` as `CORPUS_S3_*`; boto3 needs path-style +
|
||||
SigV4) and resumes from it:
|
||||
|
||||
- `unseen/<seed>/decisions/<sha256(repo+fix)>.json` — accept/reject per repository (idempotent; content dedup).
|
||||
- `unseen/<seed>/resolved/<sha256(ghsa)>.json` — per-advisory resolution.
|
||||
- `unseen/<seed>/advisories/list.json` — the enumerated advisory list.
|
||||
- `unseen/repos/<sha256(name)>.json` — get_repo cache, **not** seed-scoped (shared across seeds/runs), so the
|
||||
thousands of used-set canonicalization lookups are paid once ever.
|
||||
- `unseen/<seed>/pool/` — the frozen pool and its provenance (`pool-p1.toml`, `private-pool-p1.toml`,
|
||||
`freeze-p1.json`, `bisect-p1.json`, `protocol-p1.md`): the durable "don't redo the sourcing" archive.
|
||||
|
||||
Implementation: `benchmarks/unseen/bank.py` (`CorpusBank`, `corpus_settings`). The old local journal is a
|
||||
within-process cursor only; the bucket is the cross-run source of truth, so a restart (even after a code
|
||||
change) resumes cheaply. Still to do and consistent with the default above: replay result records
|
||||
(`replay.py --out`) currently default to `~/ousast-results/unseen/...` on local disk — point them at RustFS
|
||||
for the slice-1 baseline and after. The AX (kube) lane already uses presigned object transfer, which is the
|
||||
same principle. One in-repo code fact: `benchmarks/ax/batch.py` currently expects a public-HTTPS DNS,
|
||||
presign-capable endpoint. How the store endpoint and the cluster's access to it are provided is operator/AX-owned
|
||||
infra, not something this doc should presume.
|
||||
@@ -1,165 +0,0 @@
|
||||
---
|
||||
name: kiro-spec-batch
|
||||
description: Create complete specs (requirements, design, tasks) for all features in roadmap.md using parallel sub-agent dispatch by dependency wave.
|
||||
---
|
||||
|
||||
|
||||
# Spec Batch
|
||||
|
||||
<background_information>
|
||||
- **Success Criteria**:
|
||||
- All features have complete spec files (spec.json, requirements.md, design.md, tasks.md)
|
||||
- Dependency ordering respected (upstream specs complete before downstream)
|
||||
- Independent features processed in parallel via sub-agent dispatch
|
||||
- Cross-spec consistency verified (data models, interfaces, naming)
|
||||
- Mixed roadmap context understood without breaking `## Specs (dependency order)` parsing
|
||||
- Controller context stays lightweight (sub-agents do the heavy work)
|
||||
</background_information>
|
||||
|
||||
<instructions>
|
||||
|
||||
## Step 1: Read Roadmap and Validate
|
||||
|
||||
1. Read `.kiro/steering/roadmap.md`
|
||||
2. Parse the `## Specs (dependency order)` section to extract:
|
||||
- Feature names
|
||||
- One-line descriptions
|
||||
- Dependencies for each feature
|
||||
- Completion status (`[x]` = done, `[ ]` = pending)
|
||||
3. If present, also read for context:
|
||||
- `## Existing Spec Updates`
|
||||
- `## Direct Implementation Candidates`
|
||||
Do not include these in dependency-wave execution; they are awareness-only inputs for sequencing and consistency review.
|
||||
4. For each pending feature in `## Specs (dependency order)`, verify `.kiro/specs/<feature>/brief.md` exists
|
||||
5. If any brief.md is missing, stop and report: "Missing brief.md for: [list]. Run `/kiro-discovery` to generate briefs first."
|
||||
|
||||
## Step 2: Build Dependency Waves
|
||||
|
||||
Group pending features into waves based on dependencies:
|
||||
|
||||
- **Wave 1**: Features with no dependencies (or all dependencies already completed `[x]`)
|
||||
- **Wave 2**: Features whose dependencies are all in Wave 1 or already completed
|
||||
- **Wave N**: Features whose dependencies are all in earlier waves or already completed
|
||||
|
||||
Display the execution plan:
|
||||
```
|
||||
Spec Batch Plan:
|
||||
Wave 1 (parallel): app-foundation
|
||||
Wave 2 (parallel): block-editor, page-management
|
||||
Wave 3 (parallel): sidebar-navigation, database-views
|
||||
Wave 4 (parallel): cli-integration
|
||||
Total: 6 specs across 4 waves
|
||||
```
|
||||
|
||||
If roadmap contains `## Existing Spec Updates` or `## Direct Implementation Candidates`, mention them separately as non-batch items so the user can see the whole decomposition.
|
||||
|
||||
## Step 3: Execute Waves
|
||||
|
||||
For each wave, dispatch all features in the wave as **parallel sub-agents**.
|
||||
|
||||
**For each feature in the wave**, spawn a sub-agent with this task:
|
||||
|
||||
```
|
||||
Create a complete specification for feature "{feature-name}".
|
||||
|
||||
1. Read the brief at .kiro/specs/{feature-name}/brief.md for feature context
|
||||
2. Read the roadmap at .kiro/steering/roadmap.md for project context
|
||||
3. Execute the full spec pipeline. For each phase, read the corresponding skill's SKILL.md for complete instructions (templates, rules, review gates):
|
||||
a. Initialize: Read .opencode/skills/kiro-spec-init/SKILL.md, then create spec.json and requirements.md
|
||||
b. Generate requirements: Read .opencode/skills/kiro-spec-requirements/SKILL.md, then follow its steps
|
||||
c. Generate design: Read .opencode/skills/kiro-spec-design/SKILL.md, then follow its steps
|
||||
d. Generate tasks: Read .opencode/skills/kiro-spec-tasks/SKILL.md, then follow its steps
|
||||
4. Set all approvals to true in spec.json (auto-approve mode, equivalent of -y flag)
|
||||
5. Report completion with file list and task count
|
||||
```
|
||||
|
||||
If multi-agent is not available, execute features in the wave sequentially.
|
||||
|
||||
**After all sub-agents in the wave complete**:
|
||||
1. Verify each feature has: spec.json, requirements.md, design.md, tasks.md
|
||||
2. If any feature failed, report the error and continue with features that succeeded
|
||||
3. Display wave completion: "Wave N complete: [features]. Files verified."
|
||||
4. Proceed to next wave
|
||||
|
||||
## Step 4: Cross-Spec Review
|
||||
|
||||
After all waves complete, spawn a **single sub-agent** for cross-spec consistency review. This is the highest-value quality gate -- it catches issues that per-spec review gates cannot.
|
||||
|
||||
**Sub-agent task**:
|
||||
|
||||
Read ALL generated specs and check for consistency across the entire project:
|
||||
- `.kiro/specs/*/design.md` (primary: contains interfaces, data models, architecture)
|
||||
- `.kiro/specs/*/requirements.md` (for scope and acceptance criteria)
|
||||
- `.kiro/specs/*/tasks.md` (for boundary annotations only -- read _Boundary:_ lines, skip task descriptions)
|
||||
- `.kiro/steering/roadmap.md`
|
||||
|
||||
Reading priority: Focus on design.md files (they contain interfaces, data models, architecture). For requirements.md, focus on section headings and acceptance criteria. For tasks.md, focus on _Boundary:_ annotations.
|
||||
|
||||
Check:
|
||||
1. **Data model consistency**: Same entities defined consistently across specs (field names, types, relationships)
|
||||
2. **Interface alignment**: Where spec A outputs what spec B consumes, do contracts match exactly?
|
||||
3. **No duplicate functionality**: Any capability specified in more than one spec?
|
||||
4. **Dependency completeness**: Every design.md references correct upstream specs? Implicit dependencies not in roadmap?
|
||||
5. **Naming conventions**: Component names, file paths, API routes, table names consistent across specs?
|
||||
6. **Shared infrastructure**: Shared concerns (auth, error handling, logging) handled in one spec and correctly referenced?
|
||||
7. **Task boundary alignment**: Task _Boundary:_ annotations partition codebase cleanly? No files claimed by multiple specs?
|
||||
8. **Roadmap boundary continuity**: If roadmap includes `Existing Spec Updates` or `Direct Implementation Candidates`, do the generated new specs avoid absorbing that work by accident?
|
||||
9. **Architecture boundary integrity**: Do the specs preserve clean responsibility seams, avoid shared ownership, keep dependency direction coherent, and include enough revalidation triggers to catch downstream impact?
|
||||
10. **Change-friendly decomposition**: Has any spec absorbed multiple independent seams that should probably be split instead of kept together?
|
||||
|
||||
Output: CONSISTENT areas + ISSUES with (which specs, what's inconsistent, suggested fix).
|
||||
|
||||
**After the review sub-agent returns**:
|
||||
- **Critical/important issues found**: Dispatch fix sub-agents for each affected spec to apply the suggested fixes. If the issue is really a decomposition problem (for example boundary overlap or one spec carrying multiple independent seams), stop and return to roadmap/discovery instead of papering over it locally. Re-run cross-spec review after fixes (max 3 remediation rounds).
|
||||
- **Minor issues only**: Report them for user awareness, proceed to Step 5.
|
||||
- **No issues**: Proceed to Step 5.
|
||||
|
||||
## Step 5: Finalize
|
||||
|
||||
1. Scan `.kiro/specs/*/tasks.md` to verify all specs exist
|
||||
2. For each completed spec, read spec.json to confirm phase and approvals
|
||||
3. Update roadmap.md: mark completed specs as `[x]`
|
||||
4. If roadmap.md includes `Existing Spec Updates` or `Direct Implementation Candidates`, leave them untouched and mention them as remaining follow-up items unless already explicitly completed elsewhere
|
||||
|
||||
Display final summary:
|
||||
```
|
||||
Spec Batch Complete:
|
||||
✓ app-foundation: X requirements, Y design components, Z tasks
|
||||
✓ block-editor: ...
|
||||
✓ page-management: ...
|
||||
...
|
||||
Total: N specs created, M tasks generated
|
||||
Cross-spec review: PASSED / N issues found (M fixed)
|
||||
Existing spec updates pending: <count or none>
|
||||
Direct implementation candidates pending: <count or none>
|
||||
|
||||
Next: Review generated specs, then start implementation with /kiro-impl <feature>
|
||||
```
|
||||
|
||||
</instructions>
|
||||
|
||||
## Critical Constraints
|
||||
- **Controller stays lightweight**: Only read roadmap.md and brief.md existence checks in main context. All spec generation happens in sub-agents.
|
||||
- **Wave ordering is strict**: Never start a wave until all features in previous waves are complete.
|
||||
- **Parallel within waves**: All features in the same wave should be dispatched in parallel if multi-agent is available.
|
||||
- **No partial waves**: If a feature in a wave fails, still complete the other features in that wave before reporting.
|
||||
- **Skip completed specs**: Features with `[x]` in roadmap.md or existing tasks.md are skipped.
|
||||
- **`## Specs (dependency order)` remains authoritative for batch execution**: Other roadmap sections are context, not wave inputs.
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
**Sub-agent failure**:
|
||||
- Log the error, skip the failed feature
|
||||
- Continue with remaining features in the wave
|
||||
- Report failed features in the summary
|
||||
- Suggest: "Run `/kiro-spec-quick <feature> --auto` manually for failed features."
|
||||
|
||||
**Circular dependencies**:
|
||||
- If dependency graph has cycles, report the cycle and stop
|
||||
- Suggest: "Fix dependency ordering in roadmap.md"
|
||||
|
||||
**Roadmap not found**:
|
||||
- Stop and report: "No roadmap.md found. Run `/kiro-discovery` first."
|
||||
|
||||
**All specs already complete**:
|
||||
- Report: "All specs in roadmap.md are already complete. Nothing to do."
|
||||
@@ -1,256 +0,0 @@
|
||||
---
|
||||
name: kiro-spec-quick
|
||||
description: Quick spec generation with interactive or automatic mode
|
||||
---
|
||||
|
||||
# Quick Spec Generator
|
||||
|
||||
<instructions>
|
||||
## CRITICAL: Automatic Mode Execution Rules
|
||||
|
||||
**If `--auto` flag is present in `$ARGUMENTS`, you are in AUTOMATIC MODE.**
|
||||
|
||||
In Automatic Mode:
|
||||
- Execute ALL 4 phases in a continuous loop without stopping
|
||||
- Display progress after each phase (e.g., "Phase 1/4 complete: spec initialized")
|
||||
- IGNORE any "Next Step" messages from Phase 2-4 (they are for standalone usage)
|
||||
- After Phase 4, run the final sanity review before exiting
|
||||
- Stop ONLY after the sanity review completes or if error occurs
|
||||
|
||||
---
|
||||
|
||||
## Core Task
|
||||
Execute 4 spec phases sequentially. In automatic mode, execute all phases without stopping. In interactive mode, prompt user for approval between phases.
|
||||
|
||||
Before claiming quick generation is complete, run one lightweight sanity review over the generated requirements, design, and tasks. If the host supports fresh sub-agents, use one. Otherwise run the sanity review inline.
|
||||
|
||||
## Execution Steps
|
||||
|
||||
### Step 1: Parse Arguments and Initialize
|
||||
|
||||
Parse `$ARGUMENTS`:
|
||||
- If contains `--auto`: **Automatic Mode** (execute all 4 phases)
|
||||
- Otherwise: **Interactive Mode** (prompt at each phase)
|
||||
- Extract description (remove `--auto` flag if present)
|
||||
|
||||
Example:
|
||||
```
|
||||
"User profile with avatar upload --auto" → mode=automatic, description="User profile with avatar upload"
|
||||
"User profile feature" → mode=interactive, description="User profile feature"
|
||||
```
|
||||
|
||||
Display mode banner and proceed to Step 2.
|
||||
|
||||
### Step 2: Execute Phase Loop
|
||||
|
||||
Execute these 4 phases in order:
|
||||
|
||||
---
|
||||
|
||||
#### Phase 1: Initialize Spec (Direct Implementation)
|
||||
|
||||
**Core Logic**:
|
||||
|
||||
1. **Check for Brief**:
|
||||
- If `.kiro/specs/{feature-name}/brief.md` exists (created by `/kiro-discovery`), read it for discovery context (problem, approach, scope, constraints)
|
||||
- Use brief content as the project description instead of `$ARGUMENTS`
|
||||
|
||||
2. **Generate Feature Name**:
|
||||
- Convert description to kebab-case
|
||||
- Example: "User profile with avatar upload" → "user-profile-avatar-upload"
|
||||
- Keep name concise (2-4 words ideally)
|
||||
|
||||
3. **Check Uniqueness**:
|
||||
- Use Glob to check `.kiro/specs/*/`
|
||||
- If directory exists with only `brief.md` (no `spec.json`), use that directory (discovery created it)
|
||||
- Otherwise if feature name exists, append `-2`, `-3`, etc.
|
||||
|
||||
4. **Create Directory**:
|
||||
- Use Bash: `mkdir -p .kiro/specs/{feature-name}` (skip if already exists from discovery)
|
||||
|
||||
5. **Initialize Files from Templates**:
|
||||
|
||||
a. Read templates:
|
||||
```
|
||||
- .kiro/settings/templates/specs/init.json
|
||||
- .kiro/settings/templates/specs/requirements-init.md
|
||||
```
|
||||
|
||||
b. Replace placeholders:
|
||||
```
|
||||
{{FEATURE_NAME}} → feature-name
|
||||
{{TIMESTAMP}} → current ISO 8601 timestamp (use `date -u +"%Y-%m-%dT%H:%M:%SZ"`)
|
||||
{{PROJECT_DESCRIPTION}} → description
|
||||
en → language code (detect from user's input language, default to `en`)
|
||||
```
|
||||
|
||||
c. Write files using Write tool:
|
||||
```
|
||||
- .kiro/specs/{feature-name}/spec.json
|
||||
- .kiro/specs/{feature-name}/requirements.md
|
||||
```
|
||||
|
||||
6. **Output Progress**: "Phase 1/4 complete: Spec initialized at .kiro/specs/{feature-name}/"
|
||||
|
||||
**Automatic Mode**: IMMEDIATELY continue to Phase 2.
|
||||
|
||||
**Interactive Mode**: Prompt "Continue to requirements generation? (yes/no)"
|
||||
- If "no": Stop, show current state
|
||||
- If "yes": Continue to Phase 2
|
||||
|
||||
---
|
||||
|
||||
#### Phase 2: Generate Requirements
|
||||
|
||||
Invoke `/kiro-spec-requirements {feature-name}`.
|
||||
|
||||
Wait for completion. IGNORE any "Next Step" message (it is for standalone usage).
|
||||
|
||||
**Output Progress**: "Phase 2/4 complete: Requirements generated"
|
||||
|
||||
**Automatic Mode**: IMMEDIATELY continue to Phase 3.
|
||||
|
||||
**Interactive Mode**: Prompt "Continue to design generation? (yes/no)"
|
||||
- If "no": Stop, show current state
|
||||
- If "yes": Continue to Phase 3
|
||||
|
||||
---
|
||||
|
||||
#### Phase 3: Generate Design
|
||||
|
||||
Invoke `/kiro-spec-design {feature-name} -y`. The `-y` flag auto-approves requirements.
|
||||
|
||||
Wait for completion. IGNORE any "Next Step" message.
|
||||
|
||||
**Output Progress**: "Phase 3/4 complete: Design generated"
|
||||
|
||||
**Automatic Mode**: IMMEDIATELY continue to Phase 4.
|
||||
|
||||
**Interactive Mode**: Prompt "Continue to tasks generation? (yes/no)"
|
||||
- If "no": Stop, show current state
|
||||
- If "yes": Continue to Phase 4
|
||||
|
||||
---
|
||||
|
||||
#### Phase 4: Generate Tasks
|
||||
|
||||
Invoke `/kiro-spec-tasks {feature-name} -y`. The `-y` flag auto-approves requirements, design, and tasks.
|
||||
|
||||
Wait for completion.
|
||||
|
||||
**Output Progress**: "Phase 4/4 complete: Tasks generated"
|
||||
|
||||
#### Final Sanity Review
|
||||
|
||||
After Phase 4, run a lightweight sanity review before claiming completion.
|
||||
|
||||
- Review `requirements.md`, `design.md`, and `tasks.md` directly from disk. If `brief.md` exists, use it only as supporting context.
|
||||
- Prefer a fresh review sub-agent when the host supports it. Pass only file paths and the review objective; the reviewer should read the generated files itself.
|
||||
- Review focus:
|
||||
- Do requirements, design, and tasks tell a coherent story?
|
||||
- Are there obvious contradictions, missing prerequisites, or missing task coverage for required design work?
|
||||
- Are `_Depends:_`, `_Boundary:_`, and `(P)` markers plausible for implementation?
|
||||
- If the review finds only task-plan-local issues, repair or update the generated `tasks.md` once, then re-run the sanity review.
|
||||
- If the review finds a real requirements/design gap or contradiction, stop and report follow-up instead of claiming the quick spec is implementation-ready.
|
||||
|
||||
**All 4 phases plus sanity review complete.**
|
||||
|
||||
Output final completion summary (see Output Description section) and exit.
|
||||
|
||||
---
|
||||
|
||||
## Important Constraints
|
||||
|
||||
### Error Handling
|
||||
- Any phase failure stops the workflow
|
||||
- Display error and current state
|
||||
- Suggest manual recovery command
|
||||
|
||||
</instructions>
|
||||
|
||||
## Output Description
|
||||
|
||||
### Mode Banners
|
||||
|
||||
**Interactive Mode**:
|
||||
```
|
||||
Quick Spec Generation (Interactive Mode)
|
||||
|
||||
You will be prompted at each phase.
|
||||
Note: Skips gap analysis and design validation.
|
||||
```
|
||||
|
||||
**Automatic Mode**:
|
||||
```
|
||||
Quick Spec Generation (Automatic Mode)
|
||||
|
||||
All phases execute automatically without prompts.
|
||||
Note: Skips optional validations (gap analysis, design review) and user approval prompts. Internal review gates still run.
|
||||
Final sanity review still runs.
|
||||
```
|
||||
|
||||
### Intermediate Output
|
||||
|
||||
After each phase, show brief progress:
|
||||
```
|
||||
Spec initialized at .kiro/specs/{feature}/
|
||||
Requirements generated → Continuing to design...
|
||||
Design generated → Continuing to tasks...
|
||||
```
|
||||
|
||||
### Final Completion Summary
|
||||
|
||||
Provide output in the language specified in `spec.json`:
|
||||
|
||||
```
|
||||
Quick Spec Generation Complete!
|
||||
|
||||
## Generated Files:
|
||||
- .kiro/specs/{feature}/spec.json
|
||||
- .kiro/specs/{feature}/requirements.md ({X} requirements)
|
||||
- .kiro/specs/{feature}/design.md ({Y} components, {Z} endpoints)
|
||||
- .kiro/specs/{feature}/tasks.md ({N} tasks)
|
||||
|
||||
Quick generation skipped:
|
||||
- `/kiro-validate-gap` - Gap analysis (integration check)
|
||||
- `/kiro-validate-design` - Design review (architecture validation)
|
||||
|
||||
Sanity review: PASSED | FOLLOW-UP REQUIRED
|
||||
|
||||
## Next Steps:
|
||||
1. Review generated specs (especially design.md)
|
||||
2. Optional validation:
|
||||
- `/kiro-validate-gap {feature}` - Check integration with existing codebase
|
||||
- `/kiro-validate-design {feature}` - Verify architecture quality
|
||||
3. Start implementation: `/kiro-impl {feature}`
|
||||
|
||||
```
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
### Error Scenarios
|
||||
|
||||
**Template Missing**:
|
||||
- Check `.kiro/settings/templates/specs/` exists
|
||||
- Report specific missing file
|
||||
- Exit with error
|
||||
|
||||
**Directory Creation Failed**:
|
||||
- Check permissions
|
||||
- Report error with path
|
||||
- Exit with error
|
||||
|
||||
**Phase Execution Failed** (Phase 2-4):
|
||||
- Stop workflow
|
||||
- Show current state and completed phases
|
||||
- Suggest: "Continue manually from `/kiro-spec-{next-phase} {feature}`"
|
||||
|
||||
**Sanity Review Failed**:
|
||||
- Stop workflow
|
||||
- Report the exact contradiction, missing prerequisite, or task-plan issue
|
||||
- Suggest targeted follow-up with `/kiro-spec-design {feature}`, `/kiro-spec-tasks {feature}`, or manual edits depending on the finding
|
||||
|
||||
**User Cancellation** (Interactive Mode):
|
||||
- Stop gracefully
|
||||
- Show completed phases
|
||||
- Suggest manual continuation
|
||||
@@ -1,145 +0,0 @@
|
||||
---
|
||||
name: kiro-steering-custom
|
||||
description: Create custom steering documents for specialized project contexts
|
||||
metadata:
|
||||
shared-rules: "steering-principles.md"
|
||||
---
|
||||
|
||||
|
||||
# Kiro Custom Steering Creation
|
||||
|
||||
<background_information>
|
||||
**Role**: Create specialized steering documents beyond core files (product, tech, structure).
|
||||
|
||||
**Mission**: Help users create domain-specific project memory for specialized areas.
|
||||
|
||||
**Success Criteria**:
|
||||
- Custom steering captures specialized patterns
|
||||
- Follows same granularity principles as core steering
|
||||
- Provides clear value for specific domain
|
||||
</background_information>
|
||||
|
||||
<instructions>
|
||||
### Step 1: Gather Context
|
||||
|
||||
If steering context is already available from conversation, skip redundant file reads.
|
||||
Otherwise:
|
||||
- Check `.kiro/settings/templates/steering-custom/` for available templates
|
||||
- Read `rules/steering-principles.md` from this skill's directory for steering principles
|
||||
|
||||
## Workflow
|
||||
|
||||
1. **Ask user** for custom steering needs:
|
||||
- Domain/topic (e.g., "API standards", "testing approach")
|
||||
- Specific requirements or patterns to document
|
||||
|
||||
2. **Check if template exists**:
|
||||
- Load from `.kiro/settings/templates/steering-custom/{name}.md` if available
|
||||
- Use as starting point, customize based on project
|
||||
|
||||
3. **Analyze codebase** (JIT) for relevant patterns:
|
||||
|
||||
#### Parallel Research
|
||||
|
||||
The following research areas are independent and can be executed in parallel:
|
||||
1. **Template & principles**: Load matching template and steering-principles.md
|
||||
2. **Domain patterns**: Analyze codebase for domain-specific patterns using Glob/Grep/Read
|
||||
|
||||
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
|
||||
|
||||
After all parallel research completes, synthesize findings for steering document.
|
||||
|
||||
4. **Generate custom steering**:
|
||||
- Follow template structure if available
|
||||
- Apply principles from `rules/steering-principles.md` from this skill's directory
|
||||
- Focus on patterns, not exhaustive lists
|
||||
- Keep to 100-200 lines (2-3 minute read)
|
||||
|
||||
5. **Create file** in `.kiro/steering/{name}.md`
|
||||
|
||||
## Available Templates
|
||||
|
||||
Templates available in `.kiro/settings/templates/steering-custom/`:
|
||||
|
||||
1. **api-standards.md** - REST/GraphQL conventions, error handling
|
||||
2. **testing.md** - Test organization, mocking, coverage
|
||||
3. **security.md** - Auth patterns, input validation, secrets
|
||||
4. **database.md** - Schema design, migrations, query patterns
|
||||
5. **error-handling.md** - Error types, logging, retry strategies
|
||||
6. **authentication.md** - Auth flows, permissions, session management
|
||||
7. **deployment.md** - CI/CD, environments, rollback procedures
|
||||
|
||||
Load template when needed, customize for project.
|
||||
|
||||
## Steering Principles
|
||||
|
||||
From `rules/steering-principles.md` (in this skill's directory):
|
||||
|
||||
- **Patterns over lists**: Document patterns, not every file/component
|
||||
- **Single domain**: One topic per file
|
||||
- **Concrete examples**: Show patterns with code
|
||||
- **Maintainable size**: 100-200 lines typical
|
||||
- **Security first**: Never include secrets or sensitive data
|
||||
|
||||
</instructions>
|
||||
|
||||
## Tool guidance
|
||||
|
||||
- **Read**: Load template, analyze existing code
|
||||
- **Glob**: Find related files for pattern analysis
|
||||
- **Grep**: Search for specific patterns
|
||||
- **Bash** with `ls`: Understand relevant structure
|
||||
|
||||
**JIT Strategy**: Load template only when creating that type of steering.
|
||||
|
||||
## Output description
|
||||
|
||||
Chat summary with file location (file created directly).
|
||||
|
||||
```
|
||||
✅ Custom Steering Created
|
||||
|
||||
## Created:
|
||||
- .kiro/steering/api-standards.md
|
||||
|
||||
## Based On:
|
||||
- Template: api-standards.md
|
||||
- Analyzed: src/api/ directory patterns
|
||||
- Extracted: REST conventions, error format
|
||||
|
||||
## Content:
|
||||
- Endpoint naming patterns
|
||||
- Request/response format
|
||||
- Error handling conventions
|
||||
- Authentication approach
|
||||
|
||||
Review and customize as needed.
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Success: API Standards
|
||||
**Input**: "Create API standards steering"
|
||||
**Action**: Load template, analyze src/api/, extract patterns
|
||||
**Output**: api-standards.md with project-specific REST conventions
|
||||
|
||||
### Success: Testing Strategy
|
||||
**Input**: "Document our testing approach"
|
||||
**Action**: Load template, analyze test files, extract patterns
|
||||
**Output**: testing.md with test organization and mocking strategies
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
- **No template**: Generate from scratch based on domain knowledge
|
||||
- **Security**: Never include secrets (load principles)
|
||||
- **Validation**: Ensure doesn't duplicate core steering content
|
||||
|
||||
## Notes
|
||||
|
||||
- Templates are starting points, customize for project
|
||||
- Follow same granularity principles as core steering
|
||||
- All steering files loaded as project memory
|
||||
- Custom files equally important as core files
|
||||
- Avoid documenting agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
|
||||
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
|
||||
|
||||
@@ -1,90 +0,0 @@
|
||||
# Steering Principles
|
||||
|
||||
Steering files are **project memory**, not exhaustive specifications.
|
||||
|
||||
---
|
||||
|
||||
## Content Granularity
|
||||
|
||||
### Golden Rule
|
||||
> "If new code follows existing patterns, steering shouldn't need updating."
|
||||
|
||||
### ✅ Document
|
||||
- Organizational patterns (feature-first, layered)
|
||||
- Naming conventions (PascalCase rules)
|
||||
- Import strategies (absolute vs relative)
|
||||
- Architectural decisions (state management)
|
||||
- Technology standards (key frameworks)
|
||||
|
||||
### ❌ Avoid
|
||||
- Complete file listings
|
||||
- Every component description
|
||||
- All dependencies
|
||||
- Implementation details
|
||||
- Agent-specific tooling directories (e.g. `.cursor/`, `.gemini/`, `.claude/`)
|
||||
- Detailed documentation of `.kiro/` metadata directories (settings, automation)
|
||||
|
||||
### Example Comparison
|
||||
|
||||
**Bad** (Specification-like):
|
||||
```markdown
|
||||
- /components/Button.tsx - Primary button with variants
|
||||
- /components/Input.tsx - Text input with validation
|
||||
- /components/Modal.tsx - Modal dialog
|
||||
... (50+ files)
|
||||
```
|
||||
|
||||
**Good** (Project Memory):
|
||||
```markdown
|
||||
## UI Components (`/components/ui/`)
|
||||
Reusable, design-system aligned primitives
|
||||
- Named by function (Button, Input, Modal)
|
||||
- Export component + TypeScript interface
|
||||
- No business logic
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
Never include:
|
||||
- API keys, passwords, credentials
|
||||
- Database URLs, internal IPs
|
||||
- Secrets or sensitive data
|
||||
|
||||
---
|
||||
|
||||
## Quality Standards
|
||||
|
||||
- **Single domain**: One topic per file
|
||||
- **Concrete examples**: Show patterns with code
|
||||
- **Explain rationale**: Why decisions were made
|
||||
- **Maintainable size**: 100-200 lines typical
|
||||
|
||||
---
|
||||
|
||||
## Preservation (when updating)
|
||||
|
||||
- Preserve user sections and custom examples
|
||||
- Additive by default (add, don't replace)
|
||||
- Add `updated_at` timestamp
|
||||
- Note why changes were made
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- Templates are starting points, customize as needed
|
||||
- Follow same granularity principles as core steering
|
||||
- All steering files loaded as project memory
|
||||
- Light references to `.kiro/specs/` and `.kiro/steering/` are acceptable; avoid other `.kiro/` directories
|
||||
- Custom files equally important as core files
|
||||
|
||||
---
|
||||
|
||||
## File-Specific Focus
|
||||
|
||||
- **product.md**: Purpose, value, business context (not exhaustive features)
|
||||
- **tech.md**: Key frameworks, standards, conventions (not all dependencies)
|
||||
- **structure.md**: Organization patterns, naming rules (not directory trees)
|
||||
- **Custom files**: Specialized patterns (API, testing, security, etc.)
|
||||
@@ -1,103 +0,0 @@
|
||||
---
|
||||
name: kiro-validate-design
|
||||
description: Interactive technical design quality review and validation
|
||||
metadata:
|
||||
shared-rules: "design-review.md"
|
||||
---
|
||||
|
||||
|
||||
# Technical Design Validation
|
||||
|
||||
<background_information>
|
||||
- **Mission**: Conduct interactive quality review of technical design to ensure readiness for implementation
|
||||
- **Success Criteria**:
|
||||
- Critical issues identified (maximum 3 most important concerns)
|
||||
- Balanced assessment with strengths recognized
|
||||
- Clear GO/NO-GO decision with rationale
|
||||
- Actionable feedback for improvements if needed
|
||||
</background_information>
|
||||
|
||||
<instructions>
|
||||
## Core Task
|
||||
Interactive design quality review for feature **$1** based on approved requirements and design document.
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. **Gather Context**:
|
||||
- Read `.kiro/specs/$1/spec.json` for language and metadata
|
||||
- Read `.kiro/specs/$1/requirements.md` for requirements
|
||||
- Read `.kiro/specs/$1/design.md` for design document
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Additional steering files only when directly relevant to architecture boundaries, integrations, runtime prerequisites, domain rules, security/performance constraints, or team conventions that affect implementation readiness
|
||||
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide review-relevant context
|
||||
|
||||
#### Parallel Research
|
||||
|
||||
The following research areas are independent and can be executed in parallel:
|
||||
1. **Context & rules loading**: Spec documents, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and `rules/design-review.md` from this skill's directory for review criteria
|
||||
2. **Codebase pattern survey**: Gather existing architecture patterns, naming conventions, and component structure from the codebase to use as reference during review
|
||||
|
||||
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
|
||||
|
||||
After all parallel research completes, synthesize findings for review.
|
||||
|
||||
2. **Execute Design Review**:
|
||||
- Reference conversation history when available: leverage prior requirements discussion and user's stated design intent
|
||||
- Follow design-review.md process: Analysis → Critical Issues → Strengths → GO/NO-GO
|
||||
- Limit to 3 most important concerns
|
||||
- Engage interactively with user — ask clarifying questions, propose alternatives
|
||||
- Use language specified in spec.json for output
|
||||
|
||||
3. **Decision and Next Steps**:
|
||||
- Clear GO/NO-GO decision with rationale
|
||||
- Provide specific actionable next steps (see Next Phase below)
|
||||
|
||||
## Important Constraints
|
||||
- **Quality assurance, not perfection seeking**: Accept acceptable risk
|
||||
- **Critical focus only**: Maximum 3 issues, only those significantly impacting success
|
||||
- **Conversation-aware**: Leverage discussion history for requirements context and user intent when available
|
||||
- **Interactive approach**: Engage in dialogue, ask clarifying questions, propose alternatives
|
||||
- **Balanced assessment**: Recognize both strengths and weaknesses
|
||||
- **Actionable feedback**: All suggestions must be implementable
|
||||
- **Context Discipline**: Start with core steering and expand only with review-relevant steering or use-case-aligned local agent skills/playbooks
|
||||
</instructions>
|
||||
|
||||
## Tool Guidance
|
||||
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before review
|
||||
- **Grep if needed**: Search codebase for pattern validation or integration checks
|
||||
- **Interactive**: Engage with user throughout the review process
|
||||
|
||||
## Output Description
|
||||
Provide output in the language specified in spec.json with:
|
||||
|
||||
1. **Review Summary**: Brief overview (2-3 sentences) of design quality and readiness
|
||||
2. **Critical Issues**: Maximum 3, following design-review.md format
|
||||
3. **Design Strengths**: 1-2 positive aspects
|
||||
4. **Final Assessment**: GO/NO-GO decision with rationale and next steps
|
||||
|
||||
**Format Requirements**:
|
||||
- Use Markdown headings for clarity
|
||||
- Follow design-review.md output format
|
||||
- Keep summary concise
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
### Error Scenarios
|
||||
- **Missing Design**: If design.md doesn't exist, stop with message: "Run `/kiro-spec-design $1` first to generate design document"
|
||||
- **Design Not Generated**: If design phase not marked as generated in spec.json, warn but proceed with review
|
||||
- **Empty Steering Directory**: Warn user that project context is missing and may affect review quality
|
||||
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
|
||||
|
||||
### Next Phase: Task Generation
|
||||
|
||||
**If Design Passes Validation (GO Decision)**:
|
||||
- Review feedback and apply changes if needed
|
||||
- Run `/kiro-spec-tasks $1` to generate implementation tasks
|
||||
- Or `/kiro-spec-tasks $1 -y` to auto-approve and proceed directly
|
||||
|
||||
**If Design Needs Revision (NO-GO Decision)**:
|
||||
- Address critical issues identified
|
||||
- Re-run `/kiro-spec-design $1` with improvements
|
||||
- Re-validate with `/kiro-validate-design $1`
|
||||
|
||||
**Note**: Design validation is recommended but optional. Quality review helps catch issues early.
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
name: kiro-validate-gap
|
||||
description: Analyze implementation gap between requirements and existing codebase
|
||||
metadata:
|
||||
shared-rules: "gap-analysis.md"
|
||||
---
|
||||
|
||||
|
||||
# Implementation Gap Validation
|
||||
|
||||
<background_information>
|
||||
- **Mission**: Analyze the gap between requirements and existing codebase to inform implementation strategy
|
||||
- **Success Criteria**:
|
||||
- Comprehensive understanding of existing codebase patterns and components
|
||||
- Clear identification of missing capabilities and integration challenges
|
||||
- Multiple viable implementation approaches evaluated
|
||||
- Technical research needs identified for design phase
|
||||
</background_information>
|
||||
|
||||
<instructions>
|
||||
## Core Task
|
||||
Analyze implementation gap for feature **$1** based on approved requirements and existing codebase.
|
||||
|
||||
## Execution Steps
|
||||
|
||||
1. **Load Context**:
|
||||
- Read `.kiro/specs/$1/spec.json` for language and metadata
|
||||
- Read `.kiro/specs/$1/requirements.md` for requirements
|
||||
- Core steering context: `product.md`, `tech.md`, `structure.md`
|
||||
- Additional steering files only when directly relevant to the feature's domain rules, integrations, runtime prerequisites, compliance/security constraints, or existing product boundaries
|
||||
- Relevant local agent skills or playbooks only when they clearly match the feature's host environment or use case and provide analysis-relevant context
|
||||
|
||||
2. **Read Analysis Guidelines**:
|
||||
- Read `rules/gap-analysis.md` from this skill's directory for comprehensive analysis framework
|
||||
|
||||
#### Parallel Research
|
||||
|
||||
The following research areas are independent and can be executed in parallel:
|
||||
1. **Codebase analysis**: Existing implementations, architecture patterns, integration points, extension possibilities
|
||||
2. **External dependency research**: Dependency compatibility, version constraints, known integration challenges (when needed)
|
||||
3. **Context loading**: Requirements, core steering, task-relevant extra steering, relevant local agent skills/playbooks, and gap-analysis rules
|
||||
|
||||
If multi-agent is enabled, spawn sub-agents for each area above. Otherwise execute sequentially.
|
||||
|
||||
After all parallel research completes, synthesize findings for gap analysis.
|
||||
|
||||
3. **Execute Gap Analysis**:
|
||||
- Follow gap-analysis.md framework for thorough investigation
|
||||
- Analyze existing codebase using Grep and Read tools
|
||||
- Use WebSearch/WebFetch for external dependency research if needed
|
||||
- Evaluate multiple implementation approaches (extend/new/hybrid)
|
||||
- Use language specified in spec.json for output
|
||||
|
||||
4. **Generate Analysis Document**:
|
||||
- Create comprehensive gap analysis following the output guidelines in gap-analysis.md
|
||||
- Present multiple viable options with trade-offs
|
||||
- Flag areas requiring further research
|
||||
|
||||
5. **Write Gap Analysis to Disk**:
|
||||
|
||||
**Write the gap analysis to disk so it survives session boundaries and can be referenced during design phase.**
|
||||
|
||||
- Save the gap analysis to `.kiro/specs/$1/research.md`
|
||||
- If the file already exists, append the new analysis (separated by a horizontal rule `---`) rather than overwriting previous research
|
||||
- Verify the file was written by reading it back
|
||||
|
||||
## Important Constraints
|
||||
- **Information over Decisions**: Provide analysis and options, not final implementation choices
|
||||
- **Multiple Options**: Present viable alternatives when applicable
|
||||
- **Thorough Investigation**: Use tools to deeply understand existing codebase
|
||||
- **Explicit Gaps**: Clearly flag areas needing research or investigation
|
||||
- **Context Discipline**: Start with core steering and expand only with analysis-relevant steering or use-case-aligned local agent skills/playbooks
|
||||
</instructions>
|
||||
|
||||
## Tool Guidance
|
||||
- **Read first**: Load spec, core steering, relevant local playbooks/agent skills, and rules before analysis
|
||||
- **Grep extensively**: Search codebase for patterns, conventions, and integration points
|
||||
- **WebSearch/WebFetch**: Research external dependencies and best practices when needed
|
||||
- **Write last**: Generate analysis only after complete investigation
|
||||
|
||||
## Output Description
|
||||
Provide output in the language specified in spec.json with:
|
||||
|
||||
1. **Analysis Summary**: Brief overview (3-5 bullets) of scope, challenges, and recommendations
|
||||
2. **Document Status**: Confirm analysis approach used
|
||||
3. **Next Steps**: Guide user on proceeding to design phase
|
||||
|
||||
**Format Requirements**:
|
||||
- Use Markdown headings for clarity
|
||||
- Keep summary concise (under 300 words)
|
||||
- Detailed analysis follows gap-analysis.md output guidelines
|
||||
|
||||
## Safety & Fallback
|
||||
|
||||
### Error Scenarios
|
||||
- **Missing Requirements**: If requirements.md doesn't exist, stop with message: "Run `/kiro-spec-requirements $1` first to generate requirements"
|
||||
- **Requirements Not Approved**: If requirements not approved, warn user but proceed (gap analysis can inform requirement revisions)
|
||||
- **Empty Steering Directory**: Warn user that project context is missing and may affect analysis quality
|
||||
- **Complex Integration Unclear**: Flag for comprehensive research in design phase rather than blocking
|
||||
- **Language Undefined**: Default to English (`en`) if spec.json doesn't specify language
|
||||
|
||||
### Next Phase: Design Generation
|
||||
|
||||
**If Gap Analysis Complete**:
|
||||
- Review gap analysis insights
|
||||
- Run `/kiro-spec-design $1` to create technical design document
|
||||
- Or `/kiro-spec-design $1 -y` to auto-approve requirements and proceed directly
|
||||
|
||||
**Note**: Gap analysis is optional but recommended for brownfield projects to inform design decisions.
|
||||
@@ -1,26 +0,0 @@
|
||||
---
|
||||
name: openultrasast-fix-audit
|
||||
description: OpenUltraSAST patch, fix audit, differential review, adversarial review, and patch validation workflow. Use when proposing or auditing fixes for OpenUltraSAST findings from opencode.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Fix Audit
|
||||
|
||||
Use this skill to steer fixes for OpenUltraSAST findings.
|
||||
|
||||
## Fix Lifecycle
|
||||
|
||||
1. Intake: restate the finding ID, evidence level, affected path, and acceptance criteria.
|
||||
2. Plan: choose the smallest defensive change that addresses the root cause.
|
||||
3. Implement: modify only files required by the finding.
|
||||
4. Adversarial review: look for bypasses, regressions, false assumptions, and unrelated edits.
|
||||
5. Reconcile: either fix the review finding or document why it is not applicable.
|
||||
6. Verify: rerun the reproducer or affected checks.
|
||||
7. Ready: only then consider advancing to `patch_validated`.
|
||||
|
||||
## Guardrails
|
||||
|
||||
- Do not rewrite modules when a local fix is enough.
|
||||
- Do not change public APIs unless the vulnerability requires it.
|
||||
- Do not mark a patch validated from model reasoning alone.
|
||||
- Use differential review for risky changes in auth, crypto, config, parsers, and sandbox behavior.
|
||||
- Run Semgrep or CodeQL delta checks when those analyzers supplied the original evidence.
|
||||
@@ -1,35 +0,0 @@
|
||||
---
|
||||
name: openultrasast-scan
|
||||
description: OpenUltraSAST scan, index, quick, standard, deep, SARIF, and report workflows. Use when running or planning `ousast scan`, repository indexing, or evidence-gated SAST analysis from opencode.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Scan
|
||||
|
||||
Use this skill to steer OpenUltraSAST scan workflows from opencode.
|
||||
|
||||
## Workflow
|
||||
|
||||
1. Confirm the target is a local repository path.
|
||||
2. Run commands through uv, never by activating a virtualenv manually.
|
||||
3. Start with quick mode unless the task explicitly requires standard or deep analysis.
|
||||
4. Treat quick findings as `static_corroboration`, not verified vulnerabilities.
|
||||
5. Preserve run artifacts under `.openultrasast/runs/<scan-id>/`.
|
||||
6. Report `scan_id`, `run_dir`, finding count, and the strongest available evidence level.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
uv run ousast scan <repo> --mode quick
|
||||
uv run ousast index <repo>
|
||||
uv run pytest
|
||||
```
|
||||
|
||||
## Evidence Rules
|
||||
|
||||
- `suspicion` means a model or weak signal raised a hypothesis.
|
||||
- `static_corroboration` means deterministic code evidence exists.
|
||||
- `crash_reproduced` requires an artifact-backed reproducer.
|
||||
- `root_cause_explained` requires independent verifier reasoning.
|
||||
- `patch_validated` requires rerunning relevant checks after the patch.
|
||||
|
||||
Do not describe a finding as verified unless its evidence level meets the configured report threshold.
|
||||
@@ -1,34 +0,0 @@
|
||||
---
|
||||
name: openultrasast-triage
|
||||
description: OpenUltraSAST false-positive elimination, ranking calibration, evidence ladder, verifier triage, and finding adjudication. Use when reviewing findings, reducing false positives, or deciding whether evidence is enough.
|
||||
---
|
||||
|
||||
# OpenUltraSAST Triage
|
||||
|
||||
Use this skill when triaging OpenUltraSAST findings from opencode.
|
||||
|
||||
## Triage Loop
|
||||
|
||||
1. Identify the finding, path, rule, severity, confidence, and evidence level.
|
||||
2. Separate deterministic evidence from model assumptions.
|
||||
3. Check attacker control, reachability, sanitizer behavior, duplicates, impact, and contradictory evidence.
|
||||
4. If rejected or unresolved, record the false-positive reason and scope.
|
||||
5. Prefer scoped demotion over deleting a vulnerability class globally.
|
||||
6. Feed the outcome into ranking calibration and prompt constraints.
|
||||
|
||||
## False-Positive Reasons
|
||||
|
||||
- `unreachable_path`
|
||||
- `missing_attacker_control`
|
||||
- `sanitizer_disproved`
|
||||
- `static_rule_mismatch`
|
||||
- `incorrect_model_assumption`
|
||||
- `duplicate`
|
||||
- `insufficient_impact`
|
||||
- `unsupported`
|
||||
- `contradicted`
|
||||
- `unverified`
|
||||
|
||||
## Decision Standard
|
||||
|
||||
If evidence is uncertain, mark the finding as requiring next evidence instead of escalating severity. Difficult or high-impact disagreements should trigger fusion/deeper reasoning when available.
|
||||
@@ -28,17 +28,14 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
|
||||
- Think in English, generate responses in English. All Markdown content written to project files (e.g., requirements.md, design.md, tasks.md, research.md, validation reports) MUST be written in the target language configured for this specification (see spec.json.language).
|
||||
|
||||
## Minimal Workflow
|
||||
- Phase 0 (optional): `/kiro-steering`, `/kiro-steering-custom`
|
||||
- Phase 0 (optional): `/kiro-steering` (bootstrap, sync, or a custom steering document)
|
||||
- Discovery: `/kiro-discovery "idea"` — determines action path, writes brief.md + roadmap.md for multi-spec projects
|
||||
- Phase 1 (Specification):
|
||||
- Single spec: `/kiro-spec-quick {feature} [--auto]` or step by step:
|
||||
- `/kiro-spec-init "description"`
|
||||
- `/kiro-spec-requirements {feature}`
|
||||
- `/kiro-validate-gap {feature}` (optional: for existing codebase)
|
||||
- `/kiro-spec-design {feature} [-y]`
|
||||
- `/kiro-validate-design {feature}` (optional: design review)
|
||||
- `/kiro-spec-tasks {feature} [-y]`
|
||||
- Multi-spec: `/kiro-spec-batch` — creates all specs from roadmap.md in parallel by dependency wave
|
||||
- Phase 1 (Specification), one spec at a time, each phase human-reviewed:
|
||||
- `/kiro-spec-init "description"`
|
||||
- `/kiro-spec-requirements {feature}`
|
||||
- `/kiro-spec-design {feature} [-y | --gap | --review]` (`--gap`: optional gap analysis against the existing code first; `--review`: optional interactive review of an existing design; the design-review gate always runs)
|
||||
- `/kiro-spec-tasks {feature} [-y]`
|
||||
- Multi-spec projects: the specs of roadmap.md in dependency order, with the same steps each
|
||||
- Phase 2 (Implementation): `/kiro-impl {feature} [tasks]`
|
||||
- Without task numbers: autonomous mode (subagent per task + independent review + final validation)
|
||||
- With task numbers: manual mode (selected tasks in main context, still reviewer-gated before completion)
|
||||
@@ -46,14 +43,17 @@ Project memory keeps persistent guidance (steering, specs notes, component docs)
|
||||
- Progress check: `/kiro-spec-status {feature}` (use anytime)
|
||||
|
||||
## Skills Structure
|
||||
Skills are located in `.opencode/skills/kiro-*/SKILL.md`
|
||||
Skills are located in `.agents/skills/*/SKILL.md` (the plural directory OpenCode searches by default; moved from `.opencode/skills` on 2026-10-02)
|
||||
- Each skill is a directory with a `SKILL.md` file
|
||||
- Use `/skills` to inspect currently available skills
|
||||
- Invoke a skill directly with `/kiro-<skill-name>`
|
||||
- **If there is even a 1% chance a skill applies to the current task, invoke it.** Do not skip skills because the task seems simple.
|
||||
- Workflow skills: `kiro-steering`, `kiro-discovery`, `kiro-spec-init`, `kiro-spec-requirements`, `kiro-spec-design`, `kiro-spec-tasks`, `kiro-impl`, `kiro-validate-impl`, `kiro-spec-status`
|
||||
- `kiro-review` — task-local adversarial review protocol used by reviewer subagents
|
||||
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents
|
||||
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims
|
||||
- `kiro-debug` — root-cause-first debug protocol used by debugger subagents; carries the instrument rule below
|
||||
- `kiro-verify-completion` — fresh-evidence gate before success or completion claims; carries the instrument rule below
|
||||
- Product skills for users of the tool (the only ones user documentation names): `openultrasast-scan`, `openultrasast-triage`
|
||||
- Reviewed 2026-10-02: `kiro-spec-quick`, `kiro-spec-batch` removed (no fast-track or parallel spec creation); `kiro-validate-gap` and `kiro-validate-design` merged into `kiro-spec-design` (`--gap`, `--review`); `kiro-steering-custom` merged into `kiro-steering`; `openultrasast-fix-audit` merged into `openultrasast-triage`
|
||||
|
||||
## Development Rules
|
||||
- 3-phase approval workflow: Requirements → Design → Tasks → Implementation
|
||||
@@ -81,5 +81,5 @@ So, before reporting any measurement:
|
||||
|
||||
## Steering Configuration
|
||||
- Load entire `.kiro/steering/` as project memory
|
||||
- Default files: `product.md`, `tech.md`, `structure.md`
|
||||
- Custom files are supported (managed via `/kiro-steering-custom`)
|
||||
- Core files in this repository: `overview.md` (current state wins over dated history), `roadmap.md`, `safety-net.md`
|
||||
- Custom files may exist beside them and carry equal weight (created via `/kiro-steering`, templates under `.kiro/settings/templates/steering-custom/`)
|
||||
|
||||
+19
-40
@@ -1,24 +1,14 @@
|
||||
# OpenUltraSAST with its CPG engine, in one image.
|
||||
#
|
||||
# The model layer's arbiter is Joern, and `openultrasast.cpg.backend` reaches it by subprocess. Shipping both
|
||||
# in one image is what makes that simple: Joern is on PATH inside, so there is no container-to-container call,
|
||||
# no docker socket, no path translation between host and container, and no second code path in the backend to
|
||||
# get wrong. The alternative -- tool and engine as sibling services -- needs the tool to exec into the other
|
||||
# container, which means mounting the docker socket, and that is a large amount of privilege for a linter.
|
||||
#
|
||||
# The image is big (a JRE plus ~2 GB of Joern). That is the honest cost of bundling an engine, and it buys a
|
||||
# contributor a working tool from `docker compose run` with nothing installed on their machine.
|
||||
|
||||
FROM eclipse-temurin:21-jre-noble
|
||||
|
||||
# syntax=docker/dockerfile:1
|
||||
# Supply digests from plane/task-base.digest; deliberately no floating fallback.
|
||||
ARG TASK_BASE_TAG=base-unpublished
|
||||
ARG TASK_BASE_DIGEST
|
||||
FROM ghcr.io/norandom/ousast-task-base:${TASK_BASE_TAG}@${TASK_BASE_DIGEST} AS joern-build
|
||||
ENV PATH="/opt/joern-cli:${PATH}"
|
||||
ARG JOERN_VERSION=v4.0.625
|
||||
ARG ARCHIVE=joern-cli-linux-x86_64.zip
|
||||
|
||||
# php-cli is required by php2cpg, which drives PHP-Parser and shells out to a real interpreter. Without it a
|
||||
# PHP tree fails with an opaque "Process exited with code 1" -- measured on this project, not assumed.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
python3 python3-venv python3-pip unzip curl ca-certificates git php-cli \
|
||||
&& apt-get install -y --no-install-recommends curl unzip \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
RUN set -eux; \
|
||||
@@ -30,39 +20,28 @@ RUN set -eux; \
|
||||
unzip -q "${ARCHIVE}" -d /opt; \
|
||||
rm -f "${ARCHIVE}" "${ARCHIVE}.sha512" checksum.sha512; \
|
||||
/opt/joern-cli/joern-parse --help > /dev/null
|
||||
ENV PATH="/opt/joern-cli:${PATH}"
|
||||
|
||||
# This release ships the JS launcher with a .sh suffix; the backend's direct fallback uses
|
||||
# the same extensionless command convention as the other frontends.
|
||||
RUN if [ ! -e /opt/joern-cli/jssrc2cpg ] && [ -x /opt/joern-cli/jssrc2cpg.sh ]; then \
|
||||
ln -s jssrc2cpg.sh /opt/joern-cli/jssrc2cpg; \
|
||||
fi
|
||||
|
||||
COPY ops/frontend-retention/install.py /tmp/frontend-retention-install.py
|
||||
RUN python3 /tmp/frontend-retention-install.py /opt/joern-cli \
|
||||
&& rm /tmp/frontend-retention-install.py
|
||||
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md ./
|
||||
COPY src ./src
|
||||
COPY benchmarks ./benchmarks
|
||||
|
||||
# The core install is zero-dependency by design; `semantic` adds the tree-sitter grammars the candidate
|
||||
# enumerator uses. The LLM endpoint stays optional and is configured by environment, never baked in.
|
||||
RUN python3 -m venv /venv \
|
||||
&& /venv/bin/pip install --no-cache-dir -e ".[semantic]"
|
||||
ENV PATH="/venv/bin:${PATH}"
|
||||
|
||||
# Analysis never needs root.
|
||||
# Noble may already provide the ubuntu account at UID 1000. Reuse that identity rather than
|
||||
# failing installation or creating a second account with the same UID.
|
||||
FROM ghcr.io/norandom/ousast-task-base:${TASK_BASE_TAG}@${TASK_BASE_DIGEST} AS engine
|
||||
ENV PATH="/opt/joern-cli:${PATH}"
|
||||
COPY --from=joern-build /opt/joern-cli /opt/joern-cli
|
||||
# Create runtime identity before the application layers.
|
||||
RUN if getent passwd 1000 > /dev/null; then \
|
||||
usermod --login ousast --home /home/ousast --move-home "$(getent passwd 1000 | cut -d: -f1)"; \
|
||||
else \
|
||||
useradd --create-home --uid 1000 ousast; \
|
||||
fi \
|
||||
&& chown -R ousast /app
|
||||
else useradd --create-home --uid 1000 ousast; fi
|
||||
WORKDIR /app
|
||||
COPY pyproject.toml README.md LICENSE ./
|
||||
COPY src ./src
|
||||
RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
/venv/bin/pip install --no-deps --no-cache-dir .
|
||||
COPY --chown=1000 benchmarks ./benchmarks
|
||||
RUN chown -R ousast /app
|
||||
USER ousast
|
||||
|
||||
ENTRYPOINT ["python", "-m", "openultrasast.cli"]
|
||||
CMD ["--help"]
|
||||
|
||||
@@ -0,0 +1,202 @@
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
APPENDIX: How to apply the Apache License to your work.
|
||||
|
||||
To apply the Apache License to your work, attach the following
|
||||
boilerplate notice, with the fields enclosed by brackets "[]"
|
||||
replaced with your own identifying information. (Don't include
|
||||
the brackets!) The text should be enclosed in the appropriate
|
||||
comment syntax for the file format. We also recommend that a
|
||||
file or class name and description of purpose be included on the
|
||||
same "printed page" as the copyright notice for easier
|
||||
identification within third-party archives.
|
||||
|
||||
Copyright [yyyy] [name of copyright owner]
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
@@ -1,239 +1,344 @@
|
||||
# OpenUltraSAST
|
||||
|
||||
**v2.0.0 (Python package `2.0.0`), state as of 2026-10-02.** An experimental static
|
||||
security analyser. The pre-push safety net is **NO-GO** for rollout: no hook capability is
|
||||
enabled and no independent population has passed the qualification gates (see
|
||||
[docs/evaluation.md](docs/evaluation.md)). See the [release notes](RELEASE_NOTES.md).
|
||||
OpenUltraSAST is an experimental static security analyser. It is meant to be a safety net on
|
||||
repositories it has never seen. It scans a checkout, or the commits a `git push` would publish. It
|
||||
keeps what it can prove apart from what it only suspects.
|
||||
|
||||
OpenUltraSAST separates what it can prove from what it merely suspects, and says which is which.
|
||||
A scan combines language-scoped pattern rules, a semantic overlay, a Joern-based taint engine
|
||||
and optional model calls; every finding carries the evidence that earned its place, and a scan
|
||||
that could not look at something records that as a degradation instead of reporting a clean
|
||||
result.
|
||||
A scan combines up to four parts:
|
||||
|
||||
`ousast` is the interface. Model-driven work over whole repositories runs on a separate
|
||||
agentic plane (google/ax), and a learned decision engine that weighs all the signals is in
|
||||
development; neither is needed for a scan.
|
||||
- language-scoped pattern rules;
|
||||
- a tree-sitter overlay (a syntax tree of each file, built by the tree-sitter parser);
|
||||
- a Joern code property graph (a graph of the program's syntax, control flow and data flow);
|
||||
- optionally, a model. It is asked only the question the graph cannot settle.
|
||||
|
||||
## Install and quickstart
|
||||
Every finding carries the evidence that earned its place. When a scan could not look at
|
||||
something, it records a degradation. It does not report a clean result.
|
||||
|
||||
Where a repository's application can be built and run, an experimental proof arm goes further. It
|
||||
demonstrates a finding by executing it: the issue triggers on the vulnerable revision and not on
|
||||
the fixed one, so a demonstrated finding is a reproducible fact, not a score. This arm is in pilot
|
||||
and being qualified on repositories the tool has never seen; see [Status](#status).
|
||||
|
||||
`ousast` is the interface. Model work over whole repositories runs on a separate agentic plane
|
||||
(google/ax). A learned decision engine is in development. A scan needs neither of them.
|
||||
|
||||
This file is the front door. Each section links to its page of the documentation site.
|
||||
The site is published at <https://norandom.github.io/OpenUltraSAST/>. Its sources are in `docs/` and
|
||||
`mkdocs.yml`; to preview it locally, run `uv sync --extra docs && uv run mkdocs serve`.
|
||||
|
||||
## Status
|
||||
|
||||
**v2.0.1, state as of 2026-10-08.** The pre-push safety net is **not yet released**. Blocking on
|
||||
the static analyser by hand is **NO-GO** against every M4 gate, and the project no longer tries to
|
||||
reach them that way.
|
||||
|
||||
- No blocking hook capability is enabled; the pre-push hook is advisory only.
|
||||
- Two independent populations (case sets frozen before any scan) have been spent on the static
|
||||
engine: recall was 1 of 11, then 0 of 17. The learned decision engine ranks but blocks for no
|
||||
family.
|
||||
- Why blocking is hard: static abstract interpretation could not prove the precision a block
|
||||
needs. So the current work is a **dynamic proof arm**. It builds and runs a repository's app and
|
||||
demonstrates a finding by a differential: the issue triggers on the vulnerable revision and not
|
||||
on the fixed one. A demonstrated finding is ground truth by execution, so it can earn a block
|
||||
without calibration. Pilot demonstrations exist, including one executed on the cluster; the arm
|
||||
is being qualified, not shipped.
|
||||
- The release gate is a **measured** catch-rate and false-alarm number on repositories the tool
|
||||
has never seen (the unseen-repo evaluation), produced rather than asserted. Until that number
|
||||
exists, treat all output as advisory.
|
||||
|
||||
Every detection change has to win a controlled measurement on held-out repositories. One page
|
||||
states where things stand, with the record behind each figure:
|
||||
[docs/where-we-stand.md](docs/where-we-stand.md). For developers, the pre-push check and its
|
||||
honest limits: [docs/pre-push-for-developers.md](docs/pre-push-for-developers.md); the
|
||||
catch–prove–fix–re-verify loop: [docs/use-case-fix-and-reverify.md](docs/use-case-fix-and-reverify.md).
|
||||
The release history is in [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
|
||||
## Install and first scan
|
||||
|
||||
Pick the tier you need:
|
||||
|
||||
1. Tier 1: a deterministic scan of a checkout, or a CI gate.
|
||||
2. Tier 2: the Joern engine (`standard`, `deep`).
|
||||
3. Tier 3, for maintainers: model work over whole repositories on the plane.
|
||||
4. Tier 4 does not exist yet.
|
||||
|
||||
### Tier 1: quick scan, no Docker
|
||||
|
||||
The core install depends only on PyYAML. From a checkout, `uv sync` (or `pip install .`) installs
|
||||
the `ousast` entry point. `uv sync --group dev` adds the test and lint tools.
|
||||
|
||||
```bash
|
||||
uv sync --group dev
|
||||
|
||||
# Deterministic scan, no model calls
|
||||
uv sync
|
||||
uv run ousast scan /path/to/repo --mode quick
|
||||
|
||||
# CI: exit 1 on any evidence-verified finding
|
||||
uv run ousast scan . --mode quick --fail-on verified
|
||||
uv run ousast scan . --mode quick --fail-on verified # CI: exit 1 on any evidence-verified finding
|
||||
```
|
||||
|
||||
The core install depends only on PyYAML. Optional extras: `semantic` (tree-sitter grammars
|
||||
for the overlay) and `s3` (the plane's memory store on an S3-compatible bucket, boto3).
|
||||
This runs the language-scoped pattern rules, entry-point reachability hints, ranking,
|
||||
verification and scoring. It is deterministic and reproducible. It needs no network and no keys.
|
||||
|
||||
For `standard` and `deep` scans, use the Docker image, which ships Joern (with `php-cli` for
|
||||
the PHP frontend) next to the tool. Source the shell wrapper once and `ousast` runs in the
|
||||
container with your directory mounted read-only:
|
||||
The same install gives you the advisory pre-push hook for any repository. It runs the quick rules
|
||||
on the lines a push changes, names every check it could not run, and never blocks a push:
|
||||
|
||||
```bash
|
||||
cd /path/to/your/repository && ousast pre-push install # remove: ousast pre-push uninstall
|
||||
```
|
||||
|
||||
What it cannot do:
|
||||
|
||||
- There is no engine. Nothing follows a value across statements or files.
|
||||
- The PHP quick rules are a low-precision, in-sample development set. Per-rule precision lower
|
||||
bounds run from 0.03 to 1.0 on the real pins
|
||||
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`).
|
||||
|
||||
The optional `semantic` extra (`uv sync --extra semantic`) installs the tree-sitter grammars for
|
||||
the overlay.
|
||||
|
||||
### Tier 2: standard and deep scans (Docker)
|
||||
|
||||
`standard` and `deep` need Joern. The Docker image ships three things in one image: the tool,
|
||||
Joern v4.0.625 and `php-cli` (the PHP frontend needs it). Each release publishes it to GHCR. It is
|
||||
big: a JRE plus about 2 GB of Joern. Pull it instead of building, and tag it with the name the
|
||||
shell wrapper uses:
|
||||
|
||||
```bash
|
||||
docker pull ghcr.io/norandom/openultrasast:2.0.1
|
||||
docker tag ghcr.io/norandom/openultrasast:2.0.1 openultrasast:dev # or build: docker compose build
|
||||
```
|
||||
|
||||
The shell wrapper runs `ousast` in the container. It builds the image only if `openultrasast:dev`
|
||||
is missing. Your directory is mounted read-only and the container network is off. For the pre-push
|
||||
hook, use `ousast pre-push install --docker`: Git cannot call a shell function, so it installs the
|
||||
script `ousast-docker` beside the hook.
|
||||
|
||||
```bash
|
||||
source /path/to/OpenUltraSAST/ops/shell/ousast.sh # PowerShell: ops/shell/ousast.ps1
|
||||
ousast scan . --mode standard
|
||||
ousast scan . --mode standard # the graph decides; no model is asked
|
||||
ousast-with-judge scan . --mode standard # network on, the model is asked (DEEPSEEK_API_KEY)
|
||||
```
|
||||
|
||||
Details, including a pinned host install of Joern, are in [ops/README.md](ops/README.md).
|
||||
`docker compose build` and `TARGET=/path/to/repo docker compose run --rm ousast scan /target` do
|
||||
the same without the wrapper.
|
||||
|
||||
Every run writes auditable artifacts under `<target>/.openultrasast/runs/<scan-id>/`:
|
||||
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`,
|
||||
`report.sarif` and `trace/events.jsonl`, plus the stage artifacts of the mode.
|
||||
`deep` adds REGRESS. REGRESS runs each reproduction snippet with `docker run` from wherever the
|
||||
scan runs. The engine container has no Docker socket. So run `deep` in one of two ways:
|
||||
|
||||
- from a host install with Joern on the PATH (`pyinfra @local ops/joern.py`, pinned and checksum-
|
||||
verified) and a working `docker`;
|
||||
- or accept the recorded `sandbox_unavailable` degradation.
|
||||
|
||||
Details, including the host install, are in [ops/README.md](ops/README.md).
|
||||
|
||||
Keys and store settings live in `.env` in the working directory. The file is gitignored;
|
||||
`.env.example` lists the names.
|
||||
|
||||
- `DEEPSEEK_API_KEY` serves every LLM call.
|
||||
- `OPENROUTER_API_KEY` is used for embeddings only (`ousast learn memory embed`).
|
||||
- `.env` never overrides a variable already exported in your shell. If a key change seems to have
|
||||
no effect, `unset` the stale export.
|
||||
- The plane's memory store defaults to a local file store under `~/ousast-results/plane/memory`.
|
||||
- An S3-compatible bucket is optional: `uv sync --extra s3`, then `OUSAST_MEMORY=s3://<bucket>`
|
||||
with `S3_ENDPOINT` and the AWS credentials in `.env`. Nothing in a scan needs it.
|
||||
|
||||
### Tier 3: the plane on ax (maintainers)
|
||||
|
||||
Tier 1 and 2 need none of this. Today's profile is a single-node kind cluster on the maintainer's
|
||||
host.
|
||||
|
||||
- `ops/ax/up.sh` needs `docker`, `go`, `kubectl`, `kind`, `ko` and `ax` on the PATH. It refuses to
|
||||
start when one is missing.
|
||||
- It builds Agent Substrate and ax from their checkouts under `~/.cache/ousast/ax-src/`.
|
||||
- Measured idle footprint with two workers: the kind node takes 1.7 GiB of memory (27% of the
|
||||
7.7 GiB host) and 4.9 GB of disk for images. The build uses several GB more for Go caches.
|
||||
- The worker pool is sized for that 7 GB host.
|
||||
- The verify and roles tasks need `DEEPSEEK_API_KEY`.
|
||||
- Memory lives in the local file store or the S3 store. RustFS is the tested server.
|
||||
|
||||
Bring-up, doctor and the smoke Run: [ops/ax/README.md](ops/ax/README.md).
|
||||
|
||||
### Tier 4: Kubernetes (k3s), planned
|
||||
|
||||
Nothing has been deployed to any cluster other than that kind cluster. The target is two profiles
|
||||
of one code base: `kind` locally and `k3s` in production. The same `ousast plane run` would
|
||||
submit a Run in a remote execution mode. It needs:
|
||||
|
||||
- a kubeconfig context or the cluster's ingress (and an `imagePullSecret` on the worker
|
||||
ServiceAccount for a private package);
|
||||
- runner and engine images published to GHCR and pinned by digest;
|
||||
- the S3 store as the artifact path, with no receiver on a laptop.
|
||||
|
||||
The plan, and the code changes it needs with file paths, are in
|
||||
[docs/deployment.md](docs/deployment.md#production-topology-ax-on-a-kubernetes-cluster).
|
||||
|
||||
## Scan modes
|
||||
|
||||
| Mode | What runs | Needs |
|
||||
| --- | --- | --- |
|
||||
| `quick` | Language-scoped pattern rules, entry-point reachability hints, ranking, verification, scoring. Deterministic and reproducible. | nothing |
|
||||
| `standard` | `quick` plus the MAP stage: complexity map, semantic overlay, authorization obligations, and the model layer, which builds one Joern code property graph per repository and decides taint, guard-dominance and configuration questions on it. An LLM is asked only the residual question the graph cannot settle (a `suspicion`); `[models] hunter` adds the tool hunter. | Joern for the model layer (else a `cpg_unavailable` degradation); a provider key for the LLM parts (else they are skipped and recorded) |
|
||||
| `deep` | The MAP stage of `standard` plus REGRESS: promoted candidates are loaded by a small snippet inside a Docker sandbox (no network, read-only source, non-root, memory/pid/time limits) and get a `triggerable` / `not_triggerable` / ... verdict. | a working `docker` (else a `sandbox_unavailable` degradation) |
|
||||
There are three modes. Each adds to the one before.
|
||||
|
||||
`--fail-on never|findings|verified|worth-fixing` sets the exit code: `findings` fails on any
|
||||
reported finding, `verified` on any evidence-verified one, `worth-fixing` on a deep-mode
|
||||
verdict that is worth fixing. `--config openultrasast.toml` loads settings.
|
||||
- `quick` runs the pattern rules, hints, ranking, verification and score. It needs nothing.
|
||||
- `standard` adds the MAP stage: complexity map, semantic overlay, authorization obligations and
|
||||
the Joern model layer. The model layer builds one code property graph per repository. It decides
|
||||
taint, guard-dominance and configuration questions on that graph. A model is asked only the
|
||||
residual question. Without a key, that question is skipped and only the graph's entailed
|
||||
findings are reported.
|
||||
- `deep` adds REGRESS. Promoted candidates are loaded by a small snippet in a Docker sandbox. Each
|
||||
gets a `triggerable` or `not_triggerable` verdict.
|
||||
|
||||
## Detection coverage (quick rules)
|
||||
A missing engine is a recorded degradation (`cpg_unavailable`, `sandbox_unavailable`), never a
|
||||
clean result.
|
||||
|
||||
Rules run only against files of their language. The bundled ruleset lives in
|
||||
`src/openultrasast/ruleset/<language>/rules.toml`:
|
||||
`--fail-on never|findings|verified|worth-fixing` sets the exit code. `--config openultrasast.toml`
|
||||
loads settings. Every run writes these files under `<target>/.openultrasast/runs/<scan-id>/`:
|
||||
`manifest.json`, `findings.json`, `verification.json`, `score.json`, `report.md`, `report.sarif`
|
||||
and `trace/events.jsonl`.
|
||||
|
||||
| Language | Classes (CWE) |
|
||||
| --- | --- |
|
||||
| C / C++ | stack buffer overflow (121), format string (134), command injection (78) |
|
||||
| Python | command injection (78), code injection (95), SSTI (94), deserialization (502), SQLi (89), path traversal (22), SSRF (918), weak hash (327), insecure default (489) |
|
||||
| JavaScript / TS | command injection (78), code injection (95), XSS (79), SQLi (89), path traversal (22), SSRF (918), weak hash (327), deserialization (502) |
|
||||
| Java + Groovy templates | command injection (78), SQLi (89), weak hash (327), deserialization (502), unescaped template XSS (79) |
|
||||
| PHP | SQLi (89), command injection (78), file inclusion (98), path (22), open redirect (601), SSRF (918); code eval (95), unserialize (502) and echo XSS (79) ship as `shadow` rules |
|
||||
`ousast pre-push` (experimental) analyses the commits a push would publish against their base.
|
||||
First, the quick rules run on the changed lines. Then the engine compares head with base, and only
|
||||
new or worsened defects become candidates. It is advisory by default, and it prints one line per
|
||||
check it could not run. Its capability registry is empty, so it emits no normal alert today. An
|
||||
engine finding the push introduced is shown as advisory. Install, settings, skip reasons and a CI
|
||||
recipe: [docs/scanning.md](docs/scanning.md#first-run-install-the-hook-on-any-repository).
|
||||
|
||||
The detection gate (`uv run python -m openultrasast.gate`, also `tests/test_detection_benchmarks.py`)
|
||||
enforces the project goal of **at least 90% recall and under 10% false positives** per language
|
||||
on the bundled corpora in `benchmarks/manifests/`. Its output on this tree:
|
||||
Flows and the per-language rule table: [docs/scanning.md](docs/scanning.md).
|
||||
|
||||
| Language | Recall | False-positive rate |
|
||||
| --- | --- | --- |
|
||||
| C / C++ | 93.3% (14/15) | 0.0% |
|
||||
| Python | 93.3% (14/15) | 0.0% |
|
||||
| JavaScript | 92.3% (12/13) | 0.0% |
|
||||
| Java | 100% (4/4) | 0.0% |
|
||||
| **Overall** | **93.6% (44/47)** | **0.0%** |
|
||||
## How detection works
|
||||
|
||||
These corpora are cheat-sheet fixtures modelled on deliberately vulnerable applications. They
|
||||
show that a rule change did not break a known detection; they are not a recall estimate on
|
||||
real code. Real-code results are in [docs/evaluation.md](docs/evaluation.md).
|
||||
The tool uses three representations of code, from cheap to expensive:
|
||||
|
||||
**PHP is not in the gate, and its numbers are in-sample only.** The PHP rules were written and
|
||||
tightened while looking at the development corpus they were measured on
|
||||
(`benchmarks/measurements/2026-09-30-php-quick-rules/measurement.json`): 13 of 14 expected
|
||||
findings on the `php-vulnerable` fixture with no false positive, 2 false positives on
|
||||
`php-benign`, and per-rule precision lower bounds from 0.03 to 1.0 on the real pins. Treat
|
||||
them as development numbers, not as a holdout result.
|
||||
1. Text: regex quick rules, one statement at a time.
|
||||
2. Tree: the tree-sitter CST (concrete syntax tree) and an IR (intermediate representation) per
|
||||
function. The overlay and the obligations use it.
|
||||
3. Graph: the Joern CPG. It is the only one that follows a value across statements, calls and
|
||||
files.
|
||||
|
||||
**Framework knowledge is optional.** Rule and taint-fact entries that know a framework or
|
||||
library (WordPress, Flask, Django, Express, Spring, ...) carry a `framework` or `library` tag
|
||||
naming a row of `src/openultrasast/ruleset/frameworks.toml`. The loaders take `priors=`
|
||||
(`"all"`, the default and today's behaviour; `"off"`, language-level knowledge only; or a set
|
||||
of ids), so their contribution can be measured and switched off.
|
||||
Taint runs as fact tables, queries and verdicts on the graph. Absence bugs are operations that
|
||||
should have been guarded. They are found as obligations, dominance and configuration census. A
|
||||
model answers only the residual question.
|
||||
|
||||
## Pre-push checks (experimental)
|
||||
- [docs/detection-techniques.md](docs/detection-techniques.md) explains each technique, with the
|
||||
code it comes from and what it misses.
|
||||
- [docs/architecture.md](docs/architecture.md) gives the stage order and the evidence ladder (a
|
||||
state machine, not a label a model may assign). It also covers the central CWE policy, the
|
||||
project score and the self-improving loops (`ousast improve`, with `--memory` for proposals from
|
||||
plane rows).
|
||||
|
||||
`ousast pre-push` analyses the commits a `git push` would publish, or an explicit
|
||||
`--base`/`--head` pair, and compares the head against its base so that only new or worsened
|
||||
defects are candidates. `--mode advisory` (the default) never blocks; `--mode blocking` and
|
||||
`--incomplete-coverage block` opt into enforcement. The artifact must be written outside the
|
||||
analysed repository.
|
||||
Quick rules cover C/C++, Python, JavaScript/TypeScript, Java with Groovy templates, and PHP. Each
|
||||
rule runs only against files of its language.
|
||||
|
||||
```bash
|
||||
uv run ousast pre-push /path/to/repo --base BASE --head HEAD \
|
||||
--artifact /outside/the/repo/result.json
|
||||
```
|
||||
The detection gate (`uv run python -m openultrasast.gate`) enforces at least 90% recall and under
|
||||
10% false positives per language on the bundled cheat-sheet corpora. On this tree it reports
|
||||
93.62% recall (44/47) and 0.0% false positives. Those corpora show that a rule change did not
|
||||
break a known detection. They are not a recall estimate on real code, and PHP is not in the gate.
|
||||
|
||||
The default capability registry is empty, so no normal alert is emitted today; every
|
||||
candidate stays diagnostic in the artifact. Installation into a repository's hooks
|
||||
(`ops/install-pre-push`), caching and the optional witness model are documented in
|
||||
[ops/README.md](ops/README.md).
|
||||
Framework knowledge (WordPress, Flask, Django, Express, Spring) is tagged as an optional prior in
|
||||
`src/openultrasast/ruleset/frameworks.toml`. So its contribution can be measured and switched off.
|
||||
|
||||
## Model providers and keys
|
||||
## The plane on ax
|
||||
|
||||
Keys live in `.env` in the working directory (gitignored, never committed):
|
||||
Model-driven work over whole repositories runs as a Run manifest. A Run is a DAG of ax Tasks. Each
|
||||
Task runs in its own gVisor actor (an isolated sandbox) on Agent Substrate. ax is the only agentic
|
||||
executor; there is no local subprocess path.
|
||||
|
||||
```bash
|
||||
DEEPSEEK_API_KEY=... # every LLM call: model layer, tool hunter, plane tasks
|
||||
OPENROUTER_API_KEY=... # only OpenAI embeddings (openai/text-embedding-3-small)
|
||||
```
|
||||
- Each Task binds at most one Model and its own `usd`/`calls` budget.
|
||||
- Egress is deny-by-default per Task, by hostname only.
|
||||
- The provider key travels only in the Task's start request, never in a manifest or an image.
|
||||
|
||||
The project uses DeepSeek (`deepseek-flash`) for LLM calls and OpenRouter only for embeddings.
|
||||
Set `DEEPSEEK_API_KEY`: without it, chat calls fall back to OpenRouter when that key is set.
|
||||
|
||||
**`.env` never overrides a variable already exported in your shell.** A stale
|
||||
`export DEEPSEEK_API_KEY=...` in your profile wins over the `.env` file silently; `unset` it
|
||||
when a key change in `.env` seems to have no effect.
|
||||
|
||||
## Benchmarks and pair evaluation
|
||||
|
||||
```bash
|
||||
# Score the scan against a ground-truth manifest
|
||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||
|
||||
# Vulnerable-vs-fixed pairs: fire on the vulnerable side, stay silent on the fix
|
||||
uv run ousast pairs
|
||||
uv run ousast pairs --slice sast # OWASP Benchmark + Juliet
|
||||
uv run ousast pairs --slice vibe-py --pointers # also fetch non-vendored pointer pairs (network)
|
||||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||
```
|
||||
|
||||
Each benchmark run writes `.openultrasast/benchmarks/<id>/` (`benchmark_result.json`,
|
||||
`calibration_records.json` with every miss, `external_baseline_deltas.json`). The pair
|
||||
catalogs live in `benchmarks/pairs/`.
|
||||
|
||||
## Self-improvement
|
||||
|
||||
`ousast improve` runs a bounded loop over the governed ruleset: benchmark, propose edits to
|
||||
rule status and score constants only, validate, re-benchmark, and accept a round only if
|
||||
recall stays at least 90%, false positives under 10%, the project score does not regress and
|
||||
no matched finding is lost. Accepted rounds land in
|
||||
`<target>/.openultrasast/calibration/rule_policy.json`, which the next scan of that target loads.
|
||||
|
||||
```bash
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
With `--memory`, proposals also come from the rows plane runs stored (see below), through the
|
||||
same validator and gate; rows from holdout pairs, from the gated manifest's own cases and from
|
||||
any `--qualify-population` are dropped first. Details:
|
||||
[docs/architecture.md](docs/architecture.md#self-improvement).
|
||||
|
||||
## The agentic plane (maintainers)
|
||||
|
||||
Model work over whole repositories runs as a Run manifest on [google/ax](ops/ax/README.md)
|
||||
over Agent Substrate (a kind cluster on the maintainer's host), one isolated actor per task.
|
||||
Each task binds its own Model and its own `usd`/`calls` budget, egress is deny-by-default per
|
||||
task, and the provider key travels only in the task's start request.
|
||||
Measured on the first increment: the plane agreed on 16 of 20 declared sites at $0.0205 per
|
||||
candidate.
|
||||
|
||||
```bash
|
||||
uv run ousast plane doctor # kind, Agent Substrate, ax controller, runner image
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done
|
||||
uv run ousast plane status validation-46 # per-task status and token attribution
|
||||
uv run ousast plane run plane/runs/validation-46.yaml # a rerun skips tasks already done; --workers bounds tasks in flight
|
||||
uv run ousast plane status validation-46 # per-task status and the token attribution table
|
||||
uv run ousast plane remember validation-46 # ingest the run's rows into the memory store
|
||||
```
|
||||
|
||||
`ousast plane workspaces`, `alerts-engine` and `harvest` generate Runs and inputs. Bring-up,
|
||||
the task catalogue, the memory store (`OUSAST_MEMORY`) and the checklist for moving the plane
|
||||
to another Kubernetes cluster are in [ops/ax/README.md](ops/ax/README.md).
|
||||
`workspaces`, `harvest`, `alerts-engine` and `memory-normalise` generate Runs, their inputs and
|
||||
the store's rewrites. The flow and the task catalogue: [docs/plane.md](docs/plane.md). What runs
|
||||
where today, and what a separate cluster would take: [docs/deployment.md](docs/deployment.md).
|
||||
|
||||
> HarnessX, the earlier optional agentic extra, was retired 2026-09-30, and a leftover
|
||||
> `[harnessx]` section (retired 2026-09-30) is ignored with one warning. `[models] verifier`,
|
||||
> `[fusion] panel_model` and `[fusion] decider_model` fail with a message naming the plane
|
||||
> replacement. The LLM judge and the LLM fusion panels were retired with it; fusion is
|
||||
> deterministic. See [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
HarnessX, the earlier agentic extra, was retired 2026-09-30. A `[harnessx]` section is ignored
|
||||
with one warning. `[models] verifier`, `[fusion] panel_model` and `[fusion] decider_model` fail and
|
||||
name the plane replacement. Config keys that nothing ever read fail the same way. Fusion is
|
||||
deterministic.
|
||||
|
||||
## Decision engine (in development, not adopted)
|
||||
## Memory
|
||||
|
||||
A learned decision engine is being built to replace hand-tuned detection edits: each
|
||||
instrument (quick rules, the Joern engine, model sink classification, verify passes, repository
|
||||
facts) produces signals, never verdicts, and a compiled AI classifier with local memory turns
|
||||
them into a calibrated probability per candidate with BLOCK and ADVISORY operating points.
|
||||
No scan, pre-push or report uses it yet. Status and the first measured slice:
|
||||
[docs/decision-engine.md](docs/decision-engine.md).
|
||||
One store holds what plane runs and the decision engine learned. It is keyed by repository and pin.
|
||||
It holds two kinds of data:
|
||||
|
||||
## Agent integrations
|
||||
- rows: facts, verdicts, unit costs, alerts, coverage and feature records;
|
||||
- content-addressed blobs: excerpts, embeddings, cached responses and compiled programs.
|
||||
|
||||
An agent that can run shell commands needs nothing but the CLI. For
|
||||
[OpenCode](https://opencode.ai) there are optional project skills in `.opencode/skills/`:
|
||||
`openultrasast-scan`, `openultrasast-triage` and `openultrasast-fix-audit`.
|
||||
`OUSAST_MEMORY` selects the backend. It is either a local file store or `s3://<bucket>` on any
|
||||
S3-compatible server. The S3 server must support versioning, lifecycle rules, object tags and S3
|
||||
Select. RustFS is the tested server.
|
||||
|
||||
`ousast mcp` runs a narrow MCP server over stdio (newline-delimited JSON-RPC) with ten tools:
|
||||
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`,
|
||||
`benchmark`, `explain`, `propose_patch` (degrades visibly) and `export_report`. No tool runs a
|
||||
shell, Docker or a free-form command.
|
||||
An admin sets the bucket up once. The store verifies it at every start. If anything is missing, it
|
||||
refuses to run and names each missing piece. It does not fall back.
|
||||
|
||||
```jsonc
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
- Layout and the read and write paths: [docs/memory.md](docs/memory.md).
|
||||
- The bucket policy and admin steps: [docs/rustfs.md](docs/rustfs.md).
|
||||
- How stored rows are meant to turn into better decisions:
|
||||
[docs/memory-and-detection.md](docs/memory-and-detection.md).
|
||||
|
||||
## Decision engine and evaluation
|
||||
|
||||
A learned decision engine is being built. It is meant to replace hand-tuned detection edits. It
|
||||
follows three rules:
|
||||
|
||||
- Every instrument produces signals, never verdicts.
|
||||
- Labels come from ground truth only.
|
||||
- A compiled AI classifier with local memory turns retrieved similar cases and the candidate's
|
||||
signals into a calibrated probability. It has BLOCK and ADVISORY operating points.
|
||||
|
||||
It is **in development, not adopted**. No scan, pre-push check or report uses it.
|
||||
|
||||
It was measured on six families, on repositories the compiled program was not shown:
|
||||
|
||||
- The pooled AUC (area under the ROC curve) of its score runs from 0.77 (injection) to 0.96
|
||||
(access control).
|
||||
- The stricter within-pair AUC is 0.72 to 0.93 across the same families.
|
||||
- BLOCK is offered for no family
|
||||
(`benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`).
|
||||
|
||||
Maintainer surface: `ousast learn labels|memory|compile|evaluate|curve|experiment|audit-leaks`.
|
||||
Design, slices and the A/B protocol: [docs/decision-engine.md](docs/decision-engine.md).
|
||||
|
||||
[docs/evaluation.md](docs/evaluation.md) explains how anything is measured on code the tool was not
|
||||
tuned on. It covers the pair corpora, the independent populations and the M4 gates. It also covers
|
||||
the fold discipline that keeps learned decisions out of their own training data. Population v3
|
||||
(PHP, 15 cases) is frozen as the one-time final check. Nothing in the tree may read it.
|
||||
|
||||
```bash
|
||||
uv run ousast benchmark benchmarks/manifests/python-vulnerable.toml --mode quick
|
||||
uv run ousast pairs --slice sast # fire on the vulnerable side, stay silent on the fix
|
||||
uv run python -m openultrasast.pair_gate # CI: local pairs must all pass
|
||||
uv run ousast improve benchmarks/manifests/java-spring-boot-vulnerable.toml --dry-run --memory
|
||||
```
|
||||
|
||||
> The `kiro-*` skills, `AGENTS.md` and `.kiro/` are the maintainer's development tooling and
|
||||
> can be ignored by users.
|
||||
## Token economics
|
||||
|
||||
## Further reading
|
||||
Model tokens are the one running cost. At runtime the plane routes them:
|
||||
|
||||
Documentation site with flow diagrams (`docs/`, `mkdocs.yml`): `uv sync --extra docs && uv run mkdocs serve`.
|
||||
- one Model and one budget per Task;
|
||||
- a metered client that refuses the next call at the ceiling;
|
||||
- repository facts computed without a model and reused;
|
||||
- blobs replayed at $0, with no new model call;
|
||||
- triage before the hunt;
|
||||
- a tie-break only on disputed candidates.
|
||||
|
||||
- [docs/architecture.md](docs/architecture.md): the scan pipeline, evidence ladder, CWE policy
|
||||
and project score, and the self-improving loops.
|
||||
- [docs/evaluation.md](docs/evaluation.md): independent populations, the qualification gates,
|
||||
pair corpora and the plane increment results.
|
||||
- [docs/decision-engine.md](docs/decision-engine.md): the learned decision engine.
|
||||
- [docs/threat-model.md](docs/threat-model.md): trust boundaries, sandbox and hardening.
|
||||
- [docs/token-ergonomics.md](docs/token-ergonomics.md): model spend at runtime (budgets, reuse,
|
||||
measured costs) and in coding sessions.
|
||||
- [docs/examples.md](docs/examples.md): end-to-end walkthroughs.
|
||||
In development, 85.5% of the maintainers' coding-session tokens were tool input and output rather
|
||||
than reasoning. That is down from 90%. The remedies in `benchmarks/dev/` target that share.
|
||||
Mechanisms, the measured costs and the before-and-after table:
|
||||
[docs/token-ergonomics.md](docs/token-ergonomics.md).
|
||||
|
||||
## Development
|
||||
## Examples, threat model
|
||||
|
||||
- End-to-end walkthroughs, from a quick scan to an MCP client: [docs/examples.md](docs/examples.md).
|
||||
- What the tool trusts and what it does not: [docs/threat-model.md](docs/threat-model.md). This
|
||||
covers scanned code as untrusted input, secret redaction, per-task egress, budgets and the
|
||||
`deep` sandbox.
|
||||
|
||||
## Contributing and maintainer tooling
|
||||
|
||||
```bash
|
||||
uv run pytest # tests, incl. the 90/10 detection gate
|
||||
@@ -245,6 +350,112 @@ uv run python -m openultrasast.pair_gate # local vuln-vs-fix pairs
|
||||
uv run python dagger/ci.py # containerized CI pipeline
|
||||
```
|
||||
|
||||
Maintainer commands not covered above: `ousast repos` (pinned known-vulnerable checkouts),
|
||||
`ousast model candidates` (what the candidate enumerator can reach) and `ousast learn`
|
||||
(the decision engine's data; see [docs/decision-engine.md](docs/decision-engine.md)).
|
||||
An agent that can run shell commands needs nothing but the CLI. For [OpenCode](https://opencode.ai)
|
||||
there are two project skills in `.agents/skills/`:
|
||||
|
||||
- `openultrasast-scan`: modes and their prerequisites, `--fail-on`, the evidence ladder, recorded
|
||||
degradations and run artifacts.
|
||||
- `openultrasast-triage`: false-positive reasons, what an adjudication may feed, and fixing and
|
||||
re-scanning.
|
||||
|
||||
`ousast mcp` runs a narrow MCP (Model Context Protocol) server over stdio. It has ten tools:
|
||||
`openultrasast.scan`, `status`, `findings`, `get_finding`, `evidence`, `artifacts`, `benchmark`,
|
||||
`explain`, `propose_patch` and `export_report`. No tool runs a shell, Docker or a free-form
|
||||
command.
|
||||
|
||||
```jsonc
|
||||
{ "command": "uv", "args": ["run", "ousast", "mcp"] }
|
||||
```
|
||||
|
||||
The `kiro-*` skills, `AGENTS.md` and the Kiro-style specifications under `.kiro/` are the
|
||||
maintainer's development tooling. Users can ignore them. Other maintainer commands:
|
||||
|
||||
- `ousast repos`: pinned known-vulnerable checkouts.
|
||||
- `ousast model candidates`: what the candidate enumerator can reach.
|
||||
|
||||
## Licence
|
||||
|
||||
OpenUltraSAST is licensed under the [Apache License 2.0](LICENSE). Third-party material keeps its
|
||||
own terms:
|
||||
|
||||
- Joern (Apache-2.0) is downloaded at image build time.
|
||||
- The tree-sitter wheels of the `semantic` extra are MIT.
|
||||
- The published benchmark cases were licence-checked one by one. v1's OpenCVE case is BUSL-1.1, see [RELEASE_NOTES.md](RELEASE_NOTES.md).
|
||||
|
||||
## Engine traces on kube-ax
|
||||
|
||||
`benchmarks/learn/engine_trace.py --executor ax` submits AX v0.3.1 Tasks with
|
||||
`ax apply -f -` and removes them with `ax delete task NAME`. The operator supplies
|
||||
kubeconfig and AX access; the dispatcher creates no Kubernetes resources or secrets.
|
||||
The `ousast-engine-task` GHCR package **must be made Public once** in its package
|
||||
settings so the actors can pull it without registry credentials.
|
||||
|
||||
The task image contains the current engine and worker code. Rebuild it after code
|
||||
changes and pass its digest with `--image`; no analyzer download is needed. The host
|
||||
materializes each pin as usual and uploads source and prepared questions under
|
||||
`engine-queue/<run>/<pin>/`. Only object-scoped presigned URLs and deadline parameters
|
||||
enter Task env. Credentials stay on the coordinator. Configure `OUSAST_MEMORY=s3://…`
|
||||
and the store credentials in the coordinator's `.env` or environment, with
|
||||
`S3_ENDPOINT=https://files.because-security.com`. Presigned links expire after the
|
||||
pin deadline plus ten minutes. IP and HTTP endpoints are rejected: the operator's
|
||||
actor EgressPolicy must allow **files.because-security.com:443**.
|
||||
|
||||
Completion is the result object, even while AX reports Running. Failed phases and
|
||||
host deadlines produce explicit failure/timeout rows. Task and staging-object cleanup
|
||||
runs on success and failure; cleanup failure stops further dispatch. Per-pin files,
|
||||
`--rerun-status`, resume and `progress.json` retain the existing trace-runner behavior.
|
||||
AX results include `executor: "ax"`, `worker_ip`, and the image digest. Placement is
|
||||
random; `--parallel` defaults to two AX tasks and cannot exceed two.
|
||||
|
||||
AX workers use `--heap-profile cluster` with
|
||||
`-Xms1700m -Xmx1700m -XX:MaxMetaspaceSize=256m -XX:ReservedCodeCacheSize=128m -XX:MaxDirectMemorySize=256m -Xss512k -XX:ActiveProcessorCount=1 -XX:+UseG1GC`
|
||||
on every frontend/query JVM; instruments record the flags. A `JVM out of memory`
|
||||
result is automatically re-queued to one local Docker lane with `--heap-profile vm`
|
||||
(2560m), recording `fallback: "vm-oom"` and the AX attempt. `mixed-ax` also uses this
|
||||
one Docker lane for overflow, prioritizing pending OOM fallbacks. Thus a mixed run has
|
||||
up to two AX tasks plus one Docker container. Use `--docker-image` for the locally
|
||||
available VM image; Docker never pulls it implicitly. A STOP file prevents new
|
||||
launches; interrupted fallback queues are recovered on resume.
|
||||
|
||||
Coordinator commands, from this worktree (the AX binary and kubeconfig remain
|
||||
configurable). Build/push the actual uncommitted tree and read its digest:
|
||||
|
||||
```bash
|
||||
cd /home/mc/Source/OpenUltraSAST/.claude/worktrees/engine-ax
|
||||
export PATH=/home/mc/Source/OpenUltraSAST/.venv/bin:$PATH
|
||||
export PYTHONPATH="$PWD/src"
|
||||
TASK_TAG="ghcr.io/norandom/ousast-engine-task:sha-$(git rev-parse --short=7 HEAD)"
|
||||
docker buildx build --platform linux/amd64 -f plane/Dockerfile.engine-task \
|
||||
-t "$TASK_TAG" -t ghcr.io/norandom/ousast-engine-task:main \
|
||||
--push --metadata-file /tmp/ousast-engine-task-build.json .
|
||||
export TASK_IMAGE="ghcr.io/norandom/ousast-engine-task@$(python -c 'import json; print(json.load(open("/tmp/ousast-engine-task-build.json"))["containerimage.digest"])')"
|
||||
printf '%s\n' "$TASK_IMAGE"
|
||||
docker build -f Dockerfile -t openultrasast:engine-ax .
|
||||
```
|
||||
|
||||
The Engine image workflow also builds/pushes both images and prints each digest;
|
||||
CI needs the changes published before dispatch. The local build above includes
|
||||
uncommitted changes without publishing a git commit.
|
||||
|
||||
With `KUBECONFIG` pointing to the operator's kube-ax config and store settings in
|
||||
`.env`, run a one-pin smoke using the default experiment units and local example
|
||||
memory (or add `--examples /path/to/frozen-examples.jsonl`):
|
||||
|
||||
```bash
|
||||
: "${KUBECONFIG:?Set KUBECONFIG to the operator kube-ax configuration}"
|
||||
python benchmarks/learn/engine_trace.py --executor ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 1 --limit 1 --deadline 900 \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-smoke"
|
||||
```
|
||||
|
||||
Then run the mixed lanes (repeat to resume; defaults re-run failed/timeout pins):
|
||||
|
||||
```bash
|
||||
python benchmarks/learn/engine_trace.py --executor mixed-ax \
|
||||
--ax-bin "$HOME/.local/ax-v0.3.1/ax" --kubeconfig "$KUBECONFIG" \
|
||||
--atespace default --image "$TASK_IMAGE" --docker-image openultrasast:engine-ax \
|
||||
--parallel 2 --deadline 900 --rerun-status failed,timeout \
|
||||
--out "$HOME/ousast-results/plane/engine-ax-mixed"
|
||||
```
|
||||
|
||||
@@ -1,3 +1,41 @@
|
||||
# v2.0.1 (2026-10-02)
|
||||
|
||||
v2.0.0 was tagged but never published: its release job stopped at the format check (see below). v2.0.1 is
|
||||
the first published 2.x release and carries everything listed under v2.0.0 plus the following.
|
||||
|
||||
- **Licence: Apache-2.0.** `LICENSE` added; `pyproject.toml` declares `license = "Apache-2.0"`; both images
|
||||
carry the `org.opencontainers.image.licenses` label.
|
||||
- **Container images on GHCR.** Each version tag pushes `ghcr.io/norandom/openultrasast` (the scan image:
|
||||
Joern and php-cli) and `ghcr.io/norandom/openultrasast-runner` (the plane's task runner), tagged with the
|
||||
version and, for full releases, `latest`. The digests are in the release job's summary; pin by digest
|
||||
(Agent Substrate rejects tags).
|
||||
- **CI green again.** CI and the release job check formatting and lint over the whole repository; eight
|
||||
benchmark scripts had drifted. They were reformatted (two `noqa` comments, one renamed loop variable; no
|
||||
script computes anything differently), and the local gate now runs the same whole-tree commands.
|
||||
- **Front-door README.** Install in four tiers (quick scan without Docker; standard and deep with the scan
|
||||
image; the plane on kind; Kubernetes on k3s, planned), every claim checked against the CLI, and links to
|
||||
every documentation page.
|
||||
|
||||
## Also in this release (previously listed as Unreleased)
|
||||
|
||||
- **Documentation audited against v2.0.0** (no behaviour change). New `docs/deployment.md`: where the plane runs
|
||||
today, this host's setup, and step-by-step guidance for a separate Kubernetes cluster with what is implemented
|
||||
and what is planned; no cluster deployment has been made, and the kube context (`kind-$KIND_CLUSTER_NAME`) and
|
||||
`doctor`'s registry check (`localhost:5001`) are still hard-coded. The decision-engine pages state slice 2
|
||||
(six families, `benchmarks/measurements/2026-10-02-decision-engine-slice-2/record.json`: pooled AUC 0.77-0.96,
|
||||
within-pair 0.72-0.93, BLOCK offered nowhere, canary below 0.9 for three families) and that the engine is not
|
||||
adopted. The model layer's residual question is documented at `model/pipeline.py` (`model/judge.py` was
|
||||
deleted); a missing provider key is described as it behaves (no question asked, entailed findings reported),
|
||||
not as a recorded degradation. The `.env` reference lists the memory-store variables; `ousast plane memory-normalise` is listed.
|
||||
- **Removed: the unused vector index, `ousast index` and `[embeddings]`** (legacy cleanup, 2026-10-02). `index.py`
|
||||
keeps only the text chunker the skill router uses; the `VectorIndex` half had no caller, and nothing read the
|
||||
`chunks.json` the subcommand wrote. `[embeddings] model`/`store` fail to load with the reason, like the other
|
||||
retired keys; `OPENROUTER_EMBEDDING_MODEL` is no longer read (`ousast learn memory embed --model` names the model).
|
||||
- **Retired three config keys nothing read** (2026-10-02): `[sandbox] network`, `[sandbox] workspace_readonly` and
|
||||
`[complexity] top_k` now fail to load naming why. The sandbox always ran with `--network none` and a read-only
|
||||
workspace regardless of the first two; no stage read `top_k`. `memory_mb`, `timeout_seconds`, `pids_limit` and
|
||||
`max_hunter_hotspots` are read and stay.
|
||||
|
||||
# v2.0.0 (2026-10-02)
|
||||
|
||||
A new major release: the agentic plane runs on google/ax, HarnessX is gone, detection decisions move
|
||||
|
||||
@@ -16,7 +16,7 @@ maps each old path to the new one. The records themselves are not rewritten.
|
||||
| `model_sinks.py` | Stage-1 source-only model classification of sinks. Moved into the package as `plane/tasks/roles.py`. | `independent/selection-v2-model-sinks.json`, `measurements/2026-10-01-decision-engine-harvest/record.json`, both `2026-10-01-decision-engine-leak-audit` records | `05d5699` (named in `selection-v2-model-sinks.json`) |
|
||||
| `sink_candidates.py` | Diagnostic: vocabulary-declared sink candidates per case. Its declaration patterns live on in `plane/tasks/repo_facts.py`. | `independent/results-v2-model-pipeline.json` ("vocabulary sink-first: 2/17") | no commit recorded |
|
||||
| `hunt.py` | The v2 tool-hunter run over the population (protocol `hunter-protocol-v2.md`, including its `FAMILY_WORDS` table). Replaced by the plane's `verify` / `agree` tasks. | `independent/hunter-protocol-v2.md`, `independent/results-v2-hunter.json` (a label source in `learn/sources.toml`) | `6bfb05f`, the commit that introduced the protocol, which pins the runner |
|
||||
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed. `tests/test_harnessx_removal_equality.py` still checks that the comparison detects differences. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
|
||||
| `harnessx_removal_equality.py` | One-off equality proof for `harnessx-removal` (record and compare deterministic outputs). The spec is closed; its unit test was dropped on 2026-10-02 so the archived copy is reference only like the rest. | `measurements/2026-09-30-harnessx-removal-baseline/`, `measurements/2026-09-30-harnessx-removal-equality/` | baseline `ca5caf7`, candidate `b9afb66` |
|
||||
|
||||
The six `independent` scripts still read the population, `evaluate.py` and the results from
|
||||
`benchmarks/independent/`. Their `HERE` points there, so outputs land where they always did.
|
||||
|
||||
@@ -40,8 +40,34 @@ from openultrasast.preprocess import build_file_target, enumerate_source_files
|
||||
# The model names a family in its own words; the protocol scores three. Fixed before any v2 pin is hunted.
|
||||
FAMILY_WORDS = {
|
||||
"untrusted_destination": ("ssrf", "server-side request", "open redirect", "redirect", "url fetch", "request forgery"),
|
||||
"config_secrets": ("cors", "debug", "secret", "hardcoded", "hard-coded", "credential", "misconfig", "configuration", "tls", "certificate"),
|
||||
"injection": ("inject", "sql", "sqli", "nosql", "command", "rce", "code execution", "eval", "template", "ssti", "xss", "deserializ", "ldap", "xpath"),
|
||||
"config_secrets": (
|
||||
"cors",
|
||||
"debug",
|
||||
"secret",
|
||||
"hardcoded",
|
||||
"hard-coded",
|
||||
"credential",
|
||||
"misconfig",
|
||||
"configuration",
|
||||
"tls",
|
||||
"certificate",
|
||||
),
|
||||
"injection": (
|
||||
"inject",
|
||||
"sql",
|
||||
"sqli",
|
||||
"nosql",
|
||||
"command",
|
||||
"rce",
|
||||
"code execution",
|
||||
"eval",
|
||||
"template",
|
||||
"ssti",
|
||||
"xss",
|
||||
"deserializ",
|
||||
"ldap",
|
||||
"xpath",
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
@@ -182,7 +208,16 @@ def main() -> int:
|
||||
result["root"] = "/case"
|
||||
out.write_text(json.dumps(result, indent=2) + "\n")
|
||||
print(
|
||||
json.dumps({"case": case["id"], "pin": label, "findings": len(result["findings"]), "hunts": f"{result['completed']}/{result['questions']}", "usd": result["cost_usd"], "total_usd": round(sum(spent), 4)}),
|
||||
json.dumps(
|
||||
{
|
||||
"case": case["id"],
|
||||
"pin": label,
|
||||
"findings": len(result["findings"]),
|
||||
"hunts": f"{result['completed']}/{result['questions']}",
|
||||
"usd": result["cost_usd"],
|
||||
"total_usd": round(sum(spent), 4),
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0
|
||||
|
||||
@@ -37,7 +37,9 @@ from openultrasast.config import load_config # noqa: E402
|
||||
from openultrasast.model.endpoint import DEFAULT_DETECTOR_MODEL # noqa: E402
|
||||
from openultrasast.preprocess import detect_language, enumerate_source_files # noqa: E402
|
||||
|
||||
NOT_PRODUCT = re.compile(r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/")
|
||||
NOT_PRODUCT = re.compile(
|
||||
r"(^|/)(tests?|__tests__|spec|specs|docs?|examples?|fixtures?|e2e|cypress|storybook)/|\.(test|spec|stories)\.[jt]sx?$|_test\.py$|(^|/)test_[^/]*\.py$|\.min\.js$|(^|/)dist/|(^|/)build/"
|
||||
)
|
||||
OPERATIONS = {
|
||||
"injection": (
|
||||
"executes a database query (SQL, NoSQL, JCR/graph query), an operating-system command or shell, code or "
|
||||
@@ -69,7 +71,9 @@ def product_files(root: Path, languages: set[str]) -> list[Path]:
|
||||
|
||||
def chunks(text: str, size: int) -> list[str]:
|
||||
lines = text.splitlines()
|
||||
return ["\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)]
|
||||
return [
|
||||
"\n".join(f"{n + 1:5d}| {line}" for n, line in enumerate(lines[i : i + size], start=i)) for i in range(0, max(len(lines), 1), size)
|
||||
]
|
||||
|
||||
|
||||
class Meter:
|
||||
@@ -85,7 +89,9 @@ class Meter:
|
||||
return sum(float(c.cost_usd()) for c in self.clients)
|
||||
|
||||
|
||||
def classify(root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local) -> tuple[list[tuple[str, str, int]], int, int]:
|
||||
def classify(
|
||||
root: Path, path: Path, family: str, args: argparse.Namespace, meter: Meter, local: threading.local
|
||||
) -> tuple[list[tuple[str, str, int]], int, int]:
|
||||
from openultrasast import tool_hunter
|
||||
|
||||
if not hasattr(local, "client"):
|
||||
@@ -99,7 +105,9 @@ def classify(root: Path, path: Path, family: str, args: argparse.Namespace, mete
|
||||
asked += 1
|
||||
prompt = PROMPT.format(operation=OPERATIONS[family], path=relative, code=code)
|
||||
try:
|
||||
response = local.client.complete(model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
|
||||
response = local.client.complete(
|
||||
model=args.model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True
|
||||
)
|
||||
items = json.loads(response.content or "{}").get("functions", [])
|
||||
except Exception: # noqa: BLE001 -- counted as unclassified, never as "no sinks"
|
||||
failed += 1
|
||||
@@ -138,7 +146,7 @@ def main() -> int:
|
||||
files = product_files(checkout, {case["language"], *case.get("also", [])})
|
||||
found, failed, asked = [], 0, 0
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=args.workers) as pool:
|
||||
for f, fail, ask in pool.map(lambda p: classify(checkout, p, case["family"], args, meter, local), files):
|
||||
for f, fail, ask in pool.map(lambda p, c=checkout, fam=case["family"]: classify(c, p, fam, args, meter, local), files):
|
||||
found += f
|
||||
failed += fail
|
||||
asked += ask
|
||||
|
||||
@@ -66,7 +66,8 @@ def main() -> int:
|
||||
"agreed_findings": len(vulnerable["findings"]), "disputed_findings": len(vulnerable.get("disputed", [])),
|
||||
"triaged_out": vulnerable.get("triaged_out"), "candidates": vulnerable.get("questions"),
|
||||
"fixed_side_alerts": fixed_alerts, "fixed_agreed_total": len(fixed["findings"]) if fixed else None,
|
||||
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None, "usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
|
||||
"removed_by_fix": fixed.get("removed_by_fix") if fixed else None,
|
||||
"usd": vulnerable.get("usd"), "usage": vulnerable.get("usage"),
|
||||
}
|
||||
) # fmt: skip
|
||||
pool += [{"case": case["id"], **f} for f in vulnerable["findings"]]
|
||||
|
||||
@@ -36,7 +36,9 @@ from openultrasast.preprocess import detect_language, enumerate_source_files #
|
||||
DECLARATION = {
|
||||
"python": re.compile(r"^\s*(?:async\s+)?def\s+(\w+)"),
|
||||
"php": re.compile(r"^\s*(?:(?:public|private|protected|static|final|abstract)\s+)*function\s+&?(\w+)"),
|
||||
"javascript": re.compile(r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"),
|
||||
"javascript": re.compile(
|
||||
r"^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:(?:public|private|protected|static|async)\s+)*(\w+)\s*\([^;]*\)\s*(?::[^{=]+)?\{\s*$|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=\s*(?:async\s*)?(?:function|\()"
|
||||
),
|
||||
}
|
||||
DECLARATION["typescript"] = DECLARATION["javascript"]
|
||||
KEYWORDS = {"if", "for", "while", "switch", "catch", "return", "function", "else"}
|
||||
|
||||
@@ -101,13 +101,22 @@ def triage(root: Path, path: str, candidates: list[list], family: str, model: st
|
||||
except OSError:
|
||||
return {}
|
||||
names = sorted({fn for _, fn, _ in candidates})
|
||||
prompt = TRIAGE.format(operation=OPERATIONS[family], functions=json.dumps(names), path=path, code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines))
|
||||
prompt = TRIAGE.format(
|
||||
operation=OPERATIONS[family],
|
||||
functions=json.dumps(names),
|
||||
path=path,
|
||||
code=_relevant_code(text, [ln for _, _, ln in candidates], chunk_lines),
|
||||
)
|
||||
response = client.complete(model=model, messages=[{"role": "user", "content": prompt}], tools=[], timeout_seconds=90, json_object=True)
|
||||
try:
|
||||
items = json.loads(response.content or "{}").get("functions", [])
|
||||
except (json.JSONDecodeError, AttributeError):
|
||||
return {}
|
||||
return {str(i.get("function")): str(i.get("data_origin", "")).strip().lower() for i in items if isinstance(i, dict) and i.get("function") in names}
|
||||
return {
|
||||
str(i.get("function")): str(i.get("data_origin", "")).strip().lower()
|
||||
for i in items
|
||||
if isinstance(i, dict) and i.get("function") in names
|
||||
}
|
||||
|
||||
|
||||
def hunt(root: Path, path: str, group: list[list], family: str, model: str, steps: int) -> tuple[list[dict], dict]:
|
||||
@@ -120,8 +129,30 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
|
||||
f"Operations to judge, all in `{path}`, each of which {OPERATIONS[family]}:\n{listing}\n"
|
||||
"The file is above. For each, trace where the operation's data comes from and whether it is guarded."
|
||||
)
|
||||
spots = [Hotspot(path=path, function_name=fn, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=()) for _, fn, _ in group]
|
||||
found = tool_hunter.run_tool_hunter(root, spots, client=recording, model=model, max_steps=steps, system_prompt=SYSTEM, user_prompt=prompt, context_files=[path], tags=("sink-verifier",))
|
||||
spots = [
|
||||
Hotspot(
|
||||
path=path,
|
||||
function_name=fn,
|
||||
score=1.0,
|
||||
band="candidate",
|
||||
signals={},
|
||||
rationale="model-classified sink",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(),
|
||||
)
|
||||
for _, fn, _ in group
|
||||
]
|
||||
found = tool_hunter.run_tool_hunter(
|
||||
root,
|
||||
spots,
|
||||
client=recording,
|
||||
model=model,
|
||||
max_steps=steps,
|
||||
system_prompt=SYSTEM,
|
||||
user_prompt=prompt,
|
||||
context_files=[path],
|
||||
tags=("sink-verifier",),
|
||||
)
|
||||
names = {fn for _, fn, _ in group}
|
||||
rows = []
|
||||
for f in found:
|
||||
@@ -131,7 +162,16 @@ def hunt(root: Path, path: str, group: list[list], family: str, model: str, step
|
||||
# The verdict is about the candidate operation, so the site is the candidate's file and function; the
|
||||
# model's own location is kept beside it (it often names the file where the input enters instead).
|
||||
line = f.line if f.path == path else next(c[2] for c in group if c[1] == fn)
|
||||
rows.append({"candidate": f"{path}::{fn}", "site": f"{path}:{line or 0}:{fn}", "reported_at": f"{f.path}:{f.line or 0}", "family": family, "title": f.title, "witness": f.rationale[:600]})
|
||||
rows.append(
|
||||
{
|
||||
"candidate": f"{path}::{fn}",
|
||||
"site": f"{path}:{line or 0}:{fn}",
|
||||
"reported_at": f"{f.path}:{f.line or 0}",
|
||||
"family": family,
|
||||
"title": f.title,
|
||||
"witness": f.rationale[:600],
|
||||
}
|
||||
)
|
||||
return rows, recording.usage()
|
||||
|
||||
|
||||
@@ -143,7 +183,12 @@ def verify_file(root: str, path: str, candidates: list[list], family: str, args_
|
||||
try:
|
||||
if not args_["no_triage"]:
|
||||
record["triage"] = triage(Path(root), path, candidates, family, args_["model"], args_["chunk_lines"])
|
||||
kept = [c for c in candidates if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"} or record["triage"].get(c[1]) not in {"internal", "constant"}]
|
||||
kept = [
|
||||
c
|
||||
for c in candidates
|
||||
if record["triage"].get(c[1], "unclassified") in VERIFY | {"unclassified"}
|
||||
or record["triage"].get(c[1]) not in {"internal", "constant"}
|
||||
]
|
||||
record["kept"] = [c[1] for c in kept]
|
||||
usage: dict[str, int] = {}
|
||||
for _pass in range(args_["passes"]):
|
||||
@@ -169,14 +214,16 @@ def agreement(record: dict) -> tuple[list[dict], list[dict]]:
|
||||
some = set.union(*(set(b) for b in by_pass))
|
||||
first = by_pass[0]
|
||||
pick = lambda c: next(b[c] for b in by_pass if c in b) # noqa: E731
|
||||
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)]
|
||||
return [dict(first[c], passes=len(by_pass)) for c in sorted(every)], [
|
||||
dict(pick(c), passes=sum(c in b for b in by_pass)) for c in sorted(some - every)
|
||||
]
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
parser.add_argument("--population", type=Path, default=HERE / "population-v2.toml")
|
||||
parser.add_argument("--only", default="")
|
||||
parser.add_argument("--subset", type=Path, help='JSON {case_id: [[path, function], ...]}: verify only these candidates')
|
||||
parser.add_argument("--subset", type=Path, help="JSON {case_id: [[path, function], ...]}: verify only these candidates")
|
||||
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
|
||||
parser.add_argument("--max-steps", type=int, default=6)
|
||||
parser.add_argument("--per-hunt", type=int, default=6)
|
||||
@@ -206,7 +253,11 @@ def main() -> int:
|
||||
if out.is_file():
|
||||
continue
|
||||
log = out_dir / f"{case['id']}--{label}.jsonl"
|
||||
done = {r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r} if log.is_file() else {}
|
||||
done = (
|
||||
{r["path"]: r for r in (json.loads(row) for row in log.read_text().splitlines()) if "unverified" not in r}
|
||||
if log.is_file()
|
||||
else {}
|
||||
)
|
||||
spent += sum(float(r.get("usd", 0.0)) for r in done.values())
|
||||
record = json.loads((candidates_dir / f"{case['id']}.json").read_text())
|
||||
unique = list({(p, fn): [p, fn, ln] for p, fn, ln in sorted(record["candidates"])}.values())
|
||||
@@ -265,19 +316,41 @@ def main() -> int:
|
||||
agreed += a
|
||||
disputed += d
|
||||
triaged_out = sum(len(r["candidates"]) - len(r.get("kept", [])) for r in done.values())
|
||||
usage = {k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values()) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")}
|
||||
usage = {
|
||||
k: sum(int(r.get("usage", {}).get(k, 0)) for r in done.values())
|
||||
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens", "calls")
|
||||
}
|
||||
complete = len(done) >= len(by_file) and not account_error
|
||||
result = {
|
||||
"root": "/case", "families": [case["family"]], "questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
|
||||
"root": "/case", "families": [case["family"]],
|
||||
"questions": len(unique), "completed": sum(len(r["candidates"]) for r in done.values()),
|
||||
"completed_regions": sorted(f"{c[0]}:{c[1]}" for r in done.values() for c in r["candidates"]),
|
||||
"triaged_out": triaged_out, "passes": args.passes, "usage": usage, "usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
|
||||
"seconds": round(time.monotonic() - started, 1), "findings": sorted(agreed, key=lambda f: f["site"]), "disputed": sorted(disputed, key=lambda f: f["site"]),
|
||||
"triaged_out": triaged_out, "passes": args.passes, "usage": usage,
|
||||
"usd": round(sum(float(r.get("usd", 0.0)) for r in done.values()), 4),
|
||||
"seconds": round(time.monotonic() - started, 1),
|
||||
"findings": sorted(agreed, key=lambda f: f["site"]),
|
||||
"disputed": sorted(disputed, key=lambda f: f["site"]),
|
||||
} # fmt: skip
|
||||
if args.fixed:
|
||||
result["removed_by_fix"] = removed
|
||||
if complete:
|
||||
out.write_text(json.dumps(result, indent=1) + "\n")
|
||||
print(json.dumps({"case": case["id"], "files": f"{len(done)}/{len(by_file)}", "candidates": len(unique), "triaged_out": triaged_out, "agreed": len(agreed), "disputed": len(disputed), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"case": case["id"],
|
||||
"files": f"{len(done)}/{len(by_file)}",
|
||||
"candidates": len(unique),
|
||||
"triaged_out": triaged_out,
|
||||
"agreed": len(agreed),
|
||||
"disputed": len(disputed),
|
||||
"usd": result["usd"],
|
||||
"total_usd": round(spent, 4),
|
||||
"complete": complete,
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
if account_error:
|
||||
print(json.dumps({"stopped": "account", "error": account_error}), flush=True)
|
||||
return 2
|
||||
|
||||
@@ -104,7 +104,16 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
|
||||
client = _client()
|
||||
before = float(client.cost_usd())
|
||||
path, function, line = candidate
|
||||
spot = Hotspot(path=path, function_name=function, score=1.0, band="candidate", signals={}, rationale="model-classified sink", test_hint=None, inventory_finding_ids=())
|
||||
spot = Hotspot(
|
||||
path=path,
|
||||
function_name=function,
|
||||
score=1.0,
|
||||
band="candidate",
|
||||
signals={},
|
||||
rationale="model-classified sink",
|
||||
test_hint=None,
|
||||
inventory_finding_ids=(),
|
||||
)
|
||||
prompt = (
|
||||
f"Operation to judge: in `{path}`, function `{function}`, around line {line}, code that {OPERATIONS[family]}. "
|
||||
"The file is above. Trace where the operation's data comes from and whether it is guarded."
|
||||
@@ -120,9 +129,19 @@ def verify(root: str, candidate: list, family: str, model: str, max_steps: int,
|
||||
second = recording.usage()
|
||||
usage = {k: usage.get(k, 0) + second.get(k, 0) for k in set(usage) | set(second)}
|
||||
except Exception as error: # noqa: BLE001 -- recorded; an unverified candidate is not a clean one
|
||||
return {"candidate": candidate, "unverified": f"{type(error).__name__}: {str(error)[:120]}", "usd": float(client.cost_usd()) - before}
|
||||
return {
|
||||
"candidate": candidate,
|
||||
"unverified": f"{type(error).__name__}: {str(error)[:120]}",
|
||||
"usd": float(client.cost_usd()) - before,
|
||||
}
|
||||
findings = [
|
||||
{"site": f"{f.path}:{f.line or line}:{f.function_name or function}", "family": family, "title": f.title, "witness": f.rationale[:600], "candidate": f"{path}::{function}"}
|
||||
{
|
||||
"site": f"{f.path}:{f.line or line}:{f.function_name or function}",
|
||||
"family": family,
|
||||
"title": f.title,
|
||||
"witness": f.rationale[:600],
|
||||
"candidate": f"{path}::{function}",
|
||||
}
|
||||
for f in found
|
||||
]
|
||||
return {"candidate": candidate, "findings": findings, "usd": float(client.cost_usd()) - before, "escalated": escalated, "usage": usage}
|
||||
@@ -134,7 +153,12 @@ def main() -> int:
|
||||
parser.add_argument("--only", default="")
|
||||
parser.add_argument("--model", default=DEFAULT_DETECTOR_MODEL)
|
||||
parser.add_argument("--max-steps", type=int, default=6)
|
||||
parser.add_argument("--escalate-steps", type=int, default=0, help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)")
|
||||
parser.add_argument(
|
||||
"--escalate-steps",
|
||||
type=int,
|
||||
default=0,
|
||||
help="rerun a hunt that used every step without a verdict; 0 disables (measured: 88% escalate, +45% cost)",
|
||||
)
|
||||
parser.add_argument("--workers", type=int, default=4)
|
||||
parser.add_argument("--budget-usd", type=float, default=40.0)
|
||||
parser.add_argument("--fixed", action="store_true", help="recheck detected sites at the fixed pin")
|
||||
@@ -185,7 +209,9 @@ def main() -> int:
|
||||
with concurrent.futures.ProcessPoolExecutor(max_workers=args.workers) as pool, log.open("a") as sink:
|
||||
futures = []
|
||||
for candidate in pending:
|
||||
futures.append(pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps))
|
||||
futures.append(
|
||||
pool.submit(verify, str(checkout), list(candidate), case["family"], args.model, args.max_steps, args.escalate_steps)
|
||||
)
|
||||
for future in concurrent.futures.as_completed(futures):
|
||||
if future.cancelled(): # cancelled at the ceiling: never run, nothing to record
|
||||
continue
|
||||
@@ -219,9 +245,29 @@ def main() -> int:
|
||||
} # fmt: skip
|
||||
if complete:
|
||||
out.write_text(json.dumps(result, indent=1) + "\n")
|
||||
tokens = {k: sum(int(d.get("usage", {}).get(k, 0)) for d in done) for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")}
|
||||
tokens = {
|
||||
k: sum(int(d.get("usage", {}).get(k, 0)) for d in done)
|
||||
for k in ("prompt_tokens", "prompt_cache_hit_tokens", "completion_tokens")
|
||||
}
|
||||
result["usage"] = tokens | {"escalated": sum(1 for d in done if d.get("escalated"))}
|
||||
print(json.dumps({"case": case["id"], "pin": label, "candidates": len(todo), "verified": len(verified), "findings": len(findings), "unverified": len(unverified), "escalated": result["usage"]["escalated"], "cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2), "usd": result["usd"], "total_usd": round(spent, 4), "complete": complete}), flush=True)
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"case": case["id"],
|
||||
"pin": label,
|
||||
"candidates": len(todo),
|
||||
"verified": len(verified),
|
||||
"findings": len(findings),
|
||||
"unverified": len(unverified),
|
||||
"escalated": result["usage"]["escalated"],
|
||||
"cache_share": round(tokens["prompt_cache_hit_tokens"] / max(tokens["prompt_tokens"], 1), 2),
|
||||
"usd": result["usd"],
|
||||
"total_usd": round(spent, 4),
|
||||
"complete": complete,
|
||||
}
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
if account_error:
|
||||
print(json.dumps({"stopped": "account", "error": account_error, "spent_usd": round(spent, 4)}), flush=True)
|
||||
return 2
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user