mirror of
https://github.com/norandom/OpenUltraSAST.git
synced 2026-10-17 03:59:28 +00:00
feat(ops): ship the tool and its engine in one image
A contributor should be able to run this with nothing installed but Docker: docker compose build TARGET=/path/to/repo docker compose run --rm ousast scan /target One image rather than two services, and the reason is the seam. The model layer reaches Joern by subprocess; putting both in the same image means Joern is simply on PATH, so there is no container-to-container call, no docker socket to mount, no host/container path translation, and no second code path in cpg/backend.py to get wrong. Sibling services would need the tool to exec into the engine's container, which means handing a linter the docker socket -- a large amount of privilege for something that reads source code. The image is big (a JRE plus ~2 GB of Joern); that is the honest cost of bundling an engine. I had started building the two-service version with a docker-wrapper mode in the backend and dropped it, tests included: once the engine is on PATH the wrapper is complexity with no purpose. Defaults chosen so an unfamiliar analyser is safe to try: the target mounts READ-ONLY, /work is a tmpfs so CPG scratch never outlives the run or touches the host, network_mode is `none` unless the LLM judge is wanted, and API keys pass through the environment rather than into an image layer. mem_limit is set because two Joern JVMs at once is what triggered an out-of-memory kill on a 7.7 GB machine during this feature's measurements. pyinfra deploy kept for persistent host installs, with the same version pin and the same checksum verification -- done as an explicit `sha512sum -c` step because pyinfra's files.download offers sha384/sha256/sha1/md5 and Joern publishes sha512. That mismatch was caught by validating against the installed pyinfra rather than trusting the API from memory. Both include php-cli: php2cpg drives PHP-Parser and shells out to a real interpreter, and without it a PHP tree fails with an opaque "Process exited with code 1" -- measured here, which is what makes a WordPress corpus feasible. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01T7XD7x5KonMdmGRkp63EQG
This commit is contained in:
co-authored by
Claude Opus 5
parent
3fe7b9ded7
commit
46232fe3e6
@@ -0,0 +1,89 @@
|
||||
"""pyinfra deploy for the Joern CPG engine (model-grounded-detection, Req 4).
|
||||
|
||||
Replaces the ad-hoc `curl | unzip` this engine was first installed with. Joern is the model layer's arbiter
|
||||
substrate and the only thing standing between a `suspicion` and a `model_entailed`, so how it reaches a
|
||||
machine should be reproducible and stated rather than remembered.
|
||||
|
||||
Deliberately narrow:
|
||||
|
||||
* **It installs an engine, it does not become one.** No sandbox, no container, no service.
|
||||
``openultrasast.cpg.backend`` reaches Joern by subprocess and nothing here changes that.
|
||||
* **The checksum is verified, in a shell step rather than by ``files.download``.** Upstream publishes a
|
||||
``.sha512`` beside the archive and pyinfra's download operation offers sha384/sha256/sha1/md5 but not
|
||||
sha512, so the verification is done with ``sha512sum -c`` against the published file. An unverified 1.8 GB
|
||||
download that then parses your source tree is not something to wave through.
|
||||
* **Idempotent.** A version stamp short-circuits the download and extraction, so a re-run costs a file read.
|
||||
* **Nothing the tool needs at runtime.** The core install stays ``dependencies = []``; without Joern the model
|
||||
layer degrades to ``suspicion`` with a recorded reason, which is a supported state rather than a broken one.
|
||||
|
||||
Usage:
|
||||
|
||||
uv pip install pyinfra # tooling only, never a project dependency
|
||||
pyinfra @local ops/joern.py # this machine
|
||||
pyinfra inventory.py ops/joern.py # anywhere else
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pyinfra import host
|
||||
from pyinfra.operations import files, server
|
||||
|
||||
# Pinned. An engine that silently changes version changes every verdict it produces, and the ceiling
|
||||
# artifacts under benchmarks/measurements are only comparable against a fixed one.
|
||||
JOERN_VERSION = host.data.get("joern_version", "v4.0.623")
|
||||
PREFIX = host.data.get("joern_prefix", "/opt/joern")
|
||||
ARCHIVE = "joern-cli-linux-x86_64.zip"
|
||||
BASE = f"https://github.com/joernio/joern/releases/download/{JOERN_VERSION}"
|
||||
STAMP = f"{PREFIX}/.joern-version"
|
||||
|
||||
# Joern ships Scala but not a JVM. `php2cpg` additionally shells out to a real PHP interpreter (it drives
|
||||
# PHP-Parser), so without `php` on PATH a PHP tree fails to build with an opaque "Process exited with code 1"
|
||||
# -- measured on this machine before the package was added. `php_frontend` is opt-in because most targets do
|
||||
# not need it and it pulls a second runtime.
|
||||
PACKAGES = ["openjdk-21-jre-headless", "unzip"]
|
||||
if host.data.get("php_frontend", False):
|
||||
PACKAGES.append("php-cli")
|
||||
|
||||
server.packages(
|
||||
name="Install a JVM, unzip, and (optionally) the PHP frontend's interpreter",
|
||||
packages=PACKAGES,
|
||||
present=True,
|
||||
_sudo=True,
|
||||
)
|
||||
|
||||
files.directory(
|
||||
name=f"Create {PREFIX}",
|
||||
path=PREFIX,
|
||||
present=True,
|
||||
_sudo=True,
|
||||
)
|
||||
|
||||
server.shell(
|
||||
name=f"Install Joern {JOERN_VERSION} (skipped when the stamp already matches)",
|
||||
commands=[
|
||||
# One guarded block: a matching stamp skips the download, the checksum and the extraction alike.
|
||||
f"""
|
||||
if [ -f {STAMP} ] && grep -qx {JOERN_VERSION} {STAMP}; then
|
||||
echo "joern {JOERN_VERSION} already installed"
|
||||
else
|
||||
set -e
|
||||
cd /tmp
|
||||
curl -fsSL -o {ARCHIVE} {BASE}/{ARCHIVE}
|
||||
curl -fsSL -o {ARCHIVE}.sha512 {BASE}/{ARCHIVE}.sha512
|
||||
# Verify before unpacking, never after: the archive is what we are choosing to trust.
|
||||
sha512sum -c {ARCHIVE}.sha512
|
||||
rm -rf {PREFIX}/joern-cli
|
||||
unzip -q -o {ARCHIVE} -d {PREFIX}
|
||||
echo {JOERN_VERSION} > {STAMP}
|
||||
# 1.8 GB in /tmp is how a disk fills up.
|
||||
rm -f {ARCHIVE} {ARCHIVE}.sha512
|
||||
fi
|
||||
"""
|
||||
],
|
||||
_sudo=True,
|
||||
)
|
||||
|
||||
server.shell(
|
||||
name="Verify joern-parse runs",
|
||||
commands=[f"{PREFIX}/joern-cli/joern-parse --help > /dev/null"],
|
||||
)
|
||||
Reference in New Issue
Block a user