Write script tests for a package

Write a Starlark script that asserts on a package’s behavior, then run it with ocx package test --script. You need a built archive and its identifier, as in Test a package before you push it. The script runs the same on Linux, macOS and Windows, with no host shell.

Why scripts instead of a command

The -- CMD form works well when the package ships its own test runner. Packages of standalone executables such as cmake, shellcheck and goreleaser do not. They need sh -c '...' on the host, which breaks on Windows without WSL or Git Bash.

--script PATH solves this. Instead of exec’ing a command, OCX interprets a Starlark script against the materialized package environment. The script has no access to the host shell or runtime. It runs identically on linux/*, macos/*, and windows/*.

Run a script

# Read script from a file
ocx package test -p linux/amd64 -i shfmt/shfmt:3.8.0 shfmt.tar.xz --script smoke.star

# Read script source from stdin (the value `-` means stdin)
printf 'r = ocx.run("shfmt", "--version")\nexpect.ok(r)\n' \
  | ocx package test -p linux/amd64 -i shfmt/shfmt:3.8.0 shfmt.tar.xz --script -

--script and -- CMD are mutually exclusive. Supplying both exits with code 64, and so does supplying neither. With --script -, OCX reads the script source from stdin. A read failure such as a broken pipe or a closed stream exits with code 74.

A typical smoke test runs the binary, checks its output and checks the env the package exports. Save this as smoke.star, the file the run command above passes:

r = ocx.run("shfmt", "--version")
expect.ok(r)
expect.contains(r.stdout, "v3")

# Verify the package exported the expected env var
expect.true(ocx.env("SHFMT_ROOT") != None, "SHFMT_ROOT should be set")

# Branch on the platform the package was built for
if ocx.target_platform.os == ocx.os.Linux:
    expect.eq(ocx.target_platform.arch, ocx.arch.Amd64)

Read paths are relative. They resolve against the scratch root first, then the bundle’s content_root, so a file the package ships at bin/javac is read as bin/javac:

if ocx.exists("bin/javac"):
    expect.ok(ocx.run("javac", "-version"))

Every ocx.* function and expect.* assertion, with its arguments and return types, is listed in the Script Host API reference.

Sandbox model

The script runs inside a bounded environment:

Writable area: a scratch directory created as a sibling of the package root. The ocx.scratch_root attribute holds its path. Files written here survive --keep.
Read area: the bundle’s own files (ocx.content_root) are readable but not writable. The package directory around them is storage structure, not a script contract, so a script never has to name it.
Symlink containment: every path is validated for symlink escape after lexical normalization. A symlink inside scratch_root that points outside is refused on access, not just at creation.
Path portability: use / as the separator on all platforms. Absolute paths and .. escapes are rejected, including an absolute path built from ocx.content_root. Use the relative spelling.

Re-entrant ocx invocations are refused in v1. ocx.run("ocx", ...) exits with code 1 and a message explaining the limitation.

ocx.run resolves a bare program name the way the trailing-command form does, against the package first. A name your package ships but cannot execute fails the script instead of running the host’s copy. Names the package does not ship still resolve on the host PATH.

Output format

Pass --format json to get a structured envelope alongside the exit code:

ocx package test -p linux/amd64 -i shfmt/shfmt:3.8.0 shfmt.tar.xz \
  --script smoke.star --format json

The envelope has three top-level keys, all part of the stable v1 contract:

{
  "status": "passed|failed|usage|script_error|io|timeout",
  "assertion": { "kind": "ok|eq|ne|true|false|contains|matches|fail|other|unknown", "message": "…",
                 "location": { "file": "smoke.star", "line": 12, "column": 5 } },
  "run":       { "exit_code": 0, "stdout": "…", "stderr": "…", "duration_ms": 12, "truncated": false }
}

assertion and run are null when not applicable. For example, assertion is null on a passing run. assertion.kind names the expect.* function that triggered the failure and is the stable machine field for tooling. The assertion.message prose is not stable.

assertion.location points at the failing statement, 1-indexed. Its file is the --script argument verbatim, or <stdin>. It is omitted when the failure has no source position, such as a timeout. The exit code remains the primary machine signal.

Exit codes

Code	Meaning
0	All expectations passed
1	An expectation failed, `expect.fail` was called, or a host API returned a failure
64	Usage error: both `--script` and `-- CMD` supplied, neither supplied, or script file not found
65	Script syntax, type, or arity error
74	I/O error: stdin read failure (`--script -`) scratch directory creation failure, or a `--junit` report that cannot be written after an otherwise green run

JUnit reports for CI

--junit PATH writes a JUnit XML report beside the JSON envelope. A CI system can then surface the run as a test result instead of a bare exit code:

ocx package test -p linux/amd64 -i shfmt/shfmt:3.8.0 shfmt.tar.xz \
  --script smoke.star --junit build/junit/linux-amd64.xml

Parent directories are created, so the pipeline needs no mkdir step. --junit requires --script and is not available for the trailing -- CMD form.

The file holds one <testsuite name="ocx package test <identifier>"> with one <testcase classname="<identifier>" name="<platform>">. One invocation tests one identifier on one platform.

A failing assertion becomes a <failure> element. Its type= holds the failing expect.* kind and its message= holds the first diagnostic line. The element body holds the full multi-line diagnostic.

The failure also carries file and line pointing at the failing statement in the script. A script read from --script - has no source file, so both attributes are omitted.

A script that never delivered a verdict becomes an <error> instead. That covers a script that is unreadable, syntactically broken or timed out. Captured stdout and stderr are attached as system-out and system-err on every run that produced output.

The report is written whenever the scripted run is reached, including a red run and an unreadable --script path. A failure resolving the package beforehand writes none. The JSON envelope on stdout and the exit code are unchanged either way.

One path per platform

An existing file is truncated, never merged: ocx writes the one testcase it just produced and nothing else. Give every matrix leg its own path. Pointing every leg at one path keeps only the last leg’s result.

test:
  parallel:
    matrix:
      - PLATFORM: [linux/amd64, linux/arm64, darwin/arm64]
  script:
    - ocx package test -p "$PLATFORM" -i shfmt/shfmt:3.8.0 shfmt.tar.xz
        --script smoke.star --junit "build/junit/${PLATFORM//\//-}.xml"
  artifacts:
    reports:
      junit: build/junit/*.xml

What the artifacts:reports:junit glob buys you differs by CI system. GitLab reads the <testsuite name=> attribute for internal grouping only and never displays it. Its test report UI labels each suite by testcase classname. ocx sets that to the package identifier, the same value as <testsuite name>. A GitLab reader therefore still sees the identifier as the suite label.

Each parallel: matrix leg is its own job with its own report. The glob only collects the files as job artifacts and does not merge them. The three platforms above therefore show up as three separate entries in GitLab’s test report UI, one per leg.

The shared suite name and per-platform case name serve the other consumer. A GitHub test-reporter action downloads the JUnit artifacts from every matrix leg and parses them together. It sees one suite (ocx package test <identifier>) with one <testcase> per platform. Nothing in the file distinguishes the leg except the platform-specific name=. Design for that shape on GitHub, and plan for one suite per leg on GitLab.

Editor integration

For .star syntax highlighting in VS Code, add the vscode-bazel extension to your workspace.