Testing Architecture
Poly’s testing pyramid climbs from pure C++ unit tests on the engine core up through browser-driven Playwright specs against the deployed guide site, with a scripted post-deploy verification loop tying every layer together. Each layer catches a specific class of regression the layer above cannot see. This appendix walks the pyramid top-down through the actual test code and gate scripts, not descriptions of them.
Layer 0 — Engine unit tests
Section titled “Layer 0 — Engine unit tests”The engine ships a Google Test suite that runs entirely off-host, without the VST3 SDK, without a DAW, and without an audio device. Every algorithmic core — the Björklund Euclidean generator, envelope evaluation, macro resolution, constraint enforcement, scene interpolation, swing/humanization, sanitization — has its own unit-test file under tests/.
Unit tests are small, deterministic, and assert known-good outputs for known inputs. The Björklund tests, for example, encode culturally-specific rhythms as string patterns and check the algorithm reproduces them exactly:
TEST(Euclidean, Tresillo_3_8) {
std::array<bool, poly::kMaxSteps> p{};
poly::euclidean(3, 8, 0, p);
EXPECT_EQ(countHits(p, 8), 3);
EXPECT_EQ(patternStr(p, 8), "10010010");
}Similar tests cover cinquillo (5-in-8), rotation invariants (rotating by n returns to the original), zero-pulse and over-full edge cases, and the boundary condition where k > n. A single ctest run exercises hundreds of these across the engine’s algorithmic surface.
Layer 0 — Golden determinism tests
Section titled “Layer 0 — Golden determinism tests”Unit tests prove each component is correct in isolation. Golden tests prove the engine as a whole is a pure function of transport plus groove state — same inputs, same outputs, byte-for-byte, forever. This is the invariant that makes VST3 preset compatibility possible and that CI enforces on every commit.
TEST(GoldenDeterminism, SamePatchSameSeed) {
poly::Engine engine;
auto state = makeTestState();
auto run1 = renderSorted(engine, state, 0.0, 16.0, 0.5);
auto run2 = renderSorted(engine, state, 0.0, 16.0, 0.5);
ASSERT_EQ(run1.size(), run2.size());
EXPECT_EQ(serialize(run1), serialize(run2));
}The suite covers the full determinism surface: same seed on two runs, different block sizes producing identical events, loop restarts reproducing the pre-loop range, position jumps not accumulating state, tempo independence, and polymetric phase variation. A single serialization comparison catches any accidental introduction of shared mutable state anywhere in the engine.
Layer 1 — Real-time safety static scanner
Section titled “Layer 1 — Real-time safety static scanner”Google Test cannot detect a push_back() that will only misbehave under real-time contention. scripts/check-realtime-safety.sh catches it statically by scanning the audio-thread call chain for known-unsafe patterns and refusing to let the commit land.
# Files in the audio-thread call chain:
# process() -> renderRange() -> euclidean(), deterministicRand()
# process() -> resolveMacros(), resolveConstraints(), interpolateGrooveState(),
# SceneChainState::update() (plugin/source/processor.cpp process())
# renderRange() -> computeEnvelopePhase(), evaluateShapeFull() (engine.cpp per-envelope loop)
# M049 S07 (E8): scene/macro/constraint/envelope added — all four are live on the
# audio thread but were previously ungated.
RT_FILES=(
plugin/source/processor.cpp
plugin/source/processor.h
engine/src/engine.cpp
engine/src/euclidean.cpp
engine/src/scene.cpp
engine/src/macro.cpp
engine/src/constraint.cpp
engine/src/envelope.cpp
engine/include/poly/engine.h
engine/include/poly/euclidean.h
engine/include/poly/rng.h
engine/include/poly/types.h
)
PATTERNS=(
'\bnew\b'
'\bdelete\b'
'\bmalloc\b'
'\bfree\b'
'\bmake_unique\b'
'\bmake_shared\b'
'\bmutex\b'
'\block_guard\b'
'\bunique_lock\b'
'\bscoped_lock\b'
'\bthrow\b'
'\bcatch\s*\('
'\bcout\b'
'\bcerr\b'
'\bprintf\b'
'\bfprintf\b'
'\bfopen\b'
'\bfwrite\b'
'\bpush_back\b'
'\bemplace_back\b'
'\bresize\b'
'\breserve\b'
'\bstd::string\b'
'\ballocateMessage\b'
'\bsendMessage\b'
)The scanner is scope-aware only at the file level — it does not track which functions in processor.cpp run on which thread. Non-audio-thread methods (notify(), setState() outside process()) that legitimately need to allocate get flagged; a // RT-SAFE-OK comment on the same line suppresses the finding while leaving a record of the intent. Every suppression must justify itself in code review.
Layer 2 — CodeSnippet region gate
Section titled “Layer 2 — CodeSnippet region gate”The three architecture appendices — plugin, website, and this one — pull their code blocks from live source files at build time via <CodeSnippet />. If a source region is renamed or deleted, every appendix pointing at that region silently rots. The region-check gate makes that a build failure instead.
# grep across all mdx files, then parse each match line.
# `-H` prefixes filename, `-n` prefixes line number. Format: `mdx:LINE:...`.
while IFS= read -r hit; do
mdx_path="${hit%%:*}"
rest="${hit#*:}"
mdx_line="${rest%%:*}"
tag_body="${rest#*:}"
# Extract file="..." and region="..." from the tag body.
file_attr=$(printf '%s' "$tag_body" | sed -n 's/.*file="\([^"]*\)".*/\1/p')
region_attr=$(printf '%s' "$tag_body" | sed -n 's/.*region="\([^"]*\)".*/\1/p')
if [ -z "$file_attr" ] || [ -z "$region_attr" ]; then
log_fail " ${mdx_path}:${mdx_line}: could not parse file/region from tag"
continue
fi
src_path="$REPO_ROOT/$file_attr"
if [ ! -f "$src_path" ]; then
log_fail " ${mdx_path}:${mdx_line}: source file missing: ${file_attr} (region=${region_attr})"
continue
fi
# Two accepted marker styles per side; grep -F for literal match, anchored via -x on the trimmed line.
# Using awk to trim leading/trailing whitespace before comparison so indented markers count.
start_ok=$(awk -v r="$region_attr" '
{
line = $0
sub(/^[[:space:]]+/, "", line)
sub(/[[:space:]]+$/, "", line)
if (line == "// region:" r || line == "# region:" r) { print "1"; exit }
}
' "$src_path")
end_ok=$(awk -v r="$region_attr" '
{
line = $0
sub(/^[[:space:]]+/, "", line)
sub(/[[:space:]]+$/, "", line)
if (line == "// endregion:" r || line == "# endregion:" r) { print "1"; exit }
}
' "$src_path")
if [ "$start_ok" != "1" ]; then
log_fail " ${mdx_path}:${mdx_line}: (file=${file_attr}, region=${region_attr}) missing start marker in source"
fi
if [ "$end_ok" != "1" ]; then
log_fail " ${mdx_path}:${mdx_line}: (file=${file_attr}, region=${region_attr}) missing end marker in source"
fi
done < <(grep -rHnE "$PATTERN" "$DOCS_DIR" --include='*.mdx' || true)The gate runs in both pre-commit and pre-push. Renaming a region marker fails the pre-push hook at step [3/4] with the precise (mdx:line, file, region) triple, so a broken snippet reference is a fix-it-now blocker instead of a next-week discovery.
Layer 2 — Sample manifest gate
Section titled “Layer 2 — Sample manifest gate”The site’s sample library lives under site/public/samples/ with a JSON manifest describing every file’s role, MIDI-note assignments, and license. Drift between the filesystem and the manifest silently breaks preset playback on the deployed site.
STRICT=0
COVERAGE=0
for arg in "$@"; do
case "$arg" in
--strict) STRICT=1 ;;
--coverage) COVERAGE=1 ;;
-h|--help)
echo "Usage: $0 [--strict] [--coverage]"
exit 0
;;
*)
echo "unknown argument: $arg" >&2
exit 2
;;
esac
doneThe --coverage mode cross-references the manifest against docs/midi-note-mapping.md and exits non-zero if any documented GM note has zero registered samples — catching a sample deletion before it reaches production.
Layer 2 — Pre-push and pre-commit gates
Section titled “Layer 2 — Pre-push and pre-commit gates”The pre-push hook is the local enforcement point for everything above. It refuses to push if any of four gates fails, so a red CI run is not the first signal of a formatting or safety regression:
echo "[0/9] Build config check..."
CACHE_FILE="build/CMakeCache.txt"
NEEDS_RECONFIG=0
RECONFIG_REASONS=()
if [ -f "$CACHE_FILE" ]; then
if grep -q '^POLY_ENGINE_ONLY:BOOL=ON' "$CACHE_FILE"; then
NEEDS_RECONFIG=1
RECONFIG_REASONS+=("POLY_ENGINE_ONLY=ON (host tests would be skipped)")
fi
# BUILD_INTERACTION_TESTS gates poly_interaction_tests (and by extension the
# POLY_PLUGIN_SOURCES link check). If it's OFF, a missing source in that list
# only trips on CI — see the T01 fixup (controller_base.cpp) that shipped
# a red build to CI because local hooks compiled without this target.
if ! grep -q '^BUILD_INTERACTION_TESTS:BOOL=ON' "$CACHE_FILE"; then
NEEDS_RECONFIG=1
RECONFIG_REASONS+=("BUILD_INTERACTION_TESTS!=ON (CI-parity link check would be skipped)")
fi
fi
if [ "$NEEDS_RECONFIG" -eq 1 ]; then
for reason in "${RECONFIG_REASONS[@]}"; do
echo " build/ needs reconfigure: $reason"
done
if ! cmake -S . -B build -DPOLY_ENGINE_ONLY=OFF -DBUILD_INTERACTION_TESTS=ON >/dev/null; then
echo "FAIL: could not reconfigure build/ with POLY_ENGINE_ONLY=OFF -DBUILD_INTERACTION_TESTS=ON."
FAILED=1
fi
fi
echo "[1/9] clang-format..."
# Prefer running on staged files (fast, no full-tree traversal) when we have any;
# fall back to --all-files if git diff --cached is empty (e.g. hook invoked
# outside a staged context, or the whole tree just got reformatted).
STAGED_CPP=$(git diff --cached --name-only --diff-filter=ACMR 2>/dev/null | grep -E '\.(cpp|h|hpp|cc)$' || true)
if [ -n "$STAGED_CPP" ]; then
if ! pre-commit run clang-format --files $STAGED_CPP; then
echo "FAIL: clang-format found formatting issues (staged files)."
FAILED=1
fi
else
if ! pre-commit run clang-format --all-files; then
echo "FAIL: clang-format found formatting issues (full tree)."
FAILED=1
fi
fi
# pre-push-token: rt-safety
echo "[2/9] RT safety..."
if ! scripts/check-realtime-safety.sh; then
echo "FAIL: RT safety check failed."
FAILED=1
fi
# pre-push-token: snippet-regions
echo "[3/9] CodeSnippet region markers..."
if ! scripts/check-snippet-regions.sh; then
echo "FAIL: CodeSnippet region check failed."
FAILED=1
fi
# pre-push-token: unit
echo "[4/9] Build + test..."
if ! cmake --build build --config Release --parallel 2>/dev/null; then
echo "FAIL: Build failed."
FAILED=1
elif ! ctest --test-dir build --build-config Release --output-on-failure 2>/dev/null; then
echo "FAIL: Tests failed."
FAILED=1
fi
echo "[5/9] pluginval (strictness=${PLUGINVAL_STRICTNESS})..."
if ! command -v pluginval >/dev/null 2>&1; then
echo " SKIP: pluginval not on PATH. Install with: bash scripts/install-pluginval.sh"
else
VST3_PATH=$(find build -type d -name '*.vst3' -not -name '*probe*' 2>/dev/null | head -1)
if [ -z "$VST3_PATH" ]; then
echo " SKIP: no .vst3 bundle in build/ (engine-only build produces none)."
else
echo " Validating $VST3_PATH..."
if ! pluginval --strictness-level "$PLUGINVAL_STRICTNESS" --skip-gui-tests --timeout-ms 60000 \
--validate "$VST3_PATH"; then
echo "FAIL: pluginval reported errors at strictness $PLUGINVAL_STRICTNESS."
echo " For CI-parity strictness: PLUGINVAL_STRICTNESS=8 bash scripts/pre-push-check.sh"
FAILED=1
fi
fi
fi
# pre-push-token: doc-conformance
echo "[6/9] Doc-conformance + audit-ledger guardrail suite..."
# Single source of truth with the CI site-lint job: both invoke
# scripts/check-doc-conformance.sh (asserted by
# site/tests/doc-conformance-wiring.test.mjs). The runner imports js-yaml from
# site/node_modules, so gate on its presence — matching the runner's documented
# contract that pre-push callers verify deps before invoking.
if [ ! -d "site/node_modules" ]; then
echo " SKIP: site/node_modules missing. Install with: (cd site && npm ci)"
else
if ! bash scripts/check-doc-conformance.sh; then
echo "FAIL: doc-conformance guardrail suite failed (see the named test file/case above)."
FAILED=1
fi
# pre-push-token: site-unit
echo "[7/9] Site unit tests..."
if ! npm --prefix site test; then
echo "FAIL: site unit tests failed. Reproduce with: npm --prefix site test"
FAILED=1
fi
fi
# pre-push-token: doc-discipline
echo "[8/9] Doc discipline..."
if ! bash scripts/check-doc-discipline.sh; then
echo "FAIL: doc-discipline failed. Reproduce with: bash scripts/check-doc-discipline.sh"
FAILED=1
fi
# pre-push-token: guards
echo "[9/9] Repo guards..."
if ! bash scripts/check-guards.sh; then
echo "FAIL: one or more repo guards failed. Reproduce with: bash scripts/check-guards.sh"
FAILED=1
fiThe pre-commit hooks share the same script surface plus the standard hygiene hooks — trailing whitespace, end-of-file, YAML validity, gitleaks, clang-format, and the local scripts registered as pass_filenames: false so they always run against the whole tree:
- repo: local
hooks:
- id: check-pragma-once
name: Check pragma once in headers
entry: scripts/check-pragma-once.sh
language: script
files: '\.(h|hpp)$'
- id: check-realtime-safety
name: Check real-time safety
entry: scripts/check-realtime-safety.sh
language: script
pass_filenames: false
- id: check-snippet-regions
name: Check CodeSnippet region markers exist
entry: scripts/check-snippet-regions.sh
language: script
pass_filenames: false
- id: pre-push-check
name: Pre-push quality gate
entry: scripts/pre-push-check.sh
language: script
pass_filenames: false
stages: [pre-push]The ci skip list at the bottom of that file prevents the local-only hooks from firing in pre-commit.ci (where they would double-run against what CI executes directly).
Layer 3 — Site unit tests
Section titled “Layer 3 — Site unit tests”The website’s audio stack — sample loading, preset-pattern resolution, SMF writer, dump-mode plumbing — has its own Node-based test suite under site/tests/. These use the built-in node:test runner plus node-web-audio-api for a headless OfflineAudioContext, so the tests decode real sample files but never touch the browser or a speaker.
test('loadNotes decodes AudioBuffers for kick / snare / hat notes', async () => {
const manifest = await readManifest();
const context = new OfflineAudioContext(2, 48000, 48000);
const fetcher = filesystemFetcher(SAMPLES_ROOT);
const loader = createSampleLoader({ manifest, context, fetcher });
const buffers = await loader.loadNotes([36, 38, 42]);
assert.equal(buffers.size, 3, 'expected 3 decoded buffers');
for (const note of [36, 38, 42]) {
const buf = buffers.get(note);
assert.ok(buf, `missing buffer for note ${note}`);
assert.ok(buf.length > 0, `note ${note} buffer has zero length`);
assert.ok(
buf.numberOfChannels >= 1,
`note ${note} buffer has no channels`,
);
assert.ok(buf.sampleRate > 0, `note ${note} buffer has zero sampleRate`);
}
});Unit tests here focus on the pure data layer: loader memoization, note-to-sample resolution, preferred-role hints, SMF-writer byte layout, and preset-pattern schema conformance. Browser-only surfaces (WebAudio graph wiring, DOM event handlers) get covered at the next layer up.
Layer 3 — Playwright end-to-end
Section titled “Layer 3 — Playwright end-to-end”The Playwright suite under site/tests-e2e/ drives a real Chromium against a locally-built preview site. It covers every gate that only manifests once the full stack is running: audio-gate probes (card Play triggers real WebAudio nodes), preset-consistency parity (card path and Try It modal path must resolve to the same engine, lanes, and samples), sample-selection equivalence (both paths pick the same file for every note), macro-diff regression (macro=0 vs macro=1 must produce different SMFs for complexity, density, and swing), lane-mute, control-audit, and a zero-console-error walk of every published page.
The keystone is the preset-consistency spec, which cross-checks the card path and the WASM modal path against each other rather than against a hand-written expectation:
// ---------- Structural agreement ----------
expect(
cardResolved!.engineName,
`[${chapter.preset}] engineName mismatch: card="${cardResolved!.engineName}" wasm="${wasmResolved!.engineName}"`,
).toBe(wasmResolved!.engineName);
expect(
cardResolved!.lanes.length,
`[${chapter.preset}] lane count mismatch: card=${cardResolved!.lanes.length} wasm=${wasmResolved!.lanes.length}`,
).toBe(wasmResolved!.lanes.length);
for (let i = 0; i < cardResolved!.lanes.length; i++) {
const cl = cardResolved!.lanes[i];
const wl = wasmResolved!.lanes[i];
expect(
cl.noteNumber,
`[${chapter.preset}] lane ${i} noteNumber mismatch: card=${cl.noteNumber} wasm=${wl.noteNumber}`,
).toBe(wl.noteNumber);
expect(
cl.roleLabel,
`[${chapter.preset}] lane ${i} roleLabel mismatch: card="${cl.roleLabel}" wasm="${wl.roleLabel}"`,
).toBe(wl.roleLabel);
}Failures name the chapter, the diverging field, and the two values, so a broken preset is diagnosable from the CI log alone.
Layer 3 — Post-deploy verification pipeline
Section titled “Layer 3 — Post-deploy verification pipeline”scripts/site-verify-local.sh sequences all eight Playwright gates against a fresh local WASM + Astro build. Its remote sibling, scripts/site-verify-remote.sh, hits the deployed Pages URL after each release with the same gate lineup — proving the deployed site behaves like the built site, not just that the built site behaves like the source.
Playwright clears test-results/ at the start of every npx playwright test invocation, so the pipeline captures each gate’s summary JSON into .gsd/artifacts/ immediately after the spec runs — never batched at the end, where every summary but the last would be lost:
# Each `npx playwright test` clears test-results/ at start, so each gate's
# summary must be copied into .gsd/artifacts/ before the next spec runs.
capture_summary() {
local src="$1" dst="$2"
if [ -f "${src}" ]; then
cp "${src}" "${dst}"
echo " wrote ${dst}"
else
echo " (no summary at ${src} — spec may have crashed before writing it)" >&2
fi
}The pipeline’s exit code is the OR of every gate’s exit code, so a single failing spec fails the whole run and the printed banner names which gate blocked.
Layer 4 — Continuous integration
Section titled “Layer 4 — Continuous integration”The ci.yml workflow runs the full pipeline on every push to main and every PR. The site-e2e job is the browser-driven half — it provisions Emscripten, Node, and Playwright, then hands off to scripts/site-verify-local.sh and archives the reports whether the run passes or fails:
site-e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v5
- name: Setup Emscripten
uses: mymindstorm/setup-emsdk@4528d102f7230f0e7b276855c01ea1159be0e984 # v16
with:
version: 3.1.61
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: '24'
cache: npm
cache-dependency-path: site/package-lock.json
- name: Install site dependencies
working-directory: site
run: npm ci
- name: Get Playwright version
id: pw-version
working-directory: site
run: echo "version=$(npx playwright --version | awk '{print $2}')" >> "$GITHUB_OUTPUT"
- name: Cache Playwright browsers
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/ms-playwright
key: playwright-${{ steps.pw-version.outputs.version }}
- name: Install Playwright browsers
working-directory: site
run: npx playwright install --with-deps chromium
- name: Run local site-verify pipeline (S10 + S11 + S13 + sample-equivalence + S14 + S15-S16 macro-diff + S06 first-bar-parity + S18 gates)
run: bash scripts/site-verify-local.shThe parallel build matrix runs the C++ tests across macOS-14, ubuntu-latest, and windows-2022, so an MSVC-only transitive-include bug can’t hide behind a Clang-only PR check. The pluginval job runs the JUCE validator against the built .vst3 on both macOS and Linux runners.
Visual and interaction regression for the plugin’s web UI runs in the Playwright suite (webui/tests/, Chromium); the former dedicated-machine stage for native VSTGUI pixel-baseline visual regression was retired when the native editor was decommissioned in M053 S05.
What the pyramid does not cover
Section titled “What the pyramid does not cover”Cubase-in-the-loop integration testing (an actual DAW hosting the plugin under automated control) is deliberately outside this test surface — it is deferred to a later testing milestone because the harness for that is non-trivial and was out of scope for M044.
Everything else — engine correctness, audio-thread safety, preset data integrity, site-deployed audio behavior, cross-surface parity, and the web UI’s Playwright interaction/visual coverage — is caught by one of the layers above before a change can reach main.
Preview audio uses CC0 and CC-BY drum samples. Every sample is credited on theCredits & Licenses page.