Add integration tests + testcases/ browser MCP specs (testing pyramid) #11
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Context
Only unit CRUD tests exist (
crates/hero_livekit_server/src/livekit/tests.rs). Pertesting_suite, every Hero service ships a 7-layer pyramid: unit / smoke / API / E2E / Playwright regression / adversarial / visual. None of the higher layers exist today.Goals
crates/hero_livekit_examples/tests/integration.rsthat spins up server + UI via the CLI (#1) and exercises SDK methods (health, discover, create/list/delete room) end-to-end.tests/smoke/bash curl scripts covering/health,/openrpc.json,/.well-known/heroservice.jsonpertests_smoketest.openrpc.json.testcases/directory with hero_browser MCP test specs for the dashboard (perhero_ui_test_create).make test-allruns every layer; CI green.Related skills:
testing_suite,tests_smoketest,hero_ui_test_create,hero_browser_mcp.Implementation Spec for Issue #11 — Testing Pyramid
Objective
Add the missing higher layers of the Hero 7-layer testing pyramid to
hero_livekitso the repo ships with smoke (layer 2), API integration (layer 3), E2E integration (layer 4), and hero_browser MCP visual tests (layer 5/7) alongside the existing unit layer. Provide a unifiedmake test-allentrypoint driven byscripts/run-tests-all.shthat startshero_livekit_server+hero_livekit_uivia the existing singlebin--startCLI, runs every layer against the running stack, and tears down cleanly. Wire CI (.forgejo/workflows/build.yml) to invokemake test-allso every PR enforces the full pyramid.Requirements
crates/hero_livekit_server/src/livekit/tests.rs(reference only, keep as-is).tests/smoke/*.shbash+curl scripts against the running stack covering/health,/openrpc.json,/.well-known/heroservice.jsonon both the serverrpc.sockand UIui.sock, pertests_smoketest.tests/api/*.shbash scripts that call every method incrates/hero_livekit_server/src/livekit/core/openrpc.json(12 LiveKitService methods + the auto-generated CRUD surface forRoom,Participant,AccessToken) over the UI's/rpcHTTP proxy.crates/hero_livekit_examples/tests/integration.rsRust#[tokio::test]suite that:hero_livekit_serverandhero_livekit_ui(via binaries fromtarget/debug/or~/hero/bin/, matching whatmake installproduces).openrpc_client!macro (same pattern asexamples/basic_usage.rsandexamples/health.rs) to exercisestatus → configure → create_room → list_rooms → delete_room → issue_token.testcases/directory followinghero_ui_test_createconventions with test specs for the admin dashboard (Rooms tab, Join tab, connection dot, theme toggle, room CRUD via UI), atestcases/run_all.mdmaster runner, and a repo-local.claude/skills/run_ui_tests/SKILL.mdthat describes how to execute them.make test-allsingle target chaining all layers;scripts/run-tests-all.shhandles server/UI lifecycle.make test-all(new step, not replacingmake test).Files to Modify/Create
Test layer files (create):
tests/smoke/smoke.sh— curl-based smoke checks against server rpc.sock + UI ui.sock endpoints (/health,/openrpc.json,/.well-known/heroservice.json, plus UI HTTP index).tests/smoke/helpers.sh— shared bash helpers (pass/fail/skip,check_get,check_status,check_unix_get,check_unix_post) reused by smoke and API scripts.tests/api/rpc.sh— JSON-RPC 2.0 coverage for everyLiveKitService.*method (12 methods) and the auto-CRUD domain methods forRoom/Participant/AccessToken.tests/api/openrpc_coverage.sh— parsesopenrpc.jsonwithjq, asserts everymethods[].namewas exercised bytests/api/rpc.sh(prevents coverage drift).tests/README.md— describes the four test layers, how to run each individually, and troubleshooting.Lifecycle/orchestration (create):
scripts/run-tests-all.sh— boots server, starts UI in background, waits for~/hero/var/sockets/hero_livekit/rpc.sockandui.sock, runs smoke → api → cargo integration tests → optional browser pass, tears down, propagates first non-zero exit code.scripts/run-smoke-tests.sh— thin wrapper invoked bymake smoke-test, shared lifecycle with run-tests-all but runs only layer 2.scripts/run-api-tests.sh— thin wrapper invoked bymake api-test, shared lifecycle but runs only layer 3.Integration test (create):
crates/hero_livekit_examples/tests/integration.rs— Rust#[tokio::test]E2E suite usingopenrpc_client!; covers health/discover/create-room/list-rooms/delete-room/issue-token end-to-end. Uses shared helper module.crates/hero_livekit_examples/tests/common/mod.rs— helpers: resolve binary paths (target/debug/hero_livekit_server,target/debug/hero_livekit_ui),spawn_server(),spawn_ui(),wait_for_socket(),teardown()withDropguard.Browser MCP specs (create — follow
hero_ui_test_createtemplate):testcases/run_all.md— master runner, parallel execution plan, retry policy, tool cheat sheet.testcases/01_dashboard_load/01_dashboard_load.md— navbar renders, version badge,rooms_countrendered.testcases/02_connection_status/02_connection_status.md—#status-dot/#status-labelreflect service state (online/offline).testcases/03_tab_navigation/03_tab_navigation.md— Rooms ↔ Join tab switching.testcases/04_theme_toggle/04_theme_toggle.md—#theme-toggleflipsdata-bs-theme.testcases/05_room_create/05_room_create.md— create room via#room-nameinput → verify card appears, verify viarpc('LiveKitService.list_rooms', {}).testcases/06_room_delete/06_room_delete.md— delete a test-prefixed room, verify removal in DOM + store.testcases/07_room_advanced_options/07_room_advanced_options.md—#room-max+#room-timeoutinputs round-trip.testcases/08_join_start_meeting/08_join_start_meeting.md—Start meetingbutton opens/join/{room}with auto-issued token.testcases/09_join_with_code/09_join_with_code.md— enter name + room code, verify navigation to/join/{room}?identity=....testcases/10_rpc_proxy/10_rpc_proxy.md—js_evaluatehits/rpcdirectly, verifies JSON-RPC envelope.testcases/11_backend_autostart/11_backend_autostart.md— kill server rpc.sock, reload dashboard, verifyPOST /backend/startauto-spawns..claude/skills/run_ui_tests/SKILL.md— repo-local execution skill (adapted from the template inhero_ui_test_create).Orchestration/CI (modify):
Makefile— add targets:smoke-test,api-test,e2e-test,test-all, update.PHONY..forgejo/workflows/build.yml— add a new stepTest-all (pyramid)after the existingTeststep that runsmake test-all. Hard-fail on failure.crates/hero_livekit_examples/Cargo.toml— add[dev-dependencies]block.README/docs (modify):
README.md— add "Testing" section listing the seven layers and which are implemented.tests/README.md(new) — layer-by-layer docs.Implementation Plan
Step 1: Shared bash helpers + smoke tests (Layer 2)
Files:
tests/smoke/helpers.sh(new)tests/smoke/smoke.sh(new)Actions:
tests/smoke/helpers.shwith color vars,pass/fail/skipcounters, and helpers:check_unix_get <desc> <socket> <path> <expected_substring>.check_unix_status <desc> <socket> <path> <expected_http>.check_unix_post_json <desc> <socket> <path> <json_body> <expected_substring>.smoke.shsourceshelpers.sh, definesRPC_SOCK,UI_SOCK, then asserts:/health,/openrpc.json,/.well-known/heroservice.jsonresponses./health,/healthz,/.well-known/heroservice.json,/returns HTML.exit 1on failure.Dependencies: none. Can run in parallel with: Step 2, Step 5.
Step 2: API layer — RPC coverage (Layer 3)
Files:
tests/api/rpc.sh(new)tests/api/openrpc_coverage.sh(new)Actions:
rpc.shsourcestests/smoke/helpers.sh. Send JSON-RPC 2.0 requests for every method in openrpc.json; assert valid JSON-RPC envelopes (eitherresultorerror.codepresent).openrpc_coverage.shusesjq -r '.methods[].name'to enumerate methods and grepsrpc.shfor each — fails on any method missing.Dependencies: Step 1 (for
helpers.sh). Can run in parallel with: Step 5.Step 3: E2E integration test (Layer 4)
Files:
crates/hero_livekit_examples/tests/common/mod.rs(new)crates/hero_livekit_examples/tests/integration.rs(new)crates/hero_livekit_examples/Cargo.toml(modify — add dev-dependencies)Actions:
[dev-dependencies]block (tokio, anyhow, serde_json, dirs, hero_rpc_derive, hero_rpc_openrpc).common/mod.rs:resolve_bin(),spawn_server(),spawn_ui(),wait_for_socket(),TestStackstruct withDropimpl.integration.rs#[tokio::test] full_pyramid():openrpc_client!against openrpc.json.room.set,room.list,room.get,room.delete,room.exists) which does not require livekit-server children.issue_tokenafter minimalconfigure, asserts JWT shape./healthover Unix socket.openrpc_discovery()— fetches/openrpc.jsonover rpc.sock, asserts method count.Dependencies: none. Can run in parallel with: Step 1, Step 2, Step 5.
Step 4: Orchestration script + Makefile targets
Files:
scripts/run-tests-all.sh(new)scripts/run-smoke-tests.sh(new)scripts/run-api-tests.sh(new)Makefile(modify)Actions:
run-tests-all.sh:set -euo pipefail, optionallymake install, start server + UI in background, poll for sockets (max 10s),trap EXIT INT TERMto cleanup, run smoke → api → openrpc_coverage → cargo integration, aggregate exit codes.run-smoke-tests.sh,run-api-tests.sh: thin wrappers with same lifecycle but only one phase.Makefile: addsmoke-test,api-test,e2e-test,test-alltargets; update.PHONY.test-allrunscargo test --workspacefirst, thenscripts/run-tests-all.sh.Dependencies: Step 1, Step 2, Step 3. Must run after them.
Step 5: Browser MCP testcases + repo-local skill (Layer 5/7)
Files:
testcases/run_all.md(new)testcases/01_dashboard_load/01_dashboard_load.md(new)testcases/02_connection_status/02_connection_status.md(new)testcases/03_tab_navigation/03_tab_navigation.md(new)testcases/04_theme_toggle/04_theme_toggle.md(new)testcases/05_room_create/05_room_create.md(new)testcases/06_room_delete/06_room_delete.md(new)testcases/07_room_advanced_options/07_room_advanced_options.md(new)testcases/08_join_start_meeting/08_join_start_meeting.md(new)testcases/09_join_with_code/09_join_with_code.md(new)testcases/10_rpc_proxy/10_rpc_proxy.md(new)testcases/11_backend_autostart/11_backend_autostart.md(new).claude/skills/run_ui_tests/SKILL.md(new)Actions:
hero_ui_test_createtemplate exactly (Purpose, Prerequisites, Steps, Assertions, Recovery, Cleanup).templates/index.html+dashboard.js:#status-dot,#status-label,#theme-toggle,#room-name,#room-max,#room-timeout,#rooms-grid,.nav-tabs-custom .nav-link,#start-identity,#join-identity,#join-room, etc.test-lk-<NN>.rpc('LiveKitService.delete_room', {name: 'test-lk-NN'})viajs_evaluate.run_all.md: prereqs, parallel plan, retry policy, tool cheat sheet..claude/skills/run_ui_tests/SKILL.md: repo-local template filled forhero_livekit_ui.Dependencies: none. Can run in parallel with: Step 1, Step 2, Step 3.
Step 6: CI workflow + README docs
Files:
.forgejo/workflows/build.yml(modify)README.md(modify)tests/README.md(new)Actions:
Test-all (pyramid)step tobuild.ymlafterTest; installcurl/socat/jq; runmake test-all; hard-fail.tests/README.md.README.md: add "Testing" section listing seven layers with green/manual status.tests/README.md: one section per layer; note thatLiveKitService.install/startare skipped on CI (download upstream binaries + need Redis).Dependencies: Step 1-5.
Acceptance Criteria
make test-allexits 0 on a clean checkout with Redis not required.tests/smoke/smoke.shasserts/health,/openrpc.json,/.well-known/heroservice.jsonon both sockets plus UI HTML index + 404 path.tests/api/openrpc_coverage.shproves every method in openrpc.json is exercised bytests/api/rpc.sh.crates/hero_livekit_examples/tests/integration.rsruns undercargo test -p hero_livekit_examples --test integrationwithout Redis.testcases/contains at least 11 test specs +run_all.md+ repo-local skill, all following thehero_ui_test_createtemplate..forgejo/workflows/build.ymlhas aTest-all (pyramid)step that runsmake test-allafterTest, installscurl/socat/jq, and fails the build on test failure.README.mdhas a Testing section listing the seven layers with green/manual status.Makefileaddssmoke-test,api-test,e2e-test,test-alltargets (all PHONY, all thin wrappers overscripts/*.sh).crates/hero_livekit_server/src/livekit/tests.rs.Notes
LiveKitService.install(downloads upstream binaries) andLiveKitService.start(spawns livekit-server needing Redis). Those methods are exercised only for error-path assertions.ghcr.io/despiegk/builder:latest. TheTest-allstep installscurl/socat/jq.examples/keep working — dev-dependencies are additive.hero_rpc_serverroutes (/health,/openrpc.json,/.well-known/heroservice.json,/rpc) are provided automatically byHeroLifecycle::new(...)— no new server wiring.