[hero_browser_mcp] Add basic-auth + extra-headers support so MCP browser can drive auth-gated services without nginx workarounds #42

New issue

Open

opened 2026-04-28 12:21:50 +00:00 by mik-tf · 0 comments

mik-tf commented

2026-04-28 12:21:50 +00:00

Owner

Why this matters

During the home#193 [nu-demo] sweep we hit a hard wall using Hero Browser MCP against herodemo.gent01.grid.tf (which sits behind an nginx basic-auth gate, see hero_skills/tools/modules/installers/auth.nu):

Navigating with https://admin:admin123@host/... gets the document past nginx, but Chrome (since v59+) refuses to construct fetch() calls from any URL whose document URL contains embedded credentials. Symptom: Failed to execute 'fetch' on 'Window': Request cannot be constructed from a URL that includes credentials: /….
WASM shells (hero_os) and Askama UIs that call /rpc from JS therefore look like the page "loaded" but their backends never connect — connection errors only, no UI state.
Sub-resource preload + <script type="module"> requests with crossorigin="" also strip URL credentials and silently 401.

We worked around this by SSHing to herodemo and temporarily commenting out the auth_basic directives in /etc/nginx/sites-enabled/hero_demo, running the test, and restoring. That works for one-shot runs but is unacceptable as a general workflow — exposes the host briefly, requires SSH access, and breaks for any contributor without root on the demo VM.

What the canonical pattern looks like

Puppeteer / Playwright / chromiumoxide all expose this as a first-class feature:

Library	API
Puppeteer	`await page.authenticate({ username, password })`
Playwright	`browser.newContext({ httpCredentials: { username, password } })`
chromiumoxide (already a hero_browser dep — `crates/hero_browser_core/Cargo.toml:12`)	direct CDP `Network.setExtraHTTPHeaders` + `Fetch.continueWithAuth`

Under the hood all of them call CDP Fetch.continueWithAuth to respond to Fetch.authRequired events, OR pre-set an Authorization: Basic … header via Network.setExtraHTTPHeaders.

Proposed surface

Two new MCP tools in hero_browser_mcp (matching the existing tool naming style):

mcp__hero_browser__set_basic_auth
  params: { browser_id, page_id, username, password }
  effect: enable Fetch.authRequired interception, respond with provided creds.
          Persists across navigation within the page.

mcp__hero_browser__set_extra_headers
  params: { browser_id, page_id, headers: { [name: string]: string } }
  effect: Network.setExtraHTTPHeaders — applies to every request the page makes,
          including the document load and all sub-resources.

Either alone would unblock the herodemo case. Both together cover the broader "any HTTP-header-gated service" use case (bearer tokens, custom session headers, etc.).

Impl notes

crates/hero_browser_core/src/browser/page.rs already wraps the chromiumoxide Page. Adding set_basic_auth(&self, user: &str, pass: &str) and set_extra_headers(&self, headers: HashMap<String, String>) is local to that file plus matching dispatch in the MCP/tool layer.
chromiumoxide exposes Page::execute(...) for raw CDP commands — Network.setExtraHTTPHeaders is straightforward; Fetch.enable + Fetch.continueWithAuth for the basic-auth path is slightly more ceremony (needs an event subscriber).
Existing crates/hero_browser_core/src/browser/credentials.rs already declares a CredentialStore for cookies + storage — basic-auth creds could live alongside as a third type for persistence.

Why this is high-leverage

The whole point of Hero Browser MCP is autonomous AI-driven verification of Hero services. Today, the moment any service sits behind basic auth, that promise breaks. With these two tools added, Hero Browser becomes the reliable end-to-end verifier across the entire fleet — the canonical "layer 6 visual verification" tier from the testing pyramid (testing_suite skill).

Session case study: we just spent ~20 minutes nginx-juggling to verify home#146 (Photos <img> not rendering — turned out to be a duplicate of home#156's leading-slash bug) and home#154 (purple dock highlight remains after closing apps). With native auth support both would have been minutes, no SSH, no exposure window.

Filed 2026-04-27 from the home#193 close-out work. Signed-off-by: mik-tf

Originally filed as home#194 on 2026-04-27 by mik-tf — moved to hero_demo as part of consolidating issue tracking.

## Why this matters During the home#193 [nu-demo] sweep we hit a hard wall using Hero Browser MCP against `herodemo.gent01.grid.tf` (which sits behind an nginx basic-auth gate, see `hero_skills/tools/modules/installers/auth.nu`): - Navigating with `https://admin:admin123@host/...` gets the document past nginx, but Chrome (since v59+) refuses to construct `fetch()` calls from any URL whose document URL contains embedded credentials. Symptom: `Failed to execute 'fetch' on 'Window': Request cannot be constructed from a URL that includes credentials: /…`. - WASM shells (hero_os) and Askama UIs that call `/rpc` from JS therefore look like the page "loaded" but their backends never connect — connection errors only, no UI state. - Sub-resource preload + `<script type="module">` requests with `crossorigin=""` also strip URL credentials and silently 401. We worked around this by SSHing to herodemo and temporarily commenting out the `auth_basic` directives in `/etc/nginx/sites-enabled/hero_demo`, running the test, and restoring. That works for one-shot runs but is unacceptable as a general workflow — exposes the host briefly, requires SSH access, and breaks for any contributor without root on the demo VM. ## What the canonical pattern looks like Puppeteer / Playwright / chromiumoxide all expose this as a first-class feature: | Library | API | |---------|-----| | Puppeteer | `await page.authenticate({ username, password })` | | Playwright | `browser.newContext({ httpCredentials: { username, password } })` | | chromiumoxide (already a hero_browser dep — `crates/hero_browser_core/Cargo.toml:12`) | direct CDP `Network.setExtraHTTPHeaders` + `Fetch.continueWithAuth` | Under the hood all of them call CDP `Fetch.continueWithAuth` to respond to `Fetch.authRequired` events, OR pre-set an `Authorization: Basic …` header via `Network.setExtraHTTPHeaders`. ## Proposed surface Two new MCP tools in `hero_browser_mcp` (matching the existing tool naming style): ``` mcp__hero_browser__set_basic_auth params: { browser_id, page_id, username, password } effect: enable Fetch.authRequired interception, respond with provided creds. Persists across navigation within the page. mcp__hero_browser__set_extra_headers params: { browser_id, page_id, headers: { [name: string]: string } } effect: Network.setExtraHTTPHeaders — applies to every request the page makes, including the document load and all sub-resources. ``` Either alone would unblock the herodemo case. Both together cover the broader "any HTTP-header-gated service" use case (bearer tokens, custom session headers, etc.). ## Impl notes - `crates/hero_browser_core/src/browser/page.rs` already wraps the chromiumoxide `Page`. Adding `set_basic_auth(&self, user: &str, pass: &str)` and `set_extra_headers(&self, headers: HashMap<String, String>)` is local to that file plus matching dispatch in the MCP/tool layer. - chromiumoxide exposes `Page::execute(...)` for raw CDP commands — `Network.setExtraHTTPHeaders` is straightforward; `Fetch.enable` + `Fetch.continueWithAuth` for the basic-auth path is slightly more ceremony (needs an event subscriber). - Existing `crates/hero_browser_core/src/browser/credentials.rs` already declares a `CredentialStore` for cookies + storage — basic-auth creds could live alongside as a third type for persistence. ## Why this is high-leverage The whole point of Hero Browser MCP is autonomous AI-driven verification of Hero services. Today, the moment any service sits behind basic auth, that promise breaks. With these two tools added, Hero Browser becomes the reliable end-to-end verifier across the entire fleet — the canonical "layer 6 visual verification" tier from the testing pyramid (`testing_suite` skill). Session case study: we just spent ~20 minutes nginx-juggling to verify home#146 (Photos `<img>` not rendering — turned out to be a duplicate of home#156's leading-slash bug) and home#154 (purple dock highlight remains after closing apps). With native auth support both would have been minutes, no SSH, no exposure window. Filed 2026-04-27 from the home#193 close-out work. Signed-off-by: mik-tf --- *Originally filed as [home#194](https://forge.ourworld.tf/lhumina_code/home/issues/194) on 2026-04-27 by mik-tf — moved to hero_demo as part of consolidating issue tracking.*