Generator: OSchema marker parser uses substring matching, miscategorizes adjacent markers #67

New issue

Closed

opened 2026-05-19 15:30:37 +00:00 by timur · 1 comment

timur commented

2026-05-19 15:30:37 +00:00

Owner

Context

Discovered by the #261 agent while bootstrapping the hero_service template repo.

The OSchema marker parser identifies section markers via substring matching, which means markers that share a prefix can be matched against the wrong section (e.g., a enum block being read as a fragment of a longer marker, or two adjacent block markers colliding when one is a prefix of the other).

What to do

Locate the marker-parsing logic in crates/oschema/ (or wherever section markers are tokenised).
Replace substring contains / starts_with checks with whole-line / delimiter-aware matching.
Add regression tests covering: adjacent marker collision, prefix-collision (e.g. @root vs @rootobject), markers inside string literals (must not match).

Acceptance

Existing schemas in example/, hero_compute/, hero_osis/, hero_service all parse identically post-fix.
New regression tests cover the failure modes the #261 agent hit.

hero_skills#261 — source of the discovery.
Parent META: hero_skills#262.

## Context Discovered by the [#261 agent](https://forge.ourworld.tf/lhumina_code/hero_skills/issues/261#issuecomment-34264) while bootstrapping the `hero_service` template repo. The OSchema marker parser identifies section markers via substring matching, which means markers that share a prefix can be matched against the wrong section (e.g., a `enum` block being read as a fragment of a longer marker, or two adjacent block markers colliding when one is a prefix of the other). ## What to do - Locate the marker-parsing logic in `crates/oschema/` (or wherever section markers are tokenised). - Replace substring `contains` / `starts_with` checks with whole-line / delimiter-aware matching. - Add regression tests covering: adjacent marker collision, prefix-collision (e.g. `@root` vs `@rootobject`), markers inside string literals (must not match). ## Acceptance - Existing schemas in `example/`, `hero_compute/`, `hero_osis/`, `hero_service` all parse identically post-fix. - New regression tests cover the failure modes the #261 agent hit. ## Related - [hero_skills#261](https://forge.ourworld.tf/lhumina_code/hero_skills/issues/261) — source of the discovery. - Parent META: [hero_skills#262](https://forge.ourworld.tf/lhumina_code/hero_skills/issues/262).

timur referenced this issue from lhumina_code/hero_skills

2026-05-19 15:31:15 +00:00

[META] Standardize Hero service architecture — template, hero_rpc alignment, framework hoist, skills #262

timur referenced this issue from lhumina_code/hero_skills

2026-05-20 00:15:53 +00:00

[META] Standardize Hero service architecture — template, hero_rpc alignment, framework hoist, skills #262

timur referenced this issue from a commit

2026-05-20 00:15:56 +00:00

fix(oschema): delimiter-aware marker parser (closes #67)

timur referenced this issue from a commit

2026-05-20 00:21:54 +00:00

test(oschema): regression tests for marker failure modes + route generator through shared parser (#67)

timur referenced this issue from a pull request that will close it,

2026-05-20 00:25:40 +00:00

fix(oschema): delimiter-aware marker parser (closes #67) #78

timur commented

2026-05-20 00:25:54 +00:00

Author

Owner

PR #78 opened against development.

Root cause: c.text.contains("[rootobject]") (in oschema/parser.rs and generator/schemas/oschema.rs) and source.find("# ===SCHEMA===") (in oschema/parser.rs) are raw substring lookups. Adjacent markers, prefix-colliding names, markers mentioned inside string literals, and any longer comment line that happens to contain the header delimiter could all be miscategorised.

Fix: new crates/oschema/src/markers.rs with a delimiter- and string-literal-aware parse_markers / has_marker / strip_markers / is_header_delimiter_line. The oschema parser uses it for [rootobject] detection and a new split_header helper that requires the # ===SCHEMA=== delimiter to be a whole comment line. The generator (schemas/oschema.rs, generator.rs, domain.rs) is routed through the same helpers so the two crates cannot diverge.

Regression coverage

11 unit tests in markers.rs — adjacent markers, prefix-collision, string-literal escapes, case-insensitivity, whole-line delimiter.
4 parser-level tests in parser.rs exercising the failure modes hit by the #261 agent: marker_regression_adjacent_markers, marker_regression_prefix_collision, marker_regression_string_literal_mention_does_not_match, marker_regression_header_delimiter_must_be_whole_line.
7 integration tests in oschema/tests/real_schemas.rs parsing every .oschema shipped in the repo (basic OSIS example, recipe_server schemas/ + src/recipes/ + src/recipes/core/, example/recipe_server) and asserting the set of root objects is unchanged.

Acceptance: 79 oschema lib + 7 integration tests pass; 125 generator tests pass; all real schemas parse identically (same root-object sets).

PR #78 opened against `development`. **Root cause:** `c.text.contains("[rootobject]")` (in `oschema/parser.rs` and `generator/schemas/oschema.rs`) and `source.find("# ===SCHEMA===")` (in `oschema/parser.rs`) are raw substring lookups. Adjacent markers, prefix-colliding names, markers mentioned inside string literals, and any longer comment line that happens to contain the header delimiter could all be miscategorised. **Fix:** new `crates/oschema/src/markers.rs` with a delimiter- and string-literal-aware `parse_markers` / `has_marker` / `strip_markers` / `is_header_delimiter_line`. The oschema parser uses it for `[rootobject]` detection and a new `split_header` helper that requires the `# ===SCHEMA===` delimiter to be a whole comment line. The generator (`schemas/oschema.rs`, `generator.rs`, `domain.rs`) is routed through the same helpers so the two crates cannot diverge. **Regression coverage** - 11 unit tests in `markers.rs` — adjacent markers, prefix-collision, string-literal escapes, case-insensitivity, whole-line delimiter. - 4 parser-level tests in `parser.rs` exercising the failure modes hit by the #261 agent: `marker_regression_adjacent_markers`, `marker_regression_prefix_collision`, `marker_regression_string_literal_mention_does_not_match`, `marker_regression_header_delimiter_must_be_whole_line`. - 7 integration tests in `oschema/tests/real_schemas.rs` parsing every `.oschema` shipped in the repo (basic OSIS example, recipe_server `schemas/` + `src/recipes/` + `src/recipes/core/`, `example/recipe_server`) and asserting the set of root objects is unchanged. **Acceptance:** 79 oschema lib + 7 integration tests pass; 125 generator tests pass; all real schemas parse identically (same root-object sets).

timur closed this issue

2026-05-20 05:55:04 +00:00

timur referenced this issue from a commit

2026-05-20 05:55:05 +00:00

fix(oschema): delimiter-aware marker parser (closes #67) (#78)

timur referenced this issue from lhumina_code/hero_skills

2026-05-20 05:57:18 +00:00

[META] Standardize Hero service architecture — template, hero_rpc alignment, framework hoist, skills #262