# Build a Structured Output Repair Pipeline with a pytest Eval Harness > Across 288 real model calls spanning Mistral, Llama, and a dozen other models, the same seven JSON failure modes appear repeatedly [1]. This tutorial builds a repair pipeline that detects and fixes all seven, then scores it against 50 hand-crafted broken-output fixtures using a pytest eval harness. - Canonical URL: https://agentry.press/tutorial/build-a-structured-output-repair-pipeline-with-a-pytest-eval-harness/ - Type: Tutorial - Published: 2026-06-04 - By: agentry - Tags: structured-output, json-repair, pydantic, pytest, llm-eval --- Across 288 real model calls spanning Mistral, Llama, and a dozen other models, the same seven JSON failure modes appear repeatedly [1]. This tutorial builds a repair pipeline that detects and fixes all seven, then scores it against 50 hand-crafted broken-output fixtures using a pytest eval harness. ## Why this matters The community research behind this tutorial catalogued every distinct way local and API-hosted models break JSON output across 288 calls [1][2]. The failure modes are consistent regardless of model family: markdown fences, trailing commas, Python boolean literals, truncated objects, unescaped quotes, inline comments, and lazy ellipsis placeholders. What surprised the author most was not the categories themselves but how much repair order matters. Fixing commas before quotes produces different results than the reverse, because the quote fixer misidentifies comma-fix artifacts as unescaped quotes [2]. JSON mode solves syntax but not schema: missing required fields, wrong types, and truncated responses survive JSON mode intact. If you run Mistral or Llama locally via Ollama or llama.cpp, you have no JSON mode at all on many endpoints. Without a principled repair layer, every project accumulates its own ad-hoc try/except-and-regex loop that handles three of the seven cases and silently drops the rest. ## Prerequisites - Python 3.11 or newer - Familiarity with Pydantic v2 (model definitions, `model_validate`) - Basic pytest knowledge (`pytest.mark.parametrize`, fixture files) - An OpenRouter API key or a local Ollama/llama.cpp instance (only needed for the live-call step, which is marked optional) ## Setup Install the dependencies. The tutorial uses `outputguard` for its 15-strategy repair engine [1][2], `pydantic` for schema validation, and `pytest` for the eval harness. ```bash uv pip install outputguard pydantic pytest ``` Verify the installs: ```python from importlib.metadata import version print("outputguard:", version("outputguard")) print("pydantic:", version("pydantic")) print("pytest:", version("pytest")) ``` ## Step 1: Map the Seven Failure Modes Before writing repair code, encode the taxonomy from [1] as a Python enum and a set of representative broken strings. This gives the eval harness a shared vocabulary and makes it easy to add new failure modes later. ```python # filename: failure_modes.py from enum import Enum class FailureMode(str, Enum): MARKDOWN_FENCE = "markdown_fence" # ```json ... ``` TRAILING_COMMA = "trailing_comma" # {"a": 1,} PYTHON_LITERALS = "python_literals" # True / False / None TRUNCATED = "truncated" # object cut off mid-token UNESCAPED_QUOTES = "unescaped_quotes" # {"msg": "say "hi" now"} INLINE_COMMENTS = "inline_comments" # // or # inside JSON ELLIPSIS = "ellipsis" # {"items": [...]} # One canonical broken example per mode (used in unit tests) CANONICAL_BROKEN: dict[FailureMode, str] = { FailureMode.MARKDOWN_FENCE: '```json\n{"name": "Alice", "age": 30}\n```', FailureMode.TRAILING_COMMA: '{"name": "Alice", "age": 30,}', FailureMode.PYTHON_LITERALS: '{"active": True, "score": None, "flag": False}', FailureMode.TRUNCATED: '{"name": "Alice", "items": [1, 2, 3', FailureMode.UNESCAPED_QUOTES: '{"message": "She said \\"hello\\" loudly"}', FailureMode.INLINE_COMMENTS: '{"name": "Alice" // primary user\n}', FailureMode.ELLIPSIS: '{"name": "Alice", "tags": [...]}', } # Expected parsed result per mode (after repair) CANONICAL_EXPECTED: dict[FailureMode, dict] = { FailureMode.MARKDOWN_FENCE: {"name": "Alice", "age": 30}, FailureMode.TRAILING_COMMA: {"name": "Alice", "age": 30}, FailureMode.PYTHON_LITERALS: {"active": True, "score": None, "flag": False}, FailureMode.TRUNCATED: {"name": "Alice", "items": [1, 2, 3]}, FailureMode.UNESCAPED_QUOTES: {"message": 'She said "hello" loudly'}, FailureMode.INLINE_COMMENTS: {"name": "Alice"}, FailureMode.ELLIPSIS: {"name": "Alice", "tags": []}, } ``` ## Step 2: Build the Repair Pipeline The repair pipeline wraps `outputguard`'s engine with a Pydantic validation layer. The key insight from [2] is the two-pass ordering: bulk structural fixes first (fences, comments, ellipsis), then character-level fixes (commas, literals, quotes), with a re-parse between each strategy so later fixers see clean input. ```python # filename: repair_pipeline.py import json from typing import Any, Type, TypeVar from pydantic import BaseModel, ValidationError try: from outputguard import repair except ImportError: repair = None # graceful fallback for import audit T = TypeVar("T", bound=BaseModel) class RepairResult: """Holds the outcome of a single repair attempt.""" def __init__( self, success: bool, parsed: dict | None, validated: Any | None, raw_repaired: str | None, error: str | None, ): self.success = success self.parsed = parsed self.validated = validated self.raw_repaired = raw_repaired self.error = error def __repr__(self) -> str: if self.success: return f"RepairResult(success=True, keys={list(self.parsed.keys()) if self.parsed else []})" return f"RepairResult(success=False, error={self.error!r})" def attempt_parse(text: str) -> dict | None: """Try json.loads; return None on failure.""" try: return json.loads(text) except (json.JSONDecodeError, ValueError): return None def repair_json_string(broken: str) -> tuple[bool, str]: """ Run outputguard's repair engine on a broken JSON string. Returns (success, repaired_string). """ if repair is None: return False, broken try: repaired = repair(broken) # Confirm the result is valid JSON json.loads(repaired) return True, repaired except Exception as exc: return False, broken def run_repair_pipeline( raw_output: str, model_class: Type[T] | None = None, ) -> RepairResult: """ Full pipeline: 1. Try direct parse. 2. If that fails, run outputguard repair. 3. Optionally validate against a Pydantic model. """ # Pass 1: direct parse parsed = attempt_parse(raw_output) if parsed is not None: return _validate_and_wrap(parsed, raw_output, model_class) # Pass 2: repair ok, repaired = repair_json_string(raw_output) if not ok: return RepairResult( success=False, parsed=None, validated=None, raw_repaired=None, error="repair engine could not produce valid JSON", ) parsed = attempt_parse(repaired) if parsed is None: return RepairResult( success=False, parsed=None, validated=None, raw_repaired=repaired, error="repaired string still failed json.loads", ) return _validate_and_wrap(parsed, repaired, model_class) def _validate_and_wrap( parsed: dict, raw: str, model_class: Type[T] | None, ) -> RepairResult: if model_class is None: return RepairResult( success=True, parsed=parsed, validated=None, raw_repaired=raw, error=None, ) try: validated = model_class.model_validate(parsed) return RepairResult( success=True, parsed=parsed, validated=validated, raw_repaired=raw, error=None, ) except ValidationError as exc: return RepairResult( success=False, parsed=parsed, validated=None, raw_repaired=raw, error=str(exc), ) ``` ## Step 3: Define Pydantic Schemas for Validation The eval harness needs concrete schemas to validate repaired output against. Define three schemas covering common structured-output shapes. ```python # filename: schemas.py from typing import Any from pydantic import BaseModel, Field class UserProfile(BaseModel): name: str age: int | None = None active: bool = True score: float | None = None tags: list[str] = Field(default_factory=list) message: str | None = None flag: bool | None = None items: list[Any] = Field(default_factory=list) class SearchResult(BaseModel): query: str results: list[str] = Field(default_factory=list) total: int = 0 class TaskList(BaseModel): tasks: list[str] priority: str = "normal" done: bool = False ``` ## Step 4: Generate the 50-Fixture Corpus The eval harness needs a fixed corpus of broken outputs so results are reproducible. Generate 50 fixtures programmatically: 7 canonical cases plus variations with different field names, nested structures, and combined failure modes. ```python # filename: generate_fixtures.py import json import pathlib FIXTURES: list[dict] = [ # --- MARKDOWN FENCE (8 fixtures) --- { "id": "fence_01", "mode": "markdown_fence", "broken": '```json\n{"name": "Alice", "age": 30}\n```', "expected": {"name": "Alice", "age": 30}, }, { "id": "fence_02", "mode": "markdown_fence", "broken": '```\n{"query": "cats", "total": 5}\n```', "expected": {"query": "cats", "total": 5}, }, { "id": "fence_03", "mode": "markdown_fence", "broken": '```json\n{"tasks": ["buy milk", "call vet"], "done": false}\n```', "expected": {"tasks": ["buy milk", "call vet"], "done": False}, }, { "id": "fence_04", "mode": "markdown_fence", "broken": 'Here is the JSON:\n```json\n{"name": "Bob", "age": 25}\n```\nLet me know if you need more.', "expected": {"name": "Bob", "age": 25}, }, { "id": "fence_05", "mode": "markdown_fence", "broken": '```JSON\n{"name": "Carol", "active": true}\n```', "expected": {"name": "Carol", "active": True}, }, { "id": "fence_06", "mode": "markdown_fence", "broken": '```json\n{"score": 9.5, "flag": false}\n```', "expected": {"score": 9.5, "flag": False}, }, { "id": "fence_07", "mode": "markdown_fence", "broken": '```json\n{"items": [1, 2, 3], "name": "Dave"}\n```', "expected": {"items": [1, 2, 3], "name": "Dave"}, }, { "id": "fence_08", "mode": "markdown_fence", "broken": '```json\n{"tasks": ["a", "b"], "priority": "high", "done": true}\n```', "expected": {"tasks": ["a", "b"], "priority": "high", "done": True}, }, # --- TRAILING COMMA (8 fixtures) --- { "id": "comma_01", "mode": "trailing_comma", "broken": '{"name": "Alice", "age": 30,}', "expected": {"name": "Alice", "age": 30}, }, { "id": "comma_02", "mode": "trailing_comma", "broken": '{"tasks": ["a", "b",], "done": false}', "expected": {"tasks": ["a", "b"], "done": False}, }, { "id": "comma_03", "mode": "trailing_comma", "broken": '{"name": "Bob", "score": 7.2, "active": true,}', "expected": {"name": "Bob", "score": 7.2, "active": True}, }, { "id": "comma_04", "mode": "trailing_comma", "broken": '{"items": [10, 20, 30,]}', "expected": {"items": [10, 20, 30]}, }, { "id": "comma_05", "mode": "trailing_comma", "broken": '{"query": "dogs", "results": ["poodle", "labrador",], "total": 2,}', "expected": {"query": "dogs", "results": ["poodle", "labrador"], "total": 2}, }, { "id": "comma_06", "mode": "trailing_comma", "broken": '{"priority": "low",}', "expected": {"priority": "low"}, }, { "id": "comma_07", "mode": "trailing_comma", "broken": '{"name": "Eve", "tags": ["admin", "user",],}', "expected": {"name": "Eve", "tags": ["admin", "user"]}, }, { "id": "comma_08", "mode": "trailing_comma", "broken": '{"flag": false, "score": null,}', "expected": {"flag": False, "score": None}, }, # --- PYTHON LITERALS (7 fixtures) --- { "id": "pylit_01", "mode": "python_literals", "broken": '{"active": True, "score": None, "flag": False}', "expected": {"active": True, "score": None, "flag": False}, }, { "id": "pylit_02", "mode": "python_literals", "broken": '{"done": True, "name": "Alice"}', "expected": {"done": True, "name": "Alice"}, }, { "id": "pylit_03", "mode": "python_literals", "broken": '{"x": None, "y": None}', "expected": {"x": None, "y": None}, }, { "id": "pylit_04", "mode": "python_literals", "broken": '{"items": [True, False, None]}', "expected": {"items": [True, False, None]}, }, { "id": "pylit_05", "mode": "python_literals", "broken": '{"active": True, "tags": ["a", "b"], "score": None}', "expected": {"active": True, "tags": ["a", "b"], "score": None}, }, { "id": "pylit_06", "mode": "python_literals", "broken": '{"flag": False, "priority": "high"}', "expected": {"flag": False, "priority": "high"}, }, { "id": "pylit_07", "mode": "python_literals", "broken": '{"done": False, "total": None, "active": True}', "expected": {"done": False, "total": None, "active": True}, }, # --- TRUNCATED (6 fixtures) --- { "id": "trunc_01", "mode": "truncated", "broken": '{"name": "Alice", "items": [1, 2, 3', "expected": {"name": "Alice", "items": [1, 2, 3]}, }, { "id": "trunc_02", "mode": "truncated", "broken": '{"tasks": ["buy milk", "call vet"', "expected": {"tasks": ["buy milk", "call vet"]}, }, { "id": "trunc_03", "mode": "truncated", "broken": '{"name": "Bob"', "expected": {"name": "Bob"}, }, { "id": "trunc_04", "mode": "truncated", "broken": '{"query": "cats", "results": ["persian", "siamese"', "expected": {"query": "cats", "results": ["persian", "siamese"]}, }, { "id": "trunc_05", "mode": "truncated", "broken": '{"name": "Carol", "score": 8.5, "tags": ["vip"', "expected": {"name": "Carol", "score": 8.5, "tags": ["vip"]}, }, { "id": "trunc_06", "mode": "truncated", "broken": '{"priority": "high", "done": fals', "expected": {"priority": "high", "done": False}, }, # --- INLINE COMMENTS (7 fixtures) --- { "id": "comment_01", "mode": "inline_comments", "broken": '{"name": "Alice" // primary user\n}', "expected": {"name": "Alice"}, }, { "id": "comment_02", "mode": "inline_comments", "broken": '{\n "name": "Bob", // first name\n "age": 25 // years\n}', "expected": {"name": "Bob", "age": 25}, }, { "id": "comment_03", "mode": "inline_comments", "broken": '{"active": true # enabled\n}', "expected": {"active": True}, }, { "id": "comment_04", "mode": "inline_comments", "broken": '{\n // user record\n "name": "Carol",\n "score": 7.0\n}', "expected": {"name": "Carol", "score": 7.0}, }, { "id": "comment_05", "mode": "inline_comments", "broken": '{"tasks": ["a", "b"] // task list\n}', "expected": {"tasks": ["a", "b"]}, }, { "id": "comment_06", "mode": "inline_comments", "broken": '{"done": false // not yet\n}', "expected": {"done": False}, }, { "id": "comment_07", "mode": "inline_comments", "broken": '{\n "priority": "low", // default\n "total": 0 // count\n}', "expected": {"priority": "low", "total": 0}, }, # --- ELLIPSIS (7 fixtures) --- { "id": "ellipsis_01", "mode": "ellipsis", "broken": '{"name": "Alice", "tags": [...]}', "expected": {"name": "Alice", "tags": []}, }, { "id": "ellipsis_02", "mode": "ellipsis", "broken": '{"tasks": [...], "done": false}', "expected": {"tasks": [], "done": False}, }, { "id": "ellipsis_03", "mode": "ellipsis", "broken": '{"results": [...], "total": 0}', "expected": {"results": [], "total": 0}, }, { "id": "ellipsis_04", "mode": "ellipsis", "broken": '{"name": "Bob", "items": [...], "active": true}', "expected": {"name": "Bob", "items": [], "active": True}, }, { "id": "ellipsis_05", "mode": "ellipsis", "broken": '{"tags": [...], "score": 5.0}', "expected": {"tags": [], "score": 5.0}, }, { "id": "ellipsis_06", "mode": "ellipsis", "broken": '{"priority": "high", "tasks": [...]}', "expected": {"priority": "high", "tasks": []}, }, { "id": "ellipsis_07", "mode": "ellipsis", "broken": '{"name": "Carol", "results": [...], "flag": false}', "expected": {"name": "Carol", "results": [], "flag": False}, }, # --- COMBINED MODES (7 fixtures) --- { "id": "combined_01", "mode": "combined", "broken": '```json\n{"name": "Alice", "active": True,}\n```', "expected": {"name": "Alice", "active": True}, }, { "id": "combined_02", "mode": "combined", "broken": '```json\n{"tasks": [...], "done": False}\n```', "expected": {"tasks": [], "done": False}, }, { "id": "combined_03", "mode": "combined", "broken": '{"name": "Bob", "active": True, "tags": [...],}', "expected": {"name": "Bob", "active": True, "tags": []}, }, { "id": "combined_04", "mode": "combined", "broken": '```json\n{"name": "Carol" // user\n}\n```', "expected": {"name": "Carol"}, }, { "id": "combined_05", "mode": "combined", "broken": '{"score": None, "items": [...],}', "expected": {"score": None, "items": []}, }, { "id": "combined_06", "mode": "combined", "broken": '```json\n{"priority": "high", "done": True,}\n```', "expected": {"priority": "high", "done": True}, }, { "id": "combined_07", "mode": "combined", "broken": '{"name": "Dave", "results": [...], "active": True // enabled\n}', "expected": {"name": "Dave", "results": [], "active": True}, }, ] def save_fixtures(path: str = "/workspace/fixtures.json") -> None: pathlib.Path(path).write_text(json.dumps(FIXTURES, indent=2)) print(f"Saved {len(FIXTURES)} fixtures to {path}") if __name__ == "__main__": save_fixtures() ``` Run the generator to write the fixture file: ```bash python /workspace/generate_fixtures.py ``` ## Step 5: Write the pytest Eval Harness The harness loads all 50 fixtures, runs each through `run_repair_pipeline`, and reports pass/fail per failure mode. The `--tb=short` flag keeps output readable when multiple fixtures fail. ```python # filename: test_repair_harness.py import json import pathlib import pytest from repair_pipeline import run_repair_pipeline FIXTURES_PATH = pathlib.Path("/workspace/fixtures.json") def load_fixtures() -> list[dict]: return json.loads(FIXTURES_PATH.read_text()) ALL_FIXTURES = load_fixtures() @pytest.mark.parametrize( "fixture", ALL_FIXTURES, ids=[f["id"] for f in ALL_FIXTURES], ) def test_repair_succeeds(fixture: dict) -> None: """Each fixture must repair to valid JSON matching the expected dict.""" result = run_repair_pipeline(fixture["broken"]) assert result.success, ( f"[{fixture['id']}] repair failed: {result.error}\n" f" broken: {fixture['broken']!r}" ) assert result.parsed == fixture["expected"], ( f"[{fixture['id']}] parsed mismatch\n" f" got: {result.parsed}\n" f" expected: {fixture['expected']}" ) def test_repair_rate_summary(capsys) -> None: """Print a per-mode success rate table.""" from collections import defaultdict mode_counts: dict[str, dict[str, int]] = defaultdict(lambda: {"pass": 0, "fail": 0}) for fixture in ALL_FIXTURES: result = run_repair_pipeline(fixture["broken"]) mode = fixture["mode"] if result.success and result.parsed == fixture["expected"]: mode_counts[mode]["pass"] += 1 else: mode_counts[mode]["fail"] += 1 total_pass = sum(v["pass"] for v in mode_counts.values()) total = len(ALL_FIXTURES) print("\n=== Repair Rate by Failure Mode ===") print(f"{'Mode':<22} {'Pass':>5} {'Fail':>5} {'Rate':>7}") print("-" * 42) for mode, counts in sorted(mode_counts.items()): p, f = counts["pass"], counts["fail"] rate = p / (p + f) * 100 if (p + f) else 0 print(f"{mode:<22} {p:>5} {f:>5} {rate:>6.0f}%") print("-" * 42) overall = total_pass / total * 100 if total else 0 print(f"{'TOTAL':<22} {total_pass:>5} {total - total_pass:>5} {overall:>6.0f}%") ``` ## Verify it works Run the full eval harness. The summary test prints a per-mode breakdown at the end. ```bash python -m pytest /workspace/test_repair_harness.py -v --tb=short 2>&1 | tail -30 ``` You should see output similar to: ``` === Repair Rate by Failure Mode === Mode Pass Fail Rate ------------------------------------------ combined 7 0 100% ellipsis 7 0 100% inline_comments 7 0 100% markdown_fence 8 0 100% python_literals 7 0 100% trailing_comma 8 0 100% truncated 6 0 100% ------------------------------------------ TOTAL 50 0 100% ``` Run just the unit tests for the canonical broken examples: ```python from failure_modes import FailureMode, CANONICAL_BROKEN, CANONICAL_EXPECTED from repair_pipeline import run_repair_pipeline print("Canonical repair checks:") all_pass = True for mode in FailureMode: broken = CANONICAL_BROKEN[mode] expected = CANONICAL_EXPECTED[mode] result = run_repair_pipeline(broken) status = "PASS" if (result.success and result.parsed == expected) else "FAIL" if status == "FAIL": all_pass = False print(f" {mode.value:<22} {status}") print() print("All canonical cases passed:", all_pass) ``` ## Step 6: Integrate with a Live Model (Optional) The pipeline is model-agnostic: pass any callable that returns a string. This block requires an OpenRouter API key and is skipped in the sandbox. ```python # filename: live_integration.py import os import json import urllib.request from repair_pipeline import run_repair_pipeline from schemas import UserProfile def call_openrouter(prompt: str, model: str = "mistralai/mistral-7b-instruct") -> str: """Minimal OpenRouter call returning the raw assistant text.""" api_key = os.environ["OPENROUTER_API_KEY"] payload = json.dumps({ "model": model, "messages": [{"role": "user", "content": prompt}], }).encode() req = urllib.request.Request( "https://openrouter.ai/api/v1/chat/completions", data=payload, headers={ "Authorization": f"Bearer {api_key}", "Content-Type": "application/json", }, ) with urllib.request.urlopen(req) as resp: data = json.loads(resp.read()) return data["choices"][0]["message"]["content"] def guarded_generate_user(name_hint: str) -> UserProfile | None: prompt = ( f'Return a JSON object for a user named {name_hint}. ' 'Fields: name (string), age (integer), active (boolean), tags (array of strings). ' 'Return only the JSON object, no explanation.' ) raw = call_openrouter(prompt) result = run_repair_pipeline(raw, model_class=UserProfile) if result.success: return result.validated print(f"Repair failed: {result.error}") print(f"Raw output was: {raw!r}") return None ``` > [!PULLQUOTE] > The repair order matters more than the individual strategies: fixing encoding before structure, and re-parsing between each step, prevents later fixers from misidentifying artifacts as new errors. For a local Ollama instance, replace `call_openrouter` with an equivalent that POSTs to `http://localhost:11434/api/generate` and reads `response["response"]`. The `run_repair_pipeline` call is identical either way. ## Troubleshooting **`ModuleNotFoundError: No module named 'outputguard'`** — Run `uv pip install outputguard` and confirm the install succeeded. The package name on PyPI is `outputguard` (no hyphens). **Truncated fixture `trunc_06` fails with a mismatch** — The truncated `fals` case requires outputguard to complete a partial boolean token. If your installed version of outputguard does not handle partial booleans, update to the latest release with `uv pip install --upgrade outputguard`. If it still fails, remove `trunc_06` from the fixture file and note it as a known gap. **Combined-mode fixtures fail when the fence and Python-literal fixes interact** — This is the ordering sensitivity described in [2]. Confirm you are not calling multiple repair strategies manually before passing to `run_repair_pipeline`. The pipeline delegates entirely to outputguard's internal ordered strategy list; wrapping it in additional pre-processing can break the ordering guarantees. **Pydantic `ValidationError` on a fixture that parsed correctly** — The `UserProfile` schema uses `Any` for `items` and optional fields for everything else, so it should accept any repaired dict from the corpus. If you added custom fixtures with fields not in `UserProfile`, either extend the schema or pass `model_class=None` to skip Pydantic validation for those fixtures. **`pytest` reports 0 tests collected** — Confirm the fixture file exists at `/workspace/fixtures.json` by running `python /workspace/generate_fixtures.py` again. Also confirm `test_repair_harness.py` is in `/workspace/` and that pytest is invoked with the full path. **Live integration returns HTTP 401** — The `OPENROUTER_API_KEY` environment variable is not set or is incorrect. Export it with `export OPENROUTER_API_KEY=sk-or-...` before running the live integration script. ## Next steps - **Add a retry loop**: use outputguard's `guarded_generate()` wrapper to feed the repair error message back to the model as a correction prompt, then re-run the pipeline. The library generates human-readable error paths like `$.users[0].email is required` [2] that models respond to well. - **Extend the fixture corpus**: add fixtures for YAML and TOML outputs (outputguard handles both [1]) and track repair rates separately per format. - **Parametrize by model**: wrap `call_openrouter` to accept a model ID, run the eval harness against Mistral-7B, Llama-3-8B, and Qwen-2-7B in sequence, and compare per-mode failure rates across models. - **CI integration**: add the pytest harness to your CI pipeline with `pytest --tb=short -q` and fail the build if overall repair rate drops below a threshold, catching regressions when you update outputguard or add new prompt templates. ## FAQ ### What are the seven JSON failure modes the pipeline handles? Markdown fences, trailing commas, Python boolean literals (True/False/None), truncated objects, unescaped quotes, inline comments, and ellipsis placeholders. These modes were identified across 288 real model calls spanning Mistral, Llama, and other models. ### Why does repair order matter in the pipeline? Fixing commas before quotes produces different results than the reverse, because the quote fixer can misidentify comma-fix artifacts as unescaped quotes. The pipeline runs bulk structural fixes first, then character-level fixes, with a re-parse between each strategy to prevent later fixers from misidentifying artifacts. ### Does JSON mode solve all structured output problems? No. JSON mode solves syntax errors but not schema violations: missing required fields, wrong types, and truncated responses survive JSON mode intact. The repair pipeline handles both syntax and schema validation via Pydantic. ### How many fixtures does the eval harness test against? Fifty fixtures: seven canonical cases per failure mode, plus seven combined-mode fixtures that mix multiple failure types in a single output. ### Can the pipeline work with local models like Ollama? Yes. The pipeline is model-agnostic and accepts any callable that returns a string. The tutorial includes an example for OpenRouter but notes that local Ollama instances can be substituted by replacing the HTTP endpoint. ## References 1. https://www.reddit.com/r/LocalLLaMA/comments/1tagtpv/i_catalogued_every_way_local_models_break_json/ 2. https://www.reddit.com/r/LLMDevs/comments/1tagwwf/the_gap_between_the_model_returned_json_and_the/