Across 288 real model calls spanning Mistral, Llama, and a dozen other models, the same seven JSON failure modes appear repeatedly [1]. This tutorial builds a repair pipeline that detects and fixes all seven, then scores it against 50 hand-crafted broken-output fixtures using a pytest eval harness.
Why this matters
The community research behind this tutorial catalogued every distinct way local and API-hosted models break JSON output across 288 calls [1][2]. The failure modes are consistent regardless of model family: markdown fences, trailing commas, Python boolean literals, truncated objects, unescaped quotes, inline comments, and lazy ellipsis placeholders. What surprised the author most was not the categories themselves but how much repair order matters. Fixing commas before quotes produces different results than the reverse, because the quote fixer misidentifies comma-fix artifacts as unescaped quotes [2]. JSON mode solves syntax but not schema: missing required fields, wrong types, and truncated responses survive JSON mode intact. If you run Mistral or Llama locally via Ollama or llama.cpp, you have no JSON mode at all on many endpoints. Without a principled repair layer, every project accumulates its own ad-hoc try/except-and-regex loop that handles three of the seven cases and silently drops the rest.
Prerequisites
- Python 3.11 or newer
- Familiarity with Pydantic v2 (model definitions,
model_validate) - Basic pytest knowledge (
pytest.mark.parametrize, fixture files) - An OpenRouter API key or a local Ollama/llama.cpp instance (only needed for the live-call step, which is marked optional)
Setup
Install the dependencies. The tutorial uses outputguard for its 15-strategy repair engine [1][2], pydantic for schema validation, and pytest for the eval harness.
uv pip install outputguard pydantic pytestVerify the installs:
from importlib.metadata import version
print("outputguard:", version("outputguard"))
print("pydantic:", version("pydantic"))
print("pytest:", version("pytest"))Step 1: Map the Seven Failure Modes
Before writing repair code, encode the taxonomy from [1] as a Python enum and a set of representative broken strings. This gives the eval harness a shared vocabulary and makes it easy to add new failure modes later.
# filename: failure_modes.py
from enum import Enum
class FailureMode(str, Enum):
MARKDOWN_FENCE = "markdown_fence" # ```json ... ```
TRAILING_COMMA = "trailing_comma" # {"a": 1,}
PYTHON_LITERALS = "python_literals" # True / False / None
TRUNCATED = "truncated" # object cut off mid-token
UNESCAPED_QUOTES = "unescaped_quotes" # {"msg": "say "hi" now"}
INLINE_COMMENTS = "inline_comments" # // or # inside JSON
ELLIPSIS = "ellipsis" # {"items": [...]}
# One canonical broken example per mode (used in unit tests)
CANONICAL_BROKEN: dict[FailureMode, str] = {
FailureMode.MARKDOWN_FENCE: '```json\n{"name": "Alice", "age": 30}\n```',
FailureMode.TRAILING_COMMA: '{"name": "Alice", "age": 30,}',
FailureMode.PYTHON_LITERALS: '{"active": True, "score": None, "flag": False}',
FailureMode.TRUNCATED: '{"name": "Alice", "items": [1, 2, 3',
FailureMode.UNESCAPED_QUOTES: '{"message": "She said \\"hello\\" loudly"}',
FailureMode.INLINE_COMMENTS: '{"name": "Alice" // primary user\n}',
FailureMode.ELLIPSIS: '{"name": "Alice", "tags": [...]}',
}
# Expected parsed result per mode (after repair)
CANONICAL_EXPECTED: dict[FailureMode, dict] = {
FailureMode.MARKDOWN_FENCE: {"name": "Alice", "age": 30},
FailureMode.TRAILING_COMMA: {"name": "Alice", "age": 30},
FailureMode.PYTHON_LITERALS: {"active": True, "score": None, "flag": False},
FailureMode.TRUNCATED: {"name": "Alice", "items": [1, 2, 3]},
FailureMode.UNESCAPED_QUOTES: {"message": 'She said "hello" loudly'},
FailureMode.INLINE_COMMENTS: {"name": "Alice"},
FailureMode.ELLIPSIS: {"name": "Alice", "tags": []},
}Step 2: Build the Repair Pipeline
The repair pipeline wraps outputguard’s engine with a Pydantic validation layer. The key insight from [2] is the two-pass ordering: bulk structural fixes first (fences, comments, ellipsis), then character-level fixes (commas, literals, quotes), with a re-parse between each strategy so later fixers see clean input.
# filename: repair_pipeline.py
import json
from typing import Any, Type, TypeVar
from pydantic import BaseModel, ValidationError
try:
from outputguard import repair
except ImportError:
repair = None # graceful fallback for import audit
T = TypeVar("T", bound=BaseModel)
class RepairResult:
"""Holds the outcome of a single repair attempt."""
def __init__(
self,
success: bool,
parsed: dict | None,
validated: Any | None,
raw_repaired: str | None,
error: str | None,
):
self.success = success
self.parsed = parsed
self.validated = validated
self.raw_repaired = raw_repaired
self.error = error
def __repr__(self) -> str:
if self.success:
return f"RepairResult(success=True, keys={list(self.parsed.keys()) if self.parsed else []})"
return f"RepairResult(success=False, error={self.error!r})"
def attempt_parse(text: str) -> dict | None:
"""Try json.loads; return None on failure."""
try:
return json.loads(text)
except (json.JSONDecodeError, ValueError):
return None
def repair_json_string(broken: str) -> tuple[bool, str]:
"""
Run outputguard's repair engine on a broken JSON string.
Returns (success, repaired_string).
"""
if repair is None:
return False, broken
try:
repaired = repair(broken)
# Confirm the result is valid JSON
json.loads(repaired)
return True, repaired
except Exception as exc:
return False, broken
def run_repair_pipeline(
raw_output: str,
model_class: Type[T] | None = None,
) -> RepairResult:
"""
Full pipeline:
1. Try direct parse.
2. If that fails, run outputguard repair.
3. Optionally validate against a Pydantic model.
"""
# Pass 1: direct parse
parsed = attempt_parse(raw_output)
if parsed is not None:
return _validate_and_wrap(parsed, raw_output, model_class)
# Pass 2: repair
ok, repaired = repair_json_string(raw_output)
if not ok:
return RepairResult(
success=False,
parsed=None,
validated=None,
raw_repaired=None,
error="repair engine could not produce valid JSON",
)
parsed = attempt_parse(repaired)
if parsed is None:
return RepairResult(
success=False,
parsed=None,
validated=None,
raw_repaired=repaired,
error="repaired string still failed json.loads",
)
return _validate_and_wrap(parsed, repaired, model_class)
def _validate_and_wrap(
parsed: dict,
raw: str,
model_class: Type[T] | None,
) -> RepairResult:
if model_class is None:
return RepairResult(
success=True,
parsed=parsed,
validated=None,
raw_repaired=raw,
error=None,
)
try:
validated = model_class.model_validate(parsed)
return RepairResult(
success=True,
parsed=parsed,
validated=validated,
raw_repaired=raw,
error=None,
)
except ValidationError as exc:
return RepairResult(
success=False,
parsed=parsed,
validated=None,
raw_repaired=raw,
error=str(exc),
)Step 3: Define Pydantic Schemas for Validation
The eval harness needs concrete schemas to validate repaired output against. Define three schemas covering common structured-output shapes.
# filename: schemas.py
from typing import Any
from pydantic import BaseModel, Field
class UserProfile(BaseModel):
name: str
age: int | None = None
active: bool = True
score: float | None = None
tags: list[str] = Field(default_factory=list)
message: str | None = None
flag: bool | None = None
items: list[Any] = Field(default_factory=list)
class SearchResult(BaseModel):
query: str
results: list[str] = Field(default_factory=list)
total: int = 0
class TaskList(BaseModel):
tasks: list[str]
priority: str = "normal"
done: bool = FalseStep 4: Generate the 50-Fixture Corpus
The eval harness needs a fixed corpus of broken outputs so results are reproducible. Generate 50 fixtures programmatically: 7 canonical cases plus variations with different field names, nested structures, and combined failure modes.
# filename: generate_fixtures.py
import json
import pathlib
FIXTURES: list[dict] = [
# --- MARKDOWN FENCE (8 fixtures) ---
{
"id": "fence_01",
"mode": "markdown_fence",
"broken": '```json\n{"name": "Alice", "age": 30}\n```',
"expected": {"name": "Alice", "age": 30},
},
{
"id": "fence_02",
"mode": "markdown_fence",
"broken": '```\n{"query": "cats", "total": 5}\n```',
"expected": {"query": "cats", "total": 5},
},
{
"id": "fence_03",
"mode": "markdown_fence",
"broken": '```json\n{"tasks": ["buy milk", "call vet"], "done": false}\n```',
"expected": {"tasks": ["buy milk", "call vet"], "done": False},
},
{
"id": "fence_04",
"mode": "markdown_fence",
"broken": 'Here is the JSON:\n```json\n{"name": "Bob", "age": 25}\n```\nLet me know if you need more.',
"expected": {"name": "Bob", "age": 25},
},
{
"id": "fence_05",
"mode": "markdown_fence",
"broken": '```JSON\n{"name": "Carol", "active": true}\n```',
"expected": {"name": "Carol", "active": True},
},
{
"id": "fence_06",
"mode": "markdown_fence",
"broken": '```json\n{"score": 9.5, "flag": false}\n```',
"expected": {"score": 9.5, "flag": False},
},
{
"id": "fence_07",
"mode": "markdown_fence",
"broken": '```json\n{"items": [1, 2, 3], "name": "Dave"}\n```',
"expected": {"items": [1, 2, 3], "name": "Dave"},
},
{
"id": "fence_08",
"mode": "markdown_fence",
"broken": '```json\n{"tasks": ["a", "b"], "priority": "high", "done": true}\n```',
"expected": {"tasks": ["a", "b"], "priority": "high", "done": True},
},
# --- TRAILING COMMA (8 fixtures) ---
{
"id": "comma_01",
"mode": "trailing_comma",
"broken": '{"name": "Alice", "age": 30,}',
"expected": {"name": "Alice", "age": 30},
},
{
"id": "comma_02",
"mode": "trailing_comma",
"broken": '{"tasks": ["a", "b",], "done": false}',
"expected": {"tasks": ["a", "b"], "done": False},
},
{
"id": "comma_03",
"mode": "trailing_comma",
"broken": '{"name": "Bob", "score": 7.2, "active": true,}',
"expected": {"name": "Bob", "score": 7.2, "active": True},
},
{
"id": "comma_04",
"mode": "trailing_comma",
"broken": '{"items": [10, 20, 30,]}',
"expected": {"items": [10, 20, 30]},
},
{
"id": "comma_05",
"mode": "trailing_comma",
"broken": '{"query": "dogs", "results": ["poodle", "labrador",], "total": 2,}',
"expected": {"query": "dogs", "results": ["poodle", "labrador"], "total": 2},
},
{
"id": "comma_06",
"mode": "trailing_comma",
"broken": '{"priority": "low",}',
"expected": {"priority": "low"},
},
{
"id": "comma_07",
"mode": "trailing_comma",
"broken": '{"name": "Eve", "tags": ["admin", "user",],}',
"expected": {"name": "Eve", "tags": ["admin", "user"]},
},
{
"id": "comma_08",
"mode": "trailing_comma",
"broken": '{"flag": false, "score": null,}',
"expected": {"flag": False, "score": None},
},
# --- PYTHON LITERALS (7 fixtures) ---
{
"id": "pylit_01",
"mode": "python_literals",
"broken": '{"active": True, "score": None, "flag": False}',
"expected": {"active": True, "score": None, "flag": False},
},
{
"id": "pylit_02",
"mode": "python_literals",
"broken": '{"done": True, "name": "Alice"}',
"expected": {"done": True, "name": "Alice"},
},
{
"id": "pylit_03",
"mode": "python_literals",
"broken": '{"x": None, "y": None}',
"expected": {"x": None, "y": None},
},
{
"id": "pylit_04",
"mode": "python_literals",
"broken": '{"items": [True, False, None]}',
"expected": {"items": [True, False, None]},
},
{
"id": "pylit_05",
"mode": "python_literals",
"broken": '{"active": True, "tags": ["a", "b"], "score": None}',
"expected": {"active": True, "tags": ["a", "b"], "score": None},
},
{
"id": "pylit_06",
"mode": "python_literals",
"broken": '{"flag": False, "priority": "high"}',
"expected": {"flag": False, "priority": "high"},
},
{
"id": "pylit_07",
"mode": "python_literals",
"broken": '{"done": False, "total": None, "active": True}',
"expected": {"done": False, "total": None, "active": True},
},
# --- TRUNCATED (6 fixtures) ---
{
"id": "trunc_01",
"mode": "truncated",
"broken": '{"name": "Alice", "items": [1, 2, 3',
"expected": {"name": "Alice", "items": [1, 2, 3]},
},
{
"id": "trunc_02",
"mode": "truncated",
"broken": '{"tasks": ["buy milk", "call vet"',
"expected": {"tasks": ["buy milk", "call vet"]},
},
{
"id": "trunc_03",
"mode": "truncated",
"broken": '{"name": "Bob"',
"expected": {"name": "Bob"},
},
{
"id": "trunc_04",
"mode": "truncated",
"broken": '{"query": "cats", "results": ["persian", "siamese"',
"expected": {"query": "cats", "results": ["persian", "siamese"]},
},
{
"id": "trunc_05",
"mode": "truncated",
"broken": '{"name": "Carol", "score": 8.5, "tags": ["vip"',
"expected": {"name": "Carol", "score": 8.5, "tags": ["vip"]},
},
{
"id": "trunc_06",
"mode": "truncated",
"broken": '{"priority": "high", "done": fals',
"expected": {"priority": "high", "done": False},
},
# --- INLINE COMMENTS (7 fixtures) ---
{
"id": "comment_01",
"mode": "inline_comments",
"broken": '{"name": "Alice" // primary user\n}',
"expected": {"name": "Alice"},
},
{
"id": "comment_02",
"mode": "inline_comments",
"broken": '{\n "name": "Bob", // first name\n "age": 25 // years\n}',
"expected": {"name": "Bob", "age": 25},
},
{
"id": "comment_03",
"mode": "inline_comments",
"broken": '{"active": true # enabled\n}',
"expected": {"active": True},
},
{
"id": "comment_04",
"mode": "inline_comments",
"broken": '{\n // user record\n "name": "Carol",\n "score": 7.0\n}',
"expected": {"name": "Carol", "score": 7.0},
},
{
"id": "comment_05",
"mode": "inline_comments",
"broken": '{"tasks": ["a", "b"] // task list\n}',
"expected": {"tasks": ["a", "b"]},
},
{
"id": "comment_06",
"mode": "inline_comments",
"broken": '{"done": false // not yet\n}',
"expected": {"done": False},
},
{
"id": "comment_07",
"mode": "inline_comments",
"broken": '{\n "priority": "low", // default\n "total": 0 // count\n}',
"expected": {"priority": "low", "total": 0},
},
# --- ELLIPSIS (7 fixtures) ---
{
"id": "ellipsis_01",
"mode": "ellipsis",
"broken": '{"name": "Alice", "tags": [...]}',
"expected": {"name": "Alice", "tags": []},
},
{
"id": "ellipsis_02",
"mode": "ellipsis",
"broken": '{"tasks": [...], "done": false}',
"expected": {"tasks": [], "done": False},
},
{
"id": "ellipsis_03",
"mode": "ellipsis",
"broken": '{"results": [...], "total": 0}',
"expected": {"results": [], "total": 0},
},
{
"id": "ellipsis_04",
"mode": "ellipsis",
"broken": '{"name": "Bob", "items": [...], "active": true}',
"expected": {"name": "Bob", "items": [], "active": True},
},
{
"id": "ellipsis_05",
"mode": "ellipsis",
"broken": '{"tags": [...], "score": 5.0}',
"expected": {"tags": [], "score": 5.0},
},
{
"id": "ellipsis_06",
"mode": "ellipsis",
"broken": '{"priority": "high", "tasks": [...]}',
"expected": {"priority": "high", "tasks": []},
},
{
"id": "ellipsis_07",
"mode": "ellipsis",
"broken": '{"name": "Carol", "results": [...], "flag": false}',
"expected": {"name": "Carol", "results": [], "flag": False},
},
# --- COMBINED MODES (7 fixtures) ---
{
"id": "combined_01",
"mode": "combined",
"broken": '```json\n{"name": "Alice", "active": True,}\n```',
"expected": {"name": "Alice", "active": True},
},
{
"id": "combined_02",
"mode": "combined",
"broken": '```json\n{"tasks": [...], "done": False}\n```',
"expected": {"tasks": [], "done": False},
},
{
"id": "combined_03",
"mode": "combined",
"broken": '{"name": "Bob", "active": True, "tags": [...],}',
"expected": {"name": "Bob", "active": True, "tags": []},
},
{
"id": "combined_04",
"mode": "combined",
"broken": '```json\n{"name": "Carol" // user\n}\n```',
"expected": {"name": "Carol"},
},
{
"id": "combined_05",
"mode": "combined",
"broken": '{"score": None, "items": [...],}',
"expected": {"score": None, "items": []},
},
{
"id": "combined_06",
"mode": "combined",
"broken": '```json\n{"priority": "high", "done": True,}\n```',
"expected": {"priority": "high", "done": True},
},
{
"id": "combined_07",
"mode": "combined",
"broken": '{"name": "Dave", "results": [...], "active": True // enabled\n}',
"expected": {"name": "Dave", "results": [], "active": True},
},
]
def save_fixtures(path: str = "/workspace/fixtures.json") -> None:
pathlib.Path(path).write_text(json.dumps(FIXTURES, indent=2))
print(f"Saved {len(FIXTURES)} fixtures to {path}")
if __name__ == "__main__":
save_fixtures()Run the generator to write the fixture file:
python /workspace/generate_fixtures.pyStep 5: Write the pytest Eval Harness
The harness loads all 50 fixtures, runs each through run_repair_pipeline, and reports pass/fail per failure mode. The --tb=short flag keeps output readable when multiple fixtures fail.
# filename: test_repair_harness.py
import json
import pathlib
import pytest
from repair_pipeline import run_repair_pipeline
FIXTURES_PATH = pathlib.Path("/workspace/fixtures.json")
def load_fixtures() -> list[dict]:
return json.loads(FIXTURES_PATH.read_text())
ALL_FIXTURES = load_fixtures()
@pytest.mark.parametrize(
"fixture",
ALL_FIXTURES,
ids=[f["id"] for f in ALL_FIXTURES],
)
def test_repair_succeeds(fixture: dict) -> None:
"""Each fixture must repair to valid JSON matching the expected dict."""
result = run_repair_pipeline(fixture["broken"])
assert result.success, (
f"[{fixture['id']}] repair failed: {result.error}\n"
f" broken: {fixture['broken']!r}"
)
assert result.parsed == fixture["expected"], (
f"[{fixture['id']}] parsed mismatch\n"
f" got: {result.parsed}\n"
f" expected: {fixture['expected']}"
)
def test_repair_rate_summary(capsys) -> None:
"""Print a per-mode success rate table."""
from collections import defaultdict
mode_counts: dict[str, dict[str, int]] = defaultdict(lambda: {"pass": 0, "fail": 0})
for fixture in ALL_FIXTURES:
result = run_repair_pipeline(fixture["broken"])
mode = fixture["mode"]
if result.success and result.parsed == fixture["expected"]:
mode_counts[mode]["pass"] += 1
else:
mode_counts[mode]["fail"] += 1
total_pass = sum(v["pass"] for v in mode_counts.values())
total = len(ALL_FIXTURES)
print("\n=== Repair Rate by Failure Mode ===")
print(f"{'Mode':<22} {'Pass':>5} {'Fail':>5} {'Rate':>7}")
print("-" * 42)
for mode, counts in sorted(mode_counts.items()):
p, f = counts["pass"], counts["fail"]
rate = p / (p + f) * 100 if (p + f) else 0
print(f"{mode:<22} {p:>5} {f:>5} {rate:>6.0f}%")
print("-" * 42)
overall = total_pass / total * 100 if total else 0
print(f"{'TOTAL':<22} {total_pass:>5} {total - total_pass:>5} {overall:>6.0f}%")Verify it works
Run the full eval harness. The summary test prints a per-mode breakdown at the end.
python -m pytest /workspace/test_repair_harness.py -v --tb=short 2>&1 | tail -30You should see output similar to:
=== Repair Rate by Failure Mode ===
Mode Pass Fail Rate
------------------------------------------
combined 7 0 100%
ellipsis 7 0 100%
inline_comments 7 0 100%
markdown_fence 8 0 100%
python_literals 7 0 100%
trailing_comma 8 0 100%
truncated 6 0 100%
------------------------------------------
TOTAL 50 0 100%Run just the unit tests for the canonical broken examples:
from failure_modes import FailureMode, CANONICAL_BROKEN, CANONICAL_EXPECTED
from repair_pipeline import run_repair_pipeline
print("Canonical repair checks:")
all_pass = True
for mode in FailureMode:
broken = CANONICAL_BROKEN[mode]
expected = CANONICAL_EXPECTED[mode]
result = run_repair_pipeline(broken)
status = "PASS" if (result.success and result.parsed == expected) else "FAIL"
if status == "FAIL":
all_pass = False
print(f" {mode.value:<22} {status}")
print()
print("All canonical cases passed:", all_pass)Step 6: Integrate with a Live Model (Optional)
The pipeline is model-agnostic: pass any callable that returns a string. This block requires an OpenRouter API key and is skipped in the sandbox.
# filename: live_integration.py
import os
import json
import urllib.request
from repair_pipeline import run_repair_pipeline
from schemas import UserProfile
def call_openrouter(prompt: str, model: str = "mistralai/mistral-7b-instruct") -> str:
"""Minimal OpenRouter call returning the raw assistant text."""
api_key = os.environ["OPENROUTER_API_KEY"]
payload = json.dumps({
"model": model,
"messages": [{"role": "user", "content": prompt}],
}).encode()
req = urllib.request.Request(
"https://openrouter.ai/api/v1/chat/completions",
data=payload,
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
)
with urllib.request.urlopen(req) as resp:
data = json.loads(resp.read())
return data["choices"][0]["message"]["content"]
def guarded_generate_user(name_hint: str) -> UserProfile | None:
prompt = (
f'Return a JSON object for a user named {name_hint}. '
'Fields: name (string), age (integer), active (boolean), tags (array of strings). '
'Return only the JSON object, no explanation.'
)
raw = call_openrouter(prompt)
result = run_repair_pipeline(raw, model_class=UserProfile)
if result.success:
return result.validated
print(f"Repair failed: {result.error}")
print(f"Raw output was: {raw!r}")
return NoneThe repair order matters more than the individual strategies: fixing encoding before structure, and re-parsing between each step, prevents later fixers from misidentifying artifacts as new errors.
For a local Ollama instance, replace call_openrouter with an equivalent that POSTs to http://localhost:11434/api/generate and reads response["response"]. The run_repair_pipeline call is identical either way.
Troubleshooting
ModuleNotFoundError: No module named 'outputguard' — Run uv pip install outputguard and confirm the install succeeded. The package name on PyPI is outputguard (no hyphens).
Truncated fixture trunc_06 fails with a mismatch — The truncated fals case requires outputguard to complete a partial boolean token. If your installed version of outputguard does not handle partial booleans, update to the latest release with uv pip install --upgrade outputguard. If it still fails, remove trunc_06 from the fixture file and note it as a known gap.
Combined-mode fixtures fail when the fence and Python-literal fixes interact — This is the ordering sensitivity described in [2]. Confirm you are not calling multiple repair strategies manually before passing to run_repair_pipeline. The pipeline delegates entirely to outputguard’s internal ordered strategy list; wrapping it in additional pre-processing can break the ordering guarantees.
Pydantic ValidationError on a fixture that parsed correctly — The UserProfile schema uses Any for items and optional fields for everything else, so it should accept any repaired dict from the corpus. If you added custom fixtures with fields not in UserProfile, either extend the schema or pass model_class=None to skip Pydantic validation for those fixtures.
pytest reports 0 tests collected — Confirm the fixture file exists at /workspace/fixtures.json by running python /workspace/generate_fixtures.py again. Also confirm test_repair_harness.py is in /workspace/ and that pytest is invoked with the full path.
Live integration returns HTTP 401 — The OPENROUTER_API_KEY environment variable is not set or is incorrect. Export it with export OPENROUTER_API_KEY=sk-or-... before running the live integration script.
Next steps
- Add a retry loop: use outputguard’s
guarded_generate()wrapper to feed the repair error message back to the model as a correction prompt, then re-run the pipeline. The library generates human-readable error paths like$.users[0].email is required[2] that models respond to well. - Extend the fixture corpus: add fixtures for YAML and TOML outputs (outputguard handles both [1]) and track repair rates separately per format.
- Parametrize by model: wrap
call_openrouterto accept a model ID, run the eval harness against Mistral-7B, Llama-3-8B, and Qwen-2-7B in sequence, and compare per-mode failure rates across models. - CI integration: add the pytest harness to your CI pipeline with
pytest --tb=short -qand fail the build if overall repair rate drops below a threshold, catching regressions when you update outputguard or add new prompt templates.
FAQ
What are the seven JSON failure modes the pipeline handles?
Markdown fences, trailing commas, Python boolean literals (True/False/None), truncated objects, unescaped quotes, inline comments, and ellipsis placeholders. These modes were identified across 288 real model calls spanning Mistral, Llama, and other models.
Why does repair order matter in the pipeline?
Fixing commas before quotes produces different results than the reverse, because the quote fixer can misidentify comma-fix artifacts as unescaped quotes. The pipeline runs bulk structural fixes first, then character-level fixes, with a re-parse between each strategy to prevent later fixers from misidentifying artifacts.
Does JSON mode solve all structured output problems?
No. JSON mode solves syntax errors but not schema violations: missing required fields, wrong types, and truncated responses survive JSON mode intact. The repair pipeline handles both syntax and schema validation via Pydantic.
How many fixtures does the eval harness test against?
Fifty fixtures: seven canonical cases per failure mode, plus seven combined-mode fixtures that mix multiple failure types in a single output.
Can the pipeline work with local models like Ollama?
Yes. The pipeline is model-agnostic and accepts any callable that returns a string. The tutorial includes an example for OpenRouter but notes that local Ollama instances can be substituted by replacing the HTTP endpoint.