The short answer
Freeze the old and new tool schemas, callers, validators, policy rules, and downstream effects. Test required and optional fields, types, enums, defaults, nulls, unknown fields, version negotiation, permission checks, idempotency, error handling, and representative side effects. Reject incompatible calls before any external action.
Key takeaways
Inventory every party to the tool contract
Record tool name and version, schema, model/provider and prompt version, orchestration code, deterministic validator, policy engine, authentication scope, downstream API version, idempotency rule, error mapping, observability fields, and rollback dependency.
Treat model output as untrusted input. Schema validation belongs in ordinary software before authorization and execution.
Test compatibility as behavior, not syntax
Use sandbox or non-consequential fixtures where possible.
| Case | Expected validator behavior | Execution check | Failure evidence |
|---|---|---|---|
| Old caller → new tool | Accept only documented compatible fields | Same bounded effect | Version or semantic drift |
| New caller → old tool | Reject or downgrade explicitly | No partial external write | Unknown field ignored silently |
| Missing/empty/null | Apply documented distinction | No unsafe default | Required intent inferred |
| Enum/type boundary | Reject invalid values deterministically | No coercion into different action | Provider accepted unintended value |
| Duplicate/retry | Stable action identity | One effect or reconciled no-op | Second charge/message/work order |
| Unauthorized field | Policy denies despite schema validity | No external call | Schema bypasses authority |
Retain an acceptance packet for the deployed pair
Store schema hashes or versions, supported compatibility pairs, fixtures, expected and actual results, validator and policy versions, external receipts for safe tests, known limitations, approval, deployment boundary, and rollback trigger.
Re-run the packet when schema, prompt, model, policy, tool code, or external API semantics change. An unchanged schema does not prove unchanged behavior if defaults or provider interpretation changed.
Operational checklist
Mark your progress, then save a working copy. Selections reset when you leave this page. A checked box is not an approval or evidence of completion.
0 of 7 marked
Edge cases
Sources and references
Follow each source to check the underlying claim. Access checks and professional review are different steps.