The short answer
Freeze the old and new tool schemas, callers, validators, policy rules, and downstream effects. Test required and optional fields, types, enums, defaults, nulls, unknown fields, version negotiation, permission checks, idempotency, error handling, and representative side effects. Reject incompatible calls before any external action.
Key takeaways
- Valid JSON is not necessarily a valid or authorized action.
- Defaults and removed fields can change behavior silently.
- Test negative and side-effect cases for every supported version path.
Inventory every party to the tool contract
Record tool name and version, schema, model/provider and prompt version, orchestration code, deterministic validator, policy engine, authentication scope, downstream API version, idempotency rule, error mapping, observability fields, and rollback dependency.
Treat model output as untrusted input. Schema validation belongs in ordinary software before authorization and execution.
Test compatibility as behavior, not syntax
Use sandbox or non-consequential fixtures where possible.
| Case | Expected validator behavior | Execution check | Failure evidence |
|---|---|---|---|
| Old caller → new tool | Accept only documented compatible fields | Same bounded effect | Version or semantic drift |
| New caller → old tool | Reject or downgrade explicitly | No partial external write | Unknown field ignored silently |
| Missing/empty/null | Apply documented distinction | No unsafe default | Required intent inferred |
| Enum/type boundary | Reject invalid values deterministically | No coercion into different action | Provider accepted unintended value |
| Duplicate/retry | Stable action identity | One effect or reconciled no-op | Second charge/message/work order |
| Unauthorized field | Policy denies despite schema validity | No external call | Schema bypasses authority |
Retain an acceptance packet for the deployed pair
Store schema hashes or versions, supported compatibility pairs, fixtures, expected and actual results, validator and policy versions, external receipts for safe tests, known limitations, approval, deployment boundary, and rollback trigger.
Re-run the packet when schema, prompt, model, policy, tool code, or external API semantics change. An unchanged schema does not prove unchanged behavior if defaults or provider interpretation changed.
Operational checklist
Mark your progress, then save a working copy. Selections reset when you leave this page. A checked box is not an approval or evidence of completion.
☐
Old/new schemas and callers frozen
☐
Validator and policy versions recorded
☐
Defaults/nulls/unknowns tested
☐
Negative permission cases passed
☐
Retry and idempotency tested
☐
External effect verified safely
☐
Supported version pairs and rollback retained
0 of 7 marked
Edge cases
- A field is optional syntactically but required by policy: reject before execution.
- The provider accepts extra fields: do not treat permissive parsing as compatibility.
- A schema rename maps to a changed unit: test semantic conversion, not only type.
Sources and references
Follow each source to check the underlying claim. Access checks and professional review are different steps.
Continue the workflow
AI model and provider change acceptance gateProperty-management integration reliability checklistDesign reversible AI property workflowsRevision history
2026-09-18
Initial Phase 4 operational article with a distinct evidence artifact, failure states, source limits, and AI-assisted technical review.