Pilot plan and acceptance criteria

Define the pilot before installing

Choose one team, one supported integration, and one action to govern in a test environment. Write down the business risk, control owner, technical owner, authorized approvers, permitted data, and criteria for expanding or stopping the pilot. Confirm the features available in the pilot workspace; the Pro trial does not include every enterprise capability.

An example objective is: “Changes to a deployment workflow through the connected coding agent require human review; ordinary repository inspection remains available.” Use a disposable repository and harmless edits to test it.

Phase 1: establish the baseline

Follow Your first governed workflow. Record the agent, platform, client version, hook or endpoint, workspace, environment, and active policy revision. Begin with synthetic data and deliberate shadow mode. Confirm that the integration observes the actions you expect.

Capture representative work that should proceed and a nearby case that should match the proposed restriction. Use these as repeatable test cases throughout the pilot.

Phase 2: test the control

Test	Expected observation	Evidence to retain
Allowed action	The harmless operation completes under the intended policy	Action result and correlated decision
Denied action	The targeted operation does not produce its intended side effect	Denial, matched rule, and independent target check
Approved HOLD	The action waits, an authorized human approves, and the supported resume flow proceeds	Checkpoint, resolution, and final operation result
Denied or expired HOLD	The caller applies the denied or configured timeout outcome	Terminal state, effective decision, and target check
Benign near-match	Ordinary work remains usable	Input, observed result, and explanation of the exemption
Policy change and rollback	The intended revision takes effect and the prior revision can be restored	Revisions, authorized change record, and repeated test
Unavailable service or disconnected hook	Behavior matches the documented integration failure posture	Client result, failure signal, and residual-risk decision
Credential or membership revocation	The relevant Kastra access is rejected; underlying tool access is reviewed separately	Authentication/authorization result and offboarding record
Evidence review	A reviewer can explain the recorded action and verification scope	Export or decision record, verification result, and coverage notes

Test failure behavior in an isolated environment using your team’s approved method. Run no destructive command merely to demonstrate blocking. For an application dispatcher, additionally test that partial or held model output never triggers a tool.

Phase 3: measure operational fit

Agree on acceptance thresholds before measuring. Useful measures include the proportion of expected test actions observed, false positives among reviewed cases, approval handling time, added end-to-end latency, unresolved failures, and the time a reviewer needs to reconstruct a decision.

Define each denominator and test period. A lower number of recorded events can reflect caching, missing integration coverage, or collection failures. Document those causes instead of treating an incomplete count as reduced risk. Local measurements are evidence for the tested setup, not a service-level commitment.

Make the rollout decision

Summarize the scope tested, results, exceptions, remaining controls, data-use approval, operational owners, and rollback plan. Review the Trust Center and assurance material alongside the pilot results. Expand only to the agent, platform, data, and workflow combinations your team has accepted.

Operational rollout · Discuss a pilot with Kastra