What this service solves
Turn concerns about an existing assistant into checks your team can repeat. The evaluation covers representative requests and boundary cases, including when to abstain, which material the current user may access, and which actions require approval. Findings connect observed behavior to a practical next control.
What you get
Reproducible evaluation suite
Test cases describe the input, expected response or action, and the conditions needed to rerun them against a reviewed version.
Risk register and priorities
Document mismatches with the affected scenario, consequence, and a suggested control or investigation.
What’s included
Build a representative evaluation set
Select up to 30 approved tests that represent the assistant's actual use, including common requests and cases where it should hand off.
Test abstention, access and approval behavior
Observe whether the assistant abstains or escalates when expected, respects role boundaries, and waits for approval where a change requires it.
Document issues and recommended controls
Record the observed behavior, conditions, and suggested controls. Separate repeatable findings from uncertain or intermittent behavior.
How implementation works
Set expected behavior
Agree on expected results, permitted test inputs, roles, and the assistant configuration under review.
Execute tests and triage results
Run the agreed cases in one round and triage deviations with the owner. Record where the available evidence is insufficient.
Present controls and next steps
Present the suite and findings, and agree which cases the team should repeat after a change.
Technical considerations
- Reproducible test conditions
- Record the assistant version or available configuration identifiers, role, approved data fixture, and relevant retrieval or tool settings for the evaluation. A later comparison needs enough context to distinguish behavior changes from changes to the inputs.
Systems, inputs & prerequisites
Test access, approved prompts, expected results, and an owner for acceptance criteria.
- Authorized test access and the observed configuration or version of the assistant under review.
- Approved prompts, expected results, role examples, and a named owner to resolve unclear acceptance rules.
Starting scope & pricing
Base scope
One existing assistant, up to 30 approved test cases, and one evaluation round.
Illustrative workflow
Set expectation
The owner specifies a fallback for the restricted request.
Run case
The test uses the designated account and records the response.
Review result
The team records a deviation or retains the passing case for a future run.
Validation & handover
- Review a test set with the owner and confirm that each approved case describes the input, user role where relevant, expected behavior, and observed result.
- Record the tested configuration and any cases blocked by missing access. Distinguish repeated deviations from outcomes that varied during the observed round.
- Review the prioritized findings and leave the owner enough context to rerun relevant cases after a configuration change.
Limitations & possible extras
Limitations
The evidence applies to the approved cases and the system versions observed. Remediation, further rounds, and external security assessment are separate engagements.
Coverage is limited to the agreed sample and observed system versions.
Possible extras
Extended testing, remediation, or external security assessment.
Questions about this service
Can we reuse the evaluation after a model change?
Yes. Record the new configuration, rerun the relevant cases, and compare both outcomes and citations or action decisions where applicable.
How do we select up to 30 test cases?
Prioritize actual work requests and boundary cases with meaningful consequences. Include expected answers or actions, plus deliberate examples where the assistant should hand off, abstain, or require approval.
How is an answer that varies between runs reported?
Record the inputs, observed variations, and any known configuration details. Classify variable behavior separately so the owner can decide whether a follow-up round is warranted.
Does the evaluation include fixing the issues found?
The engagement delivers evidence, prioritized findings, and recommended controls for the tested scope. Remediation, expanded testing, and external security assessment are separately scoped.