
When an AI output has no single correct answer, replace exact-match assertions with three checks: metamorphic testing (the output keeps an expected relationship when the input is changed in a controlled way), property-based testing (rules such as a valid schema hold across many generated inputs) and evals (outputs are scored against defined quality criteria). Use human review for high-impact or ambiguous cases. An AI system can return a different answer to the same prompt without necessarily being broken. It can also return a fluent, convincing answer that is factually wrong. For QA teams, the difficult question is no longer just whether the output matches an expected string. It is whether the system behaves correctly across different inputs, contexts, and repeated runs.
Metamorphic testing helps address this challenge by checking relationships between outputs instead of requiring an exact expected answer for every input. Combined with property-based testing and LLM evaluations (evals), it gives testers a practical way to detect failures in AI-powered features—even when a perfect test oracle is unavailable.
What Is the Test Oracle Problem?
The test oracle problem occurs when a tester cannot easily determine the correct output for a given input. Traditional tests often compare actual results against known expected values, but that approach becomes difficult when outputs are complex, subjective, or generated probabilistically.
Consider an AI support assistant that summarizes customer conversations. You may not have one uniquely correct summary for every conversation. Two summaries could use different wording while preserving all important facts. However, a summary that invents a refund approval is clearly unacceptable.
The challenge is to distinguish harmless variation from a meaningful failure.
This is especially important when testing AI outputs, recommendation systems, search results, image recognition, and other systems where the expected result is difficult to specify exactly.
A useful response is not to abandon assertions. It is to replace a single exact-output assertion with several checks based on what the system must preserve, what it must never do, and how its behavior should change when the input changes.
How Metamorphic Testing Works
Metamorphic testing checks whether a system maintains a known relationship between outputs when the inputs are transformed in a controlled way. That relationship is called a metamorphic relation.
Instead of asking, “Is this answer exactly correct?”, the tester asks, “When I change the input in this particular way, does the output still satisfy the expected relationship?”
For example, suppose an AI assistant answers questions about a company’s refund policy. You might test the same question with different capitalization or harmless whitespace. The wording may vary, but the policy facts and eligibility conditions should remain consistent.
A metamorphic test has four main parts:
- Original input: A prompt or data sample sent to the system.
- Input transformation: A deliberate change, such as rephrasing the question or adding irrelevant text.
- Expected relation: The property that should remain true or change in a defined way.
- Relation check: An assertion or evaluation that detects a violation.
Practical metamorphic testing examples
| AI feature | Input transformation | Expected relationship | Potential failure |
| Policy assistant | Rephrase a question without changing its meaning | Key policy facts remain consistent | Eligibility rules change |
| Document summarizer | Reorder paragraphs while preserving their content | Important facts remain represented | A critical exception disappears |
| Product recommendation | Change irrelevant formatting in the request | Relevant recommendations remain stable | Recommendations change for no meaningful reason |
| Data extraction | Add unrelated text outside the target record | Extracted target fields remain unchanged | Fields are taken from unrelated content |
| Translation assistant | Change punctuation without changing the intended meaning | Core meaning is preserved | A negation or important condition is lost |
These are examples of relations to validate, not universal rules. For instance, paragraph order can matter in chronological summaries, and recommendations can legitimately change when context affects relevance. A tester must establish the relationship from the feature’s requirements before treating a difference as a defect.
How to Design a Useful Metamorphic Relation
A weak relation creates noisy tests. A good one reflects a meaningful product requirement.
Use this process:
- Step 1: Identify a behavior that is difficult to verify directly. Choose a feature where there is no simple, reliable expected answer for every possible input.
- Step 2: Identify an input transformation. Decide what can change without changing the intended meaning or what change should produce a predictable effect.
- Step 3: Define what must be preserved or changed. Specify observable properties such as factual consistency, valid structure, required fields, or a numerical relationship.
- Step 4: Run the original and transformed inputs. Keep the model version, system instructions, retrieval configuration, and other relevant conditions recorded so that comparisons are meaningful.
- Step 5: Check the relation and investigate violations. A violation is evidence of a possible failure, not automatic proof of a product defect. Confirm that the transformation was valid and that the expected relation actually applies.
For example, if an invoice-extraction model returns a different invoice number after unrelated text is added to the prompt, the result may indicate a context-isolation problem. The team should preserve both inputs, both outputs, the model configuration, and the relation that failed.
Where Property-Based Testing Fits
Property-based testing checks whether general properties hold across many generated inputs rather than relying only on a small set of manually selected examples. It is particularly useful when inputs have many combinations, edge cases, or unexpected formats.
For an AI application, property-based checks often validate the software surrounding the model as well as the model’s response.
Imagine a support assistant that must return a structured JSON object containing category, priority, and summary. A property-based test could generate varied customer messages—including empty strings, long text, Unicode, unusual punctuation, and repeated characters—and verify that the application handles them safely.
Possible properties include:
- The response can be parsed according to the required schema.
- Required fields are present and have valid types.
- Priority values belong to the permitted set.
- Invalid input does not crash the service.
- Sensitive fields are not copied into outputs when the requirements prohibit them.
These checks do not prove that the model’s classification is semantically correct. They establish that the output satisfies specific, testable constraints.
Property-based testing can also generate inputs for metamorphic tests. For example, generate many valid support messages, apply a meaning-preserving transformation to each, and check whether the required classification remains consistent within an explicitly defined tolerance.
Tools such as Hypothesis support property-based testing in Python. The key is to define meaningful properties first; generating thousands of inputs cannot compensate for a property that does not reflect the intended behavior.
LLM Evaluation (Evals): Measuring Output Quality
Metamorphic testing checks relationships between executions. LLM evaluation checks whether outputs meet defined quality criteria across a curated dataset of examples.
An evaluation dataset might contain real or synthetic prompts, relevant context, expected properties, reference answers where appropriate, and labels for known failure modes. Graders then assess outputs using exact checks, programmatic rules, semantic comparisons, or model-assisted judgments.
For example, a retrieval-based assistant might be evaluated for:
- Groundedness: Are factual claims supported by the supplied documents?
- Completeness: Does the answer include the information required by the task?
- Instruction compliance: Does it follow the specified output format and constraints?
- Safety: Does it avoid disallowed disclosures or actions?
- Task success: Does the answer actually resolve the user’s request?
Not every criterion should use an LLM as a judge. Exact schema validation is better handled programmatically. Known factual answers can often be checked against references. Subjective criteria may benefit from a model-assisted grader, but its decisions should be calibrated against human-reviewed examples.
A score is not a guarantee of correctness. A grader can miss a subtle factual error, reward persuasive wording, or disagree with human reviewers. Track false passes and false failures, review borderline cases, and keep human judgment in the loop for high-impact decisions.
For practical guidance, see the OpenAI evaluation best practices.
Metamorphic Testing vs. Property-Based Testing vs. Evals
These approaches solve related but different problems. A robust AI testing strategy uses each for the kind of claim it can actually verify.
| Technique | Main question | Best use | Important limitation |
| Metamorphic testing | Does behavior maintain the expected relationship after a controlled input change? | Consistency and transformation-based checks | A poorly chosen relation can flag valid behavior |
| Property-based testing | Does a defined invariant hold across generated inputs? | Input edge cases, structural constraints, and invariants | It cannot establish subjective correctness by itself |
| LLM evals | Does output meet defined task-quality criteria across representative cases? | Groundedness, relevance, task success, and regressions | Graders and datasets can be incomplete or biased |
| Human review | Is the result acceptable in its real business context? | Ambiguous, high-impact, or subjective decisions | Expensive to scale and requires clear review criteria |
These techniques complement rather than replace one another. Metamorphic tests can expose inconsistencies; property-based checks can uncover boundary failures; evals can measure task quality; and human review can investigate cases where automated judgments are uncertain.
A Practical Workflow for Testing AI Outputs
Teams do not need to build a complex testing platform before getting started. Begin with one high-value AI feature and create a repeatable workflow.
- Define the failure that matters
Choose a concrete risk, such as an assistant inventing refund eligibility, an extraction model assigning data to the wrong customer, or a summarizer omitting a critical exception.
Describe the impact in product terms. “The response changed” is not sufficient; “the response changed a policy condition after an irrelevant formatting change” is actionable.
- Build a representative test dataset
Include normal cases, ambiguous requests, edge cases, adversarial inputs, and examples of previous failures. Record relevant context and expected behavior. Remove or protect sensitive data before using real customer examples.
- Create explicit checks
Use deterministic assertions for schema, required fields, numeric ranges, and other exact constraints. Define metamorphic relations for transformations that should preserve or predictably change behavior. Add eval criteria for aspects such as factual support and task completion.
- Run tests against controlled versions
Record the prompt or system-instruction version, model identifier, application build, retrieval configuration, and relevant parameters. Run repeated trials where variability matters. A single passing run can hide an intermittent failure.
- Triage failures instead of blindly counting them
Classify each failure as a confirmed defect, an invalid test assumption, an evaluator disagreement, or an inconclusive result. For stochastic systems, record how often the failure occurs across repeated runs and whether its impact warrants blocking release.
- Retest after changes
When a prompt, model, retrieval pipeline, or application component changes, rerun the relevant regression set. Compare not only aggregate scores but also critical failure categories. A better overall score can conceal a regression in a high-risk behavior.
Example: Turning an AI Failure Into an Actionable Bug Report
Suppose a policy assistant correctly states that refunds are available within 30 days. When the question is rephrased, it says refunds are available within 60 days.
A useful report would contain:
- Title: Refund eligibility changes after a meaning-preserving prompt rephrase
- Preconditions: Record the model version, policy source, system instructions, and retrieval configuration.
- Original input and output: Include the prompt and the 30-day answer.
- Transformed input and output: Include the rephrased prompt and the 60-day answer.
- Expected behavior: The stated refund window remains consistent because the policy and intended question have not changed.
- Actual behavior: The eligibility period changes without a supporting policy change.
- Evidence: Save both runs, timestamps, relevant retrieved passages, and the test identifier.
- Impact: Users may receive incorrect eligibility information.
The QA team can then investigate whether the cause is retrieval inconsistency, prompt handling, model behavior, or a faulty test assumption. Keeping the evidence connected to the defect makes it easier to reproduce, prioritize, and verify the fix.
Common Mistakes When Testing Non-Deterministic Systems
Treating every output difference as a bug. Different wording is often acceptable. Define which facts, constraints, and outcomes must remain stable.
Trusting an LLM judge without validation. Compare grader decisions with human-reviewed cases and inspect disagreements. Never assume that an evaluator is correct simply because it is another model.
Writing relations that are too broad. Reordering text, changing context, or removing details can legitimately alter results. Apply transformations only when the expected relationship is justified.
Testing only average scores. A strong aggregate score can hide rare but serious failures. Track critical categories and inspect individual regressions.
Ignoring reproducibility. Save the inputs, outputs, model and application versions, configuration, and relevant context. Without these details, a failure may be difficult to investigate later.
Assuming these techniques remove the need for an oracle. They reduce dependence on exact expected outputs; they do not remove the need to define acceptable behavior.
Where Bugasura Fits
Finding a failure is only one part of testing. Teams also need to document evidence, assign ownership, prioritize impact, and confirm that fixes resolve the original problem without introducing regressions.
Bugasura’s test management page describes capabilities for organizing test plans, tracking test execution, and linking failures to defects. Its issue tracker and product features provide a relevant starting point for managing the defects uncovered by your AI testing workflow.
Use your existing evaluation or automation setup to execute the checks, then bring confirmed issues into the team’s defect-resolution process. Do not assume that a bug tracker itself replaces a dedicated evaluation harness or automatically validates AI output quality.
Start Building a More Traceable QA Workflow
Testing AI outputs requires more than collecting pass/fail results. Teams need a clear path from a failed check to a reproducible defect and a verified fix.
Start free with Bugasura to organize bug tracking and test management in your QA workflow.
