<!-- Google Tag Manager (noscript) -->
	<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-P44THP6"
	height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
<!-- End Google Tag Manager (noscript) -->{"id":5786,"date":"2026-11-12T17:45:45","date_gmt":"2026-11-12T12:15:45","guid":{"rendered":"https:\/\/bugasura.io\/blog\/?p=5786"},"modified":"2026-11-12T17:45:45","modified_gmt":"2026-11-12T12:15:45","slug":"metamorphic-testing-property-based-checks-ai-evals","status":"publish","type":"post","link":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/","title":{"rendered":"Testing Without an Oracle: Metamorphic Testing, Property-Based Checks and Evals for AI Outputs"},"content":{"rendered":"<span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 8<\/span> <span class=\"rt-label rt-postfix\">minute read<\/span><\/span><p><span data-contrast=\"auto\"><img data-recalc-dims=\"1\" decoding=\"async\" class=\"alignnone wp-image-5787 size-full lazyload\" data-src=\"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=1080%2C441&#038;ssl=1\" alt=\"\" width=\"1080\" height=\"441\" data-srcset=\"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?w=1080&amp;ssl=1 1080w, https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=300%2C123&amp;ssl=1 300w, https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=1024%2C419&amp;ssl=1 1024w, https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=768%2C314&amp;ssl=1 768w, https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=1536%2C628&amp;ssl=1 1536w, https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?resize=400%2C164&amp;ssl=1 400w\" data-sizes=\"(max-width: 1080px) 100vw, 1080px\" src=\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\" style=\"--smush-placeholder-width: 1080px; --smush-placeholder-aspect-ratio: 1080\/441;\" \/><\/span><\/p>\n<p><span data-contrast=\"auto\">When an AI output has no single correct answer, replace exact-match assertions with three checks: metamorphic testing (the output keeps an expected relationship when the input is changed in a controlled way), property-based testing (rules such as a valid schema hold across many generated inputs) and evals (outputs are scored against defined quality criteria). Use human review for high-impact or ambiguous cases. An AI system can return a different answer to the same prompt without necessarily being broken. It can also return a fluent, convincing answer that is factually wrong. For QA teams, the difficult question is no longer just whether the output matches an expected string. It is whether the system behaves correctly across different inputs, contexts, and repeated runs.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Metamorphic testing<\/span><\/b><span data-contrast=\"auto\"> helps address this challenge by checking relationships between outputs instead of requiring an exact expected answer for every input. Combined with property-based testing and LLM evaluations (evals), it gives testers a practical way to detect <\/span><a href=\"https:\/\/bugasura.io\/blog\/ai-testing-life-cycle\/\"><span data-contrast=\"none\">failures in AI-powered features<\/span><\/a><span data-contrast=\"auto\">\u2014even when a perfect test oracle is unavailable.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">What Is the Test Oracle Problem?<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">The test oracle problem occurs when a tester cannot easily determine the correct output for a given input. Traditional tests often compare actual results against known expected values, but that approach becomes difficult when outputs are complex, subjective, or generated probabilistically.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Consider an AI support assistant that summarizes customer conversations. You may not have one uniquely correct summary for every conversation. Two summaries could use different wording while preserving all important facts. However, a summary that invents a refund approval is clearly unacceptable.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">The challenge is to distinguish harmless variation from a meaningful failure.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">This is especially important when <\/span><a href=\"https:\/\/bugasura.io\/blog\/generative-ai-in-software-testing\/\"><span data-contrast=\"none\">testing AI outputs<\/span><\/a><span data-contrast=\"auto\">, recommendation systems, search results, image recognition, and other systems where the expected result is difficult to specify exactly.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">A useful response is not to abandon assertions. It is to replace a single exact-output assertion with several checks based on what the system must preserve, what it must never do, and how its behavior should change when the input changes.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">How Metamorphic Testing Works<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Metamorphic testing checks whether a system maintains a known relationship between outputs when the inputs are transformed in a controlled way. That relationship is called a <\/span><i><span data-contrast=\"auto\">metamorphic relation<\/span><\/i><span data-contrast=\"auto\">.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Instead of asking, \u201cIs this answer exactly correct?\u201d, the tester asks, \u201cWhen I change the input in this particular way, does the output still satisfy the expected relationship?\u201d<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">For example, suppose an AI assistant answers questions about a company&#8217;s refund policy. You might test the same question with different capitalization or harmless whitespace. The wording may vary, but the policy facts and eligibility conditions should remain consistent.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">A metamorphic test has four main parts:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol>\n<li aria-setsize=\"-1\" data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Original input:<\/span><\/b><span data-contrast=\"auto\"> A prompt or data sample sent to the system.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Input transformation:<\/span><\/b><span data-contrast=\"auto\"> A deliberate change, such as rephrasing the question or adding irrelevant text.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Expected relation:<\/span><\/b><span data-contrast=\"auto\"> The property that should remain true or change in a defined way.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"%1.\" data-font=\"Aptos\" data-listid=\"1\" data-list-defn-props=\"{&quot;335552541&quot;:0,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769242&quot;:[65533,0],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;%1.&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Relation check:<\/span><\/b><span data-contrast=\"auto\"> An assertion or evaluation that detects a violation.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ol>\n<h3 aria-level=\"3\"><b><span data-contrast=\"none\">Practical metamorphic testing examples<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:281,&quot;335559739&quot;:281}\">\u00a0<\/span><\/h3>\n<table data-tablestyle=\"MsoNormalTable\" data-tablelook=\"1696\" aria-rowcount=\"6\" aria-colcount=\"4\">\n<tbody>\n<tr aria-rowindex=\"1\">\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">AI feature<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Input transformation<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Expected relationship<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Potential failure<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"2\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Policy assistant<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Rephrase a question without changing its meaning<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Key policy facts remain consistent<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Eligibility rules change<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"3\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Document summarizer<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Reorder paragraphs while preserving their content<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Important facts remain represented<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">A critical exception disappears<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"4\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Product recommendation<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Change irrelevant formatting in the request<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Relevant recommendations remain stable<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Recommendations change for no meaningful reason<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"5\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Data extraction<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Add unrelated text outside the target record<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Extracted target fields remain unchanged<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Fields are taken from unrelated content<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"6\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Translation assistant<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Change punctuation without changing the intended meaning<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Core meaning is preserved<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">A negation or important condition is lost<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span data-contrast=\"auto\">These are examples of relations to validate, not universal rules. For instance, paragraph order can matter in chronological summaries, and recommendations can legitimately change when context affects relevance. A tester must establish the relationship from the feature&#8217;s requirements before treating a difference as a defect.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">How to Design a Useful Metamorphic Relation<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">A weak relation creates noisy tests. A good one reflects a meaningful product requirement.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Use this process:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li><b><span data-contrast=\"auto\">Step 1: Identify a behavior that is difficult to verify directly.<\/span><\/b><span data-contrast=\"auto\"> Choose a feature where there is no simple, reliable expected answer for every possible input.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li><b><span data-contrast=\"auto\">Step 2: Identify an input transformation.<\/span><\/b><span data-contrast=\"auto\"> Decide what can change without changing the intended meaning or what change should produce a predictable effect.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li><b><span data-contrast=\"auto\">Step 3: Define what must be preserved or changed.<\/span><\/b><span data-contrast=\"auto\"> Specify observable properties such as factual consistency, valid structure, required fields, or a numerical relationship.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li><b><span data-contrast=\"auto\">Step 4: Run the original and transformed inputs.<\/span><\/b><span data-contrast=\"auto\"> Keep the model version, system instructions, retrieval configuration, and other relevant conditions recorded so that comparisons are meaningful.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li><b><span data-contrast=\"auto\">Step 5: Check the relation and investigate violations.<\/span><\/b><span data-contrast=\"auto\"> A violation is evidence of a possible failure, not automatic proof of a product defect. Confirm that the transformation was valid and that the expected relation actually applies.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">For example, if an invoice-extraction model returns a different invoice number after unrelated text is added to the prompt, the result may indicate a context-isolation problem. The team should preserve both inputs, both outputs, the model configuration, and the relation that failed.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Where Property-Based Testing Fits<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Property-based testing checks whether general properties hold across many generated inputs rather than relying only on a small set of manually selected examples. It is particularly useful when inputs have many combinations, <\/span><a href=\"https:\/\/bugasura.io\/blog\/test-case-design-techniques\/\"><span data-contrast=\"none\">edge cases<\/span><\/a><span data-contrast=\"auto\">, or unexpected formats.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">For an AI application, property-based checks often validate the software surrounding the model as well as the model&#8217;s response.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Imagine a support assistant that must return a structured JSON object containing <\/span><span data-contrast=\"auto\">category<\/span><span data-contrast=\"auto\">, <\/span><span data-contrast=\"auto\">priority<\/span><span data-contrast=\"auto\">, and <\/span><span data-contrast=\"auto\">summary<\/span><span data-contrast=\"auto\">. A property-based test could generate varied customer messages\u2014including empty strings, long text, Unicode, unusual punctuation, and repeated characters\u2014and verify that the application handles them safely.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Possible properties include:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">The response can be parsed according to the required schema.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Required fields are present and have valid types.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Priority values belong to the permitted set.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Invalid input does not crash the service.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"2\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><span data-contrast=\"auto\">Sensitive fields are not copied into outputs when the requirements prohibit them.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">These checks do not prove that the model&#8217;s classification is semantically correct. They establish that the output satisfies specific, testable constraints.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Property-based testing can also generate inputs for metamorphic tests. For example, generate many valid support messages, apply a meaning-preserving transformation to each, and check whether the required classification remains consistent within an explicitly defined tolerance.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Tools such as <\/span><a href=\"https:\/\/hypothesis.readthedocs.io\/en\/latest\/tutorial\/introduction.html\"><span data-contrast=\"none\">Hypothesis<\/span><\/a><span data-contrast=\"auto\"> support property-based testing in Python. The key is to define meaningful properties first; generating thousands of inputs cannot compensate for a property that does not reflect the intended behavior.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">LLM Evaluation (Evals): Measuring Output Quality<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Metamorphic testing checks relationships between executions. LLM evaluation checks whether outputs meet defined quality criteria across a curated dataset of examples.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">An evaluation dataset might contain real or synthetic prompts, relevant context, expected properties, reference answers where appropriate, and labels for known failure modes. Graders then assess outputs using exact checks, programmatic rules, semantic comparisons, or model-assisted judgments.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">For example, a retrieval-based assistant might be evaluated for:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Groundedness:<\/span><\/b><span data-contrast=\"auto\"> Are factual claims supported by the supplied documents?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Completeness:<\/span><\/b><span data-contrast=\"auto\"> Does the answer include the information required by the task?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Instruction compliance:<\/span><\/b><span data-contrast=\"auto\"> Does it follow the specified output format and constraints?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Safety:<\/span><\/b><span data-contrast=\"auto\"> Does it avoid disallowed disclosures or actions?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"3\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Task success:<\/span><\/b><span data-contrast=\"auto\"> Does the answer actually resolve the user&#8217;s request?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">Not every criterion should use an LLM as a judge. Exact schema validation is better handled programmatically. Known factual answers can often be checked against references. Subjective criteria may benefit from a model-assisted grader, but its decisions should be calibrated against human-reviewed examples.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">A score is not a guarantee of correctness. A grader can miss a subtle factual error, reward persuasive wording, or disagree with human reviewers. Track false passes and false failures, review borderline cases, and keep human judgment in the loop for high-impact decisions.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">For practical guidance, see the <\/span><a href=\"https:\/\/developers.openai.com\/api\/docs\/guides\/evaluation-best-practices\"><span data-contrast=\"none\">OpenAI evaluation best practices<\/span><\/a><span data-contrast=\"auto\">.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Metamorphic Testing vs. Property-Based Testing vs. Evals<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">These approaches solve related but different problems. A robust AI testing strategy uses each for the kind of claim it can actually verify.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<table data-tablestyle=\"MsoNormalTable\" data-tablelook=\"1696\" aria-rowcount=\"5\" aria-colcount=\"4\">\n<tbody>\n<tr aria-rowindex=\"1\">\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Technique<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Main question<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Best use<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><b><span data-contrast=\"auto\">Important limitation<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335551550&quot;:2,&quot;335551620&quot;:2,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"2\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Metamorphic testing<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Does behavior maintain the expected relationship after a controlled input change?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Consistency and transformation-based checks<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">A poorly chosen relation can flag valid behavior<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"3\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Property-based testing<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Does a defined invariant hold across generated inputs?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Input edge cases, structural constraints, and invariants<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">It cannot establish subjective correctness by itself<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"4\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">LLM evals<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Does output meet defined task-quality criteria across representative cases?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Groundedness, relevance, task success, and regressions<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Graders and datasets can be incomplete or biased<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<tr aria-rowindex=\"5\">\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Human review<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Is the result acceptable in its real business context?<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Ambiguous, high-impact, or subjective decisions<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<td data-celllook=\"0\"><span data-contrast=\"auto\">Expensive to scale and requires clear review criteria<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:0,&quot;335559739&quot;:0}\">\u00a0<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><span data-contrast=\"auto\">These techniques complement rather than replace one another. Metamorphic tests can expose inconsistencies; property-based checks can uncover boundary failures; evals can measure task quality; and human review can investigate cases where automated judgments are uncertain.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">A Practical Workflow for Testing AI Outputs<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Teams do not need to build a complex testing platform before getting started. Begin with one high-value AI feature and create a repeatable workflow.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol>\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Define the failure that matters<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Choose a concrete risk, such as an assistant inventing refund eligibility, an extraction model assigning data to the wrong customer, or a summarizer omitting a critical exception.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Describe the impact in product terms. \u201cThe response changed\u201d is not sufficient; \u201cthe response changed a policy condition after an irrelevant formatting change\u201d is actionable.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol start=\"2\">\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Build a representative test dataset<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Include normal cases, ambiguous requests, edge cases, adversarial inputs, and examples of previous failures. Record relevant context and expected behavior. Remove or protect sensitive data before using real customer examples.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol start=\"3\">\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Create explicit checks<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Use deterministic assertions for schema, required fields, numeric ranges, and other exact constraints. Define metamorphic relations for transformations that should preserve or predictably change behavior. Add eval criteria for aspects such as factual support and task completion.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol start=\"4\">\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Run tests against controlled versions<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Record the prompt or system-instruction version, model identifier, application build, retrieval configuration, and relevant parameters. Run repeated trials where variability matters. A single passing run can hide an intermittent failure.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol start=\"5\">\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Triage failures instead of blindly counting them<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">Classify each failure as a confirmed defect, an invalid test assumption, an evaluator disagreement, or an inconclusive result. For stochastic systems, record how often the failure occurs across repeated runs and whether its impact warrants blocking release.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ol start=\"6\">\n<li aria-level=\"3\"><b><span data-contrast=\"none\"> Retest after changes<\/span><\/b><\/li>\n<\/ol>\n<p><span data-contrast=\"auto\">When a prompt, model, retrieval pipeline, or application component changes, rerun the relevant regression set. Compare not only aggregate scores but also critical failure categories. A better overall score can conceal a regression in a high-risk behavior.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Example: Turning an AI Failure Into an Actionable Bug Report<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Suppose a policy assistant correctly states that refunds are available within 30 days. When the question is rephrased, it says refunds are available within 60 days.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">A useful report would contain:<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<ul>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Title:<\/span><\/b><span data-contrast=\"auto\"> Refund eligibility changes after a meaning-preserving prompt rephrase<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Preconditions:<\/span><\/b><span data-contrast=\"auto\"> Record the model version, policy source, system instructions, and retrieval configuration.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Original input and output:<\/span><\/b><span data-contrast=\"auto\"> Include the prompt and the 30-day answer.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Transformed input and output:<\/span><\/b><span data-contrast=\"auto\"> Include the rephrased prompt and the 60-day answer.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Expected behavior:<\/span><\/b><span data-contrast=\"auto\"> The stated refund window remains consistent because the policy and intended question have not changed.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Actual behavior:<\/span><\/b><span data-contrast=\"auto\"> The eligibility period changes without a supporting policy change.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Evidence:<\/span><\/b><span data-contrast=\"auto\"> Save both runs, timestamps, relevant retrieved passages, and the test identifier.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<li aria-setsize=\"-1\" data-leveltext=\"\uf0b7\" data-font=\"Symbol\" data-listid=\"4\" data-list-defn-props=\"{&quot;335552541&quot;:1,&quot;335559685&quot;:720,&quot;335559991&quot;:360,&quot;469769226&quot;:&quot;Symbol&quot;,&quot;469769242&quot;:[8226],&quot;469777803&quot;:&quot;left&quot;,&quot;469777804&quot;:&quot;\uf0b7&quot;,&quot;469777815&quot;:&quot;hybridMultilevel&quot;}\" data-aria-posinset=\"1\" data-aria-level=\"1\"><b><span data-contrast=\"auto\">Impact:<\/span><\/b><span data-contrast=\"auto\"> Users may receive incorrect eligibility information.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/li>\n<\/ul>\n<p><span data-contrast=\"auto\">The QA team can then investigate whether the cause is retrieval inconsistency, prompt handling, model behavior, or a faulty test assumption. Keeping the evidence connected to the defect makes it easier to reproduce, prioritize, and verify the fix.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Common Mistakes When Testing Non-Deterministic Systems<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><b><span data-contrast=\"auto\">Treating every output difference as a bug.<\/span><\/b><span data-contrast=\"auto\"> Different wording is often acceptable. Define which facts, constraints, and outcomes must remain stable.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Trusting an LLM judge without validation.<\/span><\/b><span data-contrast=\"auto\"> Compare grader decisions with human-reviewed cases and inspect disagreements. Never assume that an evaluator is correct simply because it is another model.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Writing relations that are too broad.<\/span><\/b><span data-contrast=\"auto\"> Reordering text, changing context, or removing details can legitimately alter results. Apply transformations only when the expected relationship is justified.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Testing only average scores.<\/span><\/b><span data-contrast=\"auto\"> A strong aggregate score can hide rare but serious failures. Track critical categories and inspect individual regressions.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Ignoring reproducibility.<\/span><\/b><span data-contrast=\"auto\"> Save the inputs, outputs, model and application versions, configuration, and relevant context. Without these details, a failure may be difficult to investigate later.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><b><span data-contrast=\"auto\">Assuming these techniques remove the need for an oracle.<\/span><\/b><span data-contrast=\"auto\"> They reduce dependence on exact expected outputs; they do not remove the need to define acceptable behavior.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Where Bugasura Fits<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Finding a failure is only one part of testing. Teams also need to document evidence, assign ownership, prioritize impact, and confirm that fixes resolve the original problem without introducing regressions.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Bugasura&#8217;s <\/span><a href=\"https:\/\/bugasura.io\/test-management\"><span data-contrast=\"none\">test management<\/span><\/a><span data-contrast=\"auto\"> page describes capabilities for organizing test plans, <\/span><a href=\"https:\/\/bugasura.io\/release-notes\/visualize-test-execution-with-board-view\/\"><span data-contrast=\"none\">tracking test execution<\/span><\/a><span data-contrast=\"auto\">, and linking failures to defects. Its <\/span><a href=\"https:\/\/bugasura.io\/features\"><span data-contrast=\"none\">issue tracker and product features<\/span><\/a><span data-contrast=\"auto\"> provide a relevant starting point for managing the defects uncovered by your AI testing workflow.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><span data-contrast=\"auto\">Use your existing evaluation or automation setup to execute the checks, then bring confirmed issues into the team&#8217;s defect-resolution process. Do not assume that a bug tracker itself replaces a dedicated evaluation harness or automatically validates AI output quality.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<h2 aria-level=\"2\"><b><span data-contrast=\"none\">Start Building a More Traceable QA Workflow<\/span><\/b><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;134245418&quot;:true,&quot;134245529&quot;:true,&quot;335559738&quot;:299,&quot;335559739&quot;:299}\">\u00a0<\/span><\/h2>\n<p><span data-contrast=\"auto\">Testing AI outputs requires more than collecting pass\/fail results. Teams need a clear path from a failed check to a reproducible defect and a verified fix.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n<p><a href=\"https:\/\/my.bugasura.io\/?go=sign_up\"><b><span data-contrast=\"none\">Start free with Bugasura<\/span><\/b><\/a><span data-contrast=\"auto\"> to organize bug tracking and test management in your QA workflow.<\/span><span data-ccp-props=\"{&quot;134233117&quot;:false,&quot;134233118&quot;:false,&quot;335559738&quot;:240,&quot;335559739&quot;:240}\">\u00a0<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p><span class=\"span-reading-time rt-reading-time\" style=\"display: block;\"><span class=\"rt-label rt-prefix\"><\/span> <span class=\"rt-time\"> 8<\/span> <span class=\"rt-label rt-postfix\">minute read<\/span><\/span>When an AI output has no single correct answer, replace exact-match assertions with three checks: metamorphic testing (the output keeps an expected relationship when the input is changed in a controlled way), property-based testing (rules such as a valid schema hold across many generated inputs) and evals (outputs are scored against defined quality criteria). Use human review for high-impact or ambiguous cases. An AI system can return a different answer to the same prompt without necessarily being broken. It can also return a fluent, convincing answer that is factually wrong. For QA teams, the difficult question is no longer just [&hellip;]<\/p>\n","protected":false},"author":19,"featured_media":5787,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_post_was_ever_published":false},"categories":[139],"tags":[],"class_list":["post-5786","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-testing","post--single"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.7 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Testing Without an Oracle: Metamorphic Tests for AI Outputs<\/title>\n<meta name=\"description\" content=\"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Testing Without an Oracle: Metamorphic Tests for AI Outputs\" \/>\n<meta property=\"og:description\" content=\"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/\" \/>\n<meta property=\"og:site_name\" content=\"Bugasura Blog\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-12T12:15:45+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1080\" \/>\n\t<meta property=\"og:image:height\" content=\"441\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Natasha\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Natasha\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/\"},\"author\":{\"name\":\"Natasha\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/#\\\/schema\\\/person\\\/9f7096957533f3e9f0376aa20927933e\"},\"headline\":\"Testing Without an Oracle: Metamorphic Testing, Property-Based Checks and Evals for AI Outputs\",\"datePublished\":\"2026-11-12T12:15:45+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/\"},\"wordCount\":2245,\"image\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/bugasura.io\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1\",\"articleSection\":[\"Testing\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/\",\"url\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/\",\"name\":\"Testing Without an Oracle: Metamorphic Tests for AI Outputs\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/i0.wp.com\\\/bugasura.io\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1\",\"datePublished\":\"2026-11-12T12:15:45+00:00\",\"author\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/#\\\/schema\\\/person\\\/9f7096957533f3e9f0376aa20927933e\"},\"description\":\"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#primaryimage\",\"url\":\"https:\\\/\\\/i0.wp.com\\\/bugasura.io\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1\",\"contentUrl\":\"https:\\\/\\\/i0.wp.com\\\/bugasura.io\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/11\\\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1\",\"width\":1080,\"height\":441},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/metamorphic-testing-property-based-checks-ai-evals\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Testing Without an Oracle: Metamorphic Testing, Property-Based Checks and Evals for AI Outputs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/\",\"name\":\"Bugasura Blog\",\"description\":\"Bug reporting and bug tracking solution Bugasura is a simple to use tool helping in software bug tracking, bug reporting and development. The tool is a part of the Bugasura Platform.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/#\\\/schema\\\/person\\\/9f7096957533f3e9f0376aa20927933e\",\"name\":\"Natasha\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/ca3\\\/ca346d352d2484e446a0ffdada46c527x96.jpg\",\"url\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/ca3\\\/ca346d352d2484e446a0ffdada46c527x96.jpg\",\"contentUrl\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/wp-content\\\/wphb-cache\\\/gravatar\\\/ca3\\\/ca346d352d2484e446a0ffdada46c527x96.jpg\",\"caption\":\"Natasha\"},\"url\":\"https:\\\/\\\/bugasura.io\\\/blog\\\/author\\\/natasha\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Testing Without an Oracle: Metamorphic Tests for AI Outputs","description":"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/","og_locale":"en_US","og_type":"article","og_title":"Testing Without an Oracle: Metamorphic Tests for AI Outputs","og_description":"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.","og_url":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/","og_site_name":"Bugasura Blog","article_published_time":"2026-11-12T12:15:45+00:00","og_image":[{"width":1080,"height":441,"url":"https:\/\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg","type":"image\/jpeg"}],"author":"Natasha","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Natasha","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#article","isPartOf":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/"},"author":{"name":"Natasha","@id":"https:\/\/bugasura.io\/blog\/#\/schema\/person\/9f7096957533f3e9f0376aa20927933e"},"headline":"Testing Without an Oracle: Metamorphic Testing, Property-Based Checks and Evals for AI Outputs","datePublished":"2026-11-12T12:15:45+00:00","mainEntityOfPage":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/"},"wordCount":2245,"image":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1","articleSection":["Testing"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/","url":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/","name":"Testing Without an Oracle: Metamorphic Tests for AI Outputs","isPartOf":{"@id":"https:\/\/bugasura.io\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#primaryimage"},"image":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1","datePublished":"2026-11-12T12:15:45+00:00","author":{"@id":"https:\/\/bugasura.io\/blog\/#\/schema\/person\/9f7096957533f3e9f0376aa20927933e"},"description":"No expected answer to test against? Learn how metamorphic testing, property-based checks and evals help QA teams validate AI outputs without a test oracle.","breadcrumb":{"@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#primaryimage","url":"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1","contentUrl":"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1","width":1080,"height":441},{"@type":"BreadcrumbList","@id":"https:\/\/bugasura.io\/blog\/metamorphic-testing-property-based-checks-ai-evals\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/bugasura.io\/blog\/"},{"@type":"ListItem","position":2,"name":"Testing Without an Oracle: Metamorphic Testing, Property-Based Checks and Evals for AI Outputs"}]},{"@type":"WebSite","@id":"https:\/\/bugasura.io\/blog\/#website","url":"https:\/\/bugasura.io\/blog\/","name":"Bugasura Blog","description":"Bug reporting and bug tracking solution Bugasura is a simple to use tool helping in software bug tracking, bug reporting and development. The tool is a part of the Bugasura Platform.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/bugasura.io\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/bugasura.io\/blog\/#\/schema\/person\/9f7096957533f3e9f0376aa20927933e","name":"Natasha","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/bugasura.io\/blog\/wp-content\/wphb-cache\/gravatar\/ca3\/ca346d352d2484e446a0ffdada46c527x96.jpg","url":"https:\/\/bugasura.io\/blog\/wp-content\/wphb-cache\/gravatar\/ca3\/ca346d352d2484e446a0ffdada46c527x96.jpg","contentUrl":"https:\/\/bugasura.io\/blog\/wp-content\/wphb-cache\/gravatar\/ca3\/ca346d352d2484e446a0ffdada46c527x96.jpg","caption":"Natasha"},"url":"https:\/\/bugasura.io\/blog\/author\/natasha\/"}]}},"jetpack-related-posts":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/bugasura.io\/blog\/wp-content\/uploads\/2026\/11\/Testing-without-an-oracle-scaled.jpg?fit=1080%2C441&ssl=1","post_mailing_queue_ids":[],"_links":{"self":[{"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/posts\/5786","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/users\/19"}],"replies":[{"embeddable":true,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/comments?post=5786"}],"version-history":[{"count":0,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/posts\/5786\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/media\/5787"}],"wp:attachment":[{"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/media?parent=5786"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/categories?post=5786"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/bugasura.io\/blog\/wp-json\/wp\/v2\/tags?post=5786"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}