It Passed Locally but Failed in CI. Is It a Bug, a Flaky Test, or an Environment Problem?

11 minute read

A test that passes locally but fails in CI is a bug, a flaky test or an environment mismatch, and each needs a different fix. To tell which, save the first failure, reproduce the CI conditions, run the test alone and repeatedly, and change one variable at a time. A retry that passes does not prove the application is correct. 

Treating every failure as an application bug wastes debugging time. Dismissing every failure as a flaky test is equally dangerous because a genuine regression can be overlooked. 

The solution is to investigate the evidence systematically. This guide explains how to distinguish bugs from flaky tests and environment-specific failures, reproduce CI failures locally, design useful diagnostic test cases, and build a more reliable QA workflow. 

What Does It Mean When a Test Passes Locally but Fails in CI? 

Local testing takes place on a developer’s or tester’s machine. CI testing runs automatically in a configured environment when code is pushed. A pull request is opened, or another pipeline trigger occurs. 

The two environments may differ in operating system, browser version, available resources, dependency versions, configuration, network access, and execution order. 

A passing local test establishes only that the test passed under the conditions of that particular run. It does not prove that the application behaves correctly under every supported condition. 

For example, consider a checkout test that verifies whether an order confirmation appears after payment: 

• Locally: The developer’s browser, test database, and payment stub are configured correctly. The test passes. 
• In CI: The browser is slower, and the test checks for confirmation before the payment response is completed. The assertion fails. 
• After investigation: If the application eventually displays the confirmation correctly and the test was checking too early, the test synchronization is faulty. If the confirmation never appears because the application mishandles the payment response, there may be a genuine bug. 

The distinction matters because each cause requires a different fix. 

Flaky Test vs Bug vs Environment Problem 

1. Application bug 

The application violates an expected behavior. The failure may be consistent or triggered only by a specific input, timing condition, or supported configuration. 

Example: The checkout API returns a server error for a valid order. 

2. Flaky test 

The same test produces different outcomes without a relevant change to the application, test, or intended conditions. Uncontrolled timing, state, test order, and external dependencies are common causes. 

Example: A test sometimes clicks on a button before the page finishes loading. 

3. Environment problem 

The local and CI environments differ in a way that changes execution or prevents the test from running correctly. 

Example: CI uses a different browser version or lacks a required system dependency. 

These categories can overlap. An environment difference can expose a genuine application bug, and a test can be flaky because it depends on environment-specific timing. The goal is to identify the underlying cause, not force every failure into a single category. 

Why Do Tests Pass Locally but Fail in CI? 

1. Timing and asynchronous operations 

Modern applications frequently perform asynchronous work: API requests, database queries, background jobs, animations, and UI updates. A test that assumes an operation has finished after a fixed delay can fail when execution is slower. 

For example, a fixed wait does not guarantee that the order request has completed: 

await page.getByRole(“button”, { name: “Submit” }).click();
await page.waitForTimeout(500);
await expect(page.getByText(“Order confirmed”)).toBeVisible(); 

A better approach is to wait for a meaningful condition: 

await page.getByRole(“button”, { name: “Submit” }).click();
await expect(page.getByText(“Order confirmed”)).toBeVisible(); 

Playwright’s web assertions retry until the condition succeeds or the applicable timeout expires. This is more reliable than assuming a particular amount of time is always sufficient. The assertion must still reflect the correct expected behavior. 

2. Different environments and dependencies 

A local machine might run macOS with one browser version, while CI runs Linux with another browser build. Differences worth checking include: 

• Operating system, browser, and browser-driver versions. 
• Runtime and language versions, such as Node.js, Python, or Java. 
• Package lockfiles and installed dependency versions. 
• Locale, timezone, character encoding, and system fonts. 
• Environment variables, feature flags, and configuration files. 
• Missing browser libraries, certificates, or system packages. 

For example, a screenshot test may fail because text wraps differently under a different font. That does not automatically mean the application’s business logic is broken, but it may reveal a genuine cross-platform rendering issue if the layout is expected to be consistent. 

3. Test data and shared state 

Tests often fail when they depend on data created by another test. Imagine that a registration test always creates a user with qa@example.com. It passes when run alone. During CI, the test runs again against a database containing that email address, so registration fails with a duplicate-user error. 

The underlying problem may be test isolation rather than registration logic. Use unique test data, reset the relevant database state, and ensure each test can run independently. Check whether parallel tests modify the same account, record, file, or global variable. 

4. Parallel execution and resource contention 

CI runners may execute multiple tests at once. A suite that passes sequentially can fail when tests compete for shared resources. 

• Two tests updating the same database record. 
• Multiple tests using the same port or temporary filename. 
• A shared account being logged out by another test. 
• Browser instances competing for memory or CPU. 
• Rate limits being reached by simultaneous API calls. 

Temporarily running the affected tests sequentially can help isolate the cause. If that resolves the failure, investigate shared state and resource contention before permanently disabling parallel execution. 

5. Network services and CI configuration 

A test may rely on an external API, authentication provider, test database, or staging service. Network restrictions, expired credentials, rate limits, service outages, and incorrect endpoint URLs can all produce CI failures. 

Pipeline problems also include missing secrets, incorrect job dependencies, incomplete build steps, and using an outdated artifact. Inspect the CI logs and configuration before concluding that the application itself is defective. 

How to Debug Flaky Tests and CI Failures: A Step-by-Step Workflow 

Capture the exact failure 

Record the failing test name, assertion, stack trace, commit SHA, CI job, browser, runtime, and relevant logs. For UI tests, preserve screenshots, video, or a trace when available. 

Reproduce the CI conditions 

Check out the same commit. Match the runtime, browser, dependencies, environment variables, test data, and execution command as closely as possible. Use the same container image if the pipeline runs in one. 

Run the test in isolation 

Execute the failing test without the rest of the suite. If it passes alone but fails in the full suite, investigate test order, cleanup, shared data, and parallel execution. 

Repeat the test and observe the pattern 

Run it multiple times under controlled conditions. A mixture of passes and failures suggests intermittency, but repeated runs alone cannot establish whether the test, application, or infrastructure is responsible. 

Change one variable at a time 

Compare sequential and parallel runs, local and CI environments, or the current and previous commit. Keep other conditions as consistent as possible to make the results meaningful. 

Fix the cause and verify the fix 

Correct the application defect, improve test synchronization or isolation, or repair the environment. Rerun the targeted test, the relevant regression suite, and the original CI workflow. 

How to identify flaky tests through repeated runs 

Suppose a test is executed 20 times under comparable conditions and fails 4 times. Its observed failure rate in that sample is: 

Observed failure rate = (4 / 20) × 100 = 20% 

This indicates intermittent failure in the sample. It does not prove that the test itself is faulty, nor does it predict the exact failure rate in future runs. A real application defect triggered by a race condition can also produce intermittent results. 

For useful evidence, record the commit, environment, execution mode, failure signature, and outcome of each run. A test that fails only when run in parallel tells a different story from one that fails consistently on every CI execution. 

To keep that evidence organised, see our guide to analyzing and reporting test results. 

Practical Example: Diagnosing a Checkout Test 

Scenario: An e-commerce customer completes checkout and should see an order confirmation. 

• Input: Valid user credentials, an in-stock product, and a successful payment response from a controlled test payment service. 
• Expected result: The order is created once, and the confirmation page displays the correct order reference. 
• Actual result in CI: The test times out while waiting for the confirmation. 

Rather than immediately filing a product defect, the QA engineer investigates the evidence: 

• The payment response succeeds. This makes a payment-service failure less likely, but does not rule out an application defect in order processing. 
• The order API completes after the test’s fixed wait. The test may be checking the UI before the asynchronous workflow has completed. 
• The test passes when it waits for the confirmation condition. This supports a test synchronization issue. The team should still verify that the order is created correctly and only once. 

Now consider a different result: the order API returns HTTP 500 in CI, and application logs show a null-value exception for valid checkout data. That evidence points toward an application defect or a configuration-dependent defect. The team must investigate the server behavior rather than simply extending the test timeout. 

The key lesson is to trace the failure through the application workflow and determine where actual behavior diverges from expected behavior. 

Test Cases for Investigating Environment-Specific Failures 

Test scenario	Input or condition	Expected result	Purpose
Repeat the failing test	Same commit and environment, multiple runs	Consistent results, or intermittent failures recorded for analysis	Identify intermittent behavior
Run the test alone	Target test without the full suite	Same expected behavior as a suite run	Detect dependencies on other tests
Run tests in parallel	Same suite, parallel workers enabled	No conflicts or unexpected failures	Detect shared-state problems
Compare runtime versions	Supported local and CI runtime versions	Application behaves according to supported compatibility requirements	Identify version-related differences
Test delayed API response	Controlled response delay within supported operating conditions	UI waits for completion and shows the correct result	Identify timing assumptions
Test missing configuration	Omit a required test configuration value in a controlled test environment	Clear configuration error, not a misleading application assertion	Validate setup and failure reporting
Test a valid checkout	Valid input and successful mocked payment response	One order is created and confirmation appears	Verify business behavior
Test an invalid checkout	Invalid or incomplete required input	Appropriate validation error; no order is created	Ensure validation still works

 

These are diagnostic examples, not universal pass/fail specifications. Adapt the expected outcomes to the application’s requirements and supported environment matrix. 

How to Interpret CI Test Failures 

Failure pattern	Likely investigation area	Next action
Fails consistently locally and in CI	Application logic, test assertion, or test data	Validate expected behavior and inspect the failure
Passes locally, fails consistently in CI	Environment, configuration, dependencies, or CI-specific application behavior	Compare configurations and logs
Fails on first run, passes on retry	Timing, race conditions, state leakage, or intermittent infrastructure	Inspect first failure and retry evidence
Passes alone, fails in the full suite	Test order, shared state, cleanup, or parallel execution	Isolate dependencies and reset test state
Fails only on a specific browser or OS	Compatibility or rendering behavior	Verify platform support and inspect differences
Fails before tests execute	Build, dependency installation, credentials, or pipeline configuration	Inspect the failing CI step and setup logs

 

A retry that passes is evidence of a flaky outcome, not proof that the application is correct. Playwright, for example, classifies a test that fails initially but passes on retry as flaky. 

How to Fix CI/CD Pipeline Failures Without Hiding Real Bugs 

Fix application defects 

• Correct the failing business logic or error handling. 
• Add regression coverage for the exact input or state that exposed the problem. 
• Verify behavior across the affected environments. 
• Check for duplicate operations or inconsistent data when asynchronous requests are involved. 

Improve test reliability 

• Replace arbitrary sleeps with condition-based waits. 
• Make test data independent and repeatable. 
• Reset state before each test and clean up afterward. 
• Avoid brittle selectors and assertions tied to incidental UI details. 
• Control time, randomness, and external service responses when the test does not intend to validate those dependencies. 

Standardize the CI environment 

• Pin compatible runtime and dependency versions. 
• Use reproducible dependency installation and versioned container images where appropriate. 
• Install the browser binaries and system dependencies required by the test framework. 
• Validate required environment variables and secrets without exposing their values. 
• Save diagnostic artifacts when failures occur. 

Playwright’s CI guidance recommends installing the required browser dependencies and notes that using a single worker can improve stability and reproducibility in CI. These settings are useful diagnostic tools, but they should not replace investigation of shared-state or performance problems. 

Use retries carefully 

Retries can help identify intermittent failures and provide additional evidence. However, they should not become a permanent substitute for fixing an unreliable test or an application defect. 

If a team quarantines a flaky test to protect the main regression suite, it should document the reason, assign ownership, track the fix, and continue running the test separately when practical. Otherwise, a temporary workaround can become a permanent blind spot. 

How Bugasura Fits into the CI Failure Investigation Workflow 

Diagnosing a failure is only part of the job. QA teams also need to preserve the evidence, communicate the findings, and make sure the issue is resolved without losing the context of the original failure. 

Bugasura provides a relevant workflow for organizing test cases, tracking test runs, and connecting test findings to defect management. Its published test management documentation describes linking requirements and test cases, recording test execution outcomes, attaching supporting material, and reviewing reports. 

For a broader look at connecting QA with pipelines, read our guide to test management in CI/CD workflows. 

1. Record the failure: Capture the failing assertion, commit, CI run, browser and runtime versions, relevant logs, and any available screenshots or traces. 
2. Classify the finding: Record whether the evidence points to an application defect, test instability, an environment mismatch, or an unresolved cause. 
3. Create an actionable issue: Include reproduction steps, expected and actual results, the observed failure pattern, and evidence. Assign an owner and priority according to impact and release risk. 
4. Maintain test traceability: Where applicable, link the affected test case and requirement to the issue. 
5. Verify the resolution: Record the retest result and regression outcome. Close the issue only when the agreed acceptance criteria are met. 

Bugasura should complement, not replace, the CI runner’s execution logs and debugging artifacts. The CI system establishes what happened during execution; structured test and defect records help the team decide what to fix and verify. 

Common Mistakes to Avoid 

Assuming a retry proves the test is healthy. 

A test that passes on its second attempt may still be exposing a real race condition or intermittent application defect. 

Increasing every timeout. 

Longer timeouts can help diagnose slow operations, but they may hide missing synchronization, performance regressions, or a workflow that never completes. 

Blaming the environment without comparing it. 

Record actual version and configuration differences instead of assuming Linux, headless browsers, or CI runners are inherently responsible. 

Running the entire suite repeatedly without isolating the failure. 

Start with the failing test, then expand to the full suite after identifying a plausible cause. 

Ignoring failures classified as flaky. 

A flaky test reduces confidence in the regression suite. Track it, assign ownership, and verify the eventual fix. 

Filing vague defect reports. 

“Works locally, fails in CI” describes a symptom, not a sufficient defect report. Include the exact failure, reproduction conditions, expected behavior, and supporting evidence. Our guide to turning test failures into actionable issues shows how. 

Best Practices for Preventing False Failures in CI 

• Keep local and CI commands as consistent as practical. 
• Use reproducible dependency installation and controlled test data. 
• Prefer explicit synchronization over fixed delays. 
• Design tests to run independently and in different orders. 
• Collect useful logs and diagnostic artifacts on failure. 
• Track first-attempt failures separately from final results after retries. 
• Use a documented process for quarantining unreliable tests. 
• Review recurring failure patterns rather than treating each pipeline run as an isolated incident. 
• Keep application defects, test defects, and infrastructure incidents distinguishable in the team’s issue records. 

The goal is not to eliminate every CI failure. It is to make failures trustworthy signals that help the team detect real defects quickly.

Frequently Asked Questions

Conclusion 

When a test passes locally but fails in CI, the right response is investigation, not assumption. The failure could be a genuine application bug, an unreliable test, an environment mismatch, or a combination of these factors. 

Start by preserving the evidence, reproducing the CI conditions, and isolating the failing test. Compare repeated runs, examine asynchronous behavior and shared state, and change one variable at a time. Then fix the verified cause and rerun the relevant regression checks. 

Reliable CI depends on more than getting the pipeline to turn green. It depends on making every test result meaningful, so QA engineers and developers can trust the feedback and release software with greater confidence.