When tests pass in CI but fail in the development environment

I understand that if your app’s tests fail in the continuous integration environment but pass in your development environment, you should figure out why tests fail in CI and NOT blow off the matter.

What should you do when the reverse is true - when tests pass in CI but not in your local development environment? Do you blow off testing in the local environment, push your changes to an alternate branch, and see if things pan out in the CI environment? Maybe this is appropriate in some situations, but it sounds like a cop-out to me.

The same logic for local-PASS and CI-FAIL applies here for local-FAIL and CI-PASS: the main purpose of testing in various environments is to avoid “it works on my machine!”

Try reading some of the results from googling

test ‘fail locally’ but pass on CI -"pass locally"

and you should find that when people investigated the cause of the discrepancy between environments they usually learned how to improve their code and remove bugs by not ignoring this. Good luck!

I would check the environments are the same and try to diagnose what is different.

Start with

  • Same Ruby versions

  • Same database and database versions (especially if the failures are database ones)

Is the failures due to an area of code that uses any native gems that may have not been compiled correctly.