COMPARE WITH PREVIOUS VERSION
Pattern summary
Compare the current results with the results got by running the same tests in the previous version of the SUT
Category
Execution
Context
This pattern can be used when you have verified results for the previous release
Description
When it’s difficult to judge if the results are OK because of so many relationships, it may be much easier to just compare the current results with the results you got by running the same tests in the previous version of the SUT. If you know them to be OK then you don’t have to “understand” the contents, but just have to look for differences.
Implementation
First of all you must set-up the same initial conditions in the old and in the new version. Then run the tests on the old version and on the new one. Compare the results.
Potential problems
If you don’t have automated tests for the older version, run them manually
Issues addressed by this pattern
INEFFICIENT FAILURE ANALYSIS
HARD-TO-AUTOMATE RESULTS
Experiences
I learned this pattern by trial and error. Never run fresh test scripts with new build. (If failed you don't know if it's test script or the build)
I always run newer test on old established build, so if fails it's a test script error. (Nish Herath)
If you have used this pattern and would like to contribute your experience to the wiki, please go to Feedback to submit your experience or comment.