Regression Test Case Management
From OpenLaszlo
This is a draft description of how we might store and manage regression test cases in OpenLaszlo.
Contents |
Storage Structure
JIRA is the index for all regression test cases. Often a test case is described, pasted, or attached in the bug entry itself. In many cases, however, it can be convenient or necessary to make a test case available in a source repository. We use certain conventions to make it easy to retrieve a test case from a JIRA bug.
All test cases for active, resolved, and closed bugs reside in one place:
http://svn.openlaszlo.org/test/bugs
Each entry in this file is either a single file, named <bug-number>.<suffix>, or a folder, named <bug-number>. Examples:
test/bugs/lpp-2587.lzx test/bugs/lpp-2744.css test/bugs/lpp-2235/
Entries should not include information about bug status, since that duplicates information in JIRA, but may include information about applicability to specific OL releases or build numbers, etc. That sort of information should be either in a comment at the top of single-file test cases, or in a "readme.txt" file within test case directories.
test/bugs/lpp-2235/readme.txt
Entries must be not refer to resources or files in other test cases or in project source trees. When possible, they should use 'known-good' resources already available in the test repository at this location:
http://svn.openlaszlo.org/test/bugs/resources
This folder should be modified only with care, since each resource may be used by multiple tests.
Test cases should not use network resources unless absolutely necessary.
Test Case Migration
When a bug enters JIRA, it often includes either a description of the test case or a program or script demonstrating the problem. This test case may remain within the JIRA bug for the bug's lifetime, or it may be copied into the regression test repository test/bugs. If it is copied, the engineer must make an annotation in the bug to that effect. This is done by adding a comment of the form:
Checked test case in as http://svn.openlaszlo.org/test/bugs/lpp-2587.lzx
Note use of the full URL, which eases navigation to the test case within Subversion.
Other Notes
Duplicating Bugs Across Project Versions
When a bug is duplicated ("cloned") across project versions in JIRA, for example if a bug requires different resolution in different project versions, the duplicate must contain an annotation or link back to the original bug. (Note the "Cloners" link type is not currently enabled in JIRA so the link must be created manually. See BTS-42.)
With this audit trail, it is not necessary to copy the external test case as well. The engineer can follow the chain from duplicated bug, to original bug, to external test case. If the test case diverges with the duplicated bug, then the external test case should be duplicated as well.
Unit Tests
Sometimes it is useful or necessary to include a regression test case as a unit test. In this case, the test case should be copied into the project source tree, into whichever conventional place is used to store unit tests. Unit tests should not be run out of the test/bugs repository, only out of the project source tree itself. (QA may run comprehensive regression test runs out of the test/bugs repository, but this is separate from the developer-owned unit test system.)
