Manage test cases in Trac and Subversion

Notice: This plugin is unmaintained and available for adoption.

Description

This is a test case management tool that uses Subversion as the testcase repository and uses the ticket framework in Trac to create test runs.

Common problems in any testcase management tool are:

1. assigning a group of tests to a testing resource(s), ie a test run (this is what the plugin manages)
2. versioning of the test cases (taken care of by Subversion, because even testcases change over time)
3. reporting on results and progress (Trac reports, export to Excel, etc.)

See also: TestManagerForTracPlugin, QaTrackerPlugin

So what is a test run?

• a set of tests
• assigned to a specific tester
• for a particular product and version

In more detail

A test run is specific set of test cases for a particular milestone or version of the software under test for a particular tester.

Each tester owns a single instance of a test case in the form of a ticket. This allows you to track the progress of individual testers as they complete their set of test cases. As they complete a testing task they set the Trac ticket to closed.

Now you can take full advantage of the custom reporting engine in Trac to look for open tickets of type testcase that belong to a particular tester, and also monitor the progress of a group of testers.

A single ticket is not "owned" by multiple testers, in fact, Trac's data model makes this impossible. The testcase is in Subversion, while a test instance in the form of a Trac ticket, is in Trac.

The testcase is an xml document stored in Subversion. The plugin creates a ticket with the test case details as the ticket description and assigns that ticket to a specific tester: one ticket per tester per testcase.

The advantage of having a testcase management tool in Trac is that it removes the need to have yet another system that manages testcases, with all the usual problems of maintaining user accounts, system upgrades, and licensing issues. The other advantage is that it adds more transparency to the testing process as the tool and any testcases are for anyone to see.

Testcases are stored in a XML format.

Example

Take the testcases that you've written in a Word doc or Excel spreadsheet and put one test case into one XML document. Check the testcases into Subversion (see below). There has been some feedback that this may end up in a large set of test cases, which is only true if you write test cases in scripted form: "Testcase 1: click the submit button... end of test case".

Brief Digression

My personal philosophy is to encourage people to write work flow or task based driven testcases and have the tester find issues. I call it script based exploratory testing. Check out Exploratory testing at ​James Bach's website. If they need to look at the UI in high detail, then the test case could simply be "compare the UI to the mockup and here's the link to the screen shot".

Grouping testcases together

You can specify collection of test cases (for example a smoke test) by specifying which test cases belong to a test template. This information is also specified in an XML file.

Note this plugin is designed to work with Subversion, although no specific Subversion code is used. Potentially this plugin works with other Trac supported configuration management systems.

Example

The following example explains how to use this plugin when running a weekly smoke test.

Step 1: Create testcases

The example test cases attached at the bottom of this page are simple XML files for specifying what a test case should contain.

A test case has an Id (which probably should be the file name without any extensions), a component (matching a real component in your Trac project, case sensitive), a summary, a description, and a field for describing the expected results of the test.

Here's what a testcase might look like, remember one testcase per file:

<testcase>
  <id>
    map01_zoom
  </id>
  <component>
    map
  </component>
  <summary>
    This test verifies correct functionality of zoom on the map.  
  </summary>
  <description>
    Steps:

    1.  Zoom around on the map.

    2.  etc.  Note you can use Trac formatting in the testcase if you want.  
  </description>
  <expectedresult>
    The map should correctly zoom to the area specified by the user, etc.
  </expectedresult>
</testcase>

It helps if you give the file a name that reflects what the testcase is for, for example map01_zoom.xml.

Step 2: Group testcases

Define any required grouping of testcases in the testtemplates.xml file. An example grouping would be specifying that tests: test1, test2, and test3 all belong to the smoketest.

<templates>
 <template name="smoke">
   <testcases>
       <testcase>test1</testcase> 
       <testcase>test2</testcase> 
       <testcase>test3</testcase> 
    </testcases>
 </template>
 <template name="SomeOtherTestGrouping">
   <testcases>
       <testcase>test1</testcase>
    </testcases>
  </template>
</templates>

Note: You can have the same testcase in multiple test groupings.

Make sure the tests and template file are checked in.

Step 3: Create a testrun

1. Click on the TestManagement tab on the main Trac menu.
2. Click on the Test Run link.
3. Select users to assign testcases to, and these have to be known users to the Trac system.
4. Select the appropriate version, and/or milestone/sprint for this testrun.
5. Select the testcases, and/or the appropriate test template.
6. Click the Generate test run button. This will re-direct you to the custom ticket reporting page with a pre-built query that should show you all the test cases that you just created grouped by user. It will also pick up previously created testcases, if they also exactly match the query criteria.

Step 4: Run testcases

Open one of the created tickets and using the testcase_result custom field specify whether the test for this test run passed, failed, or was not run for whatever reason.

Close each ticket in the normal way by setting the ticket to "fixed" or one of the other appropriate resolved states.

Step 5: Reporting

For reporting on testcase progress or results, use the Trac custom query page. Just specify where ticket type = 'testcase', assuming you've added that ticket type into your Trac database. Trac is going to be adding the ability to specify which columns appear in the custom query results, so this will only improve.

Bugs/Feature Requests

Existing bugs and feature requests for TestCaseManagementPlugin are here.

If you have any issues, create a new ticket.

defect	20 / 33
enhancement	5 / 11
task	2 / 2

Download

Download the zipped source from here.

Source

You can check out TestCaseManagementPlugin from here using Subversion, or browse the source​ with Trac.

Installation

General instructions on installing Trac plugins can be found on the TracPlugins page.

Configuration

1. Create a testcase directory within an existing Subversion project. This is where your testcase and testtemplate files go. We typically structure our development projects with a main project directory and then a source and build subdirectory. So when you add the testcase directory, you might have something like this:

   project/source/ <-- checked into subversion 
   project/build/ <-- this is not checked in but created when you build the application 
   project/testcases/ <-- checked into subversion
   

   This way you add transparency to the testing process as testcases are bundled and versioned with the source code.
2. Add testcases and commit those testcases to your Subversion repository using the example format (see attachment for correct XML format of testcase files).
3. Create a testtemplate file called testtemplates.xml, then specify which tests belong to which test templates, for example the smoke-test (see attachment for correct XML format of the test templates file). Not all tests have to belong to a template, it's just a convenient way to group tests together. This file goes into the same project/testcases directory.
4. Add the following new section to the trac.ini file:

   [testManagementExtension]
   SubversionPathToTestCases = /relativePathToTestCasesInYour/Subversion/Repository
   

   This is important, because Trac can be set up against subdirectories so you often don't link Trac to your root Subversion folder. Only specify the full path from the root node if that is how Trac is set up.
5. ​Add a new ticket type called testcase.
6. Add a custom ticket field Test Case Result in the trac.ini file for reporting purposes (optional):

   [ticket-custom]
   testcase_result = select
   testcase_result.label = Test Case Result
   testcase_result.options = pass|fail|incomplete
   testcase_result.value = 
   

7. Enable the plugin either through the Trac admin plugin or by modifying the trac.ini file:

   [components]
   testManagementPlugin.* = enabled
   

8. Restrict ticket owners to only known users of the system. You do this by looking for the line restrict_owner in the trac.ini file:

   restrict_owner = true
   

Some helpful debugging/error checking has been added that should help if an error occurs. The most likely error is an incorrect specification of the relative path to the testcases.

Recent Changes

16394 by rjollos on 2017-03-25 09:20:58

Remove eggs

4349 by edunne on 2008-09-26 20:29:45

uploading a packaged (egg) version of the latest stable version of the genshi/unicode merge.

4348 by edunne on 2008-09-26 17:22:37

creating a tags folder

(more)

Author/Contributors

Author: Edunne Software Testing
Maintainer: none (needsadoption)
Contributors: A lot of people have stepped up and helped test the plugin so thanks