Changes between Version 114 and Version 115 of TestCaseManagementPlugin


Ignore:
Timestamp:
Jun 15, 2015, 7:00:25 PM (9 years ago)
Author:
figaro
Comment:

Cosmetic changes, tagged with license

Legend:

Unmodified
Added
Removed
Modified

  • TestCaseManagementPlugin

    v114 v115  
    1 [[PageOutline(1-3)]]
    2 
    3 = Manage test cases in Trac and Subversion =
    4 
    5 == NOTE (Feb. 22nd 2011) ==
    6 
    7 I haven't been able to take a look at this plugin now for over a year.  If someone wants to take this over just email me at edunneSoftwareTesting@hotmail.com and I'll see what I can do to help you out.
    8 
    9 There's also another test case management tool that integrates into TRAC that uses a different approach and it might be worth taking a look at it as an alternative. It's hosted through TracHacks and is called  TestManagerForTracPlugin.  I haven't tried it, but I read through the description and it seems interesting.
    10 
    11 
    12 
    13 == Description ==
    14 I've developed a test case management tool that uses subversion as the testcase repository and uses the ticket framework in trac to create test runs.
    15 
    16 Some of the common problems that any testcase management tool faces are:
    17  1. assigning a group of tests to a testing resource(s)...'''a test run''' (this is what the plugin manages)
    18  1. versioning of the test cases (taken care of by subversion...yep even testcases change over time)
    19  1. reporting on results and progress (trac reports/export to excel/etc.)
     1[[PageOutline(2-5,Contents,pullout)]]
     2
     3= Manage test cases in Trac and Subversion
     4
     5== Description
     6
     7This is a test case management tool that uses subversion as the testcase repository and uses the ticket framework in Trac to create test runs.
     8
     9Common problems in any testcase management tool are:
     10 1. assigning a group of tests to a testing resource(s), ie '''a test run''' (this is what the plugin manages)
     11 1. versioning of the test cases (taken care of by Subversion, because even testcases change over time)
     12 1. reporting on results and progress (Trac reports/export to Excel/etc.)
    2013
    2114See also: TestManagerForTracPlugin, [http://www.teamst.org/ Test Link] (provides Trac integration), ManualTestingPlugin, QaTrackerPlugin
    2215
    23 === So what is a test run? ===
     16=== So what is a test run?
     17
    2418 * a set of tests
    2519 * assigned to a specific tester
    2620 * for a particular product and version
    2721
    28 === In more detail... ===
     22=== In more detail
     23
    2924A test run is specific set of test cases for a particular milestone or version of the software under test for a particular tester. 
    3025
    31 Each tester owns a single '''instance of a test case in the form of a ticket'''.  This allows you to track the progress of individual testers as they complete their set of test cases.  As they complete a testing task they set the TRAC ticket to closed.   
    32 
    33 This is really powerful because you can take full advantage of the custom reporting engine in TRAC to look for open tickets of type testcase that belong to a particular tester, and also monitor the progress of a group of testers.
    34 
    35 There was some confusion about this as it seemed like a single ticket would be "owned" by multiple testers.  TRAC's data model makes this impossible (also I think that assigning test tickets in that way would be really clunky).  The testcase is in subversion, while a test instance (in the form of a TRAC ticket), is in TRAC. 
    36 
    37 Essentially what happens is the testcase is an xml document stored in subversion.  The testcasemanagementplugin creates a ticket with the test case details as the ticket description and assigns that ticket to a specific tester: one ticket per tester per testcase.
    38 
    39 The advantage of having a testcase management tool in trac is that it removes the need to have yet another system that manages testcases...with all the usual problems of maintaining user accounts, system upgrades, and licensing issues.  The other advantage is that it adds more transparency to the testing process as the tool and any testcases are right there for anyone to see.
    40 
    41 Testcases are stored in a very simple XML format. 
    42 
    43 === Example... ===
    44 Take the testcases that you've written in a word doc or excel and put one test case into one xml document.  Check the testcases into subversion (more on that below).  I've heard some feedback that potentially this ends up in a huge set of test cases.  That's only if you write test cases in the form : "Testcase 1: click the submit button...end of test case".
    45 
    46 === Brief Digression ===
    47 My personal philosohpy is to encourage people to write work flow or task based driven testcases and have the tester use their brains to find issues.  I call it script based Exploratory testing.  Check out Exploratory testing by James Bach at http://satisfice.com .  If they need to look at the UI in very high detail then the test case could simply be "compare the UI to the mockup and here's the link to the screen shot.
    48 === End Digression ===
    49 
    50 === Grouping testcases together... ===
    51 You can specify collection of test cases (for example a smoke test) by specifying which test cases belong to a test template.  This information is also specified in an xml file.
    52 
    53 Note this plugin is designed to work with Subversion, although I don't use any specific subversion code.  Potentially this plugin could work with other TRAC supported configuration management systems.  I just haven't tried it.
    54 
    55  
    56 == Configuration steps required ==
    57  1. Create a testcase directory within an existing subversion project.  This is where your testcase and testtemplate files go.  We typically structure our development projects with a main project directory and then a source and build subdirectory.  So when you add the testcase directory you might have something like this:
     26Each tester owns a single '''instance of a test case in the form of a ticket'''. This allows you to track the progress of individual testers as they complete their set of test cases. As they complete a testing task they set the Trac ticket to closed.   
     27
     28Now you can take full advantage of the custom reporting engine in Trac to look for open tickets of type testcase that belong to a particular tester, and also monitor the progress of a group of testers.
     29
     30A single ticket is not "owned" by multiple testers, in fact, Trac's data model makes this impossible. The testcase is in Subversion, while a test instance in the form of a Trac ticket, is in Trac. 
     31
     32The testcase is an xml document stored in Subversion. The plugin creates a ticket with the test case details as the ticket description and assigns that ticket to a specific tester: one ticket per tester per testcase.
     33
     34The advantage of having a testcase management tool in Trac is that it removes the need to have yet another system that manages testcases, with all the usual problems of maintaining user accounts, system upgrades, and licensing issues. The other advantage is that it adds more transparency to the testing process as the tool and any testcases are for anyone to see.
     35
     36Testcases are stored in a XML format. 
     37
     38=== Example
     39
     40Take the testcases that you've written in a Word doc or Excel spreadsheet and put one test case into one xml document. Check the testcases into Subversion (see below). There has been some feedback that this may end up in a large set of test cases, which is only true if you write test cases in scripted form: "Testcase 1: click the submit button... end of test case".
     41
     42=== Brief Digression
     43
     44My personal philosophy is to encourage people to write work flow or task based driven testcases and have the tester use their brains to find issues. I call it script based Exploratory testing. Check out Exploratory testing by James Bach at http://satisfice.com. If they need to look at the UI in very high detail then the test case could simply be "compare the UI to the mockup and here's the link to the screen shot".
     45
     46=== Grouping testcases together
     47
     48You can specify collection of test cases (for example a smoke test) by specifying which test cases belong to a test template. This information is also specified in an xml file.
     49
     50Note this plugin is designed to work with Subversion, although no specific Subversion code is used. Potentially this plugin works with other Trac supported configuration management systems.
     51
     52== Bugs/Feature Requests
     53
     54Existing bugs and feature requests for TestCaseManagementPlugin are [query:status!=closed&component=TestCaseManagementPlugin&order=priority here].
     55
     56If you have any issues, create a
     57[/newticket?component=TestCaseManagementPlugin new ticket].
     58
     59[[TicketQuery(component=TestCaseManagementPlugin&group=type,format=progress)]]
     60
     61== Download and Installation
     62
     63I've merged the unicode compliant version and the main Genshi branch into one.
     64
     65=== Genshi Migration 0.11
     66
     67You can download the source or preferably use easy_install for the 0.11.x compatible version here : 
     68
     69{{{#!sh
     70easy_install.exe http://trac-hacks.org/svn/testcasemanagementplugin/branches/testManagmentPluginGenshi
     71}}}
     72
     73If you need easy_install check: http://peak.telecommunity.com/DevCenter/EasyInstall
     74
     75=== Trac 0.10.x users
     76
     770.10.x Trac users should continue to use this version: [[BR]]
     78
     79{{{
     80easy_install.exe http://trac-hacks.org/svn/testcasemanagementplugin/branches/testManagementPlugin0.4.0
     81}}}
     82
     83Windows users should add the `python2X/scripts` directory to their path.
     84
     85== Source
     86
     87You can check out TestCaseManagementPlugin [/svn/testcasemanagementplugin using Subversion], or [source:testcasemanagementplugin browse the source] with Trac.
     88
     89== Configuration
     90
     91 1. Create a testcase directory within an existing Subversion project. This is where your testcase and testtemplate files go. We typically structure our development projects with a main project directory and then a source and build subdirectory. So when you add the testcase directory, you might have something like this:
    5892{{{
    5993project/source/ <-- checked into subversion
    … …  
    6195project/testcases/ <-- checked into subversion
    6296}}}
    63  The nice thing about this is you add a lot more transparency to the testing process as testcases are bundled and versioned with the source code.
    64  1. Add testcases and commit those testcases to your subversion repository using the example format (see attachment for correct XML format of testcase files).
    65  1. Create a testtemplate file called testtemplates.xml, then specify which tests belong to which test templates, for example the smoke-test (see attachment for correct XML format of the test templates file).  Not all tests have to belong to a template, it's just a convient way to group tests together.  This file goes into the same project/testcases directory.
    66  1. Add the following new section to the trac.ini file:
    67 {{{
    68 #!ini
     97 This way you add transparency to the testing process as testcases are bundled and versioned with the source code.
     98 1. Add testcases and commit those testcases to your Subversion repository using the example format (see attachment for correct XML format of testcase files).
     99 1. Create a testtemplate file called `testtemplates.xml`, then specify which tests belong to which test templates, for example the smoke-test (see attachment for correct XML format of the test templates file). Not all tests have to belong to a template, it's just a convenient way to group tests together. This file goes into the same `project/testcases` directory.
     100 1. Add the following new section to the `trac.ini` file:
     101 {{{#!ini
    69102[testManagementExtension]
    70103SubversionPathToTestCases = /relativePathToTestCasesInYour/Subversion/Repository
    71104}}}
    72  This is important because Trac can be set up against subdirectories so you often don't link trac to your root subversion folder, so only specify the full path from the root node if that's how trac was set up.
    73  1. '''[http://trac.edgewall.org/wiki/TracAdmin add a new ticket type] called testcase''' 
    74  1. I also add a custom ticket field '''Test Case Result''' in the trac.ini file for reporting purposes (although this is not required yet...just really useful)
    75 {{{
    76 #!ini
     105 This is important, because Trac can be set up against subdirectories so you often don't link Trac to your root Subversion folder. Only specify the full path from the root node if that is how Trac is set up.
     106 1. '''[http://trac.edgewall.org/wiki/TracAdmin Add a new ticket type] called testcase'''.
     107 1. Add a custom ticket field '''Test Case Result''' in the `trac.ini` file for reporting purposes (optional)
     108 {{{#!ini
    77109[ticket-custom]
    78110testcase_result = select
    … …  
    81113testcase_result.value =
    82114}}}
    83  1. Enable the plugin either through the trac admin plugin or by modifying the trac.ini file:
    84 {{{
    85 #!ini
     115 1. Enable the plugin either through the Trac admin plugin or by modifying the `trac.ini` file:
     116 {{{#!ini
    86117[components]
    87118testManagementPlugin.* = enabled
    88119}}}
    89  1. Restrict ticket owners to only known users of the system.  You do this by looking for the line restrict_owner in the trac.ini file.
    90 This is actually necessary because of an oversight on my part.  I incorrectly assumed that most people would have turned this on.
    91 {{{
    92 #!ini
     120 1. Restrict ticket owners to only known users of the system. You do this by looking for the line restrict_owner in the `trac.ini` file:
     121 {{{#!ini
    93122restrict_owner = true
    94123}}}
    95  A bug has been entered about this (thanks for that), and I'll be working on a solution for the next release.
    96 
    97 I've added some helpful debugging/error checking that should help if an error occurs.  Probably the most likely error is you'll specify the relative path to the testcases incorrectly.
    98 
    99 == Bugs/Feature Requests ==
    100 
    101 Existing bugs and feature requests for TestCaseManagementPlugin are [query:status!=closed&component=TestCaseManagementPlugin&order=priority here].
    102 
    103 If you have any issues, create a
    104 [/newticket?component=TestCaseManagementPlugin&owner=edunne new ticket].
    105 
    106 == Download and Install using easy_install ==
    107 
    108 
    109 Ok, I've merged the unicode compliant version and the main genshi branch into one.  Both versions have been out long enough to do that.
    110 
    111 === GENSHI Migration 0.11   ===
    112 
    113 
    114 You can download the source or preferably use easy_install for the 0.11.x compatible version here : 
    115 
    116 {{{
    117 easy_install.exe http://trac-hacks.org/svn/testcasemanagementplugin/branches/testManagmentPluginGenshi
    118 }}}
    119 
    120 if you need easy_install check: http://peak.telecommunity.com/DevCenter/EasyInstall
    121 
    122 
    123 === TRAC 0.10.x users ===
    124 0.10.x TRAC users should continue to use this version : [[BR]]
    125 (also note that I'm not going to maintain this anymore.  Unless it's a major major bug).
    126 
    127 {{{
    128 easy_install.exe http://trac-hacks.org/svn/testcasemanagementplugin/branches/testManagementPlugin0.4.0
    129 }}}
    130 
    131 Windows users should add the python2X/scripts directory to their path.
    132 
    133 
    134 == Source ==
    135 
    136 You can check out TestCaseManagementPlugin [/svn/testcasemanagementplugin using Subversion], or [source:testcasemanagementplugin browse the source] with Trac.
    137 
    138 == Example ==
    139 
    140 === Running a weekly smoke test. ===
    141 
    142 ==== Step 1 (Create testcases) ====
    143 Take a look at the example test case I attached at the bottom of this page.  It's a very simple XML format for specifying what a test case should be.
    144 
    145 Inside the xml file you need to specify specifics about each testcase.  A test case has an Id (which probably should be the file name without any extensions), a component (matching a real component in your TRAC project..case sensitive), a summary, a description, and a field for describing the expected results of the test.
    146 
    147 Here's what a testcase might look like...remember one testcase per file.  It helps if you give the file a name that reflects what the testcase is for.  For example:  map01_zoom.xml
    148 
    149 {{{
    150 #!xml
     124
     125I've added some helpful debugging/error checking that should help if an error occurs. The most likely error is specifying the relative path to the testcases incorrectly.
     126
     127== Example
     128
     129The following example explains how to use this plugin when running a weekly smoke test.
     130
     131==== Step 1: Create testcases
     132
     133The example test cases attached at the bottom of this page are simple XML files for specifying what a test case should contain.
     134
     135A test case has an Id (which probably should be the file name without any extensions), a component (matching a real component in your Trac project, case sensitive), a summary, a description, and a field for describing the expected results of the test.
     136
     137Here's what a testcase might look like, remember one testcase per file:
     138
     139{{{#!xml
    151140<testcase>
    152141  <id>
    … …  
    157146  </component>
    158147  <summary>
    159     This test verifies correct functionaliy of zoom on the map. 
     148    This test verifies correct functionality of zoom on the map. 
    160149  </summary>
    161150  <description>
    … …  
    164153    1.  Zoom around on the map.
    165154
    166     2.  etc.  Note you can use TRAC formatting in the testcase if you want. 
     155    2.  etc.  Note you can use Trac formatting in the testcase if you want. 
    167156  </description>
    168157  <expectedresult>
    169     The map should correctly zoom to the area specified by the user..etc.
     158    The map should correctly zoom to the area specified by the user, etc.
    170159  </expectedresult>
    171160</testcase>
    172161}}}
    173162
    174 ==== Step 2 (Group testcases): ====
     163It helps if you give the file a name that reflects what the testcase is for, for example `map01_zoom.xml`.
     164
     165==== Step 2: Group testcases
     166
    175167Define any required grouping of testcases in the testtemplates.xml file. An example grouping would be specifying that tests: test1, test2, and test3 all belong to the smoketest.
    176168
    177 {{{
    178 #!xml
     169{{{#!xml
    179170<templates>
    180171 <template name="smoke">
    … …  
    192183</templates>
    193184}}}
    194 ''notice you can have the same testcase in multiple test groupings''
     185
     186'''Note:''' You can have the same testcase in multiple test groupings.
    195187
    196188Make sure the tests and template file are checked in.
    197189
    198 ==== Step 3 (Create a testrun) ====
    199  1. Click on the TestManagement tab on the main trac menu.
    200  1. Click on the Test Run link
    201  1. Select users to assign testcases to (these have to be known users to the trac system).
     190==== Step 3: Create a testrun
     191
     192 1. Click on the TestManagement tab on the main Trac menu.
     193 1. Click on the Test Run link.
     194 1. Select users to assign testcases to (these have to be known users to the Trac system).
    202195 1. Select the appropriate version, and/or milestone/sprint for this testrun. 
    203196 1. Select the testcases, and/or the appropriate test template.
    204  1. Click the generate test run button.
    205  This will re-direct you to the  '''custom ticket reporting page with a pre-built query''' that should show you all the test cases that you just created grouped by user.  It will also pick up previously created testcases if they also exactly match the query criteria.
    206 
    207 ==== Step 4 (Run testcases) ====
     197 1. Click the Generate test run button.
     198 This will re-direct you to the '''custom ticket reporting page with a pre-built query''' that should show you all the test cases that you just created grouped by user. It will also pick up previously created testcases, if they also exactly match the query criteria.
     199
     200==== Step 4: Run testcases
     201
    208202Open one of the created tickets and using the testcase_result custom field specify whether the test for this test run passed, failed, or was not run for whatever reason.
    209203
    210 Close each ticket in the normal way by setting the ticket to "fixed" (or one of the other appropriate resolved states).
    211 
    212 ==== Step 5 (Reporting) ====
    213 For reporting on testcase progress or results use the TRAC custom query page.  Just specify where ticket type = 'testcase' assuming you've added that ticket type into your TRAC data base.  TRAC is going to be adding the ability to specify which columns appear in the custom query results so this will only get better.
    214 
    215 
    216 == Recent Changes ==
     204Close each ticket in the normal way by setting the ticket to "fixed" or one of the other appropriate resolved states.
     205
     206==== Step 5: Reporting
     207
     208For reporting on testcase progress or results, use the Trac custom query page. Just specify `where ticket type = 'testcase'`, assuming you've added that ticket type into your Trac database. Trac is going to be adding the ability to specify which columns appear in the custom query results, so this will only improve.
     209
     210== Recent Changes
     211
    217212[[ChangeLog(testcasemanagementplugin, 3)]]
    218213
    219 == Author/Contributors ==
     214== Author/Contributors
    220215
    221216'''Author:''' [wiki:edunneSoftwareTesting@hotmail.com] [[BR]]
    222 '''Maintainer:''' ''none''
     217'''Maintainer:''' ''none'' [[BR]]
    223218'''Contributors:''' A lot of people have stepped up and helped test the plugin so thanks