Changes between Version 91 and Version 92 of TestCaseManagementPlugin


Ignore:
Timestamp:
Aug 1, 2008 6:38:43 PM (6 years ago)
Author:
ChrisGreacen
Comment:

Cleaning up some of the WikiFormatting

Legend:

Unmodified
Added
Removed
Modified
  • TestCaseManagementPlugin

    v91 v92  
    2727 
    2828== Description == 
    29  
    30  
    3129I've developed a test case management tool that uses subversion as the testcase repository and uses the ticket framework in trac to create test runs. 
    3230 
    3331Some of the common problems that any testcase management tool faces are: 
    34  
    35 1. assigning a group of tests to a testing resource(s)...'''a test run''' (this is what the plugin manages) 
    36  
    37 2. versioning of the test cases (taken care of by subversion...yep even testcases change over time) 
    38  
    39 3. reporting on results and progress (trac reports/export to excel/etc.) 
    40  
    41 [[BR]] 
    42 '''So what is a test run?''' 
    43  
    44 -  a set of tests 
    45  
    46 -  assigned to a specific tester 
    47  
    48 -  for a particular product and version  
    49  
    50 '''In more detail...''' 
     32 1. assigning a group of tests to a testing resource(s)...'''a test run''' (this is what the plugin manages) 
     33 1. versioning of the test cases (taken care of by subversion...yep even testcases change over time) 
     34 1. reporting on results and progress (trac reports/export to excel/etc.) 
     35 
     36=== So what is a test run? === 
     37 * a set of tests 
     38 * assigned to a specific tester 
     39 * for a particular product and version  
     40 
     41=== In more detail... === 
    5142A test run is specific set of test cases for a particular milestone or version of the software under test for a particular tester.   
    5243 
     
    6354Testcases are stored in a very simple XML format.   
    6455 
    65 '''Example...''' 
     56=== Example... === 
    6657Take the testcases that you've written a in a word doc or excel and put one test case into one xml document.  Check the testcases into subversion (more on that below).  I've heard some feedback that potentially this ends up in a huge set of test cases.  That's only if you write test cases in the form : "Testcase 1: click the submit button...end of test case". 
    6758 
    68 '''Brief Digression'''[[BR]] 
     59=== Brief Digression === 
    6960My personal philosohpy is to encourage people to write work flow or task based driven testcases and have the tester use their brains to find issues.  I call it script based Exploratory testing.  Check out Exploratoy testing by James Bach at http://satisfice.com .  If they need to look at the UI in very high detail then the test case could simply be "compare the UI to the mockup and here's the link to the screen shot. 
    70 [[BR]] 
    71 '''End Digression''' 
    72  
    73 '''Grouping testcases together...''' 
     61=== End Digression === 
     62 
     63=== Grouping testcases together... === 
    7464You can specify collection of test cases (for example a smoke test) by specifying which test cases belong to a test template.  This information is also specified in an xml file. 
    7565 
     
    7868  
    7969== Configuration steps required ==  
    80  
    81 1. Create a testcase directory within an existing subversion project.  This is where your testcase and testtemplate files go.  We typically structure our development projects with a main project directory and then a source and build subdirectory.  So when you add the testcase directory you might have something like this: 
    82  
    83 project/source/ <-- checked into subversion [[BR]] 
    84 project/build/ <-- this is not checked in but created when you build the application [[BR]] 
    85 project/testcases/ <-- checked into subversion[[BR]] 
    86  
    87  
    88 The nice thing about this is you add a lot more transparency to the testing process as testcases are bundled and versioned with the source code. 
    89  
    90 2.  Add testcases and commit those testcases to your subversion repository using the example format (see attachment for correct XML format of testcase files). 
    91  
    92 3.  Create a testtemplate file called testtemplates.xml, then specify which tests belong to which test templates, for example the smoke-test (see attachment for correct XML format of the test templates file).  Not all tests have to belong to a template, it's just a convient way to group tests together.  This file goes into the same project/testcases directory. 
    93  
    94 4.  Add the following new section to the trac.ini file: 
    95  
    96 {{{ 
    97  
     70 1. Create a testcase directory within an existing subversion project.  This is where your testcase and testtemplate files go.  We typically structure our development projects with a main project directory and then a source and build subdirectory.  So when you add the testcase directory you might have something like this: 
     71{{{ 
     72project/source/ <-- checked into subversion  
     73project/build/ <-- this is not checked in but created when you build the application  
     74project/testcases/ <-- checked into subversion 
     75}}} 
     76 The nice thing about this is you add a lot more transparency to the testing process as testcases are bundled and versioned with the source code. 
     77 1. Add testcases and commit those testcases to your subversion repository using the example format (see attachment for correct XML format of testcase files). 
     78 1. Create a testtemplate file called testtemplates.xml, then specify which tests belong to which test templates, for example the smoke-test (see attachment for correct XML format of the test templates file).  Not all tests have to belong to a template, it's just a convient way to group tests together.  This file goes into the same project/testcases directory. 
     79 1. Add the following new section to the trac.ini file: 
     80{{{ 
    9881[testManagementExtension] 
    9982SubversionPathToTestCases=/relativePathToTestCasesInYour/Subversion/Repository 
    100  
    101 }}} 
    102  
    103 This is important because Trac can be set up against subdirectories so you often don't link trac to your root subversion folder, so only specify the full path from the root node if that's how trac was set up. 
    104  
    105 5. '''[http://trac.edgewall.org/wiki/TracAdmin add a new ticket type] called testcase'''   
    106  
    107 6. I also add a custom ticket type in the trac.ini file for reporting purposes (although this is not required yet...just really useful) 
    108  
     83}}} 
     84 This is important because Trac can be set up against subdirectories so you often don't link trac to your root subversion folder, so only specify the full path from the root node if that's how trac was set up. 
     85 1. '''[http://trac.edgewall.org/wiki/TracAdmin add a new ticket type] called testcase'''   
     86 1. I also add a custom ticket type in the trac.ini file for reporting purposes (although this is not required yet...just really useful) 
    10987{{{ 
    11088[ticket-custom] 
     
    11492testcase_result.value =  
    11593}}} 
    116  
    117 7. Enable the plugin either through the trac admin plugin or by modifying the trac.ini file: 
     94 1. Enable the plugin either through the trac admin plugin or by modifying the trac.ini file: 
    11895{{{ 
    11996[components] 
    12097testManagementPlugin.*=enabled 
    12198}}} 
    122  
    123 8. Restrict ticket owners to only known users of the system.  You do this by looking for the line retrict_owner in the trac.ini file. 
     99 1. Restrict ticket owners to only known users of the system.  You do this by looking for the line retrict_owner in the trac.ini file. 
    124100This is actually necessary because of an oversight on my part.  I incorrectly assumed that most people would have turned this on. 
    125101{{{ 
    126102restrict_owner = true 
    127103}}} 
    128 A bug has been entered about this (thanks for that), and I'll be working on a solution for the next release. 
    129  
     104 A bug has been entered about this (thanks for that), and I'll be working on a solution for the next release. 
    130105 
    131106I've added some helpful debugging/error checking that should help if an error occurs.  Probably the most likely error is you'll specify the relative path to the testcases incorrectly. 
     
    165140== Example == 
    166141 
    167 ''''''Running a weekly smoke test.'''''' 
    168  
    169 ''''Step 1(Create your test cases)'''' 
     142=== Running a weekly smoke test. === 
     143 
     144==== Step 1(Create your test cases) ==== 
    170145Take a look at the example test case I attached at the bottom of this page.  It's a very simple XML format for specifying what a test case should be. 
    171146 
     
    203178</testcase> 
    204179 
    205  
    206 }}} 
    207  
    208  
    209  
    210 ''''Step 2 (Define any required grouping of testcases in the testtemplates.xml file):'''' 
    211  
     180}}} 
     181 
     182==== Step 2 (Define any required grouping of testcases in the testtemplates.xml file): ==== 
    212183An example grouping would be specifying that tests: test1, test2, and test3 all belong to the smoketest. 
    213184 
    214  
    215185{{{ 
    216186<templates> 
    217 - <template name="smoke"> 
    218 -   <testcases> 
     187 <template name="smoke"> 
     188   <testcases> 
    219189       <testcase>test1</testcase>  
    220190       <testcase>test2</testcase>  
     
    222192    </testcases> 
    223193 </template> 
    224 - <template name="SomeOtherTestGrouping"> 
    225 -   <testcases> 
     194 <template name="SomeOtherTestGrouping"> 
     195   <testcases> 
    226196       <testcase>test1</testcase> 
    227197    </testcases> 
     
    233203Make sure the tests and template file are checked in. 
    234204 
    235 ''''Step 3 (Creating a test run)'''' 
    236 Click on the TestManagement tab on the main trac menu. 
    237  
    238 Click on the Test Run link 
    239  
    240 Select users to assign testcases to (these have to be known users to the trac system). 
    241  
    242 Select the appropriate version, and/or milestone/sprint for this testrun.   
    243  
    244 Select the testcases, and/or the appropriate test template. 
    245  
    246 Click the generate test run button. 
    247  
    248 This will re-direct you to the  '''custom ticket reporting page with a pre-built query''' that should show you all the test cases that you just created grouped by user.  It will also pick up previously created testcases if they also exactly match the query criteria. 
    249  
    250 ''''Step 4 (Run the testcases)'''' 
    251  
     205==== Step 3 (Creating a test run) ==== 
     206 1. Click on the TestManagement tab on the main trac menu. 
     207 1. Click on the Test Run link 
     208 1. Select users to assign testcases to (these have to be known users to the trac system). 
     209 1. Select the appropriate version, and/or milestone/sprint for this testrun.   
     210 1. Select the testcases, and/or the appropriate test template. 
     211 1. Click the generate test run button. 
     212 This will re-direct you to the  '''custom ticket reporting page with a pre-built query''' that should show you all the test cases that you just created grouped by user.  It will also pick up previously created testcases if they also exactly match the query criteria. 
     213 
     214==== Step 4 (Run the testcases) ==== 
    252215Open one of the created tickets and using the testcase_result custom field specify whether the test for this test run passed, failed, or was not run for whatever reason. 
    253216 
    254217Close each ticket in the normal way by setting the ticket to "fixed" (or one of the other appropriate resolved states). 
    255218 
    256 ''''Step 5  Reporting'''' 
    257  
     219==== Step 5  Reporting ==== 
    258220For reporting on testcase progress or results use the TRAC custom query page.  Just specify where ticket type = 'testcase' assuming you've added that ticket type into your TRAC data base.  TRAC is going to be adding the ability to specify which columns appear in the custom query results so this will only get better. 
    259221 
    260222 
    261223== Recent Changes == 
    262  
    263224[[ChangeLog(testcasemanagementplugin, 3)]] 
    264225