Changes between Version 4 and Version 5 of TestManagerForTracPluginApi


Ignore:
Timestamp:
Sep 23, 2010 4:59:02 PM (4 years ago)
Author:
seccanj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • TestManagerForTracPluginApi

    v4 v5  
    1 = Test Manager for Trac Plugin - Public API = 
    2  
    3 The TestManagerForTracPlugin plugin can be used programmatically to create, edit Test Catalogs, Test Cases and Test Plans, and to set the test execution verdicts of Test Cases in a plan. 
    4  
    5 In the following, because of the spam filter not allowing more than a few URLs in a page, I have replaced: 
     1[[PageOutline(2-5,Contents,pullout)]] 
     2 
     3= Test Manager plugin for Trac = 
     4 
     5== Description == 
     6 
     7A Trac plugin to create Test Cases, organize them in Catalogs, generate Test Plans and track their execution status and outcome. 
     8 
     9Differently from other test management plugins for Trac that use Tickets as test case holders, this one uses Wiki pages and an additional proprietary data model to store Test Cases. 
     10This allows you to not pollute your ticket lists with something that is not a ticket, and at the same time is powered by the Trac search engine and formatting syntax for Wiki pages. 
     11 
     12A set of plugins intercept requests for Wiki pages that are test cases and decorate the page with title, breadcrumbs, tree view, type-ahead search inside the catalogs, test case status semaphore and icons and buttons that allow you to create new test cases, sub-catalogs, copy and paste test cases around different catalogs and change a test case status. 
     13 
     14Multiple Test Plans can be associated to any Test Catalog, in order to keep track of the execution of the corresponding Test Cases in a particular testing context. 
     15 
     16All of the test objects, i.e. catalogs, test cases, test plans and test cases in a plan (i.e. with a status and a status change history), support: 
     17 * '''Custom properties''', which can be declared in the trac.ini file and will be available to the User for change, stored in the database and available to change listeners. 
     18 * Change history 
     19 * Listener interface to be notified of object creatio, modification and deletion 
     20 * '''Customizable Workflow''' state machine, declared in the trac.ini file, with the same syntax as for Ticket workflows (I may have reused some existing code here :-) 
     21 * '''Customizable Workflow Operations''', via a plugin api so that any component can provide its custom operations to be performed upon any workflow action, as defined in the trac.ini file. 
     22 * Workflow also supports a listener API for components interested in state transitions and actions performed 
     23 * Workflow states also support custom properties, so to be able to convey additional context information on a workflow state and use it in listeners or directly from the database. 
     24 
     25The developed workflow engine is able to work on any Trac Resource, it is not confined to this plugin ones. You can then define a workflow on any Trac resource, including Wiki pages, declaratively in the trac.ini file. 
     26You will then add a handful of custom code (for example in an ITemplateStreamFilter) to add the markup that the workflow engine generates for you to your desired Trac web page. 
     27See the enhancement tickets documentation or the README.txt file contained in the egg (if you don't wish to browse the code) for further details. 
     28 
     29The '''programmatic''' and the '''RESTful API''' are documented in details in this page TestManagerForTracPluginApi. 
     30 
     31Here follows an overview of the plugin functionalities. For a full tutorial, refer to the powerpoint presentations attached below (the ones with shorter names work). 
     32 
     33[[BR]] 
     34[[BR]] 
     35== Test Catalogs == 
     36Test catalogs contain sub-catalogs or Test Cases. A Javascript tree view displays a catalog node and its sub-tree, including all of the test cases contained. 
     37 
     38Next to each catalog (or sub-catalog) a number in brackets shows the number of test cases it contains. 
     39 
     40Notice at the top of the page breadcrumbs to easily navigate up in the catalogs tree. 
     41 
     42You can add sub-catalogs or Test Cases simply by entering a name (blanks and case is supported) and click the appropriate button. A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for editing. 
     43Be careful that the first line will always be taken as the title of the catalog (the same stands for test cases, read below). 
     44 
     45Just save the new page ("Submit Changes") and you'll have your new (sub-)catalog in place. 
     46 
     47[[BR]] 
     48[[BR]] 
     49[[Image(screen2.JPG)]] 
     50[[BR]] 
     51[[BR]] 
     52== Test Cases == 
     53Test Cases are the smallest units of test execution. 
     54 
     55They are implemented again as wiki pages, with a naming convention that allows the plugin code to recognize them and treat them appropriately. 
     56 
     57Again, notice the breadcrumbs at the top, useful to go back to the enclosing catalog or any catalog up the hierarchy. 
     58 
     59[[BR]] 
     60[[BR]] 
     61[[Image(screen3.JPG)]] 
     62[[BR]] 
     63[[BR]] 
     64== Moving Test Cases from one catalog to another == 
     65 
     66It is also possible to move a Test Case into a different catalog, with an experience similar to cut&paste. 
     67 
     68You first open a test case and click on the "Move the Test Case into another catalog" button. This is similar to a "cut" operation. 
     69 
     70You then navigate and open the destination catalog and click on "Move the copied Test Case here" button (which only appears if a Test Case has been cut first). 
     71 
     72It is also possible to cancel the operation at any time by clicking the "Cancel" button in a Gmail-type of yellow message at the top of the page. 
     73 
     74[[BR]] 
     75[[BR]] 
     76[[Image(screen6.JPG)]] 
     77[[BR]] 
     78[[BR]] 
     79== Test Plans == 
     80'''Since: 1.1.0''' 
     81 
     82A Test Plan represents a plan for a particular execution of all the Test Cases in a Test Catalog (or sub-catalog).  
     83 
     84Think for example at the build verification test following a nightly build, or, for traditional projects, Technical Test and eventually Client Test. 
     85 
     86Thus a Test Plan is associated to one Test Catalog, or sub-catalog. You can have any number of Test Plans for one Test Catalog, anyway. 
     87The list of Test Plans you generated for a Test Catalog is displayd in a table at the bottom of the same catalog, as shown in the following figure. 
     88 
     89[[BR]] 
     90[[BR]] 
     91[[Image(screen9.JPG)]] 
     92[[BR]] 
     93[[BR]] 
     94 
     95To create a Test Plan for a catalog, open the desired Test Catalog (or sub-catalog), enter the name of the new Test Plan in the appropriate test box and click "Generate a new Test Plan". 
     96 
     97The new Test Plan will be opened for display, showing all of the Test Cases in the catalog, in the "Untested" status, as shown in the figure below. 
     98 
     99[[BR]] 
     100[[BR]] 
     101[[Image(screen7.JPG)]] 
     102[[BR]] 
     103[[BR]] 
     104 
     105To track the execution status of a Test Case in a particular Test Plan, open it by clicking on the Test Case name from the Test Plan tree.  
     106Then simply click on the corresponding light in the semaphore at the bottom of the page, as shown in the following figure. 
     107 
     108You don't need to save anything, the change is immediately recorded in the database by means of an Ajax call (this API will be documented asap, to allow for setting test case execution status from external applications). 
     109 
     110The change is immediately reflected in the Test Plan. 
     111 
     112[[BR]] 
     113[[BR]] 
     114[[Image(screen8.JPG)]] 
     115[[BR]] 
     116[[BR]] 
     117 
     118When viewing a Test Case, you can open a new Ticket by means of the "Open Ticket on this Test Case" button.  
     119 
     120The new ticket will contain a link back to the corresponding Test Case and, if you were viewing it in the context of a particular Test Plan, of the Test Plan as well. 
     121 
     122This plugin also supports the TracTicketTemplatePlugin to fill a ticket template with this information. In this case, you can use the following parameters in the template to receive the information: 
     123 
     124 * testCaseNumber: The wiki page for the corresponding Test Case 
     125 * planId: The ID of the Test Plan 
     126 * planName: The name of the Test Plan 
     127 
     128For example, to get the test case number, you template will have something like: 
    6129{{{ 
    7 http://yourserver/yourproject 
     130  bleah bleah 
     131  Test Case: %(testCaseNumber)s 
     132  bleah bleah 
    8133}}} 
    9134 
    10 with: 
     135== Custom fields == 
     136'''Since: 1.2.0''' 
     137 
     138Custom fields can be added to the four test objects and to the workflow state object, by declaring them in the trac.ini file. 
     139See the README.txt file for how to recreate this example. 
     140The following screenshot shows a custom "platform" field added to the Test Case object, and how it is presented to the User for editing. 
     141 
     142[[BR]] 
     143[[BR]] 
     144[[Image(screen11.png)]] 
     145[[BR]] 
     146[[BR]] 
     147[[BR]] 
     148 
     149== Customizable Workflow == 
     150'''Since: 1.2.0''' 
     151 
     152The following figure shows a sample workflow added to Test Cases with custom sample operations. 
     153No built-in operation is currently implemented but the sample one shown here, named 'sample_operation', which logs a debug message with the text input by the User. 
     154 
     155Every object which has a workflow defined is created in a "new" state, so every transition should consider this as the first state in the state machine. 
     156 
    11157{{{ 
    12 <yourserver/yourproject> 
     158sleep = new -> asleep 
     159sleep.permissions = TEST_MODIFY 
     160 
     161sing = new -> singing 
     162sing.permissions = TEST_MODIFY 
     163sing.operations = sample_operation 
     164 
     165calmdown = new -> calm 
     166calmdown.permissions = TEST_MODIFY 
     167calmdown.operations = sample_operation 
     168 
     169kill = asleep,calm -> dead 
     170kill.permissions = TICKET_MODIFY 
    13171}}} 
    14172 
    15 [[BR]][[BR]] 
    16 == Working with Test Catalogs and Test Cases == 
    17  
    18 You can create test catalogs and cases programmatically by means of the following http requests. 
    19  
    20 [[BR]] 
    21 === Create a root test catalog === 
    22 Get the following URL: 
    23  
    24 {{{ 
    25 <yourserver/yourproject>/testcreate?type=catalog&path=TC&title=My%20new%20catalog 
    26 }}} 
    27  
    28 this will assign a unique ID (the number 0 in the URL below) to the new catalog, create the corresponding Wiki page and redirect to it.  
    29 You may discard the response if you don't need to know the catalog ID (last number in the URL): 
    30  
    31 {{{ 
    32 <yourserver/yourproject>/wiki/TC_TT0 
    33 }}} 
    34  
    35 [[BR]] 
    36 === Create a sub-catalog === 
    37 Get the wollowing URL, where "path" is the name of the parent catalog. 
    38  
    39 {{{ 
    40 <yourserver/yourproject>/testcreate?type=catalog&path=TC_TT0&title=My%20sub%20catalog 
    41 }}} 
    42  
    43 this will assign a unique ID (the number 1 in the URL below) to the new catalog, create the corresponding Wiki page and redirect to it.  
    44 You may discard the response if you don't need to know the catalog ID (last number in the URL): 
    45  
    46 {{{ 
    47 <yourserver/yourproject>/wiki/TC_TT0_TT1 
    48 }}} 
    49  
    50 [[BR]] 
    51 === Create a Test Case === 
    52 Get the following URL, where "path" is the name of the parent (sub-)catalog. 
    53  
    54 {{{ 
    55 <yourserver/yourproject>/testcreate?type=testcase&path=TC_TT0_TT1&title=My%20new%20Test%20Case 
    56 }}} 
    57  
    58 this will assign a unique ID (the number 0 in the URL below) to the new test case, create the corresponding Wiki page and redirect to it.  
    59 You may discard the response if you don't need to know the test case ID (last number in the URL): 
    60 {{{ 
    61 <yourserver/yourproject>/wiki/TC_TT0_TT1_TC0 
    62 }}} 
    63  
    64 [[BR]][[BR]] 
    65 == Working with Test Plans and setting the status of a Test Case == 
    66  
    67 You can also create a new Test Plan (e.g. for each nightly build) programmatically as follows. 
    68  
    69 [[BR]] 
    70 === Create a Test Plan from a specific catalog === 
    71  
    72 Get the following URL, where "path" is the name of the (sub-)catalog to create the test plan against. 
    73 {{{ 
    74 <yourserver/yourproject>/testcreate?type=testplan&path=TC_TT0&title=Test%20Plan%20for%2020100818 
    75 }}} 
    76  
    77 this will assign a unique ID (the number 1 in the URL below) to the new test plan and redirect to displaying the Test Plan: 
    78 You may discard the response if you don't need to know the plan ID (planid parameter in the URL): 
    79 {{{ 
    80 <yourserver/yourproject>/wiki/TC_TT0?planid=1 
    81 }}} 
    82  
    83 The Test Plan will contain all of the test cases in the specified catalog, with a status of "Untested". 
    84  
    85 '''Note:''' 
    86 As you can notice, you can always pass from a test catalog to one of its test plans by adding the "planid=<plan id>" parameter to the test catalog URL. 
    87 The same also stands for test cases. You can pass to a test case in a particular plan by adding the planid parameter to its URL. 
    88  
    89 [[BR]][[BR]] 
    90 === Set a Test Case execution verdict, in the context of a Test Plan === 
    91  
    92 Then, you can set the verdict for any test case in the plan, by means of the following. 
    93  
    94 Get the following URL, where "id" is the Test Case ID and planid is the Test Plan ID: 
    95 {{{ 
    96 <yourserver/yourproject>/teststatusupdate?id=5&planid=1&status=SUCCESSFUL 
    97 }}} 
    98  
    99 The supported statuses are currently: 
    100  - TO_BE_TESTED 
    101  - SUCCESSFUL 
    102  - FAILED 
    103  
    104 [[BR]][[BR]] 
    105 === Change a [custom] property of any test object === 
    106  
    107 Any property, either standard or custom, of any test object can be set programmatically through the RESTful API. 
    108  
    109 A test object is identified by its realm (i.e. type) and its key (i.e. in most cases the ID, for test cases in the context of a plan, also the plan ID is required). 
    110  
    111 The realms that identify the test objects are the following: 
    112  * Test Catalog: testcatalog 
    113  * Test Case: testcase 
    114  * Test Case in the context of a plan (i.e. with a status): testcaseinplan 
    115  * Test Plan: testplan 
    116  
    117 The realm must be provided to the property update service through the "realm" parameter. 
    118  
    119 The corresponding key properties, needed to identify any particular object, are: 
    120  * Test Catalog: id 
    121  * Test Case: id 
    122  * Test Case in the context of a plan (i.e. with a status): id, planid 
    123  * Test Plan: id 
    124  
    125 The key properties are provided to the property update service through the "key" parameter, in the form of a dictionary. 
    126  
    127 The next things to pass to the service are the name of the property to modify and the new value. Guess what... you use the "name" and "value" parameters, respectively. 
    128  
    129 For example, to change a Test Case - with ID 5 - custom property "platform" (which has been previously added to the test case type in the trac.ini file) to the new value "Windows", this is the URL to GET: 
    130  
    131 {{{ 
    132 <yourserver/yourproject>/testpropertyupdate?realm=testcase&key={'id':'5'}&name=platform&value=Windows 
    133 }}} 
    134  
    135  
    136 [[BR]][[BR]] 
    137 == Traceability between Test Cases and Tickets == 
    138  
    139 You can open a Ticket and have a traceback to the (e.g. failed) Test Case as follows. 
    140  
    141 [[BR]] 
    142 === Open a Ticket on a Test Case === 
    143  
    144 Whether you deploy TracTicketTemplatePlugin or not, you can get the following URL, where testCaseNumber is the Test Case complete path, planid is the Test Plan ID and planName is its name: 
    145  
    146 {{{ 
    147 <yourserver/yourproject>/newticket?testCaseNumber=TC_TT0_TC0&planId=1&planName=Test%20Plan%20for%2020100818&description=Test%20Case:%20[wiki:TC_TT0_TC0],%20Test%20Plan:%20Test%20Plan%20for%2020100818%20(1) 
    148 }}} 
    149  
    150 this will redirect to a Ticket edit page, with the Test Case in Test Plan hyperlink in the description (as Wiki page references). 
    151 You can simply post the form to create the Ticket. 
     173See the README.txt file for more details. 
     174 
     175[[BR]] 
     176[[BR]] 
     177[[Image(screen12.png)]] 
     178[[BR]] 
     179[[BR]] 
     180[[BR]] 
     181 
     182== Searching and filtering Test Cases in the tree view == 
     183A type-ahead, browser-side filtering feature allows for easily locating Test Cases matching a particular naming convention in the title, and/or a particular execution status. 
     184 
     185This is available both in the context of a Test Catalog and in Test Plans. 
     186 
     187Multiple words (or parts of) separated by blanks are supported, in AND condition. 
     188 
     189In the case of a Test Plan, you can also add a test case status to filter by this criterion.  
     190The supported statuses are (even substrings of): 
     191 * untested 
     192 * successful 
     193 * failed 
     194 
     195[[BR]] 
     196[[BR]] 
     197[[Image(screen4.JPG)]] 
     198[[BR]] 
     199[[BR]] 
     200[[BR]] 
     201 
     202== Test Management and Execution Statistics == 
     203'''Since: 1.1.1''' 
     204 
     205Charting capabilities allow for tracking the evolution of the test suites and the corresponding test plans. 
     206 
     207To access the test management statistics, click on '''Test Stats''' in the Trac toolbar on the upper right corner of the page. 
     208 
     209As shown in the next figure, a chart will be displayed with statistics about all the test cases in the system and a default period of time. 
     210 
     211By means of the filtering criteria at the bottom of the chart, you can select the desired period of time, the chart resolution (in terms of time between different samples) and the Test Plan for your chart. 
     212 
     213You can bookmark the URL named "Static URL" at the bottom of the page in order to go directly with the current selected filtering criteria. 
     214 
     215'''Note:''' You will need internet connection to be able to display this charts, since it leverages Yahoo remote charting services. 
     216 
     217[[BR]] 
     218[[BR]] 
     219[[Image(screen10.png)]] 
     220[[BR]][[BR]] 
     221== Security == 
     222 
     223The following new permissions are available to manage the Test Manager security: 
     224 
     225 * TEST_VIEW - Ability to view test catalogs and test cases 
     226 * TEST_MODIFY - Ability to create and edit test catalogs and test cases 
     227 * TEST_EXECUTE - Ability to change the status of a test case in a test plan 
     228 * TEST_DELETE - Ability to delete test cases 
     229 * TEST_PLAN_ADMIN - Ability to generate and delete test plans 
     230 * TEST_STATS_VIEW - Ability to view test management statistics  
     231 
     232[[BR]][[BR]] 
     233== Programmatic API == 
     234 
     235The '''programmatic RESTful API''' is documented in details in this page TestManagerForTracPluginApi. 
     236 
     237 
     238[[BR]][[BR]] 
     239== Tutorial (as powerpoint presentation) == 
     240'''TODO:''' These tutorials document release 1.0.3 and so need to be updated with the Test Plan and other recent features. 
     241 
     242 * attachment:"UserGuide_part_1.ppt" 
     243 * attachment:"UserGuide_part_2.ppt" 
     244 * attachment:"UserGuide_part_3.ppt" 
     245 
     246[[BR]][[BR]] 
     247== Project site == 
     248Project site: http://sourceforge.net/projects/testman4trac/ 
     249 
     250The project is also on Pypi: http://pypi.python.org/pypi/TestManager 
     251 
     252[[BR]][[BR]] 
     253== Bugs/Feature Requests ==  
     254 
     255If you have any issues, create a  
     256[http://trac-hacks.org/newticket?component=TestManagerForTracPlugin&owner=seccanj new ticket]. 
     257 
     258Existing bugs and feature requests for TestManagerForTracPlugin are  
     259[report:9?COMPONENT=TestManagerForTracPlugin here]. 
     260 
     261[[TicketQuery(component=TestManagerForTracPlugin,format=table,col=summary|status|severity|priority|milestone|type|owner|time|description)]] 
     262 
     263[[BR]][[BR]] 
     264== Download == 
     265 
     266Download the egg files, ready to be installed, from the project site [http://sourceforge.net/projects/testman4trac/files here]. 
     267 
     268[[BR]][[BR]] 
     269== Source == 
     270 
     271You can download the source code from [https://sourceforge.net/projects/testman4trac/files here]. 
     272 
     273[[BR]][[BR]] 
     274== Example == 
     275 
     276The plugin can be installed as usual, from the Trac administration panel, and requires a database upgrade. 
     277 
     278I had to drop support for Trac 0.11, couldn't afford to keep both releases compatibility with a complete rewrite to do. 
     279At the moment it has been tested only on Trac 0.12 with Python 2.6. 
     280 
     281Follow the tutorial presentation that you can download from the project site to start.  
     282The tutorial needs to be updated to the new Test Plan feature. 
     283 
     284[[BR]][[BR]] 
     285== Recent Changes == 
     286 
     287[[ChangeLog(testmanagerfortracplugin, 3)]] 
     288 
     289[[BR]][[BR]] 
     290== Author/Contributors == 
     291 
     292'''Author:''' [wiki:seccanj] [[BR]] 
     293'''Maintainer:''' [wiki:seccanj] [[BR]] 
     294'''Contributors:'''