[[PageOutline(2-5,Contents,pullout)]] = Test Manager plugin for Trac = == Description == A Trac plugin to create '''[http://trac-hacks.org/wiki/TestManagerForTracPlugin#TestCases Test Cases]''', organize them in '''[http://trac-hacks.org/wiki/TestManagerForTracPlugin#TestCatalogs Catalogs]''', generate '''[http://trac-hacks.org/wiki/TestManagerForTracPlugin#TestPlans Test Plans]''' and track their execution status and outcome. Differently from other test management plugins for Trac that use Tickets as test case holders, this one uses Wiki pages and an additional proprietary data model to store Test Cases. This allows you to not pollute your ticket lists with something that is not a ticket, and at the same time is powered by the Trac search engine and formatting syntax for Wiki pages. A set of plugins intercept requests for Wiki pages that are test cases and decorate the page with title, breadcrumbs, tree view, type-ahead search inside the catalogs, test case status semaphore and icons and buttons that allow you to create new test cases, sub-catalogs, copy and paste test cases around different catalogs and change a test case status. Multiple Test Plans can be associated to any Test Catalog, in order to keep track of the execution of the corresponding Test Cases in a particular testing context. A generic, customizable '''[wiki:TestManagerForTracPluginWorkflow Workflow Engine]''' is provided as a separate plugin in the package, which allows you to associate custom workflows to any Resource in Trac, including Wiki pages and any custom object handled through a Resource Manager. This can be done very easily by adding as little as [wiki:TestManagerForTracPluginWorkflow a few lines of code] to the component which manages the User interface. All of the test objects, i.e. catalogs, test cases, test plans and test cases in a plan (i.e. with a status and a status change history), support: * '''[http://trac-hacks.org/wiki/TestManagerForTracPlugin#Customfields Custom properties]''', which can be declared in the trac.ini file and will be available to the User for change, stored in the database and available to change listeners. * Change history * Listener interface to be notified of object creation, modification and deletion * '''[http://trac-hacks.org/wiki/TestManagerForTracPluginWorkflow Customizable Workflow]''' state machine, declared in the trac.ini file, with the same syntax as for Ticket workflows (I may have reused some existing code here :-) * '''[http://trac-hacks.org/wiki/TestManagerForTracPluginWorkflow#Howtoprovidecustomoperations Customizable Workflow Operations]''', via a plugin api so that any component can provide its custom operations to be performed upon any workflow action, as defined in the trac.ini file. * Workflow also supports a listener API for components interested in state transitions and actions performed * Workflow states also support custom properties, so to be able to convey additional context information on a workflow state and use it in listeners or directly from the database. The developed workflow engine is able to work on any Trac Resource, it is not confined to this plugin ones. You can then define a workflow on any Trac resource, including Wiki pages, declaratively in the trac.ini file. You will then add a handful of custom code (for example in an ITemplateStreamFilter) to add the markup that the workflow engine generates for you to your desired Trac web page. See the enhancement tickets documentation or the README.txt file contained in the egg (if you don't wish to actually browse the code) for further details. The '''programmatic''' and the '''RESTful API''' are documented in details in this page TestManagerForTracPluginApi. Here follows an overview of the plugin functionalities. For a brief [wiki:TestManagerForTracPlugin#Tutorialaspowerpointpresentation tutorial], refer to the powerpoint presentations attached below. Note, anyway that the tutorial was developed for release 1.0.0 and so does not cover Test Plans, Workflows and Custom properties. [[BR]] [[BR]] == Test Catalogs == Test catalogs contain sub-catalogs or Test Cases. A Javascript tree view displays a catalog node and its sub-tree, including all of the test cases contained. Next to each catalog (or sub-catalog) a number in brackets shows the number of test cases it contains. Notice at the top of the page breadcrumbs to easily navigate up in the catalogs tree. You can '''choose between a tree-like and a tabular representation''' of the catalog and all of its contained sub-catalogs and test cases, as shown in The former is useful to actually work on the Test Catalog, adding sub-catalogs and new test cases, or modifying them. The latter, which also reports a printout of all the Test Case descriptions, may be useful for a Test Engineer to save or print as a handout to actually run the test cases in the catalog. You can add sub-catalogs or Test Cases simply by entering a name (blanks and case are supported) and click the appropriate button. A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for browsing. If you wish to edit the title or add some textual contents to the (sub-)catalog, click on "Edit Page" at the bottom of the screen. '''Be careful''' that the first line (the one surrounded by '==') will always be taken as the title of the catalog (the same stands for test cases, read below), so do not remove this line. You can edit this line to change the test catalog title, anyway. Just save the page ("Submit Changes") to save your textual changes to the (sub-)catalog. To '''delete a Test Catalog''' you must delete the corresponding Wiki page. Notice that this operation does not delete all of the Test Cases contained in the Catalog. You must either delete each Test Case individually, or first move them into a different catalog first. If you '''mistakenly deleted''' a Test Catalog, you can save the day by recreating a Wiki page with the exact name. To do so, enter the desired name directly in the browser's URL, after the /wiki/ part. Trac will show you an empty page for the catalog, but already populated with all of your existing test cases and sub-catalogs. Click on "Create this page", give it a title (surrounded by '==') and submit your changes. The new (old) catalog is in place. [[BR]] [[BR]] [[Image(screen2.JPG)]] [[BR]] '''Tree View of the Test Catalog''' [[BR]] [[BR]] [[Image(screen13.png)]] [[BR]] '''Tabular view of the Test Catalog''' [[BR]] [[BR]] == Test Cases == Test Cases are the smallest units of test execution. They are implemented again as wiki pages, with a naming convention that allows the plugin code to recognize them and treat them appropriately. To add Test Cases into a Test Catalog, open the catalog, go to the bottom of the page and enter a name for the new Test Case to be created into the appropriate text box (blanks and case are supported). Then click the button "Add a Test Case". A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for editing. '''Be careful''' that the first line (the one surrounded by '==') will always be taken as the title of the Test Case, so do not remove this line. You can edit this line to change the test case title, anyway. You can then add the Test §Case description just below the first line (i.e. title), using WikiFormatting, adding attachments and everything else is supported for Wiki pages. When you are done, save the page ("Submit Changes") to save your new Test Case. To '''delete a Test Case''' you must delete the corresponding Wiki page. Again, notice the breadcrumbs at the top, useful to go back to the enclosing catalog or any catalog up the hierarchy. [[BR]] [[BR]] [[Image(screen3.JPG)]] [[BR]] [[BR]] == Moving Test Cases from one catalog to another == It is also possible to move a Test Case into a different catalog, with an experience similar to cut&paste. You first open a test case and click on the "Move the Test Case into another catalog" button. This is similar to a "cut" operation. You then navigate and open the destination catalog and click on "Move the copied Test Case here" button (which only appears if a Test Case has been cut first). It is also possible to cancel the operation at any time by clicking the "Cancel" button in a Gmail-type of yellow message at the top of the page. [[BR]] [[BR]] [[Image(screen6.JPG)]] [[BR]] [[BR]] == Test Plans == '''Since: 1.1.0''' A Test Plan represents a plan for a particular execution of all the Test Cases in a Test Catalog (or sub-catalog). Think for example at the build verification test following a nightly build, or, for traditional projects, Technical Test and eventually Client Test. Thus a Test Plan is associated to one Test Catalog, or sub-catalog. You can have any number of Test Plans for one Test Catalog, anyway. The list of Test Plans you generated for a Test Catalog is displayd in a table at the bottom of the same catalog, as shown in the following figure. [[BR]] [[BR]] [[Image(screen9.JPG)]] [[BR]] [[BR]] To create a Test Plan for a catalog, open the desired Test Catalog (or sub-catalog), enter the name of the new Test Plan in the appropriate test box and click "Generate a new Test Plan". The new Test Plan will be opened for display, showing all of the Test Cases in the catalog, in the "Untested" status. As with Test Catalogs, you can '''choose between tree-like and a tabular representation''' of the Test Plan, as shown in the figures below. The former is useful to actually work on the Test Plan and to view test cases before actually going and executing them. The latter, which indicates for each Test Case the latest status modification and its author, may be useful to keep as the documentation of a just run test plan. [[BR]] [[BR]] [[Image(screen7.JPG)]] [[BR]] '''Tree View of the Test Plan''' [[BR]] [[BR]] [[Image(screen14.png)]] [[BR]] '''Tabular view of the Test Plan''' [[BR]] [[BR]] To track the execution status of a Test Case in a particular Test Plan, open it by clicking on the Test Case name from the Test Plan tree. Then simply click on the corresponding light in the semaphore at the bottom of the page, as shown in the following figure. You don't need to save anything, the change is immediately recorded in the database by means of an Ajax call (this API will be documented asap, to allow for setting test case execution status from external applications). The change is immediately reflected in the Test Plan. [[BR]] [[BR]] [[Image(screen8.JPG)]] [[BR]] [[BR]] When viewing a Test Case, you can open a new Ticket by means of the "Open Ticket on this Test Case" button. The new ticket will contain a link back to the corresponding Test Case and, if you were viewing it in the context of a particular Test Plan, of the Test Plan as well. This plugin also supports the TracTicketTemplatePlugin to fill a ticket template with this information. In this case, you can use the following parameters in the template to receive the information: * testCaseNumber: The wiki page for the corresponding Test Case * planId: The ID of the Test Plan * planName: The name of the Test Plan For example, to get the test case number, you template will have something like: {{{ bleah bleah Test Case: %(testCaseNumber)s bleah bleah }}} == Custom fields == '''Since: 1.2.0''' Custom fields can be added to the four test objects and to the [wiki:TestManagerForTracPluginWorkflow workflow] state object, by declaring them in the trac.ini file. The syntax is the same used for custom Ticket properties, only the name of the ini file sections are specific: you must use the test artifact type name followed by "-tm_custom". The test artifacts type names are the following: * testcatalog * testcase * testplan * testcaseinplan For example, the following sections in the trac.ini file define one custom property for each of the above artifacts. {{{ [testcatalog-tm_custom] prop1 = text prop1.value = Default value [testcaseinplan-tm_custom] prop_strange = text prop_strange.value = windows [testcase-tm_custom] nice_prop = text nice_prop.value = My friend [testplan-tm_custom] good_prop = text good_prop.value = linux }}} '''Note: Only text property types are currently supported.''' Once defined in the trac.ini file as above, custom fields will be available to the User for browse and for editing in the Web page, as shown next. '''Note: Editing custom properties requires the TEST_MODIFY permission.''' The following screenshot shows a custom "platform" field added to the Test Case artifact, and how it is presented to the User for editing. The value is initially displayed read-only, as a label. Clicking on the pencil icon turns the label into an edit box, allowing the User to edit the value and also displays a "Save" button. Clicking the button immediately saves the new value into the database. [[BR]] [[BR]] [[Image(screen11.png)]] [[BR]] [[BR]] [[BR]] == Customizable Workflow == '''Since: 1.2.0''' The following figure shows a sample [wiki:TestManagerForTracPluginWorkflow workflow] added to Test Cases with custom sample operations. No built-in operation is currently implemented but the sample one shown here, named 'sample_operation', which logs a debug message with the text input by the User. Every object which has a workflow defined is created in a "new" state, so every transition should consider this as the first state in the state machine. This is a sample content of the trac.ini file to associate a workflow to the Test Case object. [[BR]] {{{ [testcase-resource_workflow] sleep = new -> asleep sleep.permissions = TEST_MODIFY sing = new -> singing sing.permissions = TEST_MODIFY sing.operations = sample_operation calmdown = singing -> calm calmdown.permissions = TEST_MODIFY kill = asleep,calm -> dead kill.permissions = TICKET_MODIFY }}} [[BR]] Refer to the [wiki:TestManagerForTracPluginWorkflow Workflow Engine] description page for more details. [[BR]] [[BR]] [[Image(screen12.png)]] [[BR]] [[BR]] [[BR]] == Searching and filtering Test Cases in the tree view == A type-ahead, browser-side filtering feature allows for easily locating Test Cases matching a particular naming convention in the title, and/or a particular execution status. This is available both in the context of a Test Catalog and in Test Plans. Multiple words (or parts of) separated by blanks are supported, in AND condition. In the case of a Test Plan, you can also add a test case status to filter by this criterion. The supported statuses are (even substrings of): * untested * successful * failed [[BR]] [[BR]] [[Image(screen4.JPG)]] [[BR]] [[BR]] [[BR]] == Test Management and Execution Statistics == '''Since: 1.1.1''' Charting capabilities allow for tracking the evolution of the test suites and the corresponding test plans. To access the test management statistics, click on '''Test Stats''' in the Trac toolbar on the upper right corner of the page. As shown in the next figure, a chart will be displayed with statistics about all the test cases in the system and a default period of time. By means of the filtering criteria at the bottom of the chart, you can select the desired period of time, the chart resolution (in terms of time between different samples) and the Test Plan for your chart. You can bookmark the URL named "Static URL" at the bottom of the page in order to go directly with the current selected filtering criteria. '''Note:''' You will need internet connection to be able to display this charts, since it leverages Yahoo remote charting services. [[BR]] [[BR]] [[Image(screen10.png)]] [[BR]][[BR]] == Security == The following new permissions are available to manage the Test Manager security: * TEST_VIEW - Ability to view test catalogs and test cases * TEST_MODIFY - Ability to create and edit test catalogs and test cases * TEST_EXECUTE - Ability to change the status of a test case in a test plan * TEST_DELETE - Ability to delete test cases * TEST_PLAN_ADMIN - Ability to generate and delete test plans * TEST_STATS_VIEW - Ability to view test management statistics [[BR]][[BR]] == Programmatic API == The '''programmatic RESTful API''' is documented in details in this page TestManagerForTracPluginApi. [[BR]][[BR]] == Tutorial (as powerpoint presentation) == '''TODO:''' These tutorials document release 1.0.3 and so need to be updated with the Test Plan and other recent features. * attachment:"UserGuide_part_1.ppt" * attachment:"UserGuide_part_2.ppt" * attachment:"UserGuide_part_3.ppt" [[BR]][[BR]] == Project site == Project site: http://sourceforge.net/projects/testman4trac/ The project is also on Pypi: http://pypi.python.org/pypi/TestManager [[BR]][[BR]] == Bugs/Feature Requests == If you have any issues, create a [http://trac-hacks.org/newticket?component=TestManagerForTracPlugin&owner=seccanj new ticket]. Existing bugs and feature requests for TestManagerForTracPlugin are [report:9?COMPONENT=TestManagerForTracPlugin here]. [[TicketQuery(component=TestManagerForTracPlugin,format=table,col=summary|status|severity|priority|milestone|type|owner|time|description)]] [[BR]][[BR]] == Download == Download the egg files, ready to be installed, from the project site [http://sourceforge.net/projects/testman4trac/files here]. [[BR]][[BR]] == Source == You can download the source code from [https://sourceforge.net/projects/testman4trac/files here]. [[BR]][[BR]] == Installation == Both '''Trac 0.11 and 0.12 are supported''', and also both '''Python 2.5 and 2.6'''. The plugin can be installed as usual, from the Trac administration panel, and requires a database upgrade. You can follow the instructions in the INSTALL.txt file contained in the binary packages. The functionalities are divided in three plugins, which must be installed in this order: 1) Trac Generic Class 2) Trac Generic Workflow 3) Test Manager An additional plugin is only useful for debugging and should not be installed in a production environment. * SQL Executor With the latter plugin, you can perform arbitrary SQL queries on the Trac database from a web browser. Just type on the URL: !http://yourserver/yourenvironment/sqlexec?sql=any SQL statement For example: !http://localhost/sampleproject/sqlexec?sql=select * from testcatalog Follow the tutorial presentation that you can download from the project site to start. The tutorial is actually very old and needs to be updated to the new Test Plans and other features, but it will get you started. [[BR]][[BR]] == Recent Changes == Note: you can '''subscribe to the [https://sourceforge.net/projects/testman4trac/forums/forum/1200598 Forum]''' to be updated on new releases. [[BR]] [[BR]] '''During the last month:''' [[BR]] '''Release 1.3.9''' (2010-11-23): o Fixed Ticket #8144 (Track-Hacks): Test statistical charts don't show successful and failed figures '''Release 1.3.8''' (2010-11-23): o Fixed Ticket #8121 (Track-Hacks): Catalog Wiki Page not added[[BR]] o Fixed Ticket #8123 (Track-Hacks): Can't move testcase more than one time into different catalog[[BR]] o Fixed Ticket #8124 (Track-Hacks): AttributeError: 'NoneType' object has no attribute 'splitlines'[[BR]] o Fixed Ticket #8125 (Track-Hacks): "duplicate test case" does not work for previously moved test case[[BR]] '''Release 1.3.7''' (2010-11-20): o Enhancement #7704 (Track-Hacks): Add ability to delete a Test Plan[[BR]] o Fixed Ticket #8084 (Track-Hacks): Ordering issue[[BR]] '''Release 1.3.6''' (2010-11-09): o Fixed Ticket #8004 (Track-Hacks): Cannot search if an admin[[BR]] '''Release 1.3.5''' (2010-10-17): o Restored compatibility with Trac 0.11. Now again both 0.11 and 0.12 are supported.[[BR]] [[BR]][[BR]] == Author/Contributors == '''Author:''' [wiki:seccanj] [[BR]] '''Maintainer:''' [wiki:seccanj] [[BR]] '''Contributors:'''