[ Home | Latest changes | Image gallery | ​Video tutorial on Youtube | Quick setup guide | ​Download | ​Source | Bugs/Feature requests ]

A full test management life-cycle plugin for Trac

Summary

This plugin implements a full test management life-cycle: definition, planning, execution, statistics.

This plugin can be used to create Test Cases, organize them in Catalogs and generate Test Plans to track their execution status and outcome.

This plugin uses Wiki pages and an additional data model to store Test Cases, rather than tickets. This allows you to separate your tickets from your test findings, while still being powered by the Trac search engine and formatting syntax for Wiki pages.

A set of plugins intercept requests for Wiki pages that match the Test Manager URL namespace and decorate them with additional information stored in the database.

Key features:

• Defining test cases and organize in test catalogs (test suites).
• Defining a hierarchy of catalogs and sub-catalogs, whatever deep. Specifying the exact order of the test cases in any particular test catalog.
• Test case and catalogs versioning and history of changes.
• Test cases and catalogs support Wiki formatting and attachments.
• Copy/move individual or a set of test cases among catalogs, or organize the catalogs contents easily using drag and drop.
• Defining one or more test plans (test rounds) for each test (sub-)catalog, based on all or a portion of the test cases in a (sub-)catalog. Specifying whether the textual contents of the test cases in a plan should be frozen at the current version, or will always be linked to the latest version.
• One-click changing status of a test case, ie marking a test case in a particular test plan as successful or failed.
• Ticket integration: opening tickets directly from a (usually failed) test case, keeping track of the relationship. Navigating from a test case to its related tickets and vice-versa.
• Test cases, catalogs and plans can have their own custom properties, defined in trac.ini like tickets. All property types are supported: text, text area, listbox, radio buttons, checkbox.
• Defining custom test outcomes (statuses), in addition to the built-in "Untested", "Success", "Failed".
• Test execution statistical charts, showing the overall testing activity, status of the test plans, statistics on the tickets opened against all or a specific test plan, using ​Flot (MIT licensed). Exporting the data to CSV.
• Choosing between Tree and Tabular view of the test catalogs or test plans, and the contained test cases.
• In the tabular view, choosing which columns to display, including any custom property, and the ability to add calculated rows, such as sums, averages, min or max.
• Type-ahead searching and filtering from the tree and tabular view.
• Breadcrumbs to easily navigate through test catalogs and plans.
• Integration with Trac search engine.
• Fine-grained security with six new Trac permissions.
• Administration panels to easily configure the test manager's behavior.
• Creating templates for test cases and test catalogs. Specifying which default template should be used for test cases in any particular catalog.
• Importing/exporting test cases and test plans from/to Excel in CSV format.
• National Language Support (NLS) and currently translated into Italian, French, German, Spanish, Russian and Korean.
• Remote APIs: XML-RPC, RESTful.
• Internal Python API.

A Programmatic API is available to interact with the test management environment with these interfaces:

• XML-RPC
• HTTP RESTful
• Python classes

The APIs are documented in detail in TestManagerForTracPluginApi.

Trac plugin developers may be interested in the plugin reusable modular architecture:

• A generic, customizable Workflow Engine is provided as a separate plugin in the package, which allows you to associate custom workflows to any Resource in Trac, including Wiki pages and any custom object handled through a Resource Manager. This can be done easily by adding a few lines of code to the component which manages the User interface.

• The data manipulation layer has been extracted into a Generic Persistent Class framework plugin, which can be easily used by any other plugin to provide persistence, security, change history, custom properties, pattern matching search, and other features to their basic data model with a handful lines of code.

• The test objects, ie catalogs, test cases, test plans and test cases in a plan (i.e. with a status and a status change history), support:
  • Custom properties, which can be declared in the trac.ini file and will be available to the User for change, stored in the database and available to change listeners.
  • Property and status change history.
  • Listener interface to be notified of object creation, modification and deletion.
  • Customizable Workflow state machine, declared in the trac.ini file, with the same syntax as for Ticket workflows.
  • Customizable Workflow Operations, via a plugin API so that any component can provide its custom operations to be performed upon any workflow action, as defined in the trac.ini file.
  • Workflow also supports a listener API for components interested in state transitions and actions performed.
  • Workflow states also support custom properties, so to be able to convey additional context information on a workflow state and use it in listeners or directly from the database.

• The developed workflow engine is able to work on any Trac resource, it is not confined to this plugin. You can then define a workflow on any Trac resource, including Wiki pages, declaratively in the trac.ini file. You then add a handful of custom code (for example in an ITemplateStreamFilter) to add the markup that the workflow engine generates for you to your desired Trac web page. See this page for further details.

For an overview of this plugin's capabilities, see this ​video.

Taxonomy of the Test Manager plugin

Test Catalogs

Test Catalogs contain Test Cases and may organize them in sub-catalogs.

They are implemented as wiki pages, with a naming convention that allows the plugin code to recognize them and treat them appropriately, and with a set of auxiliary database tables. This provides out-of the box support for rich text formatting (using Wiki syntax), attachments and versioning.

This also means that the WIKI_VIEW, in addition to the TEST_VIEW, permission is required to be able to browse test catalogs and test cases.

You can choose between a tree-like and a tabular representation of the catalog and all of its sub-catalogs and test cases, as shown in the following figures.

To switch between the two view types, click on the corresponding icons at the top-left corner. To change the default view type, ie tree or table, use the Test Manager Settings administration panel.

On a Test Catalog page, from the top down, you have:

• Some breadcrumbs, to easily locate where the current catalog is and to navigate up in the catalogs tree.
• Switch to Tabular View / Switch to Tree View icons. Use these icons to switch between the tree view and the tabular view. The former is useful to actually work on the Test Catalog, adding sub-catalogs and new test cases, or modifying them. The latter, which also reports a printout of all the Test Case descriptions, may be useful for a Test Engineer to save or print as a handout to actually run the test cases in the catalog. This is particularly true for Test Plans. The tabular view is also the only one to show all of the Custom Fields you may have added to the catalogs, plans or test cases.
• The catalog's Title. This is the text you have entered in the parent catalog's New Sub-Catalog text field when you created this catalog.
• Filter text box. Type inside the text box to filter all sub-catalogs and test cases matching your input text. See Searching and filtering Test Cases for details.
• Expand All and Collapse All. In tree view mode, use these links to expand or collapse all the tree nodes.
• Sub-catalogs and Test Cases. You can browse each sub-catalog and test case by following the corresponding link. Hovering on a Test Case, a pencil icon will appear on the right. Click on it to go to the Test Case directly into Edit mode.
• Custom fields. If you have defined any custom fields for the Test Catalog artefact, they appear here. You can then view and edit their values directly from this page. See Custom Fields for details.
• New Sub-Catalog text field and button. You can add sub-catalogs simply by entering a name (blanks, special characters and case are supported) and clicking the corresponding button. A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for browsing.
• New Test Case text field and button. You can add a new Test Case inside the currently displayed catalog by entering a name and clicking the corresponding button. As for a new catalog, a new wiki page is generated, but it is now opened for editing. Be careful that the first line (the one surrounded by '==') will always be taken as the title of the test case (the same stands for test catalogs), so do not remove this line. You can edit this line to change the test case title, anyway.
• Generate a new Test Plan button. Allows you to generate a new test execution plan for the Test Cases contained in this catalog. See Test Plans for details.
• Select Multiple Test Cases button. It will display checkboxes next to the test cases in the catalog (and su-catalogs). You may then select one or more test cases to perform operations on them. See Moving or Copying Test Cases from one catalog to another for details.
• Copy the Selected Test Cases button. You can copy (into a "virtual" clipboard) the test cases you have selected (see previous item), to be eventually pasted into some other (or the same) catalog, or sub-catalog.
• Paste the copied Test Cases here button. This button, only visible if you have previously copied some test cases into the "virtual" clipboard, allows you to paste them into the current Test Catalog.
• Move the copied Test Case here button. This button, only visible if you have previously cut a single test case into the "virtual" clipboard, allows you to paste them into the current Test Catalog.
• Import Test Cases button. Allows you to import a set of test cases into this catalog from a CSV file. See Importing Test Cases from a CSV file into a Test Catalog for details.
• Export Test Catalog button. Allows you to export the Test Catalog, including sub-catalogs and all the contained test cases, to CSV format. See Exporting Test Catalogs or Test Plans into CSV file for details.
• Organize Test Catalogs button. It will display a dialog box where you can, using simple drag and drop, reorganize the test cases in this catalog and sub-catalogs. See Organizing the contents of a Test Catalog for details.
• Available Test Plans section. The number in brackets shows the number of Test Plans that have been generated for this catalog. Expand the section to access detailed test plan information and to navigate to the individual test plans. Type inside the text box to filter all test plan information matching your input text.
• Object Change History section. In addition to the standard Wiki versioning, test objects support their own change history, which lets you inspect who changed which property and when, also showing its old and new value. This is also particularly useful for Custom Fields you may add to the test objects. See Test Artefact Change History for more details.
• Edit this page button. If you wish to edit the title of the catalog or add some textual contents to it, click on this button to enter the Wiki page editing mode. Be careful that the first line is always interpreted as the test catalog title. You can then add the Test Catalog description just below it, using WikiFormatting, adding attachments and everything else is supported for Wiki pages. When you're done, simply save the page with Submit Changes to save your textual changes to the catalog.
• Attach File button. Attach files to the Catalog page.
• Rename page button. This function is not allowed for Test Catalogs, Test Cases or Test Plans. This because the Test Manager plugin uses the page names to keep track of the test artefacts hierarchy information. Submitting a new page name will then throw you a meaningful error. To change the Catalog's title, use the Edit this page button and change the first text line. It is interpreted as the catalog title. If you wish to move the object elsewhere, instead, use the 'move' Test Manager functions.
• Delete this version button. It will delete the current Test Catalog's description Wiki page, and revert to the last one. This action will have no effect on the changes you may have done to the sub-catalogs or test cases contained therein. To undo changes to the Wiki pages of any of these latter objects, go to the corresponding page and work with the Wiki versions accordingly. As long as the Test Catalog internal organization in sub-catalogs and Test Case, Test Manager does not have a "Test Catalog snapshot" kind of functionality, which you may revert to.
• Delete Test Catalog button. Pushing this button will ask you for confirmation and, if confirmed, will delete the current Test Catalog, all the contained sub-catalogs, Test Cases, Test Plans and the status history of them. Caution: there is no undo functionality.

Tree view of the Test Catalog	Tabular view of the Test Catalog

Searching and filtering Test Cases

A type-ahead, browser-side filtering feature allows for easily locating Test Cases matching a particular naming convention in the title, and/or a particular execution status.

This is available both in the context of a Test Catalog and in Test Plans.

Multiple words (or parts of) separated by blanks are supported, in AND condition.

In the case of a Test Plan, you can also filter by test case status. The default statuses are (even substrings of) the following, but you can use Custom Test Case Outcomes (ie states, verdicts) as well:

• untested
• successful
• failed

Filtering the Test Cases in a catalog tabular view

Test Cases

Test Cases are the smallest units of test execution.

They are implemented again as wiki pages, with a naming convention that allows the plugin code to recognize them and treat them appropriately, and with a set of auxiliary database tables. This provides out-of the box support for rich text formatting (using Wiki syntax), attachments and versioning.

To add Test Cases into a Test Catalog, open the catalog, go to the bottom of the page and enter a name for the new Test Case to be created into the appropriate text box (blanks, special characters and case are supported). Then click the button Add a Test Case. A new wiki page is generated, with a naming convention allowing the plugin code to position it correctly in the catalogs tree, and opened for editing. Be careful that the first line (the one starting with '== ') will always be taken as the title of the Test Case, so do not remove this line. You can edit this line to change the test case title, anyway, at any time.

You can then add the Test Case description just below the first line, ie the title, using WikiFormatting, adding attachments and everything else is supported for Wiki pages.

When you are done, save the page Submit Changes to save your new Test Case.

A Test Case is created in the context of a Test Catalog. Think of it as the Test Case "class", or "definition". It does not carry information about whether the Test Case was executed or when, and whether it succeeded or failed.

To plan a test execution session, generate a Test Plan. A Test Plan is used to record a specific execution of a set of Test Cases. Thus, a Test Case also exists in the context of one or more Test Plans. Think of those as Test Case "instances", or "uses". In the context of the Test Manager plugin, these are called Test Cases in a Plan.

A Test Case in a Plan, in addition to a basic Test Case, also records the execution outcomes (or verdicts), ie whether a particular execution of the Test Case in the context of the containing Test Plan was successful or failing. You can also see the complete test status change history at the bottom of the Test Case in Plan page.

On a Test Case page, from the top down, you have:

• Some breadcrumbs, to easily locate where the current test case is and to navigate up in the catalogs tree, or to the containing Test Plan.
• The Test Case's Title. This is the text you have entered in the parent catalog's New Test Case text field when you created this Test Case.
• Test Case description text. This is the main test case contents, probably describing the test preconditions, the steps to be performed to run the test case and the expected results. You can edit these contents using the Edit this page button. See below.
• Custom fields. If you have defined any custom fields for the "Test Case" or "Test Case in a Plan" artefacts, they appear here. You can then view and edit their values directly from this page. See Custom Fields for details.
• Open a Ticket on this Test Case button. Allows you to open a ticket that will be related to this test case. If you are browsing a Test Case definition, the ticket will not be related to any particular Test plan. If, instead, you are browsing a Test Case in (the context of) a Plan, the ticket will be related to the execution of th Test Case in that particular Test Plan. See Linking a Ticket to a Test Case for details.
• Show Related Tickets button. Clicking this button will perform a custom query listing all the Tickets related to this Test Case and, if applicable, the containing Test plan.
• Move the Test Case into another catalog button. Only available on Test Case definitions. It allows you to cut (into a "virtual" clipboard) the test case, to be eventually pasted into some other (or the same) catalog, or sub-catalog. Push the button, then navigate to the destination catalog and push the Move the copied Test Case here button to complete the move operation. Also look at the Organize Test Catalogs functionality for an alternative way of moving Test Cases. Refer to Test Catalogs for details.
• Duplicate the Test Case button. Only available on Test Case definitions. Lets you make an exact copy of this Test Case definition. Note that the test execution statuses, ie successful, failed, etc, associated to the original Test Cases in any Test Plan are not duplicated along with the Test Case definition.
• Add to a Test Plan button. Only available on Test Case definitions. Allows you to add the Test Case to one or more suitable Test Plans, i.e. Test Plans that have been defined as not containing all the test cases (current and future) of the associated Test Catalog, but only a selected set of them.
• Object Change History section. In addition to the standard Wiki versioning, test objects support their own change history, which lets you inspect who changes which property and when, also showing its old and new value. This is also particularly useful for Custom Fields you may add to the test objects. See Test Artefact Change History for more details.
• Edit this page button. If you wish to edit the title of the Test Case or its contents, click on this button to enter the Wiki page editing mode. When you're done, simply save the page with Submit Changes to save your changes to the test case. Be careful that the first line is always interpreted as the test case title.
• Attach File button. Attach files to the Test Case page.
• Rename page button. This function is not allowed for Test Catalogs, Test Cases or Test Plans. This because the Test Manager plugin uses the page names to keep track of the test artefacts hierarchy information. Submitting a new page name will then throw you a meaningful error. To change the Test Case's title, use the Edit this page button and change the first text line. It is interpreted as the test case title. If you wish to move the object elsewhere, instead, use the 'move' Test Manager functions.
• Delete this version button. It will delete the current Test Case's description Wiki page, and revert to the last one.
• Delete page button. Pushing this button will ask you for confirmation and, if confirmed, will delete the current Test Case definition and all of the Test Case instances in any Test Plan, including their status change history. Caution: there is no undo functionality.

Sample Test Case	Late adding a Test Case to one or more Test Plans

Test Plans

Since: 1.1.0

A Test Plan represents a plan for a particular execution of all the Test Cases in a Test Catalog or sub-catalog.

Think for example at the build verification test following a nightly build, or, for traditional projects, Functional Test and eventually Customer Verification Test.

Thus a Test Plan is associated to one Test Catalog, or sub-catalog. You can have any number of Test Plans for one Test Catalog, anyway. The list of Test Plans you generated for a Test Catalog is displayed in a table at the bottom of the same catalog, as shown in the following figure.

The Test Cases associated to a Test Plan are called Test Cases in a Plan. Refer to Test Cases for details about the additional information carried by these particular artefacts.

To create a Test Plan for a catalog, open the desired Test Catalog (or sub-catalog), enter the name of the new Test Plan in the appropriate text field and click Generate a new Test Plan. A dialog box will be displayed, as show in the next figure, which lets you choose the characteristics of the Test Plan.

Available options are:

• Whether the new Test Plan will contain all the Test Cases in the catalog, or only the ones previously selected (see Test Catalogs for an explanation about how to select multiple test cases in a catalog.
• Whether the textual contents of the test cases in the new plan should be frozen at their current version, or will always be linked to the latest version. In fact, some time after the test plan has been executed it may be desired, for auditing purposes, to be able to view the exact contents of the test case description at the time the test plan was generated, disregarding all the changes that may have occurred later.

The new Test Plan will be opened for display, showing all of the Test Cases in the catalog, in the "Untested" status.

As with Test Catalogs, you can choose between tree and a tabular representation of the Test Plan, as shown in the figures below.

• The former is useful to work on the Test Plan and to view test cases before executing them. It also gives you a bird-eye view of the test execution progress and results. Additionally, you can set a test case verdict directly form the tree view, left-clicking on any Test Case's traffic light and changing its color using the pop-up menu that appears.
• The latter, which indicates for each Test Case the latest status modification and its author, may be useful to keep as the documentation of a just run test plan. It is also the only way to display the Test Case in a Plan Custom Fields you may have set. Additionally, you can set a test case verdict directly from the table view, left-clicking on any Test Case's traffic light and changing its color using the pop-up menu that appears.

On a Test Plan page, from the top down, you have:

• Back to the Catalog link. It allows you to navigate back to the Test Catalog associated to this plan.
• The Test Plan's Title. This is the text you have entered in the parent catalog's Generate a new Test Plan text field when you created this Test Plan.
• Test Plan properties. The author, creation time and basic characteristics of the Test Plan are displayed here.
• Test Catalog title. The associated Test Catalog title is reported here.
• Filter text box. Type inside the text box to filter all sub-catalogs and test cases matching your input text. In the context of a Test Plan you can also filter by the test execution status. See Searching and filtering Test Cases for details.
• Expand All and Collapse All. In tree view mode, use these links to expand or collapse all the tree nodes.
• Sub-catalogs and Test Cases.
  • A traffic light icon on a sub-catalog indicates the severity of the outcome of the test cases. Red is considered worse than yellow, which is considered worse than green.
  • You can browse each sub-catalog and Test Case instance by following the corresponding link.
  • Hovering on a Test Case, a pencil icon will appear on the right. Click on it to go to the Test Case definition directly into Edit mode. Refer to Test Cases for an explanation of the difference between Test Case definition and instance.
  • You can set a test case verdict directly from the tree or tabular view, left-clicking on any Test Case's traffic light and changing its color using the pop-up menu that comes out. If the basic three outcomes do not fit your needs, see Custom Test Case Outcomes (i.e. states, verdicts) for an explanation about how to augment them.
• Custom fields. If you have defined any custom fields for the "Test Case" or "Test Case in a Plan" artefacts, they appear here. You can then view and edit their values directly from this page. See Custom Fields for details.
• Export Test Plan button. Allows you to export the Test Plan to CSV format. See Custom Test Case Outcomes (i.e. states, verdicts) for details.
• Clone Test Plan button. This button will open a dialog where you can specify a name for the cloned Test Plan. The cloned Test Plan will inherit all the characteristics of the original one, but the contained Test Cases will all be reset to Untested, and their test status change history emptied.
• Object Change History section. In addition to the standard Wiki versioning, test objects support their own change history, which lets you inspect who changes which property and when, also showing its old and new value. This is also particularly useful for Custom Fields you may add to the test objects. See Test Artefact Change History for more details.
• Edit this page button. This button allows you to edit the Wiki page that contains the associated Test Catalog description. A Test Plan does not have itself a specific Wiki page. If you need to record specific Test Plan information, you may want to use a Custom Field, possibly of type textarea. Refer to Test Catalogs for details about editing their descriptions.
• Attach File button. Attach files to the associated Test Catalog wiki page. There is currently no way to attach files to a specific Test Plan.
• Rename page button. This function is not allowed for Test Catalogs, Test Cases or Test Plans. This because the Test Manager plugin uses the page names to keep track of the test artefacts hierarchy information. Submitting a new page name will then throw you a meaningful error.
• Delete this version button. It will delete the associated Test Catalog's description Wiki page, and revert to the last one.
• Delete page button. Pushing this button will ask you for confirmation and, if confirmed, will delete the associated Test Catalog. Refer to Test Catalogs for details about what will be actually deleted. Caution: there is no undo functionality.

You have two ways of setting the outcome of a Test Case in a particular Test Plan (see figures below):

1. Directly from the test plan tree or tabular view, by clicking on the traffic-light icon and choosing the new color from the menu that opens. Since 1.4.7.
2. By opening the test case first, by clicking on its name in the Test Plan tree or table, and then clicking on the corresponding traffic-light at the bottom of the page.

You don't need to save anything, the changes are immediately recorded into the database and reflected in the Test Plan.

The New Test Plan dialog	Tree view of a Test Plan	Tabular view of a Test Plan

Moving or Copying Test Cases from one catalog to another

It is also possible to move one or more Test Cases into a different catalog, with an experience similar to cut&paste.

As an alternative, it is also possible to graphically organize the contents of a Test Catalog using drag&drop.

Moving a single Test Case

To move a single Test Case from one catalog into another:

1. Open the Test Case.
2. Click on the "Move the Test Case into another catalog" button. This is similar to a "cut" operation, but into a "virtual" clipboard.
3. Navigate and open the destination catalog.
4. Click on "Move the copied Test Case here" button, which only appears if a Test Case has been cut first.

It is also possible to cancel the operation at any time by clicking the "Cancel" button in the yellow message at the top of the page.

Copying multiple Test Cases

To copy one or more Test Cases, even from different catalogs, and paste them all together into a destination (sub-)catalog:

1. Open the Catalog. Expand the tree nodes as you like.
2. Click on the "Select Multiple Test Cases" button. A checkbox appears next to every Test Case in the catalog and the sub-catalogs. See the next figure.
3. Select the Test Cases you wish to copy into another catalog.
4. Click on the "Copy the Selected Test Cases" button. This is similar to the usual "Copy" operation, but into a "virtual" clipboard.
5. Navigate and open the destination catalog, or create a new one.
6. Click on "Paste the copied Test Cases here" button, which only appears if Test Cases have been copied first.

Note that the test execution statuses, ie successful, failed, etc, associated with the original Test Cases in any Test Plan are not copied along with the Test Case definitions.

Organizing the contents of a Test Catalog

To (re-)organize the contents of a Test Catalog, i.e. moving test cases around and changing their intended execution order, you can click the Organize Test Catalogs button. The dialog shown in the following figure will pop-up.

Use drag and drop to move test cases into different catalogs or changing their order. As long as the Sort a test plan's test cases option in the Test Manager's Settings panel is set to User Specified, the test cases will be displayed in the specified order when browsing the catalog and any associated test plans.

Note: Currently, you can only move Test Cases but not Test Catalogs. In addition to that, copying is not supported yet.

Click Save to submit your changes.

Organize Test Catalogs dialog

Linking a Ticket to a Test Case

When viewing a Test Case definition or Test Case in a Plan, you can open a new Ticket by means of the Open Ticket on this Test Case button.

The new ticket will contain a link back to the corresponding Test Case and, if you were viewing it in the context of a particular Test Plan, of the Test Plan as well.

The association is particularly useful when browsing the Test Management and Execution Statistics.

You can specify a set of properties to control the contents of the new ticket's summary. In the [testmanager] section of the trac.ini file:

ticket_summary_option: How to generate the ticket summary. Available options are:

full_path:

the default behavior. The summary will contain a complete path in the catalogs tree, with the names of the catalogs from the root to the failing test case.

last_n_catalogs:

only the last specified number of catalogs in the path will be used to form the summary.

The number of catalogs to use must be specified using the 'ticket_summary_num_catalogs' property.

empty:

the generated summary will be empty.

fixed_text:

the specified text will be used as the ticket summary. The text to be used must be specified using the 'ticket_summary_text' property.

ticket_summary_separator: The string to be used to separate catalog names

in the generated summary.

Two sample configurations follow:

[testmanager]
ticket_summary_option = fixed_text
ticket_summary_text = Failed test case: 

[testmanager]
ticket_summary_option = last_n_catalogs
ticket_summary_num_catalogs = 2
ticket_summary_separator = ->

This plugin also supports the TracTicketTemplatePlugin to fill a ticket template with this information. In this case, you can use the following parameters in the target ticket template to receive the information:

• testcaseid: The wiki page for the corresponding Test Case
• planid: The ID of the Test Plan
• planname: The name of the Test Plan

For example, to get the test case number, your template will have something like:

  some template text
  Test Case: %(testcaseid)s
  some template text

Showing Tickets related to a Test Case

While browsing a Test Case, either inside or outside of a Test Plan, you can list all of its related Tickets, ie the Tickets that were opened using the above Open a Ticket on this Test Case button.

To do this, just browse the Test Case and click on the Show Related Tickets button.

If you are browsing a Test Case "definition", ie from a Test Catalog, you will be shown all of the Tickets opened against the Test Case in any Test Plan.

If you are instead browsing a Test Case in a specific Test Plan, you will be shown all of the Tickets opened against the Test Case in that specific Test Plan.

See the following figure for what happens when you click on the "Show Related Tickets" button from the Test Case shown at the previous section.

Trac plugin developers may be interested in the fact that a programmatic Python API is provided to retrieve all of the Tickets opened against a Test Case, in all or in a specific Test Plan.

Displaying the tickets related to a Test Case in a Plan

Test Management and Execution Statistics

Since: 1.1.1

Charting capabilities allow for tracking the evolution of the test suites, the corresponding test plans and tickets opened against test cases.

To access the test management statistics, click on Test Stats in the Trac toolbar on the upper right corner of the page.

As shown in the next figure, a chart will be displayed with statistics about all the test cases in the system and a default period of time, with more charts available in the other tabs.

The data can be exported to Excel (CSV format) by means of the links at the bottom of each chart.

If you select a specific period of time and a specific test plan, you can save the settings by bookmarking the URL displayed at the bottom of each chart.

There are three charts available:

Test activity trend chart

Displays the number of test cases created during time and the trend of the test activity, in terms of test cases to be tested, successful and failed.

By means of the filtering criteria at the bottom of the chart, you can select the desired period of time, the chart resolution (in terms of time between different samples) and the Test Plan for your chart.

You can bookmark the URL named "Static URL" at the bottom of the page in order to go directly with the current selected filtering criteria.

Test activity trend chart

Current test status chart

Shows the current status of the test activity for all test plans of a specific plan, displaying the number of tests that are successful, failed and yet to be tested.

Current test status chart

Tickets trend chart

As long as tickets have been opened from the context of a test case in a plan, by means of the appropriate Open a ticket on this test case button, this chart shows the trend in tickets associated with all test plans or a specific plan, displaying the number of total, active, ie not closed, and closed tickets.

Tickets trend chart

Configuration

You can customize the default period of time the charts will display, in terms of historical depth and scale.

To change the settings, go to the Test Manager Settings administration panel, or directly edit the trac.ini file.

Importing and Exporting

Importing Test Cases from a CSV file into a Test Catalog

If you already have Test Cases defined elsewhere you can import them into the Test Manager. While browsing a Test Catalog, click on "Import Test Cases" at the bottom of the page. The dialog box shown in the following figure will pop-up. Browse for the CSV (comma-separated values) file containing your Test Cases, specify the value separator (default is the comma character) and click "Import".

The CSV file must have the following format:

• The first row will have column names. Use lowercase identifiers, with no blanks, for them.

• The data must start from the second row. There must be at least the following required columns:
  • title
  • description

Any subsequent columns are optional, and will generate custom test case fields. To fully appreciate custom fields, switch to the Test Catalog Tabular view.

A sample CSV file may be the following:

title,description,platform,greetings
First Test Case,This is the long description for the first test case,linux,bye bye
Second Test Case,This is the long description for the second test case,windows,ciao
Third Test Case,This is the long description for the third test case,macosx,hasta la vista

Import Test Cases dialog

Exporting Test Catalogs or Test Plans into CSV file

Since: 1.4.9

You can export to CSV format a Test Catalog or a Test Plan.

In the Test Catalog and Test Plan pages you will find an "Export..." button.

The export format enables a drill down into your external statistics, but also to eventually import them back into another Test Manager plugin environment.

Inside the export dialog shown in the next figure, you can specify whether the output should contain the complete text of the Test Cases, and whether this should be pre-formatted (WYSIWYG) or in raw WikiFormatting syntax.

In addition to that, you can specify which column separator to use in the exported CSV file.

The Export Test Catalog dialog

Test Artefact Change History

Since: 1.4.8

On each test artefact page you will find a section named "Object change history", which contains a list of all the changes to any standard and, more interestingly, custom object's property.

In the following figure you can see how the "device" custom property has been changed in a Test Plan history:

Object change history for a Test Plan

Customization and Personalization

The plugin can be customized with the following:

• Adding custom fields to the test artefacts, ie Test Catalogs, Test Case definitions, Test Plans, Test Case instances. It may be useful, for example, to add a "Test Effort" field to Test Cases, to be able to better plan your testing time and resources.
• Adding calculated rows to the tabular view of Test Catalogs and Test Plans. Available operations are Sum, Average and Count of the values in any column, including custom fields. For example, if you added a "Test Effort" custom field to your Test Cases, you may want the total Test Effort calculated for each catalog and displayed at the bottom of it.
• Defining Custom Test Case Outcomes (i.e. states, verdicts) to fit your quality process.
• Adding a Workflow to any of your test artefacts. For example, you may want to have an approval workflow for new test case definitions, or be able to assign test plans or individual cases to specific test engineers to be run, and manage their completion status.
• Adding Test Artefact Templates for Test Catalogs and Test Cases, and specifying which default template should be used for new test cases in any particular catalog.

Additionally, you can personalize several aspects of the Test Manager behavior, by means of the Test Manager Settings administration panel.

Custom fields

Since: 1.2.0

Custom fields can be added to the four test objects and to the workflow state object, by declaring them in the trac.ini file.

The syntax is the same used for custom Ticket properties, only the name of the trac.ini file sections are specific: you must use the test artefact type name followed by "-tm_custom".

The test artefacts type names are the following:

• testcatalog
• testcase
• testplan
• testcaseinplan

For example, the following sections in the trac.ini file define custom properties for each of the above artefacts.

[testcatalog-tm_custom]
remarks = text
remarks.label = Remarks

[testcase-tm_custom]
component = select
component.label = Component
component.options = Framework|Backend|GUI
component.value = GUI

testeffort = text
testeffort.label = Test Effort
testeffort.value = 0

required = checkbox
required.label = Required
required.value = 0

[testplan-tm_custom]
longdescription = textarea
longdescription.label = Test Plan Description
longdescription.rows = 10
longdescription.cols = 50

[testcaseinplan-tm_custom]
operating_system = radio
operating_system.label = Operating System
operating_system.options = Windows|Linux|Macosx|Android|iOS|Other
operating_system.value = Linux

Once defined in the trac.ini file as above, custom fields will be available to the User for browse and for editing in the Web page, as shown next.

Note: Editing custom properties requires the TEST_MODIFY permission.

The following screenshot shows a custom "platform" field added to the Test Case artefact, and how it is presented to the User for editing.

The value is initially displayed read-only, as a label. Clicking on the pencil icon turns the label into an text field, allowing the User to edit the value, and also displays a Save button. Clicking the button saves the new value into the database.

Editing a custom "Operating System" property on a Test case in a Plan

Custom Test Case Outcomes (i.e. states, verdicts)

Since: 1.4.1

You can customize the Test Case outcomes (i.e. states, verdicts) to match your internal quality process.

The default outcomes are configured in trac.ini at plugin installation (or upgrade) time as follows:

[test-outcomes]
green.successful = Successful
red.failed = Failed
yellow.to_be_tested = Untested
default = to_be_tested

You can customize the outcomes at any moment by adding new outcomes, or modify the descriptions of the current ones. Do not delete previous outcomes if you have already assigned them to any of your test cases.

For example, to add a new outcome named "It's a Mess!!!" to the failures, add a line like the following in trac.ini:

red.bigmess = It's a Mess!!!

You will now be able to choose it when assigning a test case status using the traffic light, which now shows a more complete menu to choose from. See the following figure.

Setting a custom outcome to a Test Case in a Plan

Customizable Workflow

Since: 1.2.0

The following figure shows a sample workflow added to Test Cases with custom sample operations. No built-in operation is currently implemented except for the sample one shown here, named 'sample_operation', which logs a debug message with the text input by the User.

Every object which has a workflow defined is created in a "new" state, so every transition should consider this as the first state in the state machine.

This is a sample content of the trac.ini file to associate a workflow to the Test Case object:

[testcase-resource_workflow]
review = new,changes_requested -> under_review
review.permissions = TEST_MODIFY

reject = under_review -> rejected
reject.permissions = TEST_MODIFY
reject.operations = sample_operation

request_changes = under_review -> changes_requested
request_changes.permissions = TEST_MODIFY
request_changes.operations = sample_operation

accept = under_review -> accepted
accept.permissions = TEST_MODIFY

Refer to the Workflow Engine description page for more details.

A custom workflow applied to Test Cases in a Plan

Test Artefact Templates

Your quality process may require your test catalogs and test cases to follow particular templates. With the Test Manager, you can define templates for your test catalogs and your test cases. When you create a new test catalog or test case, their description will be filled with the templates you have defined.

To create templates, open the Administration tab and go to the Test Manager -> Templates section.

The Templates management section in the Test Manager Administration panel

Test Catalog Templates

In this section you can manage the templates for your Test Catalogs.

If you have any templates defined, they will be listed in this section. Otherwise, the text "No templates created yet" will be displayed.

• To create a new template, type the new template name in the text field and click Add Test Catalog Template. The template editing page will be opened.
  • Enter the main text for the Test Catalog template in the text area.
  • To preview the results, push Preview Template: the preview will be displayed right below the text area.
  • When you are satisfied with the template, push Submit Changes.
  • Push Cancel to exit without saving.
• To browse or modify a template, click on the template name. The template editing page will be opened in edit mode.
  • Follow the same options as for creating a new template.
• To delete one or more templates, select the corresponding checkboxes and push Delete Selected.

Test Case Templates

In this section you can manage the templates for your Test Cases.

If you have any templates defined, they will be listed in this section. Otherwise, the text "No templates created yet" will be displayed.

• To create a new template, type the new template name in the text field and click Add Test Catalog Template. The template editing page will be opened.
  • Enter the main text for the Test Case template in the text area.
  • To preview the results, push Preview Template: the preview will be displayed right below the text area.
  • When you are satisfied with the template, push Submit Changes.
  • Push Cancel to exit without saving.
• To browse or modify a template, click on the template name. The template editing page will be opened in edit mode.
  • Follow the same options as for creating a new template.
• To delete one or more templates, select the corresponding checkboxes and push Delete Selected.

Default template for Test Catalog

Here you can set the default template for new Test Catalogs.

Select the desired template from the list, or select None to not have any template applied, and click Save Default Test Catalog Template.

Default Test Case template for each Test Catalog

Here you can specify, for each Test Catalog you have already defined, which should be the default template for the Test Cases defined therein.

Select the desired Test Case template from the list associated to any particular Test Catalog, or select None to not have any template applied, and click Save Template Configuration.

Test Manager Settings

Several aspects of the Test Manager operations are for you to personalize. To do this, you can either use the Test Manager administration panel, or directly edit the trac.ini file.

The following paragraphs outline the administration panel sections.

When you are done with your changes, click Save at the bottom of the page.

The Settings section in the Test Manager Administration panel

Test Statistics default settings

You can personalize the default period of time the statistical charts will display, in terms of historical depth and scale:

• Number of days shown by default: determines the historical depth of the charts, in terms of days back from the current date.
• Number of days between each data point: determines the scale, or granularity, of the charts, in terms of days per single piece of data.

In alternative to using the administration panel, you can directly edit the trac.ini file:

[testmanager]
; number of days back from today
default_days_back = 14
; number of days between samples
default_interval = 1

Test Manager visualization settings

You can personalize some visualization settings:

• Sort a test plan's test cases by: Determines how the test cases in a Test Plan will be sorted.
  • User Specified: You can control the exact ordering of the test cases. To set a specific order, use the Organize Test Catalogs functionality.
  • Name: Sorts the test cases by title.
  • Modification Time: Sorts the test cases by modification time, with the most recent on top.
• Open Test Cases in a new window: Determines whether, from a test catalog or test plan view, clicking on a test case will display it in a new window or in the same window.
• Default Test Catalog view type: Determines whether the default view for test catalogs should be Tree or Table.
• Default Test Plan view type: Determines whether the default view for test plans should be Tree or Table.

Visible Test Catalog columns in table view

You can personalize the visible columns for the Test Catalog and for the Test Plan tabular views.

The two lists show the available columns, including custom fields, for the two artefact types. Use the checkbox to display or hide the corresponding column in the table.

For each column, you can choose to add a calculated "totals" row at the end of the table, showing for all the test cases:

• Sum: the sum of the non-empty values in the column.
• Average: the average of the non-empty values in the column.
• Count: the count of the non-empty values in the column.

This can be found particularly useful if you set up custom fields. Refer to the corresponding section for details and usage examples.

Security

The following new permissions are available:

• TEST_VIEW - Ability to view test catalogs and test cases
• TEST_MODIFY - Ability to create and edit test catalogs and test cases
• TEST_EXECUTE - Ability to change the status of a test case in a test plan
• TEST_DELETE - Ability to delete test cases
• TEST_PLAN_ADMIN - Ability to generate and delete test plans
• TEST_STATS_VIEW - Ability to view test management statistics
• SQL_RUN - Ability to run arbitrary queries and statements on the Trac database. The SQLExecutor plugin shipped with the Test Manager provides this feature.

Tutorial

These tutorials document release 1.0.3 and so need to be updated with the Test Plan and other recent features. Refer to the ​video tutorial on Youtube for more up-to-date information.

• attachment:"UserGuide_part_1.ppt"​
• attachment:"UserGuide_part_2.ppt"​
• attachment:"UserGuide_part_3.ppt"​

Project site

Project site (don't use it for the code repository): ​http://sourceforge.net/projects/testman4trac/

Project source code repository now on GitHub: ​https://github.com/seccanj/testman4trac

The project is also on ​pypi:TestManager

Translations

This plugin is localized and all texts and messages will be displayed in the language detected by the User's browser, if available, or in English otherwise.

You may want to do your own translation, following the ​localization cookbook page for Trac plugins.

You've done a new translation? Superb! Contributing your translation is highly appreciated.

​Contact me on the forum and I'll manage to add it to the set of available languages.

Top translations » ​Test Manager for Trac Plugin on Transifex

Kindly provided by

Bugs/Feature Requests

Existing bugs and feature requests for TestManagerForTracPlugin are here.

If you have any issues, create a new ticket.

defect	99 / 124
enhancement	61 / 78
task	2 / 2

Download

Download the egg files, ready to be installed, from the ​Sourceforge project site or from ​PyPi.

Source

You can download the source code from GitHub: ​https://github.com/seccanj/testman4trac.

Installation

Trac 0.12 and 1.4 are supported, and also Python 2.7. Trac 1.5 and above is not supported yet.

Note: If you are using MySQL or PosgreSQL, the database user must have read access to the trac.ini file.

The plugin can be installed as usual, from the Trac administration panel, and requires a database upgrade.

You can follow the instructions in the INSTALL.txt file contained in the binary packages.

The functionalities are divided in three plugins, which must be installed in this order:

1. Trac Generic Class
2. Trac Generic Workflow
3. Test Manager

An additional plugin is very useful for debugging, but also Trac data analysis and troubleshooting. It provides the ability to run arbitrary SQL queries or statements on the Trac database from your web browser (including creating, modifying and dropping tables!).

4. SQL Executor

To use it, just click on the SQL Executor button on the toolbar at the top of the page and enter your SQL in the text area, or use the REST API as follows:

http://yourserver/yourenvironment/sqlexec?sql=any SQL statement

For example:

http://localhost/sampleproject/sqlexec?sql=select * from testcatalog

Since this feature is risky, the specific Trac permission SQL_RUN has been added to protect it. You must have this permission to see and use the plugin.

Follow the quick setup guide or the ​video tutorial on YouTube to get started.

Troubleshooting

Is setuptools properly installed?

Try this from the command line:

python -c "import pkg_resources"

If you get no output, then setuptools is installed. Otherwise, you'll need to install it before plugins will work in Trac.

Did you get the correct version of the Python egg?

Python eggs have the Python version in their filename. For example, TestManager-1.0.0-py2.7.egg is an egg for Python 2.7, and will not be loaded if you're running a different Python version, such as 2.5 or 2.6.

Also, verify that the egg file you downloaded is indeed a ZIP archive and not the HTML preview page instead.

Is the plugin enabled?

If you install a plugin globally, ie not inside the plugins directory of the Trac project environment, you will have to explicitly enable it in trac.ini. Make sure that:

• you actually added the necessary lines to the [components] section:

  [components]
  tracgenericclass.* = enabled
  tracgenericworkflow.* = enabled
  testmanager.* = enabled
  

• the package/module names are correct
• the value is "enabled", not "enable"

Check the permissions on the egg file and on the trac.ini file

The system user running Trac (either tracd or Apache) must be able to read the egg files.

It has been reported that for PosgreSQL its user must also be able to read the trac.ini file.

Check the log files

Open the Trac Admin tab, enable logging and set the log level to DEBUG, then watch the log file for messages about loading plugins.

Verify you have proper permissions

Some plugins, including the Test Manager ones, require you have special permissions in order to use them.

See TestManagerForTracPlugin#Security for a description of the available/required permissions.

Is the wrong version of the plugin loading?

If you put your plugins inside plugins directories, and certainly if you have more than one project, you need to make sure that the correct version of the plugin is loading. Here are some basic rules:

• Only one version of the plugin can be loaded for each running Trac server (ie each Python process). The Python namespaces and module list will be shared, and it cannot handle duplicates. Whether a plugin is enabled or disabled makes no difference.

• A globally installed plugin, typically setup.py install, will override any version in global or project plugins directories. A plugin from the global plugins directory will be located before any project plugins directory.

• If your Trac server hosts more than one project (as with TRAC_ENV_PARENT_DIR setups), then having two versions of a plugin in two different projects will give uncertain results. Only one of them will load, and the one loaded will be shared by both projects. Trac will load the first found - basically from the project that receives the first request.

• Having more than one version listed inside Python site-packages is fine, ie installed with setup.py install, as setuptools will make sure you get the version installed most recently. However, don't store more than one version inside a global or project plugins directory: neither version number nor installed date will matter. There is no way to determine which one will be located first when Trac searches the directory for plugins.

Have you restarted Apache?

For Apache to sense a plugin upgrade (and sometimes even installation), it must be restarted.

If all of the above failed

OK, so the logs don't mention these plugins, the egg is readable, the Python version is correct and the egg has been installed globally (and is enabled in the trac.ini file), you restarted Apache and it still doesn't work or give any error messages or any other indication as to why?

Open a ticket here. The plugins maintainers watch ticket notifications continuously.

Recent Changes

Note: you can subscribe to the ​Forum to be updated on new releases.

See the complete list of changes.

Author/Contributors

Author: seccanj
Maintainer: Roberto Longobardi
Contributors: