Miscellaneous refactoring and minor clean ups

(In [11066]) Ignore .pyc files. Refs #9648.

(In [11067]) Don't know when this unnecessary blank line was introduced. Refs #9648.

(In [11068]) Fix resource name display. Refs #9648.

If no user names were configured, the resource showed up blank, not as a raw ID.

(In [11070]) White space, comments, minor refactoring. Refs #9648.

(In [11075]) Comments and white space. Refs #9648.

(In [11083]) Rework calendar offset adjustment to not go outside working hours. Refs #9648, #9533, #9300.

(In [11412]) Use [], not None, for children if parent not configured. Refs #9648.

We use the result of TracPM.children() in two ways:

• if children(ticket)
• for cid in children(ticket)

The former has the same result for None and [] but the latter needs a protecting if isCfg('parent') if children is None. Using [] saves that if.

(In [11413]) Allow configuration of start even when finish isn't. Refs #9648.

This isn't a common case but I'm trying to keep it flexible.

(In [11415]) Make TracPM DB access database agnostic. Refs #9648.

(In [11417]) Make TracPM.preQuery() safe for other plugins. Refs #9648.

When another plugin calls preQuery(), it may not have set default values for goal and root so rather than testing for the value, test to see if the value is present.

(In [11418]) It should be OK to have dependencies but not children. Refs #9648.

Showing tasks for goal= gets all predecessors and all predecessors' children. If no parent field is configured, don't try to process it.

(In [11421]) Move interface definitions to pmapi.py. Refs #9648.

(In [11425]) Computing the roots in a set of tasks is generally useful. Refs #9648.

Note that if no parent field is configured, all tickets are roots.

(In [11426]) A tree-aware task sorter. Refs #9833, #9042, #9290, #9300, #9691, #9784.

A task's priority depends on its parent's priority.

Includes a base sorter class to hold common sorter methods.

SimpleSorter? and ProjectSorter? derive from the base class.

(Copy effective priority list, just in case.)

Also remove unneeded debugging output. Refs #9648.

(In [11429]) Up version to 0.10. Refs #9648.

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

Replying to falkb:

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

That shouldn't be remarkable, of course, but I have had some missteps in the past and there was some major surgery here. I'm glad it's still working for you.

(In [11701]) Address pychecker complaints. Refs #9648.

Not all of them but at least all the local variable shadowing a built in and most of the issues with undeclared class variables.

Also renamed "cal" to "calendar" for clarity.

(In [11702]) Respond to code review feedback. Refs #9648.

(In [11703]) Don't resource level milestones. Refs #9648.

(In [12314]) Use TracPM.roots() to find root of set of tickets. Refs #9648.

(In [12315]) Allow parent-child relation. Refs #9648.

Mostly reuses logic from pred-succ relation so there isn't really any work for relation handling beyond adding the relation to the list of configurable relations; the rest is data-driven.

However, the relation handling builds lists and we previously handled parent as a singleton so some parent field handling needed tweaking:

• TracPM.parent() now returns None if there is no parent field or

relation configured or if there is no parent. Internal code was adjusted. The only user outside this module, tracjsgantt.py, works with this change.

• When creating a pseudo-ticket for milestones, set the parent list to

empty, not the previous place-holder ID 0.

• When normalizing and parsing a configured parent field, make a

one-element list of the parent ID

• When testing for children, check if the child ID is in the parent

list rather than the parent field is the child ID

Also fix a bug where parent/child field handling assumed the value of fieldName left over from a previous loop.

Note this was tested with a parent field, with a parent-child relation, and with no parent configuration.

Some plugins that support parent-child relationships may allow multiple parents (Subtickets does). Doing PM-ish things on projects broken down into phases broken down into deliverables broken down into tasks, etc. all assumes a tree structure which excludes multiple parents. If someone creates multiple parents, this will blow up. I'm OK with that.

(In [12316]) Refactor preQuery() to make _expand() usable elsewhere. Refs #9648.

Refactor _followLink() (nee _expand())) for flexibility.

• Don't include origin nodes in returned list
• Add a depth argument to control how many times to traverse link

(In [12317]) Add a function for finding related tickets. Refs #9648.

(In [12318]) Rename internal fields to avoid conflicts. Refs #9648.

Custom field names must start with a letter and have only letters, digits, and underscores. By adding a leading underscore to "calc_start" and "calc_finish", we avoid a potential conflict if the user used those as custom field names for their own purposes or even for fields.start and fields.finish values.

(In [12319]) Refactor date parsing. Refs #9648.

(In [12320]) More flexible setting of query fields. Refs #9648.

• TracPM.queryFields() lists all the fields that TracPM needs.
• Caller (in this case the Gantt chart) has to make sure to include those fields only once in query.

(In [12321]) Log some details in when scheduling goes a little wrong. Refs #9648.

(In [12322]) Make sure final tickets don't get scheduled twice. Refs #9648.

(In [12323]) Rename milestone_type to goal_ticket_type. Refs #9648.

Trying to eliminate confusion about what a "milestone" is.

(In [12324]) Assign open/closed status to milestone pseudo-tickets. Refs #9648.

(In [12330]) Refactor ticket filtering. Refs #9648.

This encapsulates a fairly long algorithm into a function for clarity.

Also, fix a bug where we sort too many times. Removing tickets in the filter shouldn't reorder them.

Also, always call _filter_tickets(); it applies omitMilestones so it has to be called even if no other display filter is specified.

please, give a sign when your series of changes has finished, and we can dare update again

Replying to anonymous:

please, give a sign when your series of changes has finished, and we can dare update again

My goal is that the on-line version is at least stable at the end of each set of updates. The parent-child relation should be usable now.

I hope to push a major feature based on this refactoring this week.

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before.

I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]]

Before your latest commit avalanche the milestone looked like:

==========
 ----
     ---
        --
      =========
          ---
             --
           ==========
                  ---
                  ---

but now the parent tickets are not ordered by time anymore and it looks like

           ==========
                  ---
                  ---
==========
 ----
     ---
        --
      =========
          ---
             --

which results in a strange snarl of dependency arrows

Odd. I don't see strange sorting here. Can you try adding order=wbs to your chart and see if that helps?

nope, doesn't help. Also the milestone is displayed between the tickets.

How can I switch on some debug tracing for you?

Replying to falkb:

How can I switch on some debug tracing for you?

I haven't added very effective run-time logging. (I tend to put it in for development and take it out before release.)

There seem to be just 10 tickets. Can you post here an overview of their relationships:

Ticket	Duration	Due Date	Blocked By	Blocking	Parent
A
B				C	A
C			B		A

Even better if you can attach it as a CSV so I can just import it into my environment.

Good idea, please look at the attached csv file. I don't have duration but startdate and enddate. The according milestone is due 10/Dec/2012.

belongs to 40

Have a look at the attachment, it's a bit more than 10 tickets.

Replying to falkb:

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before. I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]] Before your latest commit avalanche the milestone looked like: {{{ ========== ---- --- -- ========= --- -- ========== --- --- }}} but now the parent tickets are not ordered by time anymore and it looks like {{{ ========== --- --- ========== ---- --- -- ========= --- -- }}} which results in a strange snarl of dependency arrows

I don't see any strange behavior. I'll attach what I see. The summary for the tickets imported is "th <your ticket number>-<relative ticket number>"

My Gantt of tickets from imported query.csv using current code at TH.

Could you please give me an uncolapsed screenshot. Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

My Gantt with more tickets open

Replying to anonymous:

Could you please give me an uncolapsed screenshot.

Attached. I couldn't open 299 (your 2692) without going off the page and then I couldn't capture it.

Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

Yes, it might be some version dependency. Hmmm.

Can you post (or e-mail me) the embedded JavaScript? in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

OK with version before your refactoring

NOK with version after your refactoring

I analyzed a bit more (see result pictures below):

1. I used 0.12.2 again
2. I don't use my local hacks anymore, now clean from original svn sources

The related plugins I use are

• TracMasterTickets?-3.0.2-py2.5.egg
• TracSubTicketsPlugin?-0.1.0-py2.5.egg

Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: and with latest revision r12330 now:

embedded javascript from page source, requested in 44

Replying to ChrisNelson:

Can you post (or e-mail me) the embedded JavaScript? in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

Here we go: attachment:falkb-open-0.12.2-r12330_javascript.txt (obfuscated)

Replying to falkb:

I analyzed a bit more (see result pictures below): 1. I used 0.12.2 again 2. I don't use my local hacks anymore, now clean from original svn sources ... Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: ... and with latest revision r12330 now: ...

It appears to me that the tasks within each group are scheduled and ordered the same but that the top-level tickets are in a different order.

That order isn't numeric (ticket ID) or chronological (by start or finish). I'll look at the JS and see if I see a pattern.

Replying to falkb:

... The related plugins I use are * TracMasterTickets?-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

* TracSubTicketsPlugin?-0.1.0-py2.5.egg

I'm using the same.

...

Replying to ChrisNelson:

Replying to falkb:

... The related plugins I use are * TracMasterTickets?-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

* TracSubTicketsPlugin?-0.1.0-py2.5.egg

I'm using the same.

...

Are you using a field or a relation to configure the interface to those plugins? I have:

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

Replying to ChrisNelson:

Are you using a field or a relation to configure the interface to those plugins? I have: {{{ relation.parent-child = subtickets,parent,child relation.pred-succ = mastertickets,source,dest }}}

Not sure what you mean. My trac.ini just contains for those fields:

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
fields.parent = parents
fields.pred = blockedby
fields.start = userstart
fields.succ = blocking
fields.worked = totalhours
milestone_type = milestone

I don't see exactly what you see but I can make it worse with the before code (2677 ends up in the middle instead of at the end) if I use fields. Please try relations as noted above and see if it is better for you. (That is, replace the fields.parents, fields.pred, and fields.succ lines by the relation.* lines I quoted.

Do you mean from:

[TracPM]
fields.parent = parents
fields.pred = blockedby
fields.succ = blocking

to

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

or

relation.parent-child = subtickets,parents,?????
relation.pred-succ = mastertickets,blockedby,blocking

?

Replying to falkb:

Do you mean from: {{{ [TracPM] fields.parent = parents fields.pred = blockedby fields.succ = blocking }}} to {{{ relation.parent-child = subtickets,parent,child relation.pred-succ = mastertickets,source,dest }}} or {{{ relation.parent-child = subtickets,parents,????? relation.pred-succ = mastertickets,blockedby,blocking }}} ?

The former. The setting is <table>,<from>,<to>. So, In the subtickets table (private to the Subtickets plugin), the parent field is the beginning of the arc and the child field is the end.

Using

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
#fields.parent = parents
#fields.pred = blockedby
fields.start = userstart
#fields.succ = blocking
fields.worked = totalhours
relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest
milestone_type = milestone

Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

Replying to falkb:

... Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

relation wasn't supported in r12213. Are you using r12330?

yes, all my trials are with a running r12330

Replying to falkb:

yes, all my trials are with a running r12330

:-( I'll see if I can figure something out from the differences I see here.

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

Replying to falkb:

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

Yes. I checked out what I pushed to TH, rebuilt, and reinstalled and it's fine for me when I use relation. It's a little odd when I use fields. I still don't see what you see but that may be because I didn't set the task priorities so the scheduler may make slightly different choices. Sorry. I'll try to dig into this today.

The priorities are all default, except 3218 and 3456 are default+1, and 3449 is default+2

Replying to falkb:

I analyzed a bit more (see result pictures below): 1. I used 0.12.2 again 2. I don't use my local hacks anymore, now clean from original svn sources The related plugins I use are * TracMasterTickets?-3.0.2-py2.5.egg * TracSubTicketsPlugin?-0.1.0-py2.5.egg Based on 1. and 2., this is what I see with revision r12213 before the refactoring session: ... and with latest revision r12330 now: ...

While I doubt it is a significant difference, I note that in query.csv has 2682 as a child of 2681 but that is not shown on these charts.

I'll check that at work tomorrow, I think it's because I retrieved the data from and did the tests with the copy of the live system and there were activities from one day to another.

Please, follow #10686 to read the further discussion between falkb and ChrisNelson because of the chart sorting problem.

(In [12426]) Assign WBS in correct order. Refs #10868, #9648.

The order of WBS assignments was broken by r12314.

TracPM.roots() returns an unordered list of ticket IDs so traversing that list meant we were no longer computing the WBS in the correct order. Now we use that list to determine what is a root ticket but traverse the ticket list in the scheduled (sorted) order.

Thanks to falkb for his help and patience while working on this.

Today I've found another issue which is broken since your latest activities within the last 4 weeks: I have a Trac custom report [[TracJSGanttChart(format=week,order=milestone|type|wbs,res=0,colorBy=priority,owner=$USER&status!=closed)]] which gave a survey over all open tickets to the user. Now it's not working anymore. No chart appears at all. I suspect it's because of no milestone is given.

(In [12746]) Make graph augmentation a TracPM method. Refs #9648.

Need to do this before trying to reschedule in ticket change listener.

(In [12747]) Add private table creation. Refs #9648.

I'm not sure that the name of the table, schedule, is sufficiently obscure that it won't step on users or other plugins.

We're going to store the schedule in a private table. We can use IEnvironmentSetupParticipant methods to check for the table and create as needed the first time the plugin is activated.

The general principles are at http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.env.IEnvironmentSetupParticipant

The Trac schema is at http://trac.edgewall.org/browser/trunk/trac/db_default.py for comparison. I modeled schedule on ticket_change (int64 for datetimes) and ticket_custom (index by ticket to get values).

Some of the detail code came out of api.py in git://git.sixnetio.com/git/misc_tools/trac_plugin_source/trac-subtickets-plugin.git.

(In [12748]) Add a ticket change listener to detect related tickets. Refs #9648.

THIS CONTAINS SOME HARD-CODED FIELD NAMES AND WORKS ONLY WITH MASTERTICKETS AND SUBTICKETS. IT DOES NOT YET WORK WITH CHILDTICKETS OR OTHER RELATIONSHIP PLUGINS.

This doesn't do any rescheduling. It just notes how many tickets need to be rescheduled and how long it took to find them.

(In [12749]) Recompute schedule and save changes (if any) in a private table. Refs #9648.

• All datetimes used in schedule are now in local time zone.
  • Had to be "offset-aware" (to use to_utimestamp() utility function)
• TracPM.start(), .finish() now return calculated date, then stored date, then None depending on what is available.
• postQuery() now reads schedule from private table rather than having it queried from custom fields
• When the rescheduler is done calculating a schedule it uses INSERT or UPDATE to write only changed tickets back to the private schedule table.
• Changes to the schedule are stored in the schedle_history table.

This handles inserting initial schedule into DB and updating only changed tickets when rescheduling.

With the changes to TracPM.start(), .finish(), a caller (e.g., the Gantt chart) can:

• Query tickets
• Optionally schedule them
• Iterate over them getting the start and/or finish dates

If the scheduling step is skipped, the dates from the database are used.

There's a potentially inefficiency in how potentially inactive tickets are found. We might get this set by first getting all the affected tickets then following the predecessor links from the goal but there isn't a function for that yet and preQuery() is fairly efficient. I don't want to optimize prematurely.

For this to work, I have to splice the in-memory ticket graph based on ticket changes.

This has some gross code that is specific to plugins we use and makes TracPM less portable but I want to prove this works before trying to make it generic.

(In [12750]) Get actual start and finish for active and closed tickets. Refs #9648, #10717.

Use these dates in scheduling.

Note that closed tickets don't require resource leveling, display their actual start and finish times.

This defaults to disabled (0) and can be controlled by useActuals option in the TracPM section of trac.ini (1=use actual dates, 0=schedule). The useActuals parameter to the Gantt chart macro overrides the TracPM value.

(In [12753]) Add support for scheduled=1 to TracPM.query(). Refs #9648.

Allow retrieval of only tickets that have a precomputed schedule in the database.

(In [12754]) Scheduler more flexible about missing/defaulted options. Refs #9648.

The Gantt chart passes a bunch of options that come from the macro invocation or the chart configuration in Trac.ini. A lighter weight client -- like the ticket change listener -- may not have all those options and just wants default behaviors. Some missing values weren't handled as defaults before.

(In [12755]) Allow Gantt to request no scheduling (just show db values). Refs #9648.

This allows TracJSGanttChart(scheduled=1,schedule=none) to show tasks as they are in the schedule database.

(In [12757]) Improve "omitMilestones" handling. Refs #9648.

(In [12758]) Simplify Trac.pm.query() interface. Refs #9648.

(In [12759]) Move sort function to match master branch. Refs #9648.

(In [12760]) More consistent use of t.get(field) vs. t[field]. Refs #9648.

(In [12762]) White space. Refs #9648.

(In [12763]) Refine background ticket rescheduler. Refs #9648.

Move the exclusion of milestone pseudo-tickets closer to where it makes a difference (when we save the data).

This should be a no-op most of the time but without this the ticket set sometimes has dangling references which makes the plugin fragile when adding new function invocations in certain places.

(In [12764]) Bug in start/finish option experiment. Refs #9845, #9648.

(In [12969]) Background rescheduler sometimes fails to handle dependencies. Refs #9648.