Modify

Opened 3 years ago

Last modified 6 months ago

#9648 assigned defect

Miscellaneous refactoring and minor clean ups

Reported by: ChrisNelson Owned by: ChrisNelson
Priority: normal Component: TracJsGanttPlugin
Severity: normal Keywords:
Cc: Trac Release: 0.11

Description

A ticket to link various small fixes to.

Attachments (6)

query.csv (1.1 KB) - added by falkb 22 months ago.
belongs to 40
TH9648.png (72.3 KB) - added by ChrisNelson 22 months ago.
My Gantt of tickets from imported query.csv using current code at TH.
TH9648-open.png (135.3 KB) - added by ChrisNelson 22 months ago.
My Gantt with more tickets open
falkb-open-0.12.2-r12213.png (85.8 KB) - added by falkb 22 months ago.
OK with version before your refactoring
falkb-open-0.12.2-r12330.png (82.0 KB) - added by falkb 22 months ago.
NOK with version after your refactoring
falkb-open-0.12.2-r12330_javascript.txt (14.3 KB) - added by falkb 22 months ago.
embedded javascript from page source, requested in 44

Download all attachments as: .zip

Change History (116)

comment:1 Changed 3 years ago by ChrisNelson

(In [11066]) Ignore .pyc files. Refs #9648.

comment:2 Changed 3 years ago by ChrisNelson

(In [11067]) Don't know when this unnecessary blank line was introduced. Refs #9648.

comment:3 Changed 3 years ago by ChrisNelson

(In [11068]) Fix resource name display. Refs #9648.

If no user names were configured, the resource showed up blank, not as a
raw ID.

comment:4 Changed 3 years ago by ChrisNelson

(In [11070]) White space, comments, minor refactoring. Refs #9648.

comment:5 Changed 3 years ago by ChrisNelson

(In [11075]) Comments and white space. Refs #9648.

comment:6 Changed 3 years ago by ChrisNelson

(In [11083]) Rework calendar offset adjustment to not go outside working hours. Refs #9648, #9533, #9300.

comment:7 Changed 3 years ago by ChrisNelson

(In [11412]) Use [], not None, for children if parent not configured. Refs #9648.

We use the result of TracPM.children() in two ways:

  • if children(ticket)
  • for cid in children(ticket)

The former has the same result for None and [] but the latter needs
a protecting if isCfg('parent') if children is None. Using []
saves that if.

comment:8 Changed 3 years ago by ChrisNelson

(In [11413]) Allow configuration of start even when finish isn't. Refs #9648.

This isn't a common case but I'm trying to keep it flexible.

comment:9 Changed 3 years ago by ChrisNelson

(In [11415]) Make TracPM DB access database agnostic. Refs #9648.

comment:10 Changed 3 years ago by ChrisNelson

(In [11417]) Make TracPM.preQuery() safe for other plugins. Refs #9648.

When another plugin calls preQuery(), it may not have set default
values for goal and root so rather than testing for the value, test to
see if the value is present.

comment:11 Changed 3 years ago by ChrisNelson

(In [11418]) It should be OK to have dependencies but not children. Refs #9648.

Showing tasks for goal= gets all predecessors and all predecessors'
children. If no parent field is configured, don't try to process it.

comment:12 Changed 3 years ago by ChrisNelson

(In [11421]) Move interface definitions to pmapi.py. Refs #9648.

comment:13 Changed 3 years ago by ChrisNelson

(In [11425]) Computing the roots in a set of tasks is generally useful. Refs #9648.

Note that if no parent field is configured, all tickets are roots.

comment:14 Changed 3 years ago by ChrisNelson

(In [11426]) A tree-aware task sorter. Refs #9833, #9042, #9290, #9300, #9691, #9784.

A task's priority depends on its parent's priority.

Includes a base sorter class to hold common sorter methods.

SimpleSorter and ProjectSorter derive from the base class.

(Copy effective priority list, just in case.)

Also remove unneeded debugging output. Refs #9648.

comment:15 Changed 3 years ago by ChrisNelson

(In [11429]) Up version to 0.10. Refs #9648.

comment:16 follow-up: Changed 2 years ago by falkb

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

comment:17 in reply to: ↑ 16 Changed 2 years ago by ChrisNelson

  • Status changed from new to assigned

Replying to falkb:

Your commits on 03/23/12 don't have made visible trouble here. After update it still seems to work well. Thanks.

That shouldn't be remarkable, of course, but I have had some missteps in the past and there was some major surgery here. I'm glad it's still working for you.

comment:18 Changed 2 years ago by ChrisNelson

(In [11701]) Address pychecker complaints. Refs #9648.

Not all of them but at least all the local variable shadowing a built in
and most of the issues with undeclared class variables.

Also renamed "cal" to "calendar" for clarity.

comment:19 Changed 2 years ago by ChrisNelson

(In [11702]) Respond to code review feedback. Refs #9648.

comment:20 Changed 2 years ago by ChrisNelson

(In [11703]) Don't resource level milestones. Refs #9648.

comment:21 Changed 23 months ago by ChrisNelson

(In [12314]) Use TracPM.roots() to find root of set of tickets. Refs #9648.

comment:22 Changed 23 months ago by ChrisNelson

(In [12315]) Allow parent-child relation. Refs #9648.

Mostly reuses logic from pred-succ relation so there isn't really any
work for relation handling beyond adding the relation to the list of
configurable relations; the rest is data-driven.

However, the relation handling builds lists and we previously handled
parent as a singleton so some parent field handling needed tweaking:

  • TracPM.parent() now returns None if there is no parent field or

relation configured or if there is no parent. Internal code was
adjusted. The only user outside this module, tracjsgantt.py, works with
this change.

  • When creating a pseudo-ticket for milestones, set the parent list to

empty, not the previous place-holder ID 0.

  • When normalizing and parsing a configured parent field, make a

one-element list of the parent ID

  • When testing for children, check if the child ID is in the parent

list rather than the parent field is the child ID

Also fix a bug where parent/child field handling assumed the value of
fieldName left over from a previous loop.

Note this was tested with a parent field, with a parent-child relation,
and with no parent configuration.

Some plugins that support parent-child relationships may allow multiple
parents (Subtickets does). Doing PM-ish things on projects broken down
into phases broken down into deliverables broken down into tasks, etc.
all assumes a tree structure which excludes multiple parents. If
someone creates multiple parents, this will blow up. I'm OK with that.

comment:23 Changed 23 months ago by ChrisNelson

(In [12316]) Refactor preQuery() to make _expand() usable elsewhere. Refs #9648.

Refactor _followLink() (nee _expand())) for flexibility.

  • Don't include origin nodes in returned list
  • Add a depth argument to control how many times to traverse link

comment:24 Changed 23 months ago by ChrisNelson

(In [12317]) Add a function for finding related tickets. Refs #9648.

comment:25 Changed 23 months ago by ChrisNelson

(In [12318]) Rename internal fields to avoid conflicts. Refs #9648.

Custom field names must start with a letter and have only letters,
digits, and underscores. By adding a leading underscore to "calc_start"
and "calc_finish", we avoid a potential conflict if the user used those
as custom field names for their own purposes or even for fields.start
and fields.finish values.

comment:26 Changed 23 months ago by ChrisNelson

(In [12319]) Refactor date parsing. Refs #9648.

comment:27 Changed 23 months ago by ChrisNelson

(In [12320]) More flexible setting of query fields. Refs #9648.

  • TracPM.queryFields() lists all the fields that TracPM needs.
  • Caller (in this case the Gantt chart) has to make sure to include those fields only once in query.

comment:28 Changed 23 months ago by ChrisNelson

(In [12321]) Log some details in when scheduling goes a little wrong. Refs #9648.

comment:29 Changed 23 months ago by ChrisNelson

(In [12322]) Make sure final tickets don't get scheduled twice. Refs #9648.

comment:30 Changed 23 months ago by ChrisNelson

(In [12323]) Rename milestone_type to goal_ticket_type. Refs #9648.

Trying to eliminate confusion about what a "milestone" is.

comment:31 Changed 23 months ago by ChrisNelson

(In [12324]) Assign open/closed status to milestone pseudo-tickets. Refs #9648.

comment:32 Changed 23 months ago by ChrisNelson

(In [12330]) Refactor ticket filtering. Refs #9648.

This encapsulates a fairly long algorithm into a function for clarity.

Also, fix a bug where we sort too many times. Removing tickets in the
filter shouldn't reorder them.

Also, always call _filter_tickets(); it applies omitMilestones so it
has to be called even if no other display filter is specified.

comment:33 follow-up: Changed 23 months ago by anonymous

please, give a sign when your series of changes has finished, and we can dare update again

comment:34 in reply to: ↑ 33 Changed 23 months ago by ChrisNelson

Replying to anonymous:

please, give a sign when your series of changes has finished, and we can dare update again

My goal is that the on-line version is at least stable at the end of each set of updates. The parent-child relation should be usable now.

I hope to push a major feature based on this refactoring this week.

comment:35 follow-up: Changed 22 months ago by falkb

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before.

I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]]

Before your latest commit avalanche the milestone looked like:

==========
 ----
     ---
        --
      =========
          ---
             --
           ==========
                  ---
                  ---

but now the parent tickets are not ordered by time anymore and it looks like

           ==========
                  ---
                  ---
==========
 ----
     ---
        --
      =========
          ---
             --

which results in a strange snarl of dependency arrows

comment:36 Changed 22 months ago by ChrisNelson

Odd. I don't see strange sorting here. Can you try adding order=wbs to your chart and see if that helps?

comment:37 Changed 22 months ago by falkb

nope, doesn't help. Also the milestone is displayed between the tickets.

comment:38 follow-up: Changed 22 months ago by falkb

How can I switch on some debug tracing for you?

comment:39 in reply to: ↑ 38 Changed 22 months ago by ChrisNelson

Replying to falkb:

How can I switch on some debug tracing for you?

I haven't added very effective run-time logging. (I tend to put it in for development and take it out before release.)

There seem to be just 10 tickets. Can you post here an overview of their relationships:

TicketDurationDue DateBlocked ByBlockingParent
A
B C A
C B A

Even better if you can attach it as a CSV so I can just import it into my environment.

comment:40 Changed 22 months ago by falkb

Good idea, please look at the attached csv file. I don't have duration but startdate and enddate. The according milestone is due 10/Dec/2012.

Changed 22 months ago by falkb

belongs to 40

comment:41 Changed 22 months ago by falkb

Have a look at the attachment, it's a bit more than 10 tickets.

comment:42 in reply to: ↑ 35 Changed 22 months ago by ChrisNelson

Replying to falkb:

Today I've tried an update to your latest SVN revision. I noticed it's all quite mixed up now, means it's worse than before.

I have my tickets grouped with parent tickets and display my milestone this way: [[TracJSGanttChart(milestone=foo,colorBy=status,userMap=0,format=week,schedule=alap,doResourceLeveling=0)]]

Before your latest commit avalanche the milestone looked like:

==========
 ----
     ---
        --
      =========
          ---
             --
           ==========
                  ---
                  ---

but now the parent tickets are not ordered by time anymore and it looks like

           ==========
                  ---
                  ---
==========
 ----
     ---
        --
      =========
          ---
             --

which results in a strange snarl of dependency arrows

I don't see any strange behavior. I'll attach what I see. The summary for the tickets imported is "th <your ticket number>-<relative ticket number>"

Changed 22 months ago by ChrisNelson

My Gantt of tickets from imported query.csv using current code at TH.

comment:43 follow-up: Changed 22 months ago by anonymous

Could you please give me an uncolapsed screenshot. Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

Changed 22 months ago by ChrisNelson

My Gantt with more tickets open

comment:44 in reply to: ↑ 43 ; follow-up: Changed 22 months ago by ChrisNelson

Replying to anonymous:

Could you please give me an uncolapsed screenshot.

Attached. I couldn't open 299 (your 2692) without going off the page and then I couldn't capture it.

Well, I have Trac-1.0, maybe some other versions of involved plugins, and some little hacks in your code. At least my hacks shouldn't matter, I've checked it already.

Yes, it might be some version dependency. Hmmm.

Can you post (or e-mail me) the embedded JavaScript in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

Changed 22 months ago by falkb

OK with version before your refactoring

Changed 22 months ago by falkb

NOK with version after your refactoring

comment:45 follow-ups: Changed 22 months ago by falkb

I analyzed a bit more (see result pictures below):

  1. I used 0.12.2 again
  2. I don't use my local hacks anymore, now clean from original svn sources

The related plugins I use are

  • TracMasterTickets-3.0.2-py2.5.egg
  • TracSubTicketsPlugin-0.1.0-py2.5.egg

Based on 1. and 2., this is what I see with revision r12213 before the refactoring session:
OK with version before your refactoring
and with latest revision r12330 now:
NOK with version after your refactoring

Changed 22 months ago by falkb

embedded javascript from page source, requested in 44

comment:46 in reply to: ↑ 44 Changed 22 months ago by falkb

Replying to ChrisNelson:

Can you post (or e-mail me) the embedded JavaScript in the page source? (You can clip it and obfuscate the ticket summaries, if you wish.)

Here we go: attachment:falkb-open-0.12.2-r12330_javascript.txt (obfuscated)

comment:47 in reply to: ↑ 45 Changed 22 months ago by ChrisNelson

Replying to falkb:

I analyzed a bit more (see result pictures below):

  1. I used 0.12.2 again
  2. I don't use my local hacks anymore, now clean from original svn sources

...
Based on 1. and 2., this is what I see with revision r12213 before the refactoring session:
...
and with latest revision r12330 now:
...

It appears to me that the tasks within each group are scheduled and ordered the same but that the top-level tickets are in a different order.

That order isn't numeric (ticket ID) or chronological (by start or finish). I'll look at the JS and see if I see a pattern.

comment:48 in reply to: ↑ 45 ; follow-up: Changed 22 months ago by ChrisNelson

Replying to falkb:

...
The related plugins I use are

  • TracMasterTickets-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

  • TracSubTicketsPlugin-0.1.0-py2.5.egg

I'm using the same.

...

comment:49 in reply to: ↑ 48 ; follow-up: Changed 22 months ago by ChrisNelson

Replying to ChrisNelson:

Replying to falkb:

...
The related plugins I use are

  • TracMasterTickets-3.0.2-py2.5.egg

I'm using 2.2. However, I don't interact with the plugin, I just use its data.

  • TracSubTicketsPlugin-0.1.0-py2.5.egg

I'm using the same.

...

Are you using a field or a relation to configure the interface to those plugins? I have:

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

comment:50 in reply to: ↑ 49 Changed 22 months ago by falkb

Replying to ChrisNelson:

Are you using a field or a relation to configure the interface to those plugins? I have:

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

Not sure what you mean. My trac.ini just contains for those fields:

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
fields.parent = parents
fields.pred = blockedby
fields.start = userstart
fields.succ = blocking
fields.worked = totalhours
milestone_type = milestone

comment:51 Changed 22 months ago by ChrisNelson

I don't see exactly what you see but I can make it worse with the before code (2677 ends up in the middle instead of at the end) if I use fields. Please try relations as noted above and see if it is better for you. (That is, replace the fields.parents, fields.pred, and fields.succ lines by the relation.* lines I quoted.

comment:52 follow-up: Changed 22 months ago by falkb

Do you mean from:

[TracPM]
fields.parent = parents
fields.pred = blockedby
fields.succ = blocking

to

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

or

relation.parent-child = subtickets,parents,?????
relation.pred-succ = mastertickets,blockedby,blocking

?

comment:53 in reply to: ↑ 52 Changed 22 months ago by ChrisNelson

Replying to falkb:

Do you mean from:

[TracPM]
fields.parent = parents
fields.pred = blockedby
fields.succ = blocking

to

relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest

or

relation.parent-child = subtickets,parents,?????
relation.pred-succ = mastertickets,blockedby,blocking

?

The former. The setting is <table>,<from>,<to>. So, In the subtickets table (private to the Subtickets plugin), the parent field is the beginning of the arc and the child field is the end.

comment:54 follow-up: Changed 22 months ago by falkb

Using

[TracPM]
date_format = %Y-%m-%d
days_per_estimate = 0.125
fields.estimate = estimatedhours
fields.finish = userfinish
#fields.parent = parents
#fields.pred = blockedby
fields.start = userstart
#fields.succ = blocking
fields.worked = totalhours
relation.parent-child = subtickets,parent,child
relation.pred-succ = mastertickets,source,dest
milestone_type = milestone

Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

comment:55 in reply to: ↑ 54 Changed 22 months ago by ChrisNelson

Replying to falkb:

...
Uhm.. well, now it's even worse. I don't have parent tickets anymore. It's simply an unstructured list of tickets. The milestone is in the middle, ticket order is different from what I've seen the tests before but still strangly mixed up.

relation wasn't supported in r12213. Are you using r12330?

comment:56 follow-up: Changed 22 months ago by falkb

yes, all my trials are with a running r12330

comment:57 in reply to: ↑ 56 Changed 22 months ago by ChrisNelson

Replying to falkb:

yes, all my trials are with a running r12330

:-( I'll see if I can figure something out from the differences I see here.

comment:58 follow-up: Changed 22 months ago by falkb

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

comment:59 in reply to: ↑ 58 Changed 22 months ago by ChrisNelson

Replying to falkb:

Are you sure you committed everything? It's all very strange... I'm also stranded right now...

Yes. I checked out what I pushed to TH, rebuilt, and reinstalled and it's fine for me when I use relation. It's a little odd when I use fields. I still don't see what you see but that may be because I didn't set the task priorities so the scheduler may make slightly different choices. Sorry. I'll try to dig into this today.

comment:60 Changed 22 months ago by falkb

The priorities are all default, except 3218 and 3456 are default+1, and 3449 is default+2

comment:61 in reply to: ↑ 45 Changed 22 months ago by ChrisNelson

Replying to falkb:

I analyzed a bit more (see result pictures below):

  1. I used 0.12.2 again
  2. I don't use my local hacks anymore, now clean from original svn sources

The related plugins I use are

  • TracMasterTickets-3.0.2-py2.5.egg
  • TracSubTicketsPlugin-0.1.0-py2.5.egg

Based on 1. and 2., this is what I see with revision r12213 before the refactoring session:
...
and with latest revision r12330 now:
...

While I doubt it is a significant difference, I note that in query.csv has 2682 as a child of 2681 but that is not shown on these charts.

comment:62 Changed 22 months ago by anonymous

I'll check that at work tomorrow, I think it's because I retrieved the data from and did the tests with the copy of the live system and there were activities from one day to another.

comment:63 Changed 22 months ago by falkb

Please, follow #10686 to read the further discussion between falkb and ChrisNelson because of the chart sorting problem.

comment:64 Changed 22 months ago by ChrisNelson

(In [12426]) Assign WBS in correct order. Refs #10868, #9648.

The order of WBS assignments was broken by r12314.

TracPM.roots() returns an unordered list of ticket IDs so traversing
that list meant we were no longer computing the WBS in the correct
order. Now we use that list to determine what is a root ticket but
traverse the ticket list in the scheduled (sorted) order.

Thanks to falkb for his help and patience while working on this.

comment:65 Changed 22 months ago by falkb

Today I've found another issue which is broken since your latest activities within the last 4 weeks: I have a Trac custom report [[TracJSGanttChart(format=week,order=milestone|type|wbs,res=0,colorBy=priority,owner=$USER&status!=closed)]] which gave a survey over all open tickets to the user. Now it's not working anymore. No chart appears at all. I suspect it's because of no milestone is given.

comment:66 Changed 18 months ago by ChrisNelson

(In [12746]) Make graph augmentation a TracPM method. Refs #9648.

Need to do this before trying to reschedule in ticket change listener.

comment:67 Changed 18 months ago by ChrisNelson

(In [12747]) Add private table creation. Refs #9648.

I'm not sure that the name of the table, schedule, is sufficiently
obscure that it won't step on users or other plugins.

We're going to store the schedule in a private table. We can use
IEnvironmentSetupParticipant methods to check for the table and create
as needed the first time the plugin is activated.

The general principles are at http://trac.edgewall.org/wiki/TracDev/PluginDevelopment/ExtensionPoints/trac.env.IEnvironmentSetupParticipant

The Trac schema is at
http://trac.edgewall.org/browser/trunk/trac/db_default.py for
comparison. I modeled schedule on ticket_change (int64 for
datetimes) and ticket_custom (index by ticket to get values).

Some of the detail code came out of api.py in git://git.sixnetio.com/git/misc_tools/trac_plugin_source/trac-subtickets-plugin.git.

comment:68 Changed 18 months ago by ChrisNelson

(In [12748]) Add a ticket change listener to detect related tickets. Refs #9648.

THIS CONTAINS SOME HARD-CODED FIELD NAMES AND WORKS ONLY WITH
MASTERTICKETS AND SUBTICKETS. IT DOES NOT YET WORK WITH CHILDTICKETS OR
OTHER RELATIONSHIP PLUGINS.

This doesn't do any rescheduling. It just notes how many tickets need
to be rescheduled and how long it took to find them.

comment:69 Changed 18 months ago by ChrisNelson

(In [12749]) Recompute schedule and save changes (if any) in a private table. Refs #9648.

  • All datetimes used in schedule are now in local time zone.
    • Had to be "offset-aware" (to use to_utimestamp() utility function)
  • TracPM.start(), .finish() now return calculated date, then stored date, then None depending on what is available.
  • postQuery() now reads schedule from private table rather than having it queried from custom fields
  • When the rescheduler is done calculating a schedule it uses INSERT or UPDATE to write only changed tickets back to the private schedule table.
  • Changes to the schedule are stored in the schedle_history table.

This handles inserting initial schedule into DB and updating only changed tickets when rescheduling.

With the changes to TracPM.start(), .finish(), a caller (e.g., the Gantt chart) can:

  • Query tickets
  • Optionally schedule them
  • Iterate over them getting the start and/or finish dates

If the scheduling step is skipped, the dates from the database are used.

There's a potentially inefficiency in how potentially inactive tickets
are found. We might get this set by first getting all the affected
tickets then following the predecessor links from the goal but there
isn't a function for that yet and preQuery() is fairly efficient. I
don't want to optimize prematurely.

For this to work, I have to splice the in-memory ticket graph based on
ticket changes.

This has some gross code that is specific to plugins we use and makes
TracPM less portable but I want to prove this works before trying to
make it generic.

comment:70 Changed 18 months ago by ChrisNelson

(In [12750]) Get actual start and finish for active and closed tickets. Refs #9648, #10717.

Use these dates in scheduling.

Note that closed tickets don't require resource leveling, display their
actual start and finish times.

This defaults to disabled (0) and can be controlled by useActuals option in
the TracPM section of trac.ini (1=use actual dates, 0=schedule). The
useActuals parameter to the Gantt chart macro overrides the TracPM value.

comment:71 Changed 18 months ago by ChrisNelson

(In [12753]) Add support for scheduled=1 to TracPM.query(). Refs #9648.

Allow retrieval of only tickets that have a precomputed schedule in the
database.

comment:72 Changed 18 months ago by ChrisNelson

(In [12754]) Scheduler more flexible about missing/defaulted options. Refs #9648.

The Gantt chart passes a bunch of options that come from the macro
invocation or the chart configuration in Trac.ini. A lighter weight
client -- like the ticket change listener -- may not have all those
options and just wants default behaviors. Some missing values weren't
handled as defaults before.

comment:73 Changed 18 months ago by ChrisNelson

(In [12755]) Allow Gantt to request no scheduling (just show db values). Refs #9648.

This allows TracJSGanttChart(scheduled=1,schedule=none) to show tasks as
they are in the schedule database.

comment:74 Changed 18 months ago by ChrisNelson

(In [12757]) Improve "omitMilestones" handling. Refs #9648.

comment:75 Changed 18 months ago by ChrisNelson

(In [12758]) Simplify Trac.pm.query() interface. Refs #9648.

comment:76 Changed 18 months ago by ChrisNelson

(In [12759]) Move sort function to match master branch. Refs #9648.

comment:77 Changed 18 months ago by ChrisNelson

(In [12760]) More consistent use of t.get(field) vs. t[field]. Refs #9648.

comment:78 Changed 18 months ago by ChrisNelson

(In [12762]) White space. Refs #9648.

comment:79 Changed 18 months ago by ChrisNelson

(In [12763]) Refine background ticket rescheduler. Refs #9648.

Move the exclusion of milestone pseudo-tickets closer to where it makes
a difference (when we save the data).

This should be a no-op most of the time but without this the ticket set
sometimes has dangling references which makes the plugin fragile when
adding new function invocations in certain places.

comment:80 Changed 18 months ago by ChrisNelson

(In [12764]) Bug in start/finish option experiment. Refs #9845, #9648.

comment:81 Changed 18 months ago by ChrisNelson

(In [12969]) Background rescheduler sometimes fails to handle dependencies. Refs #9648.

comment:82 Changed 13 months ago by falkb

Well, after a long time I've updated this Gantt chart plugin again. But now I came across those errors. Maybe you could have a look at this:

[2013-08-28 15:19:08,296 p2464:t4356] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2772, in rescheduleTickets
    tickets = self.queryTickets(ids)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2568, in queryTickets
    return self.pm.query(options, set())
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 1212, in query
    self.postQuery(options, tickets)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 1157, in postQuery
    self.getTicketDates(tickets)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 950, in getTicketDates
    ids)
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: too many SQL variables
[2013-08-28 15:21:06,483 p2464:t4172] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2929, in rescheduleTickets
    values)
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: near ",": syntax error
[2013-08-28 15:22:28,828 p2060:t3604] main.py:549 ERROR: Internal Server Error: 
Traceback (most recent call last):
  File "build\bdist.win32\egg\trac\web\main.py", line 497, in _dispatch_request
    dispatcher.dispatch(req)
  File "build\bdist.win32\egg\trac\web\main.py", line 214, in dispatch
    resp = chosen_handler.process_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 179, in process_request
    return self._process_ticket_request(req)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 614, in _process_ticket_request
    self._do_save(req, ticket, action)
  File "build\bdist.win32\egg\trac\ticket\web_ui.py", line 1328, in _do_save
    replyto=req.args.get('replyto'))
  File "build\bdist.win32\egg\trac\ticket\model.py", line 365, in save_changes
    listener.ticket_changed(self, comment, author, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2969, in ticket_changed
    self.rescheduleTickets(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2757, in rescheduleTickets
    ids = self._findAffected(ticket, old_values)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2459, in _findAffected
    toExplore = more(border)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2418, in more
    newOwners = ownersOf(n)
  File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))
  File "build\bdist.win32\egg\trac\db\util.py", line 65, in execute
    return self.cursor.execute(sql_escape_percent(sql), args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 78, in execute
    result = PyFormatCursor.execute(self, *args)
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 56, in execute
    args or [])
  File "build\bdist.win32\egg\trac\db\sqlite_backend.py", line 48, in _rollback_on_error
    return function(self, *args, **kwargs)
OperationalError: too many SQL variables

comment:83 follow-up: Changed 13 months ago by ChrisNelson

Are using the background ticket rescheduler? If not, try disabling it.

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

I wonder if what you're seeing is another instance related to #11287.

comment:84 in reply to: ↑ 83 ; follow-up: Changed 13 months ago by falkb

Replying to ChrisNelson:

Are using the background ticket rescheduler? If not, try disabling it.

yes, I activated all of the plugin components as I noticed you've added new ones. Which ones are necessary?

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

the latest SVN revision. I just had some small local patches but they should not matter in this case.

comment:85 in reply to: ↑ 84 ; follow-up: Changed 13 months ago by ChrisNelson

Replying to falkb:

Replying to ChrisNelson:

Are using the background ticket rescheduler? If not, try disabling it.

yes, I activated all of the plugin components as I noticed you've added new ones. Which ones are necessary?

The rescheduler is definitely optional, does nothing to make the Gantt work. I wish I could make it disabled by default but people who have tracjsgantt.* enabled get it when they update. (What is does is, whenever a schedule-related ticket change occurs -- ownership, estimate, etc. -- it reschedules all the related tickets and saves the schedule in a private table. We use that to drive our Task List reports but it's insufficiently documented for the rest of the world.)

I can't tell what version you're running. Is it the most recent (r13368)? I don't see that code on those lines in that version.

the latest SVN revision. I just had some small local patches but they should not matter in this case.

Oh, OK.

Can you send me a snippet that lets me know what

File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))

refers to?

comment:86 Changed 13 months ago by ChrisNelson

Yes, that's definitely related to #11287.

comment:87 in reply to: ↑ 85 Changed 13 months ago by falkb

Replying to ChrisNelson:

The rescheduler is definitely optional, does nothing to make the Gantt work.

OK, now I've activated only

  • TracJSGanttChart
  • TracJSGanttSupport
  • ResourceScheduler
  • SimpleCalendar
  • TracPM

The other components are off. I hope this is a valid combination.

Oh, OK.

My patch is that I also support versions beside milestones as schedule points in the chart (your internal milestones)

Can you send me a snippet that lets me know what

File "build\bdist.win32\egg\tracjsgantt\tracpm.py", line 2338, in ownersOf
    list(ids))

refers to?

    def _findAffected(self, ticket, old_values):
        # Helper to find owners of tickets
        # @param ids set of tickets ID strings
        # @return set of owner strings
        def ownersOf(ids):
            db = self.env.get_db_cnx()
            inClause = "IN (%s)" % ','.join(('%s',) * len(ids))
            cursor = db.cursor()
            cursor.execute(("SELECT DISTINCT owner FROM ticket "
                            "WHERE id " + 
                           inClause),
L2338                      list(ids))
            owners = [row[0] for row in cursor]
            return set(owners)

comment:88 Changed 13 months ago by ChrisNelson

See http://trac-hacks.org/ticket/11287#comment:4 (forgot a # in the commit message).

comment:89 Changed 12 months ago by ChrisNelson

In 13401:

Trivial white space change. Refs #9648.

Need to test my ability to push to T-H.

comment:90 Changed 6 months ago by ChrisNelson

In 13811:

Clean up .gitignore Refs #9648

comment:91 Changed 6 months ago by ChrisNelson

In 13812:

Don't consider group start, finish when resource leveling. Refs #9648.

Consider one group and separate task. All tasks are one unit of work.

  • Group A is owned by Mickey
  • Tasks B, C, and D are children of A. All are owned by Minnie
  • Task E is owned by Mickey and has no dependencies.

An ASAP scheduler would try to schedule, which requires scheduling B,
then C, then D. When resource leveling, they will happen one after the
other. The way the existing algorithm bubbles up finish dates, Group A
gets a finish of 3 and that is assigned to the group owner, Mickey.

Now the scheduler tries to schedule E. Checking the resource limits, it
thinks Mickey isn't available until 3 and scheduled E from 3 to 4,
leaving Mickey idle for three units at the start of the schedule.

Since there is no scheduled work in a group, the owner's availability is
not affected by the work done in the children. This change skips
consideration and update resource limits if a task has children.

comment:92 Changed 6 months ago by ChrisNelson

In 13818:

Refactor body of ALAP function into a helper. Refs #9648.

This is the first step in making the logic common between ALAP and ASAP.

comment:93 Changed 6 months ago by ChrisNelson

In 13819:

Make ancestor limit function a parameter. Refs #9648.

comment:94 Changed 6 months ago by ChrisNelson

In 13820:

Make dependent limit function a parameter. Refs #9648.

comment:95 Changed 6 months ago by ChrisNelson

In 13821:

Change 'finish' to 'from' Refs #9648.

The ALAP scheduler works *from* the finish toward the start. The ASAP
scheduler works from the start toward the finish. "from" is always the
"leading" edge of a task determined by other constraints.

comment:96 Changed 6 months ago by ChrisNelson

In 13822:

Change 'start' to 'to'. Refs #9648.

The ALAP scheduler works from finish *to* start. The ASAP scheduler
works from start *to* finish. "to" is always the trailing edge of the
task, computed from "from" and the task duration.

comment:97 Changed 6 months ago by ChrisNelson

In 13823:

Generic task date parser. Refs #9648.

Sometimes we parse start, sometimes finish.

comment:98 Changed 6 months ago by ChrisNelson

In 13824:

Make limit comparison a function parameter. Refs #9648.

For ALAP we update resource limits going from finish to start. The
direction of comparision between a possible date and the resource limit
has a different sense than for ASAP; one uses < and > and the other uses

and <. We can't pass operators to the generic schedule function so we
pass a comparison function.

comment:99 Changed 6 months ago by ChrisNelson

In 13825:

Make logs, comments generic. Refs #9648.

Take "start" and "finish" out of text.

Also clarify resource leveling criteria.

comment:100 Changed 6 months ago by ChrisNelson

In 13826:

Make end of day wrapping a function parameter. Refs #9648.

An ALAP schedule wraps from the beginning of one day to the end of the
previous. An ASAP schedule wraps from the end of one to the beginning
of the next. This function encapsulates that logic.

comment:101 Changed 6 months ago by ChrisNelson

In 13827:

Add Doxygen to _schedule_task(). Refs #9648.

comment:102 Changed 6 months ago by ChrisNelson

In 13828:

Refactor _schedule_task_asap() using parameterized scheduler. Refs #9648.

All the heaving lifting was already done for ALAP.

comment:103 Changed 6 months ago by ChrisNelson

In 13830:

Clarify an option description. Refs #9648.

comment:104 Changed 6 months ago by ChrisNelson

In 13832:

Add a lot of logging (it's off by default). Refs #9648.

This is controlled by self.logEnabled which comes from logScheduling
in the [TracPM] section of trac.ini. It defaults to disabled if not
set.

comment:105 Changed 6 months ago by ChrisNelson

In 13837:

Use numerical precedence levels for start/finish sources. Refs #9648.

Before a date was either explicit or not. Values in the database
(actual start/finish dates, previously scheduled values, and explicit
user-supplied values) were all explicit so a test for "better" between
an old schedule table value and a new actual value were not resolved.
Now each source of a date has a different precedence so we have enough
information to resolve a conflict.

The way dates were calculated, we should not have seen conflicts in
new values, only in the explicit ones. This change allows an actual
start to be higher precedence than a previously-scheduled finish.
However, there is still the potential conflict between items of the
same precedence.

  • Two actual values should never be wrong (we can't finish before we start)
  • Two scheduled values should never be wrong if we never save bad values
  • If two user-supplied values are wrong, it's data entry, not

algorithm. We could validate this away but we don't currently even
allow explicit start dates.

comment:106 Changed 6 months ago by ChrisNelson

In 13838:

Factor out schedule and schedule_change update. Refs #9648.

This makes the code which finds affected tickets clearer, closer.

comment:107 Changed 6 months ago by ChrisNelson

In 13839:

Rewrite algorithm to identify tickets to reschedule. Refs #9648.

This is a shorter, cleaner algorithm:

  • Find what's active now
  • Find what was active
  • wasActive - isActive should be idled
  • isActive should be rescheduled
    • Save only if schedule changes

It also works. ;-) With the old algorithm if you has two goals scheduled
and interrupted the earlier one, the tasks for the second goal weren't
moved up. The code to find active and idled tickets was weak and
fragile and not worth fixing since it was also slow. In general, this
new approach handles a lot less data, defers getting ticket details as
long as possible, and gets them for many fewer tickets. In my
playground, I used to query 150 tickets (basically, everything in the
database) and now I query 4-5 tickets (the active set). I expect that
ratio to scale to production. Since querying for ticket details was the
longest part of rescheduling, this is much, much faster than the old
approach.

comment:108 Changed 6 months ago by ChrisNelson

In 13840:

Factor our graph repair. Refs #9648.

comment:109 Changed 6 months ago by ChrisNelson

In 13841:

Simplify interface to prune and repair routines. Refs #9648.

comment:110 Changed 6 months ago by ChrisNelson

In 13842:

Don't schedule closed tickets in background rescheduler. Refs #9648.

Also add some logging.

Add Comment

Modify Ticket

Action
as assigned .
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.