Ticket #388 (assigned enhancement)

Opened 2 years ago

Last modified 1 year ago

ability to to perform re-index with a quota of time / memory consumption

Reported by: patrickm Assigned to: athomas (accepted)
Priority: normal Component: RepoSearchPlugin
Severity: normal Keywords:
Cc: Trac Release: 0.9

Description

I don't think this is an easy one, but the time and resources required to re-index a large number of changes are quite significant.

Is it possible for a re-index to index a configurable number of files on each invocation? This would leave the repository search out of date, but release the server to serve the requests of other users. Susbsequent re-indexes could fill in the gap.

I'm not sure how valuable this would be if #365 were implemented, as one huge cost to re-indexing (tagging and branching) could be radically reduced. Nonetheless, repository search is transparently such a great feature is would be a shame if it became a victim of its own success when people with large-ish source trees (>1K files) have a go at using it.

Attachments

Change History

05/25/06 04:42:18 changed by athomas

  • status changed from new to assigned.

The massive consumption of memory during indexing is a definite issue. In the near future I will be porting RepoSearchPlugin to use the pyndexter indexer abstraction layer. After this is complete I will probably rewrite the default indexer (essentially what is currently used in this plugin) as a C extension. These two changes should alleviate most performance related problems, particularly if #365 is implemented.

05/27/06 18:00:52 changed by patrickm

This sounds like great stuff.

Can't wait.

01/18/07 07:33:00 changed by athomas

(In [1881]) Not for the faint of heart, but if anybody wants to test the pyndexter branch it would be most useful to get some feedback.

You will need the refactoring branch of pyndexter, available from here.

More info in the README.

References #362, #371, #385 and #388.

Add/Change #388 (ability to to perform re-index with a quota of time / memory consumption)




Change Properties
Action