Modify

Opened 8 years ago

Closed 8 years ago

Last modified 7 years ago

#630 closed enhancement (fixed)

Reduce memory consumption of initial cache population

Reported by: lewisbaker Owned by: lewisbaker
Priority: high Component: PerforcePlugin
Severity: normal Keywords:
Cc: Trac Release: 0.10

Description

The current cache syncing logic attempts to pre-fetch all change information into memory using batched Perforce commands before inserting new records into the SQL database.

Whilst this method improves performance of incremental updates to the cache it tends to consume large amounts of memory when performing the initial cache population of a large repository (>20K changes) which can swamp the web server machine making the initial cache population to take longer.

It would be better to use some form of incremental batch processing, clearing cached data in-between batches, to reduce the total amount of memory required to perform the initial cache population.

Attachments (0)

Change History (2)

comment:1 Changed 8 years ago by lewisbaker

Ticket #639 is a duplicate of this one (patch included).

comment:2 Changed 8 years ago by lewisbaker

  • Resolution set to fixed
  • Status changed from new to closed

I've updated the plugin to perform cache syncing in batches, retrieving information for 1000 changes at a time into memory and committing to the Trac database before purging the in-memory cache.

See version 0.4.2.

Some quick tests on our repository showed a significant decrease in virtual memory usage (from >1GB to <100MB) and time taken (from >1.5 hours to <25 mins).

Thanks to r.blum@… for the initial patch.

Add Comment

Modify Ticket

Action
as closed .
The resolution will be deleted. Next status will be 'reopened'.
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.