Modify

Opened 11 years ago

Closed 11 years ago

Last modified 10 years ago

#630 closed enhancement (fixed)

Reduce memory consumption of initial cache population

Reported by: Lewis Baker Owned by: Lewis Baker
Priority: high Component: PerforcePlugin
Severity: normal Keywords:
Cc: Trac Release: 0.10

Description

The current cache syncing logic attempts to pre-fetch all change information into memory using batched Perforce commands before inserting new records into the SQL database.

Whilst this method improves performance of incremental updates to the cache it tends to consume large amounts of memory when performing the initial cache population of a large repository (>20K changes) which can swamp the web server machine making the initial cache population to take longer.

It would be better to use some form of incremental batch processing, clearing cached data in-between batches, to reduce the total amount of memory required to perform the initial cache population.

Attachments (0)

Change History (2)

comment:1 Changed 11 years ago by Lewis Baker

Ticket #639 is a duplicate of this one (patch included).

comment:2 Changed 11 years ago by Lewis Baker

Resolution: fixed
Status: newclosed

I've updated the plugin to perform cache syncing in batches, retrieving information for 1000 changes at a time into memory and committing to the Trac database before purging the in-memory cache.

See version 0.4.2.

Some quick tests on our repository showed a significant decrease in virtual memory usage (from >1GB to <100MB) and time taken (from >1.5 hours to <25 mins).

Thanks to r.blum@… for the initial patch.

Modify Ticket

Action
as closed The owner will remain Lewis Baker.
The resolution will be deleted. Next status will be 'reopened'.

Add Comment


E-mail address and name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.