Encountered an KeyError while running repo search using Trac .12 version RepoSearchPlugin

[Chris Galante]

Oops… Trac detected an internal error: KeyError: This is probably a local installation issue.

Found a bug in Trac? If you think this should work and you can reproduce the problem, you should consider creating a bug report.

Note that the tracreposearch plugin seems to be involved. Please report this issue to the plugin maintainer.

Before you do that, though, please first try searching for similar issues, as it is quite likely that this problem has been reported before. For questions about installation and configuration of Trac or its plugins, please try the mailing list instead of creating a ticket.

While doing a GET operation on /search, Trac issued an internal error.


(please provide additional details here)


Request parameters:


{'q': u'setup.py', 'repo': u'on', 'noquickjump': u'1'}
User agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; MS-RTC LM 8; SFIEAUTH1)


System Information

Trac  0.12.1  
CustomFieldAdmin  0.2.2  
Genshi  0.6  
mod_wsgi  3.2 (WSGIProcessGroup WSGIApplicationGroup %{GLOBAL})  
psycopg2  2.2.2  
Pygments  1.4  
Python  2.4.3 (#1, Jun 11 2009, 14:09:37) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-44)]  
RPC  1.0.6  
setuptools  0.6c5  
Subversion  1.6.11 (r934486)  
jQuery  1.4.2  

Enabled Plugins

AnnouncerPlugin  0.2  
BreadCrumbsNavPlugin  0.1  
FlexibleAssignTo  0.6.20100524sf  
GetValidOwnerProvider  0.1.20100523sf  
GridModify  0.1.5  
HudsonTrac  0.3  
IniAdmin  0.2  
RepositoryHookSystem  0.1.1  
siteupload  0.11dev  
SvnAuthzAdminPlugin  0.1.2p  
TicketCreationStatus  0.1  
TicketImport  0.8  
ticketvalidator.admin  N/A  
ticketvalidator.core  N/A  
TracCustomFieldAdmin  0.2.2  
TracDiscussion  0.7  
TracProgressMeterMacro  0.2  
tracreposearch  0.2  
TracXMLRPC  1.0.6  
TypedTicketWorkflow-plugin  0.1dev  

Python Traceback

Traceback (most recent call last):
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 511, in _dispatch_request
  File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 237, in dispatch
  File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 96, in process_request
  File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 200, in _do_search
  File "build/bdist.linux-x86_64/egg/tracreposearch/search.py", line 80, in get_search_results
  File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 97, in wrap
  File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 247, in reindex
  File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 229, in _invalidate_file
  File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 42, in __getitem__
  File "/usr/lib64/python2.4/bsddb/__init__.py", line 219, in __getitem__
    return self.db[key]
KeyError: ''
}}}  a new bug report describing the problem and explain how to reproduce it.

Python Traceback
Most recent call last: 
File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 511, in _dispatch_request 
Code fragment:
Line  
506     try: 
507         if not env and env_error: 
508             raise HTTPInternalError(env_error) 
509         try: 
510             dispatcher = RequestDispatcher(env) 
511             dispatcher.dispatch(req) 
512         except RequestDone: 
513             pass 
514         resp = req._response or [] 
515   
516     except HTTPException, e: 

Local variables:
Name Value 
dispatcher <trac.web.main.RequestDispatcher object at 0x2aafa853bd50> 
e <exceptions.KeyError instance at 0x2aafa601f710> 
env <trac.env.Environment object at 0x2aafa4d81910> 
env_error None 
req <Request "GET '/search'"> 
resp [] 

File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 237, in dispatch 
Code fragment:
Line  
232                             msg = _('Do you have cookies enabled?') 
233                         raise HTTPBadRequest(_('Missing or invalid form token.' 
234                                                ' %(msg)s', msg=msg)) 
235   
236                 # Process the request and render the template 
237                 resp = chosen_handler.process_request(req) 
238                 if resp: 
239                     if len(resp) == 2: # Clearsilver 
240                         chrome.populate_hdf(req) 
241                         template, content_type = \ 
242                                   self._post_process_request(req, *resp) 

Local variables:
Name Value 
chosen_handler <trac.search.web_ui.SearchModule object at 0x2aafa853b950> 
chrome <trac.web.chrome.Chrome object at 0x2aafa853b1d0> 
err (<class exceptions.KeyError at 0x2aafa0cb0950>, <exceptions.KeyError ... 
handler <trac.search.web_ui.SearchModule object at 0x2aafa853b950> 
req <Request "GET '/search'"> 
self <trac.web.main.RequestDispatcher object at 0x2aafa853bd50> 

File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 96, in process_request 
Code fragment:
Line  
91             if query.startswith('!'): 
92                 query = query[1:] 
93   
94             terms = self._parse_query(req, query) 
95             if terms: 
96                 results = self._do_search(req, terms, filters) 
97                 if results: 
98                     data.update(self._prepare_results(req, filters, results)) 
99   
100         add_stylesheet(req, 'common/css/search.css') 
101         return 'search.html', data, None 

Local variables:
Name Value 
available_filters [('discussion', u'Discussion'), ('wiki', 'Wiki'), ('ticket', 'Tickets'), ... 
data {'quickjump': None, 'query': u'setup.py', 'results': [], 'filters': ... 
filters ['repo'] 
query u'setup.py' 
req <Request "GET '/search'"> 
self <trac.search.web_ui.SearchModule object at 0x2aafa853b950> 
source <tracreposearch.search.TracRepoSearchPlugin object at 0x2aafa41b4410> 
terms [u'setup.py'] 

File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 200, in _do_search 
Code fragment:
Line  
195                            num=self.min_query_length)) 
196   
197     def _do_search(self, req, terms, filters): 
198         results = [] 
199         for source in self.search_sources: 
200             results.extend(source.get_search_results(req, terms, filters) 
201                            or []) 
202         return sorted(results, key=lambda x: x[2], reverse=True) 
203   
204     def _prepare_results(self, req, filters, results): 
205         page = int(req.args.get('page', '1')) 

Local variables:
Name Value 
filters ['repo'] 
req <Request "GET '/search'"> 
results [] 
self <trac.search.web_ui.SearchModule object at 0x2aafa853b950> 
source <tracreposearch.search.TracRepoSearchPlugin object at 0x2aafa41b4410> 
terms [u'setup.py'] 

File "build/bdist.linux-x86_64/egg/tracreposearch/search.py", line 80, in get_search_results 
Code fragment:
Line  
75   
76         # Use indexer if possible, otherwise fall back on brute force search. 
77         try: 
78             from tracreposearch.indexer import Indexer 
79             self.indexer = Indexer(self.env) 
80             self.indexer.reindex() 
81             walker = lambda repo, query: [repo.get_node(filename) for filename 
82                                           in self.indexer.find_words(query)] 
83         except TracError, e: 
84             self.env.log.warning(e) 
85             self.env.log.warning('Falling back on full repository walk') 

Local variables:
Name Value 
Indexer <class tracreposearch.indexer.Indexer at 0x2aafa7b05890> 
db <trac.db.pool.PooledConnection object at 0x2aafa84cdd20> 
excludes [u'*.pyc', u'*.png', u'*.jpg', u'*.gif'] 
filters ['repo'] 
include [u'*.py', u'*.txt'] 
q u'setup.py' 
query [u'setup.py'] 
repo <trac.versioncontrol.svn_fs.SvnCachedRepository object at 0x2aafa84e7290> 
req <Request "GET '/search'"> 
self <tracreposearch.search.TracRepoSearchPlugin object at 0x2aafa41b4410> 
to_unicode <bound method Mimeview.to_unicode of <trac.mimeview.api.Mimeview object at ... 

File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 97, in wrap 
Code fragment:
Line  
92 def synchronized(f): 
93     """ Synchronization decorator. """ 
94     def wrap(*args, **kw): 
95         acquire_lock() 
96         try: 
97             return f(*args, **kw) 
98         finally: 
99             release_lock() 
100     return wrap 
101   
102 class Indexer: 

Local variables:
Name Value 
args (<tracreposearch.indexer.Indexer instance at 0x2aafa7ae0128>,) 
f <function reindex at 0x2aaaaabf85f0> 
kw {} 

File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 247, in reindex 
Code fragment:
Line  
242         for node in TracRepoSearchPlugin(self.env).walk_repo(self.repo): 
243             if node.kind != Node.DIRECTORY: 
244                 # Node has changed? 
245                 if int(self.revs.get(node.path.encode('utf-8'), -1)) != node.rev: 
246                     self.env.log.debug("Reindexing %s" % node.path) 
247                     self._invalidate_file(node.path) 
248                     self._reindex_node(node) 
249             new_files.add(node.path) 
250          
251         # All files that don't match the new filter criteria must be purged 
252         # from the index 

Local variables:
Name Value 
new_files set([u'TracFlexibleAssignToPlugin/tags/LTS-TracFlexibleAssignToPlugin-0.6.0 ... 
node <trac.versioncontrol.svn_fs.SubversionNode object at 0x2aafa84d9890> 
self <tracreposearch.indexer.Indexer instance at 0x2aafa7ae0128> 
start 1296228220.4522619 

File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 229, in _invalidate_file 
Code fragment:
Line  
224     _reindex_node = _reindex_node 
225   
226     def _invalidate_file(self, file): 
227         if file in self.files: 
228             for word in self.files[file]: 
229                 word_files = self.words[word] 
230                 word_files.discard(file) 
231                 self.words[word] = word_files 
232             self.env.log.debug("Invalidated stale index entry %s" % file) 
233   
234     def reindex(self): 

Local variables:
Name Value 
file u'TracProgressMeterMacro/branches/0.12.1/setup.py' 
self <tracreposearch.indexer.Indexer instance at 0x2aafa7ae0128> 
word u'' 

File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 42, in __getitem__ 
Code fragment:
Line  
37   
38     def __getitem__(self, key): 
39         key = key.encode('utf-8') 
40         if key in self._cache: 
41             return self._cache[key] 
42         return self._cache.setdefault(key, set(self.dbm[key].decode('utf-8').split(pathsep))) 
43   
44     def __setitem__(self, key, value): 
45         key = key.encode('utf-8') 
46         self._cache[key] = self._flush[key] = value 
47   

Local variables:
Name Value 
key '' 
self <tracreposearch.indexer.psetdict object at 0x2aafa6058f90> 

File "/usr/lib64/python2.4/bsddb/__init__.py", line 219, in __getitem__ 
Code fragment:
Line  
214         self._checkOpen() 
215         return len(self.db) 
216   
217     def __getitem__(self, key): 
218         self._checkOpen() 
219         return self.db[key] 
220   
221     def __setitem__(self, key, value): 
222         self._checkOpen() 
223         self._closeCursors() 
224         self.db[key] = value 

Local variables:
Name Value 
key '' 
self {'+189,13': ... 

File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 511, in _dispatch_request
  dispatcher.dispatch(req)
File "build/bdist.linux-x86_64/egg/trac/web/main.py", line 237, in dispatch
  resp = chosen_handler.process_request(req)
File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 96, in process_request
  results = self._do_search(req, terms, filters)
File "build/bdist.linux-x86_64/egg/trac/search/web_ui.py", line 200, in _do_search
  results.extend(source.get_search_results(req, terms, filters)
File "build/bdist.linux-x86_64/egg/tracreposearch/search.py", line 80, in get_search_results
  self.indexer.reindex()
File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 97, in wrap
  return f(*args, **kw)
File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 247, in reindex
  self._invalidate_file(node.path)
File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 229, in _invalidate_file
  word_files = self.words[word]
File "build/bdist.linux-x86_64/egg/tracreposearch/indexer.py", line 42, in __getitem__
  return self._cache.setdefault(key, set(self.dbm[key].decode('utf-8').split(pathsep)))
File "/usr/lib64/python2.4/bsddb/__init__.py", line 219, in __getitem__
  return self.db[key]


System Information:
User Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.5.30729; MS-RTC LM 8; SFIEAUTH1)

Trac 0.12.1 
CustomFieldAdmin 0.2.2 
Genshi 0.6 
mod_wsgi 3.2 (WSGIProcessGroup WSGIApplicationGroup %{GLOBAL}) 
psycopg2 2.2.2 
Pygments 1.4 
Python 2.4.3 (#1, Jun 11 2009, 14:09:37) [GCC 4.1.2 20080704 (Red Hat 4.1.2-44)] 
RPC 1.0.6 
setuptools 0.6c5 
Subversion 1.6.11 (r934486) 
jQuery 1.4.2 

Enabled Plugins:
AnnouncerPlugin  0.2 /opt/ltsapps/python/site-packages/AnnouncerPlugin-0.2-py2.4.egg 
BreadCrumbsNavPlugin  0.1 /opt/ltsapps/python/site-packages/BreadCrumbsNavPlugin-0.1-py2.4.egg 
FlexibleAssignTo  0.6.20100524sf /opt/ltsapps/python/site-packages/FlexibleAssignTo-0.6.20100524sf-py2.4.egg 
GetValidOwnerProvider  0.1.20100523sf /opt/ltsapps/python/site-packages/GetValidOwnerProvider.pyc 
GridModify  0.1.5 /opt/ltsdata/trac/april/plugins/GridModify-0.1.5-py2.4.egg 
HudsonTrac  0.3 /opt/ltsapps/python/site-packages/HudsonTrac-0.3-py2.4.egg 
IniAdmin  0.2 /opt/ltsapps/python/site-packages/IniAdmin-0.2-py2.4.egg 
RepositoryHookSystem  0.1.1 /opt/ltsapps/python/site-packages/RepositoryHookSystem-0.1.1-py2.4.egg 
siteupload  0.11dev /opt/ltsapps/python/site-packages/siteupload-0.11dev-py2.4.egg 
SvnAuthzAdminPlugin  0.1.2p /opt/ltsdata/trac/april/plugins/SvnAuthzAdminPlugin-0.1.2p-py2.4.egg 
TicketCreationStatus  0.1 /opt/ltsapps/python/site-packages/TicketCreationStatus-0.1-py2.4.egg 
TicketImport  0.8 /opt/ltsapps/python/site-packages/TicketImport-0.8-py2.4.egg 
ticketvalidator.admin  N/A /usr/lib/python2.4/site-packages/TicketValidator-0.1.20101010sf-py2.4.egg/ticketvalidator/admin.pyc 
ticketvalidator.core  N/A /usr/lib/python2.4/site-packages/TicketValidator-0.1.20101010sf-py2.4.egg/ticketvalidator/core.pyc 
TracCustomFieldAdmin  0.2.2 /opt/ltsapps/python/site-packages/TracCustomFieldAdmin-0.2.2-py2.4.egg 
TracDiscussion  0.7 /opt/ltsapps/python/site-packages/TracDiscussion-0.7-py2.4.egg 
TracProgressMeterMacro  0.2 /opt/ltsdata/trac/april/plugins/TracProgressMeterMacro-0.2-py2.4.egg 
tracreposearch  0.2 /opt/ltsdata/trac/april/plugins/tracreposearch-0.2-py2.4.egg 
TracXMLRPC  1.0.6 /opt/ltsapps/python/site-packages/TracXMLRPC-1.0.6-py2.4.egg 
TypedTicketWorkflow-plugin  0.1dev /opt/ltsapps/python/site-packages/TypedTicketWorkflow_plugin-0.1dev-py2.4.egg 

TracGuide — The Trac User and Administration Guide 

[Ryan J Ollos]

See also #7046.

[Chris Galante]

We have already consumed this change because we pulled the plugin source from the 0.12 folder.

[Ryan J Ollos]

Replying to cgalante:

We have already consumed this change because we pulled the plugin source from the 0.12 folder.

I should have been more verbose in comment:1, I only wanted to link to the related ticket which has been left open because I wasn't very confident the plugin was working on Trac 0.12. I haven't gotten much feedback in regard to this.

Unfortunately this plugin seems to need quite a bit of work, and I have other priorities at the moment. I would be very happy to apply a patch if you can figure out the issue!

[anonymous]

Below a quick hack that doesn't solve the problem, but cures the symptom: you no longer get the KeyError. It probably makes the search results incomplete. I post here again, when I discover the reason for the KeyError.

$ diff -u indexer.py indexer.py.new
--- indexer.py	2011-07-08 15:22:20.076447519 +0200
+++ indexer.py.new	2011-07-08 15:18:59.399782677 +0200
@@ -39,7 +39,10 @@
         key = key.encode('utf-8')
         if key in self._cache:
             return self._cache[key]
-        return self._cache.setdefault(key, set(self.dbm[key].decode('utf-8').split(pathsep)))
+        if self.dbm.has_key(key):
+            return self._cache.setdefault(key, set(self.dbm[key].decode('utf-8').split(pathsep)))
+        return ''
+
 
     def __setitem__(self, key, value):
         key = key.encode('utf-8')

[Ryan J Ollos]

Hello,

I took over maintainership of this plugin from athomas some time ago. There is a significant amount of work to do on this plugin, and I don't foresee having the time to do it all.

helend has written the TracSuposePlugin, which seems like a much better solution. Rather than writing the repository search functionality from scratch, a Trac interface to an existing repository search tool has been created. Rather than throwing more effort at this plugin, I'd prefer to help helend with enhancements to the TracSuposePlugin, or spend my time on other Trac plugin projects altogether.

I'd like to get some feedback and hear if anyone knows of a compelling reason to continue this project rather than moving to the TracSuposePlugin. Is there functionality in this plugin that doesn't exist in the TracSuposePlugin? I'm open to hearing all opinions and suggestions.

I'll leave these tickets open for about a week, but in all likelihood will close all of them and deprecate the plugin.

Thanks for your time,

• Ryan

[Ryan J Ollos]

Please try MultiRepoSearchPlugin.