ListTagged broken for tags starting with 'or'

[markh@…]

New Trac 0.11 install with TagsPlugin, TracAccountManager, Pygments, and TracTocMacro. I've created Tracs with both MySQL (5.1) and SQLite backends to verify it wasn't the database.

I was tagging entries with several tags, such as 'database docs howto debian4 oracle', and using ListTagged() works or all of the tags but 'oracle', which returns an empty list. I then tried 'orwich' as a tag to test, as well as 'racle', the former failing to show up and the latter working. Subsequent tests show that other tags beginning with 'or' fail to show up in a ListTagged list.

[BryanTsai]

We have also encountered the same problem here. tag 'oracle' does not work on our Trac site.

[BryanTsai]

It seems to be an regex problem. I've made the following change on my environment and it worked fine so far:

Index: tractags/query.py
===================================================================
--- tractags/query.py   (revision 4775)
+++ tractags/query.py   (working copy)
@@ -140,7 +140,7 @@

     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or)|
+        (?P<or>or\s+)|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

[otaku42]

(In [7374]) Fix bug where tags starting with "or", when passed as parameter to the
ListTagged macro, did not return any result. Thanks to BryanTsai for
the patch. Fixes #3983.

[otaku42]

Replying to otaku42:

(In [7374]) Fix bug where tags starting with "or", when passed as parameter to the
ListTagged macro, did not return any result. Thanks to BryanTsai for
the patch. Fixes #3983.

Guess I had a bad day:

I accidentally committed stuff to the tagged release v0.6 rather than to trunk, which of course was wrong. I'll thus roll back r7374 and re-apply the patch to trunk instead.

[otaku42]

(In [7378]) Fix bug where tags starting with "or", when passed as parameter to the
ListTagged macro, did not return any result. Thanks to BryanTsai for
the patch. Fixes #3983.

[osimons]

I'll reopen this actually, as it breaks the tests for the plugin. "Which tests?", I hear you say... :-) Well, query.py actually comes with doctests, and this one breaks:

----------------------------------------------------------------------
File "trunk/tractags/query.py", line 380, in tractags.query.Query._tokenise
Failed example:
    q._tokenise('one or two')
Expected:
    [(1, 'one'), (4, 'or'), (1, 'two')]
Got:
    [(1, 'one'), (4, 'or '), (1, 'two')]

----------------------------------------------------------------------

Either the fix or the test is wrong.

[hasienda]

See #3624 for another issue related to parsing/quoting.

[hasienda]

The results created with the patch applied are not exactly right, so it is certainly not awfully wrong, but still wrong.

I had a lookahead assertion in mind like this:

tagsplugin/trunk/tractags/query.py

@@ -140 +140 @@
 
     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or\s+)|
+        (?P<or>or(?=\s+))|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

but later spotted Jun Omae's solution included in his patch for #9057, that looks more elegant to my eyes:

tagsplugin/trunk/tractags/query.py

@@ -140 +140 @@
 
     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or\s+)|
+        (?P<or>or)\s+|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

Thank you Odd for your assistance with the test infrastructure for this plugin. Obviously this is already leading to improved code quality.

[hasienda]

(In [10784]) TagsPlugin: Correct tokenization for the 'or' operator, refs #3983 and #9057.

The regular expression introduced in [7378] is corrected now for tokenization
to finally match expectations with respect to the particular behavior:
All included tests pass now. Thanks to Jun Omae for the patch.