ListTagged broken for tags starting with 'or'

[markh@…]

New Trac 0.11 install with TagsPlugin, TracAccountManager, Pygments, and TracTocMacro. I've created Tracs with both MySQL (5.1) and SQLite backends to verify it wasn't the database.

I was tagging entries with several tags, such as 'database docs howto debian4 oracle', and using ListTagged() works or all of the tags but 'oracle', which returns an empty list. I then tried 'orwich' as a tag to test, as well as 'racle', the former failing to show up and the latter working. Subsequent tests show that other tags beginning with 'or' fail to show up in a ListTagged list.

[BryanTsai]

We have also encountered the same problem here. tag 'oracle' does not work on our Trac site.

[BryanTsai]

It seems to be an regex problem. I've made the following change on my environment and it worked fine so far:

Index: tractags/query.py
===================================================================
--- tractags/query.py   (revision 4775)
+++ tractags/query.py   (working copy)
@@ -140,7 +140,7 @@

     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or)|
+        (?P<or>or\s+)|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

[Michael Renzmann]

(In [7374]) Fix bug where tags starting with "or", when passed as parameter to the ListTagged macro, did not return any result. Thanks to BryanTsai for the patch. Fixes #3983.

[Michael Renzmann]

Replying to otaku42:

(In [7374]) Fix bug where tags starting with "or", when passed as parameter to the ListTagged macro, did not return any result. Thanks to BryanTsai for the patch. Fixes #3983.

Guess I had a bad day:

I accidentally committed stuff to the tagged release v0.6 rather than to trunk, which of course was wrong. I'll thus roll back r7374 and re-apply the patch to trunk instead.

[Michael Renzmann]

(In [7378]) Fix bug where tags starting with "or", when passed as parameter to the ListTagged macro, did not return any result. Thanks to BryanTsai for the patch. Fixes #3983.

[osimons]

I'll reopen this actually, as it breaks the tests for the plugin. "Which tests?", I hear you say... :-) Well, query.py actually comes with doctests, and this one breaks:

----------------------------------------------------------------------
File "trunk/tractags/query.py", line 380, in tractags.query.Query._tokenise
Failed example:
    q._tokenise('one or two')
Expected:
    [(1, 'one'), (4, 'or'), (1, 'two')]
Got:
    [(1, 'one'), (4, 'or '), (1, 'two')]

----------------------------------------------------------------------

Either the fix or the test is wrong.

[Steffen Hoffmann]

See #3624 for another issue related to parsing/quoting.

[Steffen Hoffmann]

The results created with the patch applied are not exactly right, so it is certainly not awfully wrong, but still wrong.

I had a lookahead assertion in mind like this:

tagsplugin/trunk/tractags/query.py

@@ -140 +140 @@
 
     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or\s+)|
+        (?P<or>or(?=\s+))|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

but later spotted Jun Omae's solution included in his patch for #9057, that looks more elegant to my eyes:

tagsplugin/trunk/tractags/query.py

@@ -140 +140 @@
 
     _tokenise_re = re.compile(r"""
         (?P<not>-)|
-        (?P<or>or\s+)|
+        (?P<or>or)\s+|
         \"(?P<dquote>(?:\\.|[^\"])*)\"|
         '(?P<squote>(?:\\.|[^'])*)'|
         (?P<startsub>\()|

Thank you Odd for your assistance with the test infrastructure for this plugin. Obviously this is already leading to improved code quality.

[Steffen Hoffmann]

(In [10784]) TagsPlugin: Correct tokenization for the 'or' operator, refs #3983 and #9057.

The regular expression introduced in [7378] is corrected now for tokenization to finally match expectations with respect to the particular behavior: All included tests pass now. Thanks to Jun Omae for the patch.

[Steffen Hoffmann]

In 13815:

TagsPlugin: Completing preparation for v0.7 release.

Availability of that code as stable, tagged release
closes #2429, #3359, #3610, #3624, #3677, #3754, #3864, #3947, #3983, #4078, #4277, #4503, #4799, #5523, #7787, #7857, #8638, #9057, #9058, #9059, #9060, #9061, #9062, #9063, #9149, #9210, #9521, #9630, #9636, #10032, #10416, #10636, #11096, #11147, #11152, #11274, #11302, #11658 and #11659.

Additionally there are some issues and enhancement requests showing progress,
but known to require more work to resolve them satisfactorily, specifically
refs #2804, #4200, #8747 and #9064.

Thanks to all contributors and followers, that enabled and encouraged a good
portion of this development work.