[patch] Tags with single quote not tokenised correctly

Confirmed: While the tag-cloud shows correctly formatted tag and object count (1), following the link (here: http://localhost:8000/sandbox/tags/%27single%5C%27quote%27) returns a page with title

Showing objects matching ''single\'quote''

but no matching objects.

Note: tag in tags: name space is rendered differently here (with Trac 0.13dev)

Try with "single'quote"

• in tag search box and hit <Filter Tags>, this seems to work: returned page (http://localhost:8000/sandbox/tags?q=%22single%27quote%22) has the correct object listed, but the back-link to the tag is still wrong (http://localhost:8000/sandbox/tags/%27single%5C%27quote%27)
• [[ListTagged("single'quote")]] works here as well.

Note: single double-quotes instead of pair of single quotes like above

While resolving other issues I've been stumbling over the rather strange quoting that is done by this plugin.

This is certainly connected to the behavior, that has been reported here. I'll let discussion progress a little more, but I suspect, that the sanest solutions will be to dump this silly quoting once and for all.

If a query contains \' (backslash single-quote) or ' (single-quote), it seems Query class generates wrong tokens.

>>> from tractags.query import Query
>>> Query(r"'single\'quote'")
("single\'quote")
>>> Query(r"single'quote")
(and
  ("single")
  ("quote"))

I think that the class should generate ("single'quote") from the both.

TagsPlugin with tagsplugin-r10682.diff does as the following, [[ListTagged(single'quote)]] and /tags?q='single%5C'quote' works.

>>> from tractags.query import Query
>>> Query(r"'single\'quote'")
("single'quote")
>>> Query(r"single'quote")
("single'quote")

Seems like a fairly elaborated fix. Even with test case, thank you very much. I'll check this.

(In [10784]) TagsPlugin: Correct tokenization for the 'or' operator, refs #3983 and #9057.

The regular expression introduced in [7378] is corrected now for tokenization to finally match expectations with respect to the particular behavior: All included tests pass now. Thanks to Jun Omae for the patch.

Replying to hasienda:

While resolving other issues I've been stumbling over the rather strange quoting that is done by this plugin. This is certainly connected to the behavior, that has been reported here. [...]

Disagree now with myself, but or because I've found #3624 to be an issue clearly related to tag quoting in URLs.

Anyway, I'm glad Jun's patch provides the fix for the tokenization issue reported here, as I've verified now on my own.

(In [10785]) TagsPlugin: Correct tokenization for unary quotes, refs #3624 and #9057.

TagsQuery now has got a unit test in addition to the rather old doc tests. Thanks to Jun Omae for contributing most of the code to this changeset.

Jun, would you comment on the changes your patch attempts on tractags/tests/__init__.py, please? I figure this could provide an alternative way to call the test suite, but can't figure out how exactly this should work. Tests are running fine without it, so this his really optional, right?

Replying to hasienda:

Jun, would you comment on the changes your patch attempts on tractags/tests/__init__.py, please? I figure this could provide an alternative way to call the test suite, but can't figure out how exactly this should work.

Hi Steffen,

If the tests with python tractags/tests/__init__.py, no tests run.

jun66j5@gotanda:745$ PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python tractags/tests/__init__.py

With the patch;

jun66j5@gotanda:738$ PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python tractags/tests/__init__.py
....................
----------------------------------------------------------------------
Ran 20 tests in 0.525s

OK

Tests are running fine without it, so this his really optional, right?

Right. I just understand that the tests run with setup.py test, now.

jun66j5@gotanda:742$ PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python setup.py test
running test
running egg_info
writing requirements to TracTags.egg-info/requires.txt
... (snip)
test_matches (tractags.tests.web_ui.TagRequestHandlerTestCase) ... ok
test_matches_no_permission (tractags.tests.web_ui.TagRequestHandlerTestCase) ... ok
test_init (tractags.tests.wiki.WikiTagProviderTestCase) ... ok

----------------------------------------------------------------------
Ran 20 tests in 0.525s

OK

(In [10788]) TagsPlugin: Add an alternative way to start the test suite, refs #9057.

This is a leftover from Jun's patch and allows to start a full test run with minimal output by just calling tractags/tests/__init__.py.

(In [13134]) TagsPlugin: Remove query string for simple single-tag URLs, refs #9057.

As I noticed some time ago, this is actually work to remove a regression, because it has actually been the standard behavior before tags-0.6 too.