# Ticket #9057 (assigned defect)

Opened 2 years ago

## [patch] Tags with single quote not tokenised correctly

Reported by: Assigned to: anonymous hasienda (accepted) normal TagsPlugin normal HTML links tag quote rjollos, otaku42 0.11

### Description

I added the tag "single'quote" to a wiki page.

The tag is shown in the tag-cloud, but the results for [[ListTagged(single'quote)]] and for /tags?q='single%5C'quote' (which is how the URL is rendered by default - not sure why) are empty.

I debugged this into tagsplugin/trunk/tractags/query.py Query:_tokenise() that seems to split the phrase on a single quote.

## Change History

### 08/03/11 00:59:19 changed by hasienda

• keywords set to HTML links tag quote.

Confirmed: While the tag-cloud shows correctly formatted tag and object count (1), following the link (here: http://localhost:8000/sandbox/tags/%27single%5C%27quote%27) returns a page with title

Showing objects matching ''single\'quote''

but no matching objects.

Note: tag in tags: name space is rendered differently here (with Trac 0.13dev)

Try with "single'quote"

• in tag search box and hit <Filter Tags>, this seems to work: returned page (http://localhost:8000/sandbox/tags?q=%22single%27quote%22) has the correct object listed, but the back-link to the tag is still wrong (http://localhost:8000/sandbox/tags/%27single%5C%27quote%27)
• [[ListTagged("single'quote")]] works here as well.

Note: single double-quotes instead of pair of single quotes like above

### (follow-up: ↓ 6 ) 09/25/11 18:12:04 changed by hasienda

While resolving other issues I've been stumbling over the rather strange quoting that is done by this plugin.

This is certainly connected to the behavior, that has been reported here. I'll let discussion progress a little more, but I suspect, that the sanest solutions will be to dump this silly quoting once and for all.

### 09/25/11 23:54:01 changed by jun66j5

If a query contains \' (backslash single-quote) or ' (single-quote), it seems Query class generates wrong tokens.

>>> from tractags.query import Query
>>> Query(r"'single\'quote'")
("single\'quote")
>>> Query(r"single'quote")
(and
("single")
("quote"))


I think that the class should generate ("single'quote") from the both.

TagsPlugin with tagsplugin-r10682.diff does as the following, [[ListTagged(single'quote)]] and /tags?q='single%5C'quote' works.

>>> from tractags.query import Query
>>> Query(r"'single\'quote'")
("single'quote")
>>> Query(r"single'quote")
("single'quote")


### 09/26/11 21:28:25 changed by hasienda

• status changed from new to assigned.
• summary changed from Tags with single quote not tokenised correctly to [patch] Tags with single quote not tokenised correctly.

Seems like a fairly elaborated fix. Even with test case, thank you very much. I'll check this.

### 10/15/11 15:44:17 changed by hasienda

(In [10784]) TagsPlugin: Correct tokenization for the 'or' operator, refs #3983 and #9057.

The regular expression introduced in [7378] is corrected now for tokenization to finally match expectations with respect to the particular behavior: All included tests pass now. Thanks to Jun Omae for the patch.

### (in reply to: ↑ 2 ) 10/15/11 16:04:43 changed by hasienda

• cc set to rjollos, otaku42.

While resolving other issues I've been stumbling over the rather strange quoting that is done by this plugin. This is certainly connected to the behavior, that has been reported here. [...]

Disagree now with myself, but or because I've found #3624 to be an issue clearly related to tag quoting in URLs.

Anyway, I'm glad Jun's patch provides the fix for the tokenization issue reported here, as I've verified now on my own.

### 10/15/11 22:41:05 changed by hasienda

(In [10785]) TagsPlugin: Correct tokenization for unary quotes, refs #3624 and #9057.

TagsQuery now has got a unit test in addition to the rather old doc tests. Thanks to Jun Omae for contributing most of the code to this changeset.

### (follow-up: ↓ 9 ) 10/15/11 23:06:02 changed by hasienda

Jun, would you comment on the changes your patch attempts on tractags/tests/__init__.py, please? I figure this could provide an alternative way to call the test suite, but can't figure out how exactly this should work. Tests are running fine without it, so this his really optional, right?

### (in reply to: ↑ 8 ) 10/16/11 14:40:35 changed by jun66j5

Jun, would you comment on the changes your patch attempts on tractags/tests/__init__.py, please? I figure this could provide an alternative way to call the test suite, but can't figure out how exactly this should work.

Hi Steffen,

If the tests with python tractags/tests/__init__.py, no tests run.

jun66j5@gotanda:745$PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python tractags/tests/__init__.py


With the patch;

jun66j5@gotanda:738$PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python tractags/tests/__init__.py
....................
----------------------------------------------------------------------
Ran 20 tests in 0.525s

OK


Tests are running fine without it, so this his really optional, right?

Right. I just understand that the tests run with setup.py test, now.

jun66j5@gotanda:742$PYTHONPATH=$PWD ~/venv/trac-0.12/bin/python setup.py test
running test
running egg_info
writing requirements to TracTags.egg-info/requires.txt
... (snip)
test_matches (tractags.tests.web_ui.TagRequestHandlerTestCase) ... ok
test_matches_no_permission (tractags.tests.web_ui.TagRequestHandlerTestCase) ... ok
test_init (tractags.tests.wiki.WikiTagProviderTestCase) ... ok

----------------------------------------------------------------------
Ran 20 tests in 0.525s

OK


### 10/16/11 21:00:24 changed by hasienda

(In [10788]) TagsPlugin: Add an alternative way to start the test suite, refs #9057.

This is a leftover from Jun's patch and allows to start a full test run with minimal output by just calling tractags/tests/__init__.py.

### 05/15/13 01:51:57 changed by hasienda

(In [13134]) TagsPlugin: Remove query string for simple single-tag URLs, refs #9057.

As I noticed some time ago, this is actually work to remove a regression, because it has actually been the standard behavior before tags-0.6 too.

### Add/Change #9057 ([patch] Tags with single quote not tokenised correctly)

Change Properties