ticket.query fails to return all matches

[anonymous]

Notice in the trace below (password obfuscated), an XMLRPC client call to ticket.query for all tickets of status "new" returns an array which does not contain ticket #1. When fetched directly, however, ticket 1 is returned... and it's status is, indeed, new.

irb(main):062:0> client = XMLRPC::Client.new2("http://FOO:BAR@trac.germane-software.com/TimeTracker/login/xmlrpc")
=> #<XMLRPC::Client:0xb6ffbf34 @use_ssl=false, @create=nil, @password="BAR", @http_last_response=nil, @parser=nil, @proxy_port=nil, @user="FOO", @http_header_extra=nil, @port=80, @proxy_host=nil, @auth="Basic XXYYYZZZ==", @timeout=30, @path="/TimeTracker/login/xmlrpc", @host="trac.germane-software.com", @cookie=nil, @http=#<Net::HTTP trac.germane-software.com:80 open=false>>
irb(main):063:0> client.call("ticket.query", "status=new").sort
=> [6, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108]
irb(main):064:0> client.call("ticket.get", "1")
=> ["1", #<XMLRPC::DateTime:0xb6fc6dd4 @sec=39, @month=1, @min=28, @year=2009, @hour=3, @day=29>, #<XMLRPC::DateTime:0xb6fc6690 @sec=16, @month=2, @min=2, @year=2009, @hour=9, @day=4>, {"component"=>"TimeTracker Android", "cc"=>"cupra.kev@googlemail.com", "reporter"=>"www.google.com/accounts/o8/id?id=aitoawkx8pndhvjyausyrhg8m8md8txy_7sju_w", "priority"=>"minor", "os"=>"Linux", "type"=>"enhancement", "version"=>"", "milestone"=>"2009.3", "summary"=>"Back up DB to memory card", "description"=>"Copy the whole DB to the memory card.  Optionally pack the on-phone DB, moving old times to the DB on the memory card.", "status"=>"new", "owner"=>"somebody", "keywords"=>""}]

To eliminate the possibility that the error is in the Ruby XMLRPC library, I've also tried it with the JSONRPC interface:

bean:1121)~% cat body.json 
{"params": ["status=new"], "method": "ticket.query", "id": 123}
bean:1122)~% curl -H "Content-Type: application/json" --data @body.json -u 'FOO:BAR' 'http://trac.germane-software.com/TimeTracker/login/jsonrpc'
{"id": 123, "result": [9, 6, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108], "error": null}

Again, notice that ticket is missing. Actually, there are about 41 tickets missing, as per a count on the server:

haselmaus db # sqlite3 trac.db "select count(*) from ticket where status='new'"
141

[osimons]

Note that your SQLite call is not the same as fetched via web interface with user permissions - your call just looks at the database. However, if anything in plugins (like some that add security policies) restrict users from seeing certain tickets then those tickets will not show through the API either.

You really want to compare with the tickets that show when you do a Custom Query, like /query?status=new - and when logged in as your FOO/BAR user using his/her actual permissions.

[seanerussell@…]

If that were the case, then why is the account able to fetch the missing tickets directly with ticket.get(), as demonstrated in the case?

In any case, I just verified that the tickets show up with a /query?status=new request in the web browser, both with entirely unauthenticated accounts and as the FOO user (who also happens to have admin access); see the attached screen shot.

There is nothing in the logs that would indicate an error on the server.

[seanerussell@…]

Interestingly, this problem only appears when I get more than 100 tickets in the database. When there are 10 tickets, I see ticket #1 in the results from the XMLRPC plugin. If I add 130 more tickets, about 40 of them disappear (including ticket #1).

[seanerussell@…]

More info -- I'm not sure what debugging information would be useful, but here's the behavior I'm seeing. Look for ticket #1.

bean:1393)Work/TimeTracker% curl -H "Content-Type: application/json" --data @list.json -u 'FOO:BAR' $url
{"id": 123, "result": [9, 6, 10, 12, 13, 14, 1, 3, 11, 8], "error": null}
bean:1393)Work/TimeTracker% curl -H "Content-Type: application/json" --data @get.json -u 'FOO:BAR' $url 
{"id": 124, "result": ["1", {"__jsonclass__": ["datetime", "2009-01-29T03:28:39"]}, {"__jsonclass__": ["datetime", "2009-02-04T09:02:16"]}, {"status": "new", "description": "Copy the whole DB to the memory card.  Optionally pack the on-phone DB, moving old times to the DB on the memory card.", "reporter": "xyz", "cc": "cupra.kev@googlemail.com", "os": "Linux", "component": "TimeTracker Android", "summary": "Back up DB to memory card", "priority": "minor", "keywords": "", "version": "", "milestone": "2009.3", "owner": "somebody", "type": "enhancement"}], "error": null}

Now I create 131 new tickets. Notice that the "get" command for 1 returns the identical ticket, so it hasn't changed. If I remove those 130 tickets, 1 shows back up in the list.

bean:1394)Work/TimeTracker% curl -H "Content-Type: application/json" --data @list.json -u 'FOO:BAR' $url
{"id": 123, "result": [9, 6, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108], "error": null}
bean:1395)Work/TimeTracker% curl -H "Content-Type: application/json" --data @get.json -u 'FOO:BAR' $url 
{"id": 124, "result": ["1", {"__jsonclass__": ["datetime", "2009-01-29T03:28:39"]}, {"__jsonclass__": ["datetime", "2009-02-04T09:02:16"]}, {"status": "new", "description": "Copy the whole DB to the memory card.  Optionally pack the on-phone DB, moving old times to the DB on the memory card.", "reporter": "xyz", "cc": "cupra.kev@googlemail.com", "os": "Linux", "component": "TimeTracker Android", "summary": "Back up DB to memory card", "priority": "minor", "keywords": "", "version": "", "milestone": "2009.3", "owner": "somebody", "type": "enhancement"}], "error": null}

As I said, if I delete the 130 new tickets, 1 shows back up again. It isn't just #1; the plugin is losing anything over about 101 results. I haven't tried adding 200 yet, to see if it is some sort of "only return 100 results at a time" thing, but I will do that next.

BTW, I had to obfuscate all of the URLs in this trace, because the anti-spam filter was rejecting the post :-/

[seanerussell@…]

Ok, yeah... it appears to be a limit. I've tried with 141 and 181 tickets with "new" status, and the plugin always returns a list of 100 elements. Am I missing an argument to "list" to tell it to really return all of the results?

[anonymous]

I can get around it with the max attribute, so I suppose you can close this. However, it might be useful to (a) make a comment that queries are bounded by the default max results inherent in Trac, and (b) provide some information about how to get a count of all results. For example, it is not intuitive (to me) to expect that query("status=new") would have some default bounds on it; it isn't obvious to me exactly how to find out many tickets there are that match the query; and it isn't immediately obvious to me what the sort order that Trac is using. This last point is what was causing most of my confusion here, because the plugin was returning results with occasional tickets missing, not just the first (by ticket ID) 100 tickets. Just a tiny bit of documentation may help others avoid confusion in this area.

[mark]

I'm having the same problems, my query is simple status!=closed. Expecting 266 tickets to be returned. But I only get 100 tickets. anaymous talks about 'max attribute' but I don't see where I can alter this. Does anybody have a clue?

IMHO, this ticket should not be closed yet. It is unexpected behaviour even with a workaround it should be solved if possible.

[osimons]

Replying to mark:

I'm having the same problems, my query is simple status!=closed. Expecting 266 tickets to be returned. But I only get 100 tickets. anaymous talks about 'max attribute' but I don't see where I can alter this. Does anybody have a clue?

IMHO, this ticket should not be closed yet. It is unexpected behaviour even with a workaround it should be solved if possible.

It does reflect the use of /query module, where a max setting is implemented as a default to enable paging of results and avoid queries exhausting resources (worst case). So, the idea is that you add &max=n to your query to limit results to maximum n returned results, and where you would then add &page=m to iterate through the pages. Like the query module, you can use &max=0 to turn off paging and return all results.

The RPC method is a thin layer on top of the Trac Query module, and all the arguments and their behaviour is embedded deep into this module. I don't think it makes sense for the API to change this behaviour, but I can see that it may be confusing.

As a "compromise" I've added this information to my to-do list, and will update the docs for the method to include information about the additional query arguments to use to work with result sets. Like, for your needs the real query should be status!=closed&max=0 - alternatively if you wanted to page the result for the user you could call status!=closed&max=100&page=1 and then page=2 and page=3 if the user needed to look through all the results. It all depends on how the query is to be consumed.

[osimons]

#7706 closed as duplicate.

[osimons]

(In [9261]) XmlRpcPlugin: Improve docs for query result set max for number returned by each call, and page to move through the result pages. Closes #6512