wiki.putAttachment fails for larger files

[lukasz.matecki@…]

I'm using Trac 1.0 over stunnel (tracd, https). The attachment maximum size for the environment is set to 2 MB. I've downloaded the most recent xmlrpcplugin-r12617.zip and installed it. When I'm sending a (binary) file that is 50 kB everything seems to be OK. When I try to send larger file (>100 kB) I get the following error (the file contains binary data):

Traceback (most recent call last):
  File "scripts/upload_installer.py", line 9, in <module>
    trac_rpc.upload_attachment(file)
  File "D:\workspace\utal\projects\pro_plate\trunk\scripts\trac_rpc.py", line 79, in upload_attachment
    server.wiki.putAttachment('InstallersList/{0}'.format(basename), binary)
  File "c:\Python27\lib\xmlrpclib.py", line 1224, in __call__
    return self.__send(self.__name, args)
  File "c:\Python27\lib\xmlrpclib.py", line 1575, in __request
    verbose=self.__verbose
  File "D:\workspace\utal\projects\pro_plate\trunk\scripts\trac_rpc.py", line 50, in request
    return self._internal_request(host, handler, request_body, verbose)
  File "D:\workspace\utal\projects\pro_plate\trunk\scripts\trac_rpc.py", line 41, in _internal_request
    f = self._opener.open(request)
  File "c:\Python27\lib\urllib2.py", line 394, in open
    response = self._open(req, data)
  File "c:\Python27\lib\urllib2.py", line 412, in _open
    '_open', req)
  File "c:\Python27\lib\urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "c:\Python27\lib\urllib2.py", line 1207, in https_open
    return self.do_open(httplib.HTTPSConnection, req)
  File "c:\Python27\lib\urllib2.py", line 1174, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 10054] Istniej╣ce po│╣czenie zosta│o gwa│townie zamkniŕte przez zdalnego hosta>
make: *** [upload] Error 1

I'm using the following code to be able to communicate with trac using digest authentication:

import os
import xmlrpclib
import httplib
import urllib2
import StringIO
import base64

class HTTPSDigestTransport(xmlrpclib.SafeTransport):
    """
    Transport that uses urllib2 so that we can do Digest authentication.
    
    Based upon code at http://bytes.com/topic/python/answers/509382-solution-xml-rpc-over-proxy
    """

    def __init__(self, username, pw, realm, verbose=None, use_datetime=0):
        self.__username = username
        self.__pw = pw
        self.__realm = realm
        self.verbose = verbose
        self._use_datetime = use_datetime
        self._opener = None

    def _internal_request(self, host, handler, request_body, verbose):

        url = 'https://' + host + handler
        if verbose or self.verbose:
            print "ProxyTransport URL: [%s]" % url
            
        request = urllib2.Request(url)
        request.add_data(request_body)
        # Note: 'Host' and 'Content-Length' are added automatically
        request.add_header("User-Agent", self.user_agent)
        request.add_header("Content-Type", "text/xml")  # Important

        # setup digest authentication
        if self._opener is None:
            authhandler = urllib2.HTTPDigestAuthHandler()
            authhandler.add_password(self.__realm, url, self.__username, self.__pw)
            self._opener = urllib2.build_opener(authhandler)

        f = self._opener.open(request)
        res = self.parse_response(f)
        f.close()
        return res

            
    def request(self, host, handler, request_body, verbose):
        for i in (0, 1):        
            try:
                return self._internal_request(host, handler, request_body, verbose)
            except httplib.BadStatusLine:
                if i:
                    raise
                else:
                    print("Restart...")
  
def get_server():  
    password = raw_input("Enter the repository password: ")
                       
    digestTransport = HTTPSDigestTransport("Lukasz", password, "trac")
    server = xmlrpclib.ServerProxy("....removed....", transport=digestTransport)

    return server
   
def upload_attachment(file):
    server = get_server()
    basename = os.path.basename(file)
    print file
    with open(file, "rb") as f:
        data = f.read()

    print (len(data))
    
    binary = xmlrpclib.Binary(data)

    print len(binary.data)

    server.wiki.putAttachment('InstallersList/{0}'.format(basename), binary)
    
# TODO this should be parametrized.
if __name__ == "__main__":
   pass

[Jun Omae]

Reproduced while requesting POST with large content (>= 2 MB) on tracd with HTTP authentication. However, works on tracd without HTTP authentication and modwsgi with/without HTTP authentication (verified with Trac 0.12 through 1.5).

I think that HTTP authentication module for tracd in Trac core has something wrong and that is not an XmlRpcPlugin issue.

import json, sys
if sys.version_info[0] != 2:
    import urllib.request as urllib
else:
    import urllib2 as urllib

def do_request(url, size):
    jsonrpc = 'rpc' in url
    headers = {}
    if jsonrpc:
        ctype = 'application/json'
        form_token = None
        data = json.dumps({'method': 'notfound', 'params': ['a' * size]})
    else:
        ctype = 'application/x-www-form-urlencoded'
        form_token = 'x' * 20
        headers['Cookie'] = 'trac_form_token=' + form_token
        data = '__FORM_TOKEN={}&_={}'.format(form_token, 'a' * size)
    headers['Content-Type'] = ctype
    if not isinstance(data, bytes):
        data = data.encode('utf-8')
    handler = urllib.HTTPBasicAuthHandler()
    handler.add_password('auth', url, 'admin', 'password')
    opener = urllib.build_opener(handler)
    req = urllib.Request(url, data=data, headers=headers)
    resp = opener.open(req)
    if jsonrpc:
        return json.load(resp)
    else:
        return resp.getcode()

print(do_request(sys.argv[1], int(sys.argv[2])))

$ python2.7 ./post.py http://127.0.0.1:3000/tracenv/login/rpc 5000000
Traceback (most recent call last):
  File "/dev/shm/jsonclient.py", line 35, in <module>
    print(do_request(sys.argv[1], int(sys.argv[2])))
  File "/dev/shm/jsonclient.py", line 29, in do_request
    resp = opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1235, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1205, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 32] Broken pipe>

$ python2.7 ./post.py http://127.0.0.1:3000/tracenv/rpc 5000000
{u'error': {u'message': u'RPC method "notfound" not found', u'code': -32601, u'name': u'JSONRPCError'}, u'result': None, u'id': None}

$ python2.7 ./post.py http://127.0.0.1:3000/tracenv/about 5000000
200

[Jun Omae]

That is a urllib2/urllib.request issue.

The library sends request without credentials first, and retries with credentials when 401 response is received from the remote server. However, the remote server often sends 401 and disconnects without receiving entire of the request, especially the request is large. As the result, EPIPE is raised in the client.

Workaround is to send with credentials without checking 401:

• Python 3.5+: use HTTPPasswordMgrWithPriorAuth and call .add_password(None, URL, USER, PASSWORD, is_authenticated=True)
• Python 2: set directly Authorization header instead of using HTTPBasicAuthHandler

[Jun Omae]

In 18518:

XmlRpcPlugin: add tests for adding large attachment (refs #10875)