Plugin uninstallable in Trac 0.11 - forms sample crashes wiki page, and more issues

[anonymous]

I am probably doing something wrong here, but I tried both Methods 1 & 2 and get the same crash when I put a sample (laundry list or shopping list) from the TracFormsPlugin wiki pages into my wiki page.

Any suggestion as to what I should try to get this working? We'd really like to use it. I'll put the tracback in the first comment.

[mparker@…]

Trac detected an internal error:

TypeError: 'NoneType' object is not iterable

Traceback:

Most recent call last:

    * File "/usr/lib/python2.6/dist-packages/trac/wiki/templates/wiki_view.html", line 44, in <Expression u'wiki_to_html(context, page.text)'>
      Code fragment:
        39. </table>
        40. </py:if>
        41.  
        42. <div class="wikipage searchable" py:choose="" xml:space="preserve">
        43. <py:when test="page.exists" xml:space="preserve">
        44. ${wiki_to_html(context, page.text)}
        45. </py:when>
        46. <py:otherwise>
        47. Describe ${name_of(page.resource)} here.
        48. </py:otherwise>
        49. </div>
      Local variables:
      Name	Value
      __data__ 	[{'abs_url_of': <function get_abs_url at 0x7f09f611c0c8>, 'templates': [], ...
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 1118, in format_to_html
      Code fragment:
      1113. def format_to_html(env, context, wikidom, escape_newlines=None):
      1114. if not wikidom:
      1115. return Markup()
      1116. if escape_newlines is None:
      1117. escape_newlines = context.get_hint('preserve_newlines', False)
      1118. return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
      1119.  
      1120. def format_to_oneliner(env, context, wikidom, shorten=None):
      1121. if not wikidom:
      1122. return Markup()
      1123. if shorten is None:
      Local variables:
      Name	Value
      context 	<Context <Resource u'wiki:MarlaScratch'>>
      env 	<trac.env.Environment object at 0x7f09f54a0610>
      escape_newlines 	False
      wikidom 	u"Testing page.\r\n\r\n\r\n * [[ticket_display(43)]]\r\n\r\n== Trac Forms ...
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 1073, in generate
      Code fragment:
      1068. newlines in the wikidom will be preserved if `escape_newlines` is set.
      1069. """
      1070. # FIXME: compatibility code only for now
      1071. out = StringIO()
      1072. Formatter(self.env, self.context).format(self.wikidom, out,
      1073. escape_newlines)
      1074. return Markup(out.getvalue())
      1075.  
      1076.  
      1077. class InlineHtmlFormatter(object):
      1078. """Format parsed wiki text to inline elements HTML.
      Local variables:
      Name	Value
      escape_newlines 	False
      out 	<StringIO.StringIO instance at 0x7f09f6301638>
      self 	<trac.wiki.formatter.HtmlFormatter object at 0x7f09f56db050>
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 851, in format
      Code fragment:
       846. def format(self, text, out=None, escape_newlines=False):
       847. self.reset(text, out)
       848. for line in text.splitlines():
       849. # Handle code block
       850. if self.in_code_block or line.strip() == WikiParser.STARTBLOCK:
       851. self.handle_code_block(line)
       852. continue
       853. # Handle Horizontal ruler
       854. elif line[0:4] == '----':
       855. self.close_table()
       856. self.close_paragraph()
      Local variables:
      Name	Value
      escape_newlines 	False
      line 	u'}}}'
      out 	<StringIO.StringIO instance at 0x7f09f6301638>
      result 	u''
      self 	<trac.wiki.formatter.Formatter object at 0x7f09f56db610>
      sep 	'\n'
      text 	u"Testing page.\r\n\r\n\r\n * [[ticket_display(43)]]\r\n\r\n== Trac Forms ...
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 780, in handle_code_block
      Code fragment:
       775. self.close_table()
       776. self.close_paragraph()
       777. if self.code_buf:
       778. self.code_buf.append('')
       779. code_text = os.linesep.join(self.code_buf)
       780. processed = self.code_processor.process(code_text)
       781. self.out.write(_markup_to_unicode(processed))
       782.  
       783. else:
       784. self.code_buf.append(line)
       785. elif not self.code_processor:
      Local variables:
      Name	Value
      code_text 	u"#\n# An example of a TracForm. The first lines can be commentary or #! ...
      line 	u'}}}'
      self 	<trac.wiki.formatter.Formatter object at 0x7f09f56db610>
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 180, in process
      Code fragment:
       175. if self.error:
       176. text = system_message(tag('Error: Failed to load processor ',
       177. tag.code(self.name)),
       178. self.error)
       179. else:
       180. text = self.processor(text)
       181. if not text:
       182. return ''
       183. if in_paragraph:
       184. content_for_span = None
       185. interrupt_paragraph = False
      Local variables:
      Name	Value
      in_paragraph 	False
      self 	<trac.wiki.formatter.WikiProcessor object at 0x7f09f577bc90>
      text 	u"#\n# An example of a TracForm. The first lines can be commentary or #! ...
    * File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 167, in _macro_processor
      Code fragment:
       162.  
       163. def _macro_processor(self, text):
       164. self.env.log.debug('Executing Wiki macro %s by provider %s'
       165. % (self.name, self.macro_provider))
       166. return self.macro_provider.expand_macro(self.formatter, self.name,
       167. text)
       168.  
       169. def _mimeview_processor(self, text):
       170. return Mimeview(self.env).render(self.formatter.context,
       171. self.name, text)
       172. # TODO: use convert('text/html') instead of render
      Local variables:
      Name	Value
      self 	<trac.wiki.formatter.WikiProcessor object at 0x7f09f577bc90>
      text 	u"#\n# An example of a TracForm. The first lines can be commentary or #! ...
    * File "build/bdist.linux-x86_64/egg/tracforms/macros.py", line 39, in expand_macro
      Local variables:
      Name	Value
      args 	u"#\n# An example of a TracForm. The first lines can be commentary or #! ...
      formatter 	<trac.wiki.formatter.Formatter object at 0x7f09f56db610>
      name 	u'TracForm'
      processor 	<tracforms.macros.FormProcessor object at 0x7f09f57a3210>
      self 	<tracforms.macros.TracFormMacro object at 0x7f09f57a3190>
    * File "build/bdist.linux-x86_64/egg/tracforms/macros.py", line 131, in execute
      Local variables:
      Name	Value
      args 	[]
      cmd 	u'track_fields'
      errors 	[]
      fn 	<bound method FormProcessor.cmd_track_fields of ...
      formatter 	<trac.wiki.formatter.Formatter object at 0x7f09f56db610>
      kw 	{}
      line 	u''
      name 	u'TracForm'
      realm 	'wiki'
      resource_id 	u'MarlaScratch'
      self 	<tracforms.macros.FormProcessor object at 0x7f09f57a3210>
      srciter 	<listiterator object at 0x7f09f57a32d0>
      state 	None
      textlines 	[u'', u"|| || '''Who''' || '''When''' || '''What''' ||", u'|| [tf:wash] || ...

File "/usr/lib/python2.6/dist-packages/trac/wiki/templates/wiki_view.html", line 44, in <Expression u'wiki_to_html(context, page.text)'>
  ${wiki_to_html(context, page.text)}
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 1118, in format_to_html
  return HtmlFormatter(env, context, wikidom).generate(escape_newlines)
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 1073, in generate
  escape_newlines)
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 851, in format
  self.handle_code_block(line)
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 780, in handle_code_block
  processed = self.code_processor.process(code_text)
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 180, in process
  text = self.processor(text)
File "/usr/lib/python2.6/dist-packages/trac/wiki/formatter.py", line 167, in _macro_processor
  text)
File "build/bdist.linux-x86_64/egg/tracforms/macros.py", line 39, in expand_macroFile "build/bdist.linux-x86_64/egg/tracforms/macros.py", line 131, in execute

[ewalstad@…]

It appears that our installation (I work with mparker@…) isn't running any schema migrations. It appears that this plugin requires tables that our SQLite db does not have. However, when we run trac-admin $TRACENV_DIR upgrade, trac-admin tells us that no upgrade is required

$ sudo trac-admin $TRACENV_DIR upgrade
...snip hashlib DeprecationWarnings...
Database is up to date, no upgrade necessary.

Do we need these tables ('forms', 'forms_fields' and 'forms_history')? If so, how do we convince trac-admin to run the schema migration? Our /usr/share/pyshared/trac/upgrades dir contains dbN.py files from N=3 to N=21 and tracforms/tracdb.py specifies db_version = 14. Here's the tables we have in our SQLite db:

SQLite version 3.6.22
...
sqlite> .tables
attachment         enum               revision           ticket_change    
auth_cookie        milestone          session            ticket_custom    
bill_date          node_change        session_attribute  version          
component          permission         system             wiki             
custom_report      report             ticket

Thanks,

Eric.

[Steffen Hoffmann]

I consider the plugin stable for running it with Trac 0.13dev in production for a couple of months now. However, considering your findings this may be true for Trac 0.12 and beyond only. We'll see.

First to clarify some things from previous comments:

• yes, you - or better: TracForms will need these three tables which names you already figured out correctly
• db_version is strictly related to the plugin in question (here TracForms current), only for detecting/upgrading possible previous installations of TracForms in same environment (as stored in Trac db table system) - not tracking Trac db version at all
• TracForms uses common IEnvironmentSetupParticipant methods (see tracforms/tracdb.py) to detect install/upgrade conditions and should just do the right thing (or we need to find out there, why it doesn't do as expected)
  • created the tables right away, if no previous plugin version found in system (looking for two different entries named TracFormDBComponent:version and forms_version respectively)
  • do an incremental upgrade process to the current version depending on the old version found

Neither of these seems to happen, right? As I've created two Trac 0.11 test installations I'll give it a try over there and come back at you later with any findings.

[Steffen Hoffmann]

To keep you updated I'll post my findings in this and following comments. Please confirm or correct, as you see differences to behavior of your install. I'm testing with Trac 0.11 here:

Skipping "tracforms.formdb = tracforms.formdb": (can't import "cannot import name resource_exists")

Bad, so it doesn't initially load all modules. Will need to fix this in order to proceed. As ​t:r9928 suggests: "resource_exists now also in 0.11.7.1", this is currently not backwards-compatible with (older) Trac 0.11 releases.

Is it feasible for you to upgrade to latest Trac 0.11? Meanwhile I'll investigate the implications of removing or working around this dependency.

[Steffen Hoffmann]

Ok, verbatim copy of code from 0.11-stable/trac/resource.py at Trac revision 9899 into compat.py fixes the initial error, environment upgrade gets detected and runs well by the appropriate trac-admin command.

But not finished. If you proceed with our first laundry list example (or any other form), you'll find after saving the form, that following the context navigation linked to <your_env>/form/1 throws

TypeError: __call__() got an unexpected keyword argument 'form_id'

Root cause is the i18n implementation not being capable of handling all (keyword) arguments. The immediate fix is as simple as replacing the offending tag_( with tag(_(.

But then it's going on along a long road I already know all too well. In short: There's considerably more to be done for full Trac 0.11 compatibility here. If you're curious, you'll find similar issues (and fixes in #8963) - not exactly trivial, but no magic either - just i18n related stuff.

As it's more like an enhancement I'll still stick to the initial bug label, because the plugin is explicitly announced for Trac 0.11. I'll just go on and report back by commit messages after implementation and local testing.

Most probably you'll see a number of changes here. Watch out for the next maintenance release - coming soonish. Thank you for your patience so far.

[ewalstad@…]

Replying to hasienda:

To keep you updated I'll post my findings in this and following comments. Please confirm or correct, as you see differences to behavior of your install. I'm testing with Trac 0.11 here:

OK, will-do. We are running the deb package version for Ubuntu 10.04LTS:

$ sudo dpkg --list | grep trac
...
ii  trac    0.11.7-1...

Is it feasible for you to upgrade to latest Trac 0.11?

No, we want to stay with the Ubuntu packaged version.

[anonymous]

Replying to hasienda: ...

Root cause is the i18n implementation not being capable of handling all (keyword) arguments... But then it's going on along a long road I already know all too well. In short: There's considerably more to be done for full Trac 0.11 compatibility here.

Ok.

Most probably you'll see a number of changes here. Watch out for the next maintenance release - coming soonish. Thank you for your patience so far.

Sounds good. I'll watch for ticket updates and will help out with test when/where I can. I'm a Python developer but I don't have i18n experience.

Thanks for your attention to this.

Eric.

[Steffen Hoffmann]

Replying to anonymous:

Sounds good. I'll watch for ticket updates and will help out with test when/where I can. I'm a Python developer but I don't have i18n experience.

Still sounds good to me. Encouraging to know, there's someone who cares. As a matter of fact I've put out tracforms-0.4 more than two months ago, not expecting such big no-go issues by now. Sorry.

Thanks for your attention to this.

Oh, you're welcome. As time goes by and we'll see a new revision I'll be much pleased by confirmed success at your side. That's the way to go.

[Steffen Hoffmann]

(In [10484]) TracFormsPlugin: Establish compatibility with Trac 0.11, refs #9000.

First step towards full backwards-compatibility, that did not exist for a long time, at least since starting the i18n implementation.

Now let's provide a private resource_exists for older Trac 0.11 revisions.

[Steffen Hoffmann]

You'll notice growing 0.11 compatibility in trunk development branch.

It is my personal preference to have a dedicated development stage and not fiddle with released code. I'll merge the changes back into 0.4 later or - more correctly - tag a later trunk revision as 0.4.1, without the changes, that have already been added for next feature release (0.5).

[anonymous]

Replying to hasienda:

You'll notice growing 0.11 compatibility in trunk development branch.

It is my personal preference to have a dedicated development stage and not fiddle with released code. I'll merge the changes back into 0.4 later or - more correctly - tag a later trunk revision as 0.4.1, without the changes, that have already been added for next feature release (0.5).

Sounds fine to me. When you are ready for me to test I'll check out, build and install from trunk (or a tag, if you make one).

[Steffen Hoffmann]

Ready - well almost. Let's get some sleep and re-check tomorrow for gotchas and hidden flaws/leftovers. Still I'll ask you to test trunk before publishing the 0.4.1 maintenance release.

[Steffen Hoffmann]

(In [10487]) TracFormsPlugin: Establish compatibility with Trac 0.11, refs #9000.

Now apply proper fallbacks for gettext variants similar to fixes for AccountManagerPlugin:

Genshi templates use dgettext to reliably extract and translate special content like button labels. Because for the 0.11 Trac releases this isn't included into default content of the data object passed to Genshi alongside with the template, this has to be added explicitely to each and every data object created with this plugin.

More importantly, there is some code missing to degrade more complex i18n calls like dgettext gracefully for Trac 0.11, so more code has been taken and adapted from current Trac trunk to fill in the gap here.

[Steffen Hoffmann]

(In [10488]) TracFormsPlugin: Establish compatibility with Trac 0.11, refs #9000.

Replace tag_ with more capable tag(_( call to prevent TypeError in Genshi used by any Trac 0.11 release. Add Markup to avoid wrong escape of HTML from Genshi markup.

[Steffen Hoffmann]

I've applied all changes by now. So it's time for testing, with r10489.

I'm looking forward to hear about your experiences to trigger more changes or - in case of success - the release of the appropriate maintenance version.

[ewalstad@…]

hasienda,

It seems to be working now (installed from r10489). I'm going to hand our project off to mparker@… for more testing. We'll file another ticket if we find any other issues.

Thank you very much for your quick responses - very much appreciated.

Eric.

[Steffen Hoffmann]

Replying to ewalstad@gmail.com:

It seems to be working now (installed from r10489).

Ok, so I'll proceed.

We'll file another ticket if we find any other issues.

Ok, only make sure, you always know and tell, what you're working with. For production I'd recommend to go with the stable version. I'll do the release tonight.

If you go with the development version instead, that you've been testing this time, please be aware of a gradually higher possibility of some roughness at bleeding edge. And you'll miss (wiki) documentation considerably more often than for the stable release.

Be sure to follow commit messages then. Normally I'm writing quite verbose commit messages, so you can easily get a good picture about what's going on. In the same notion I do expect everyone involved to read these statements to avoid unnecessary trouble for you and me.

Thank you very much for your quick responses - very much appreciated.

Oh, you're welcome. I do appreciate your instant feedback as well.

[anonymous]

Hasienda, EW, thank you both so much for your work on this ticket! It is working great for me. We will certainly switch to the production version once you have one out, but with what Eric installed already, I've been building and testing my form for a few hours and it works great. Thanks very much. -Marla

[Steffen Hoffmann]

(In [10490]) TracFormsPlugin: Release maintenance version 0.4.1 for compatibility with Trac 0.11, closes #9000.

These are changes cherry-picked from trunk and merged into tracforms-0.4 to establish full compatibility with Trac 0.11 on level with plugin's own claims.