= Messages encryption = I'll document the effort to add support for optionally message encryption using GnuPG. See #6773 for the corresponding ticket asking for this enhancement. [[TOC(inline,heading=page content)]] == Code structure == === Where === Where to kick in and mangle the message body is of course an essential decision. My initial assumption that I could add cryptographically functions right before inserting recipient addresses to the message was wrong. It might help a lot to look at [t:wiki:TracDev/Announcer internal structure and event flow] of AnnouncerPlugin in detail. By now I add the following into code of ./announcerplugin_trunk/announcer/distributors/mail.py from trunk of AnnouncerPlugin: {{{ return msgid def _do_send(self, transport, event, format, recipients, formatter): output = formatter.format(transport, event.realm, format, event) + + email_encrypt = True + if email_encrypt: + output = encrypt_txt(output) + self.log.debug("EmailDistributor successfully encrypted msg.") + alternate_style = formatter.alternative_style_for( transport, event.realm, }}} === What === Here is a description of what shall be done. Experts and GPG/PGP users may wish to skip that section and go to the [#AnnouncerEmailEncryption proposal for trac-specific use] right away. ==== OpenPGP principles ==== FIXME: I'll write here and cite sources for more detailed explanation of OpenPGP standard and cryptography in communication in general. ==== !AnnouncerEmailEncryption ==== Now let's look at how OpenPGP could help with Trac and especially with announcements. Imagine, you're using trac in a corporate environment, typically allowing external access to trac, repositories etc. only after authorization or not at all. However you may wish to keep business partners, support customers or even co-workers outside the tightly controlled corporate network informed about certain or all developments. Encrypting mail for external recipients will help to deal with issues as discrete as you like while still using announcements for changes in trac. To get !AnnouncerEmailEncryption up and working you'll have to do take the following steps: 1. install GnuPG on the same host along with Trac[[BR]] For Debian GNU/Linux a simple {{{apt-get install gnupg}}} will do. 2. install python-gnupg Currently there is no Debian package available. Install from source of [http://code.google.com/p/python-gnupg/downloads/list project site] is preferred. 3. configure !AnnouncerEmailEncryption in the ![announcer] section of trac.ini for the given trac environment * gnupg_dir = , will default to something like {{{/gnupg}}}[[BR]] If not existent, this directory would be created and populated with necessary (initially empty) files on next trac start with !AnnouncerEmailEncryption enabled. * msg_encryption = False|True, default to "False" Pretty self-explaining. External would opt for splitting recipients according to following rules into group require_encryption_group and allow_verbatim_msg_group and sending an local, verbatim as well as an encrypted version in parallel, if both recipient lists are not empty. * external recipient_rule = (list of e-mail addresses or regex describing range of e-mail addresses) 4. import pubkeys and associate with users * admin: * (mass-)upload from local pubkey(ring) files * upload from local or public keyserver * associate pubkey from previous upload with users User administration should allow for key import and association to users on behalf of every registered user. * user: * upload like admin, but associate only to himself/herself * select pubkey from previous upload (self or admin) Users may wish to add an OpenPGP key to their configuration. For simplicity I'd make existence of a pubkey equivalent to an "always encrypt msg for me with this key" option. For convenience it might still be possible to temporarily disable a key and re-enable it later without deletion and re-import as this is directly supported by GnuPG. 5. add an automatic signing key for the given trac environment (optional) * upload from local pubkey(ring) file or create it on-demand * use secret key, if only one is available, else provide a drop-down or similar for selection For convenience it should be possible to temporarily disable a key and re-enable it later without deleting it. Beware, that this is by now no code but pure concept and subject to change a lot, before public release of the code. As with current code for AnnouncerPlugin there'll be DEBUG logging embedded into all operations mentioned above. ==== Behind current development scope ==== There is a [t:wiki:TracDev/Proposals/Announcer proposal] to replace current Trac Notification system with AnnouncerPlugin. This will make the effort for a really clean solution even more urgent. Consider [https://subtrac.sara.nl/oss/email2trac/ticket/186 cryptography related features] for EmailtoTracScript ([https://subtrac.sara.nl/oss/email2trac current home outside of Tack-Hacks]). It could be interesting i.e. to allow only e-mails with valid signature from known senders to pass, fighting spam at another level. ==== Q&A ==== ?: ''Is it true that different users will have different keys? If so, we can add configuration to the user's preference page. We could have a big textbox for GPG key and if they have one entered, then use encryption. A: Yes, different users will (typically) have different keys. It might be desirable even to support multiple keys per user. Only in rare cases one key would be associated with different users/e-mail addresses, even if this might be technically perfectly valid and useful. But it indicates violation of the de-facto standard one-owner-per-key that abhor kind of group keys. === How === ==== Available interfaces with GnuPG ==== To make it more difficult for me to start I've found not one but several candidates for interacting with GnuPG from Python (http://wiki.python.org/moin/GnuPrivacyGuard has a listing with some more comments): * [http://code.google.com/p/python-gnupg/ python-gnupg] * PRO: no additional dependencies but pure Python, works on Windows as well as Unix/Linux, most complete set of gpg actions including key generation and management, active development - python 3 support since July 2009, latest release v0.2.4 from 01-03-2010 * CON: no Debian package? * '''TEST''': * download and local install without issues, * function list_keys() ~~doesn't "just work"~~ works on a known-good gpg keyfile directory - got it * beware: "gnupghome" directory will be created silently (including parents), if something is not there exactly as specified, init function will need to prevent creation of unwanted directories by (worst case: repeated) mis-configuration * [http://py-gnupg.sourceforge.net/ Python GnuPGInterface] * PRO: Debian package python-gnupginterface-0.3.2-9 * CON: concentrates on interacting with GnuPG via filehandles, based on Perl module GnuPG::Interface by same author, rumors about being "not very easy to use", doesn't work on Windows ([http://sourceforge.net/tracker/?func=detail&aid=1859636&group_id=29555&atid=396635 open feature request] since 2007, even has predecessor from 2002 that was plainly rejected), quite old - latest release v0.3.2 from 24-02-2002, even looks unmaintained since 2008 * [https://launchpad.net/pygpgme PyGPGME] * PRO: Debian package python-gpgme-0.1+bzr20090820-1+b1 * CON: ? * [http://pyme.sourceforge.net/ PyMe] * PRO: interface to C GPGME library, not limited to gpg by design, other backends planned, works on Windows as well as Unix/Linux, latest release v0.8.1 from 26-11-2008, Debian package python-pyme-0.8.1+clean-1 * CON: complex dependencies because built on GPGME + Python + SWIG * '''TEST''' * Debian package needed upgrade to python-pyme-0.8.1+clean-3+b1 to fix error on GnuPG interface setup call * much more complex API compared to python-gnupg * on halt for now, but still considered nice-to-have, since it would allow additional crypto-backends i.e. working with X.509 certificates etc. * [http://pypi.python.org/pypi/OpenPGP/ OpenPGP] * PRO: ? * CON: no Debian package?, no information on Windows support, quite old - latest release v0.2.3 from 01-07-2005, even looks unmaintained ([http://www.aonalu.net/openpgp project homepage] currently unreachable) * [http://www.cs.auckland.ac.nz/~pgut001/cryptlib/ cryptlib] * PRO: interface to a range of plug-in encryption modules, not only but including gnupg, language bindings for C / C++, C# / .NET, Delphi, Java, Python, and Visual Basic, re-entrant and completely thread-safe, most core algorithms implemented in assembly language, support crypto hardware acceleration facilities like in Via C3 CPU family, extensive documentation designed for cut-n-paste directly from manual * CON: no Debian package? * [http://www.dlitz.net/software/pycrypto/ PyCrypto] * PRO: ? * CON: looks incredibly complex at first glance * [http://www.freenet.org.nz/ezPyCrypto/detail/index.html ezPyCrypto], a simpler API on top of !PyCrypto ==== The choice: python-gnupg ==== '''python-gnupg''' was tested, !PyMe a little too. It became clear, that python-gnupg just worked without much hassle. Anything else had more dependencies and was more complicated i.e. by introducing GPGME. This applies to !PyMe as well as PyGPGME. GnuPGInterface, OpenPGP, cryptlib where skipped right after the initial interface research. ==== The code ==== 1. step: add some code to make encryption just work '''done'''[[BR]] * expect encryption/signing key ID hard-coded, some fixed values for variables I'd like to see as options in [annoucer] section of trac.ini and other ugliness 2. step: code evolution over time, i.e. a. put code into separate python script and import function into distributors/mail.py '''done'''[[BR]] * add new reference {{{ from announcer.util.mail_encrypt import encrypt_txt }}} * add ./announcerplugin_trunk/announcer/'''util/mail_encrypt.py''' containing new cryptographic functions b. consider invention of a new class, i.e. to allow for reusable code, gpg interface initialization before sign and encrypt actions would profit among others c. add a minimal set of new options to [annoucer] section of trac.ini and replace formerly fixed values to gain planned control about new cryptographic functions '''mandatory options''' (not needed for basic operation with common GnuPG installation ||option name ||default value ||note || ||email.encrypt ||disabled ||whole e-mail encryption disabled, to allow smooth upgrading of existing installations || ||email.sign ||disabled ||whole e-mail signing disabled || '''additional options''' ||option name ||default value ||note || ||gpg_binary ||gpg ||full path to binary, needed i.e. for custom GnuPG install or GnuPG v2 || ||gpg_home ||None ||keyring files dir, defaults to ~/.gpg (in home dir of user that is running trac) || ||email.encrypt_except ||None ||one or more e-mail-addresses or domain as comma-separated list, matching recipients that still get cleartext msg || ||email.sign_except ||None ||valid values as before, but matching recipients here still get msg without cryptographically signature || 3. step: extend web_ui of AnnouncerPlugin to remote-control new options from user and/or administration settings hints, recommendations? known-good code references or popular applications? ==== Q&A ==== [FIXME: add more Q+A here to help with code design evaluation and code review] ?: Why not implement encryption as another IAnnouncementEmailDecorator A: Decorators are called without guaranteed order. Encryption needs control, that it'll be the last message body mangling action. ''We can change this pretty easily'' - '''doki_pen''' ?: Why not implement encryption as another IAnnouncementFormatter A: Encryption is not about encoding etc. ''Formatter is more about turning an event into a message, it shouldn't be done here.'' - '''doki_pen''' ? Doesn't smtplib or any other stock python library handle encryption? A: No. Pythons smtplib is dedicated to e-mail construction including MIME, but no PGP/MIME etc. (see http://docs.python.org/library/smtplib.html). Pythons nativ [http://docs.python.org/library/crypto.html crypto utils] currently consist of secure hash and checksum generators (md5, sha). ?: What are the explicitly handled exceptions? A: For readability let's try to put this into a table. ||exception ||cause ||action/behavior || ||missing pubkey ||fingerprint in user settings but no corresponding key in pubkeyring file ||delete recipient from recipient list of event in delivery, create new event with info "specified pubkey not in Tracs keyring" to be sent to this user and project admin || ?: Does python-gnupg support GnuPG v2? A: AFAIK yes, both versions support same CLI syntax. I'll continue to test with both versions in the future to maintain compatibility. There might be even a bonus from using GnuPG v2, since it is announced to be PGP/MIME aware. However I'll still have to look into this in detail. === Sources (for ideas and code) === * GNU Privacy Guard Manual at http://www.gnupg.org/documentation/manuals/gnupg/ * Why sign&encrypt is not very secure by default see http://world.std.com/~dtd/sign_encrypt/sign_encrypt7.html (discussion about vulnerability againgst "surreptitious forwarding") * Intro to python-gnupg at http://groups.google.de/group/comp.lang.python/browse_thread/thread/f2b97a2c11e1df63 * Python Wrapper for GnuPG v0.2.4 documentation: http://www.red-dove.com/python_gnupg/index.html * Python e-mail test server http://docs.djangoproject.com/en/1.1/topics/email/#topics-email * How Django, another Python based system handles e-mail-encryption with [http://code.google.com/p/django-email-extras/ django-email-extras] ([http://code.google.com/p/django-email-extras/source/browse/trunk/email_extras/ browse the code]) * Python tutorial at http://www.python.org/doc/current/tutorial/index.html to help some newbie like me opening the world of Python code some more real-world implementations of python-gnupg, hints? * MIME-capable Python mailer script is published at http://www.physics.drexel.edu/~wking/code/python/send_pgp_mime, this look very interesting, may help with a lot of things, but may not serve as a code base (incompatible license GPL2 vs. BSD-alike), not portable (consider Linux/Unix as well as Windows) - hasienda