Unable to add users due to existing email addresses ("email already in use")

Well, this is a local installation/configuration or even a data related issue, not an AcctMgr bug. So instead of reporting a software issue to our development ticket system (this one here) I'd recommend contacting my via known contact (see setup.py or my user page at t-h.o). For the seemingly invalid claim I'll lower the priority for now. Still I'll try to help you as good as I can, if you provide enough details.

Second, what is AMPi? Avoid uncommon acronyms, please. AM as I guess is AccountManager.

Email addresses entered to tickets (in Cc) are totally unrelated here. All that AcctMgr is checking is the Trac db table session_attribute. Most probably this is polluted by other plugins or unclean user deletions. Clean it up, please. I trust in your ability to do such db operations. I'm able to help with that too, but this is totally OT here. (Ask me directly via email as mentioned above). However, if you don't care about unique emails, you shouldn't set email as a required. Just disable the account verification procedure instead.

Well I it's certainly false advertising if you can't delete an account you create in AcctMgr?. I used your database tip (thank you) to remove the account references using in sqlite:

$ sqlite3 ./trac-environment/db/trac.db sqlite> .schema session sqlite> .schema session_attribute sqlite> select * from session; sqlite> delete from session where sid like '%USER-NAME%'; sqlite> delete from session_attribute where sid like '%USER-NAME%';

This has given me a Error in _do_users "TypeError?: 'NoneType?' object is not iterable"

Thanks for any advice. I can't add user accounts so I'm marking it "critical."

Replying to anonymous:

Well I it's certainly false advertising if you can't delete an account you create in AcctMgr.

Advertising? I must have missed that commercial you refer to here. ;-)

You didn't mention, how and even that you deleted user accounts before, not with a single word.

And still you can. There are many more users out there, seemingly without this problem. So please consider this to be a local anomaly of your setup. Hence my tip to cleanup your Trac db to a sane state. At least a fault on AcctMgr's side is has to be further proven by subsequent testing. As I said, it works for me, so there need to be differences to your setup, about what you failed to present any more details by now, apart from the hint on the SQLite db backend.

I used your database tip (thank you) to remove the account references using in sqlite:

 $ sqlite3 ./trac-environment/db/trac.db
 sqlite> .schema session
 sqlite> .schema session_attribute
 sqlite> select * from session;
 sqlite> delete from session where sid like '%USER-NAME%';
 sqlite> delete from session_attribute where sid like '%USER-NAME%';

You're welcome. Glad, that worked for you.

This has given me a Error in _do_users "TypeError: 'NoneType' object is not iterable"

This is not entirely helpful. Would you offer

1. the full traceback (relevant part from trac.log while in DEBUG log mode) and
2. the exact Trac and plugin version/revision

please?

Thanks for any advice. I can't add user accounts so I'm marking it "critical."

Frankly, don't you feel you're bit near a threshold? For now I let it up to you, but don't overstress, please. These are developer tickets, that developers take care and normally set their priority, not user ones, to organize their work load. Better ask kindly, is a general good advice. We haven't signed a 24/7 service contract, have we? But it sounds a bit like that to me, and I'm not at all used to such behavior here, nor would I tolerate it for long.

Look: After your rather generic complaint you've got a best-guess within less than 1 hour(!). I don't complain about no response within 24 hours from your side. In fact I was rather lucky to spot your ticket right-away yesterday. But besides a short "thank you" in brackets your next comment was a straight follow-up with what you are concerned when proceeding.

How could you expect further help, if you don't listen carefully and fail to answer my explicit request as well as more implicit calls for details about your setup? And your comments lack basic Trac WikiFormatting like for the sqlite console log above, that is almost unreadable, right? ...

Stop it here, I just want to point out where you should really improve on your side. Now, let's go on and resolve the issue(s).

Thanks for your quick reply hasienda. Yes please direct me on the proper configuration settings. Below I'll copy our settings (from /trac-environment/conf/trac.ini) and some debug information. Essentially, the behavior is:

a.) Admin registers user through /admin/accounts/users. No email generated; Only user name visible at /admin/accounts/users. User logs in, email then visible at /admin/accounts/users.

b.) User self registers through /register. No email generated; Account immediately active despite "verify" setting. Account not visible at all. User loges in then account visible at /admin/accounts/users.

c.) Account deleted through /admin/accounts/users and username or email cannot be reused.

Oops… Trac detected an internal error: TypeError?: 'NoneType?' object is not iterable

Most recent call last:
File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 423, in _dispatch_request
  dispatcher.dispatch(req)
File "/usr/lib/python2.5/site-packages/trac/web/main.py", line 197, in dispatch
  resp = chosen_handler.process_request(req)
File "/usr/lib/python2.5/site-packages/trac/admin/web_ui.py", line 113, in process_request
  path_info)
File "build/bdist.linux-x86_64/egg/acct_mgr/admin.py", line 151, in render_admin_panelFile "build/bdist.linux-x86_64/egg/acct_mgr/admin.py", line 372, in _do_users

Settings:

[account-manager]
authentication_url = 
db_htdigest_realm = 
force_passwd_change = true
hash_method = HtPasswdHashMethod
htdigest_file = /trac-environment/trac.passwd
htdigest_realm = SCG Trac
htpasswd_file = /trac-environment/trac.passwd
htpasswd_hash_type = crypt
password_file = 
password_store = HtPasswdStore,HtDigestStore,SessionStore
persistent_sessions = true
refresh_passwd = False
user_lock_max_time = 0
verify_email = true

....
[components]
acct_mgr.admin.accountmanageradminpages = enabled
acct_mgr.api.accountmanager = enabled
acct_mgr.db.sessionstore = enabled
acct_mgr.guard.accountguard = enabled
acct_mgr.htfile.htdigeststore = enabled
acct_mgr.htfile.htpasswdstore = enabled
acct_mgr.http.httpauthstore = enabled
acct_mgr.notification.accountchangelistener = enabled
acct_mgr.notification.accountchangenotificationadminpage = enabled
acct_mgr.pwhash.htdigesthashmethod = enabled
acct_mgr.pwhash.htpasswdhashmethod = enabled
acct_mgr.svnserve.svnservepasswordstore = enabled
acct_mgr.web_ui.accountmodule = enabled
acct_mgr.web_ui.emailverificationmodule = enabled
acct_mgr.web_ui.loginmodule = enabled
acct_mgr.web_ui.registrationmodule = enabled
acct_mgr.web_ui.resetpwstore = enabled
customfieldadmin.api.customfields = enabled
customfieldadmin.customfieldadmin.customfieldadminpage = enabled
mastertickets.api.masterticketssystem = enabled
mastertickets.web_ui.masterticketsmodule = enabled
trac.web.auth.loginmodule = disabled
tracusermanager.account.admin_um.accountusermanagerpanel = disabled
tracusermanager.admin.usermanagementadminpage = disabled
tracusermanager.api.cachedsessionattributeprovider = disabled
tracusermanager.api.environmentfixknownusers = disabled
tracusermanager.api.sessionattributeprovider = disabled
tracusermanager.api.sessionuserstore = disabled
tracusermanager.api.usermanager = disabled
tracusermanager.permissions.admin_um.permissionusermanagerpanel = disabled
tracusermanager.profile.admin.userprofilefieldsadminpage = disabled
tracusermanager.profile.admin_um.userprofileusermanagerpanel = disabled
tracusermanager.profile.api.userprofilemanager = disabled
tracusermanager.profile.macros.defaultuserprofileslistcellcontributor = disabled
tracusermanager.profile.macros.teamrostermacro = disabled
tracusermanager.profile.macros.userprofileslistmacro = disabled
tracusermanager.profile.prefs.userprofilemodule = disabled

Replying to anonymous:

Thanks for your quick reply hasienda. Yes please direct me on the proper configuration settings. Below I'll copy our settings (from /trac-environment/conf/trac.ini) and some debug information. Essentially, the behavior is: a.) Admin registers user through /admin/accounts/users. No email generated; Only user name visible at /admin/accounts/users. User logs in, email then visible at /admin/accounts/users.

Ok, expected behavior. List will not show email before first login of a given user.

b.) User self registers through /register. No email generated; Account immediately active despite "verify" setting. Account not visible at all. User loges in then account visible at /admin/accounts/users.

'Verify' reads 'verify email' right? Administrative pre-verification or the like is on the ToDo list, see #843. I'm not sure, why you can't see registered users before 1st login. Will check later.

c.) Account deleted through /admin/accounts/users and username or email cannot be reused.

Same here. Most important regarding reproducible behavior: What's your Trac and plugin version?

Oops… Trac detected an internal error:

Ok, this shouldn't really happen. It bails out, because there's no user, that has been logged in by that time. Strange. How/as what user do you access the admin pages? That user should be listed at minimum.

Settings:

[account-manager]
authentication_url = 
db_htdigest_realm = 
force_passwd_change = true
hash_method = HtPasswdHashMethod
htdigest_file = /trac-environment/trac.passwd
htdigest_realm = SCG Trac
htpasswd_file = /trac-environment/trac.passwd

Do you really need both, HtDigestStore and HtPasswdStore? If so you'll need a later version to not have conflicting file settings, see #4677.

htpasswd_hash_type = crypt
password_file = 
password_store = HtPasswdStore,HtDigestStore,SessionStore
persistent_sessions = true
refresh_passwd = False
user_lock_max_time = 0
verify_email = true

Just out of curiosity, did you try with email verification disabled?

Your stance about "email not generated" sounds misleading, but is at least an annoyance indeed:

Since we rely on env.get_known_users() from Trac core here, this should only show details of users, that logged in at least once before (so have their entries in Trac db table session_details flagged with authenticated = 1.

OTOH AcctMgr inserts attributes for new users with authenticated set to 1 instead of 0. So we have seemingly all valid entries but users last_seen in Trac db table session still at 0 before the first login. Unbelievable, that this is flawed for ages (see r1111). I'll double-check and then attempt to fix it. Thanks for making me look at this more closely.

I'll apply a series of changes to trunk. Please note, while I've found a similar TypeError indeed, that it's not the same as the one reported here.

(In [10518]) AccountManagerPlugin: Fix a TypeError in account details admin page, refs #9051.

TypeError: 'NoneType' object is not iterable has been thrown, if user had never logged in before. Some always obsolete code is removed as well.

(In [10519]) AccountManagerPlugin: Make option verify_email effective for RegistrationModule too, refs #3153, #3989, #5509 and #9051.

Only module state (enabled/disabled) has been checked before, when deciding on the email address field being optional vs. required since changeset [9304].

(In [10520]) AccountManagerPlugin: Don't add a session for new users, refs #9051.

Session IDs for new users have always been added to Trac db table session. Turns out this is unneccessary at least for Trac 0.11 and beyond. Less code (complexity) is better.

Hey, think I've finally found the cause of your trouble. User account deletion was the keyword.

Meanwhile any Trac db might have retained obsolete user account data and admin stuff will need to do manual cleanup to not stumble over similar issues like you did. Depending on the number of user accounts this is bound to become a tedious procedure.

In fact I know of applications with 700+ users, so why not do the cleanup procedure with AcctMgr? After all it has always been this plugin's claim to ease account management for Trac admins. I'll try to develop such a web-UI cleanup guide within the next few days. Thank you for your patience.

(In [10526]) AccountManagerPlugin: Restore full db deletion for deleted accounts, refs #9051.

Due to SQL injections protection - hey, it works ;-) - the SQL statements responsible for purging user account information from the Trac db did not work as expected, since changeset [10283]. I meant to have fixed this a while ago. Obviously not, sorry.

I consider this a major bug, so will release another maintenance revision for the stable release as well. Thanks for reporting this issue.

I'm not yet finished with the additional table cleanup module for AcctMgr. This is little more complex than I thought before.

Anyway, I'd like to get some feedback, if recent changes really fix the reported issue(s) for you according to my claims. This is a prerequisite at least for the next maintenance release.

(In [10576]) AccountManagerPlugin: Introduce another db handler function, refs #9051.

Prepare get_user_attribute() as flexible generic wrapper for later use, and extend capability of del_user_attribute() equally to handle multiple attributes too.

(In [10577]) AccountManagerPlugin: Add new admin page for db cleanup tasks, refs #9051.

Most probably I'll rework the new db cleanup admin page into a subpage for users (accounts) admin page.

But at least there is a solution now to remove bad/unwanted user attributes without direct db access, but only for users with ACCTMGR_ADMIN for obvious reasons: You're able to delete literally any attribute, i.e. including SessionStore user passwords, just to name something really troublesome, if put into wrong hands.

Test feedback welcome for encouraging me to bring this into the next release. :-)

When clicking on the new menu cleanup I'm getting the following error:

2011-08-10 22:32:02,116 Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3dev_r10719-py2.6.egg/trac/web/main.py", line 522, in _dispatch_request
    dispatcher.dispatch(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3dev_r10719-py2.6.egg/trac/web/main.py", line 243, in dispatch
    resp = chosen_handler.process_request(req)
  File "/usr/local/lib/python2.6/dist-packages/Trac-0.12.3dev_r10719-py2.6.egg/trac/admin/web_ui.py", line 116, in process_request
    path_info)
  File "build/bdist.linux-i686/egg/acct_mgr/admin.py", line 158, in render_admin_panel
    return self._do_db_cleanup(req)
  File "build/bdist.linux-i686/egg/acct_mgr/admin.py", line 467, in _do_db_cleanup
    authenticated=None)
  File "build/bdist.linux-i686/egg/acct_mgr/api.py", line 531, in get_user_attribute
    m.update(''.join([account, str(authenticated), res_row.get('name')]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)

Perhaps a username with an unusual character? German Umlaut?

Replying to olistudent:

When clicking on the new menu cleanup I'm getting the following error:

2011-08-10 22:32:02,116 Trac[main] ERROR: Internal Server Error:
Traceback (most recent call last):
(...)
    m.update(''.join([account, str(authenticated), res_row.get('name')]))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe4' in position 1: ordinal not in range(128)

Perhaps a username with an unusual character? German Umlaut?

Well, the character mentioned in the error is a small German 'ä' (a with double dots on top) indeed. I'll look into it to change to all-Unicode input. Thanks for the report.

(In [10590]) AccountManagerPlugin: Fix UnicodeEncodeError? on non-ascii account names, refs #9051.

The hash function has been the culprit: Insisting to use string, not unicode, we have to encode it properly to something capable like UTF-8, or it bails out on dumb default ASCII encoding.

Replying to olistudent:

Perhaps a username with an unusual character? German Umlaut?

Please check this out, it should work now.

I can't confirm if it's working. The page is loading for 10 minutes now. The python process eats up all cpu and memory.

26935 www-data  20   0  764m 426m 2516 S 85.8 85.8  22:50.63 python

# cat trac.htpasswd | wc -l
5764

Mostly spammers I think. :-)

Ups, I started to think, how this would scale for a lot of accounts, next thing to do. :-(

And I see there is real need for better registration protection (see #843, #874, #2707, #2897, #5295, especially #7577 and even #8076 - well it obviously adds up, right?). Stay tuned.

(In [10592]) AccountManagerPlugin: Paginate accounts table for performance, refs #9051.

Please try this one. It's my first attempt on Trac native paginator integration and looks visually imperfect, but I'd like to hear, if it does the job before continuing with more cosmetic improvements.

Thanks for your hard work hasienda. I'm able to remove users created before the module was added and add/ remove anyone now. Works for me, cheers!

Replying to ponytrouble22@gmail.com:

[...] I'm able to remove users created before the module was added and add/ remove anyone now.

Great news. Sure, this has grown to a number of challenges indeed, not expected from the start. But many more will profit from the fixes along the way and newly added functionality, no doubt. You're welcome, but you did your share as well by testing the development code, right?

Works for me, cheers!

I do enjoy this progress a lot. Thank you for cooperation.

(In [10593]) AccountManagerPlugin: Make accounts per page adjustable in cleanup admin page, refs #9051.

With trac.util.presentation.Paginator this as become another powerful tool for review and administrativ cleanup making direct access to Trac db obsolete at many more occasions then AcctMgr allowed before.

(In [10595]) AccountManagerPlugin: Replace env.get_known_users() for account listing, refs #9051 and #9090.

With get_user_attribute() 'name' and 'email' attributes are displayed even for users, that have never been authenticated before.

(In [10598]) AccountManagerPlugin: Change cleanup admin page into users admin subpage, refs #9051.

Working against menu bloat here, and max_per_page setting is preserved now when using paginator navigation as well.

(In [10618]) AccountManagerPlugin: Publish maintenance release 0.3.2, closes #9051, #9082, #9088, #9091, #9092, #9093, #9095, #9099, #9107, #9108 and #9109.

This is an update for current stable at 0.3.1 with a number of fixes for issues reported within the last weeks.

While they will go into acct_mgr-0.4 too, current code isn't ready for release yet and will introduce a number of backwards-incompatible changes. So don't hurry for acct_mgr-0.4 right now.

Just noticed what I'd call a bug in signatures.py and removed unreasonable dependency on identical absolute path for successful check. Looks like nobody else tried this by now, right? Hey folks!

(In [10620]) AccountManagerPlugin: Fix regression in paginator navigation, refs #9051.

Changeset [10598] broke links for prev/next arrow in paginator navigation when changing cleanup admin page into users admin subpage.