#59 closed task (fixed)
Use transaction.commit_on_success in all the import scripts
Reported by: | adehnert | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | Sometime |
Component: | Infrastructure | Version: | |
Keywords: | Cc: | ||
Size: | tiny |
Description
We should be consistently using transaction.commit_on_success (https://docs.djangoproject.com/en/dev/topics/db/transactions/) in all our import scripts (and anything else running on the command line that's running lots of database queries). Not committing the transaction after each query (ie, disabling autocommit) routinely improves performance by an order of magnitude or two (on my dev setup using sqlite).
Change History (4)
comment:1 Changed 14 years ago by adehnert
- Component changed from Default to Core
- Size set to medium
comment:2 Changed 14 years ago by adehnert
comment:3 Changed 13 years ago by adehnert
- Component changed from Core to Infrastructure
- Size changed from medium to tiny
comment:4 Changed 13 years ago by adehnert
- Resolution set to fixed
- Status changed from new to closed
Fixed in commit df88ee988edcdf568f8a02e5b4954baa5d9e94a6:
commit df88ee988edcdf568f8a02e5b4954baa5d9e94a6 Author: Alex Dehnert <adehnert@mit.edu> Date: Thu May 31 02:37:46 2012 -0400 Use transaction.commit_on_success for imports Fixed the following scripts: * forms/update_validations.py (15s->1s for 70ish validations) * space/import_into_db.py (67s->1s for 75 spaces and 100 assignments) * util/sync_moira_authz.py (65s->4s for importing ~50 new people) util/update_old_previews.py was skipped: * it's dead code, I believe --- I think previews are unused * preview updating is slow, IIRC; it's dubious we'd get noticeably better runtime on the script, and there may even be utility to the progress updates not doing this transactionally supplies This closes ASA Trac #59.
(Except for util/update_old_previews.py, which was skipped per commit message.)
Note: See
TracTickets for help on using
tickets.
A good tool for this is
This, in combination with checking for calls to "save", suggests the following files:
(This fits with my guess of what writes to the DB.)