Opened 14 years ago

Closed 13 years ago

Last modified 13 years ago

#59 closed task (fixed)

Use transaction.commit_on_success in all the import scripts

Reported by: adehnert Owned by:
Priority: normal Milestone: Sometime
Component: Infrastructure Version:
Keywords: Cc:
Size: tiny

Description

We should be consistently using transaction.commit_on_success (https://docs.djangoproject.com/en/dev/topics/db/transactions/) in all our import scripts (and anything else running on the command line that's running lots of database queries). Not committing the transaction after each query (ie, disabling autocommit) routinely improves performance by an order of magnitude or two (on my dev setup using sqlite).

Change History (4)

comment:1 Changed 14 years ago by adehnert

  • Component changed from Default to Core
  • Size set to medium

comment:2 Changed 14 years ago by adehnert

A good tool for this is

git grep -E "DJANGO_SETTINGS|transaction.commit" 

This, in combination with checking for calls to "save", suggests the following files:

  • forms/update_validations.py
  • space/import_into_db.py
  • util/sync_moira_authz.py
  • util/update_old_previews.py

(This fits with my guess of what writes to the DB.)

comment:3 Changed 13 years ago by adehnert

  • Component changed from Core to Infrastructure
  • Size changed from medium to tiny

comment:4 Changed 13 years ago by adehnert

  • Resolution set to fixed
  • Status changed from new to closed

Fixed in commit df88ee988edcdf568f8a02e5b4954baa5d9e94a6:

commit df88ee988edcdf568f8a02e5b4954baa5d9e94a6
Author: Alex Dehnert <adehnert@mit.edu>
Date:   Thu May 31 02:37:46 2012 -0400

    Use transaction.commit_on_success for imports
    
    Fixed the following scripts:
    * forms/update_validations.py (15s->1s for 70ish validations)
    * space/import_into_db.py (67s->1s for 75 spaces and 100 assignments)
    * util/sync_moira_authz.py (65s->4s for importing ~50 new people)
    
    util/update_old_previews.py was skipped:
    * it's dead code, I believe --- I think previews are unused
    * preview updating is slow, IIRC; it's dubious we'd get noticeably better
      runtime on the script, and there may even be utility to the progress
      updates not doing this transactionally supplies
    
    This closes ASA Trac #59.

(Except for util/update_old_previews.py, which was skipped per commit message.)

Last edited 13 years ago by adehnert (previous) (diff)
Note: See TracTickets for help on using tickets.