Investigate South's resilience to failed migrations

[geofft]

We've seen that it's possible for South to silently ignore some migrations and carry on to future migrations (in particular see r31e4eaa4e417aa48ec6bcff79f4e792abf7a7763). We don't currently understand in perfect detail how South searches for valid migration files, and stores its current state.

It would be nice to ensure that if someone botches a migration, or if South changes in some backwards-incompatible way, manage.py won't attempt migrations out of order (because on our current database server, South goes "sorry, would you like to drop the DB and try again?", which isn't particularly practical).

[adehnert]

FWIW, ​http://south.aeracode.org/ticket/570#comment:12 is vaguely relevant here: IIRC, Django South does not store ordering of migrations in different apps. This can be an issue if you have changes that depend on each other. This mostly works if you run the migrations a couple times... They'll usually eventually finish successfully...

[adehnert]

Ah, okay, so the current code finds migration files as follows:

    if getattr(settings, "SOUTH_USE_PYC", False):
        MIGRATION_FILENAME = re.compile(r'(?!__init__)' # Don't match __init__.py
                                        r'[0-9a-zA-Z_]*' # Don't match dotfiles, or names with dots/invalid chars in them
                                        r'(\.pyc?)?$')     # Match .py or .pyc files, or module dirs
    else:
        MIGRATION_FILENAME = re.compile(r'(?!__init__)' # Don't match __init__.py
                                        r'[0-9a-zA-Z_]*' # Don't match dotfiles, or names with dots/invalid chars in them
                                        r'(\.py)?$')       # Match only .py files, or module dirs

I can't *really* imagine that regexp getting much more restrictive, so I don't foresee a repeat of the problematic commit that Geoff pointed out above.

[adehnert]

(Oh, also, South appears to yell now if you have a badly named migration, so we also won't have people screwing that one up.)

[adehnert]

Replying to adehnert:

FWIW, ​http://south.aeracode.org/ticket/570#comment:12 is vaguely relevant here: IIRC, Django South does not store ordering of migrations in different apps. This can be an issue if you have changes that depend on each other. This mostly works if you run the migrations a couple times... They'll usually eventually finish successfully...

Huh, neat, apparently South supports dependencies. We should try to habitually use dependencies between our apps (forms and groups). Relevant docs are at ​http://south.aeracode.org/docs/dependencies.html.

There's also a "migrationcheck" command in the South repo, though I don't think it's released yet. It claims that our migrations are fine, though I'm not sure I believe it. OTOH, running "./manage.py syncdb; ./manage.py migrate" on a new checkout seemed to work fine, so maybe it's not on crack.

[adehnert]

Pushing upstream to implement ​http://south.aeracode.org/ticket/509 (by which I probably mean "implement it ourself and get it accepted") may also be productive, since it'll make it easier to maintain dependency information.