Changeset 356174b

Timestamp:

Apr 29, 2015, 9:16:12 PM (11 years ago)

Author:

Alex Dehnert <adehnert@…>

Branches:

master, stable

Children:

093448c

Parents:

82e595e

git-author:

Alex Dehnert <adehnert@…> (04/29/15 21:16:12)

git-committer:

Alex Dehnert <adehnert@…> (04/29/15 21:16:12)

Message:

Improve perf and correctness of export_finboard_groups.py

export_finboard_groups.py was taking about 45 seconds to run in prod, and
including former signatories.

Changes:

• Stop using prefetch_related
  • As being used, the return value wasn't being saved as a new queryset, so it wasn't effective
  • Fixing that revealed that "officerrole" is the wrong field name
  • Finally, it requires doing filter (for role and current officers) in Python, which seemed a bit sad, and seemed like it would plausibly be appreciably slower due to needing to pull all past officers
• Don't reuse groups, because it shadows the module name
• Indent the JSON for improved readability

File:

• asadb/groups/export_finboard_groups.py (modified) (2 diffs)

asadb/groups/export_finboard_groups.py

@@ -24 +24 @@
 
 if __name__ == '__main__':
-    groups = groups.models.Group.objects.filter(group_status__is_active=True)
-    groups = groups.filter(group_funding__slug='undergrad')
-    groups = groups.filter(officeholder__role__slug__in=ROLES)
-    groups.prefetch_related('officeholder_set', 'officeholder_set__officerrole')
-    groups = groups.distinct()
+    gs = groups.models.Group.objects.filter(group_status__is_active=True)
+    gs = gs.filter(group_funding__slug='undergrad')
 
-    group_dicts = []
-    for group in groups:
-        # do the processing in python because of the prefetch_related
-        officers = list(group.officeholder_set.all())
+    officers = groups.models.OfficeHolder.current_holders
+    officers = officers.filter(group__in=gs, role__slug__in=ROLES)
+    officers = officers.select_related('role__slug', 'group')
+
+    group_dicts = {}
+    for group in gs:
         group_dict = {
             'id': group.id,
@@ -40 +39 @@
         }
         for role in ROLES:
-            group_dict[role] = [officer.person for officer in officers
-                                if officer.role.slug == role]
-        group_dicts.append(group_dict)
+            group_dict[role] = []
+        group_dicts[group.id] = group_dict
+
+    for officer in officers:
+        group_dicts[officer.group.id][officer.role.slug].append(officer.person)
 
     out = codecs.getwriter('utf-8')(sys.stdout)
     json.dump({
-        'groups': group_dicts,
+        'groups': group_dicts.values(),
         'date': datetime.datetime.now().isoformat(),
-    }, out)
+    }, out, indent=2)