4 Python Performance Techniques to Remember
Hey! A click-bait story for you to read while you’re ignoring your current task. Might as well (hopefully) learn something, right? At least it’s not pay-walled.
Well, here you go. Let’s think about some Python. Maybe these will help you out if you’re ever doing something somewhat performance-oriented.
1. LRU Caches
Situation: You have a function that does repeated computations, or maybe it does lookups, and it’s very possible for it to do the same computation or lookup multiple times in a row. This can get resource intensive fast, for your code or for the service you’re hitting for the lookups, or perhaps both.
We can solve that with LRU Caches — “Least Recently Used”. This will cache the results of your function call so that if you call it the same way again, it will grab the result from a cache instead of running the whole procedure.
Using it is easy. Let’s see how:
from functools import lru_cache# The same function twice, but one of them is cached!def uncached_lookup(big_tuple: tuple, number_to_find: int):
return number_to_find in big_tuple@lru_cache(maxsize=256)
def cached_lookup(big_tuple: tuple, number_to_find: int):
return number_to_find in big_tuple
That’s… that’s it. You just add a little decorator and choose a size, and Python takes care of the rest. There is a caveat — the types of the arguments to your function need to be hashable. So you can’t huck a dictionary in as an argument, but you can use a tuple, for example. The results are pretty staggering — I did a time test that you can find in a gist here:
mydriasis@akkad:~/Desktop$ python lru_sample.py
59000 uncached lookups took 0.8154561520ms
59000 cached lookups took 0.2635743618msWow! Good time savings.
2. Easy List De-Duping
Situation: You have a list. It has duplicates. You don’t care about the order. You don’t have to go making a new list and iterating it like this —
deduped_list = list()for item in big_list:
if item not in deduped_list:
deduped_list.append(item)
Seriously, please don’t do that. That’s horrifying. It’s also the top voted answer on the StackOverflow question about this. It’s extraordinarily unperformant, and a waste of several lines of code. Instead, just change it to a set and back:
deduped_list = list(set(big_list))That’s… that’s it. I know, these are really uh, mind blowing techniques. But boy the difference that makes. I did a time test that you can find in a gist here:
mydriasis@akkad:~/Desktop/samples$ python list_dedupe.py
Deduping 1000000-element list to 100 by creating a new list took 0.4950282574ms
Deduping 1000000-element list to 100 using set conversion took 0.0072016716msIt’s not even close. It’s not even in the same ballpark. I’m begging you, use the set casting method.
3. Membership Checking with Sets
Situation: You’ve got a big list or tuple or something, and you have some element. You want to see if the element you’ve got is in the big list or tuple.
Instead, start with a set of things, and look for membership in the set. This gives you deduplication for free, and set operations are staggeringly faster. You can still use the ‘in’ keyword as well. I did a time test that you can find in a gist here:
mydriasis@akkad:~/Desktop/samples$ python set_lookups.py
10000 lookups on list took 84.0385153294ms
10000 lookups on set took 0.0007445812msOK, maybe my setup is a touch biased. But still, if you’re doing big membership lookups, use sets. You won’t regret it. I swear.
4. Dataset XOR with Sets
Situation: You have two datasets. Maybe two lists? Two tuples? You want all of the things that are in one or the other, but not both. You don’t care about the ordering. This one’s a little trickier. You can easily write a nested loop to go over both of them:
xored_list = list()for item in big_list_1:
if item not in big_list_2:
xored_list.append(item)for item in big_list_2:
if item not in big_list_1:
xored_list.append(item)
Correction: A friend of mine pointed out that the code released with the article was wrong. I’ve gone ahead and fixed it ( I think ). Let me know if it seems legit.
That will do the trick, but as the size of the lists gets big, this gets very slow, very fast. Instead, we can use the built-in function for the set API called symmetric difference. To get the big idea, here’s the venn-diagram representation for two sets:
Behold! That’s just what we need. The code is super easy:
xored_list = list(set(big_list_1).symmetric_difference(big_list_2))That’s it. Really, that’s it. It’s even faster despite needing to cast to a new datatype. As is the theme, I did a time test that you can find in a gist here:
mydriasis@akkad:~/Desktop/article-code/perf$ python dataset_xor.py
List xor with 1000 items by iteration took 0.0177657604ms
List xor with 1000 items by set symmetric difference took 0.0002312660msHoly hell! That’s… Jeez! Yeah, don’t just iterate the lists. That’s rude. Maybe there’s another method that’s similar in performance to the set symmetric difference, but I can’t think of it off the top of my head. Also, these numbers make my article look better.
Conclusion
Well, those are uh, some things you could do. Maybe you’ll remember them the next time you’re writing some Python and you need to do… these specific things. Wait, where are you going? You weren’t impressed? Hey, don’t leave. Come on, I was as bored as you on a Saturday afternoon. It wasn’t that bad an article, was it? Oh you already knew about the list set dedupe thing? Well what about the symmetric difference thing, that was cool, right?
Resources
Set Symmetric Difference graphic: kismalac, CC0, via Wikimedia Commons
