How I Assess the Memory Consumption of My Python Code
We used to focus on performance when we were writing code. It won't be difficult for us to get thousands of articles if we search "python performance". However, I rarely see articles that tell us to optimise the memory consumption of our code.
Of course, performance is critical in programming, but memory consumption will decide how much money we need to spend on hardware. Consider that we are going to train a model on a big dataset. We may need a large VM with 64GB RAM to do the job. However, if we can optimise our code, 32GB RAM may be able to achieve a similar elapsed time. In that case, we may save lots of cost for the hardware resource.
Before I can optimise our code to reduce memory consumption, we need some effective ways to measure it. Also, by knowing the approaches to measuring memory consumption, we will be able to quantify and benchmark it. So, in this article, the common memory-measuring approaches that I use to leverage will be introduced.
1. Python Built-In "Get Size Of"
Before introducing any fancy 3rd party libraries, do you know we have a Python built-in function that can measure the memory consumption of a variable? That is inside the sys module.
import sys
x = [1, 2, 3, 4, 5]
print(f"Size of list: {sys.getsizeof(x)} bytes")
In the above code, the function sys.getsizeof(x) will give us the memory consumption of the list x straightaway. However, we have to be careful that this function will only return the size of the variable (in this case, it is the list x) but not include the items it refers to (in this case, the integers).
If you want to know more about the details of this function, please check out the following article.
Anyway, when we only want to compare and benchmark the memory consumption, this function might be enough. See the example below.
So, the items (those integers) are exactly the same in x and y, but as a set, y needs more memory than x. This is enough information to know.
Please let me know if you are interested in why a set requires much more memory than a list. I will consider to share for the next
