Want to keep learning?

This content is taken from the Partnership for Advanced Computing in Europe (PRACE)'s online course, Python in High Performance Computing. Join the course to learn more.

Skip to 0 minutes and 8 seconds Let’s see how to use the cProfile with a simple example. In the most basic use case one can measure the profile of the whole program with the cProfile by starting to python under the cProfile module with -m cProfile After that with -o we can provide the name of the file where we want to store the profiling results. In this case I choose just profile.dat, it can be any file. And finally we give the name of the file of our program we want to execute. So, we can see that it executes just as normally and this nice figure of Mandelbrot fractal appears in the screen, and we can see also printed result from the internal timer.

Skip to 1 minute and 10 seconds Now, in order to investigate the profile we use the pstats module. So we can start python with the module pstats and provide just the name of the file where the results were stored. This brings up now the interactive profile statistics browser. With help we can see information about all the possible commands that are available here. For example, we can look stats command which is used to printing out statistics and we can just say there stats and then print out then functions and modules. As you can see, the output is a bit messy here so we in many cases want to remote full path names of the modules here with the strip command.

Skip to 2 minutes and 10 seconds Also, the functions normally appear in a random order so we typically want to sort them according to some specified key. By typing sort alone, we can see all the possible keys that are available. Here we want to see how much time is spent in each function so we sort according the time which measures the time spent in the function itself. If we now print out stats with 10 again that shows us 10 most time-consuming functions in the program.

Skip to 2 minutes and 47 seconds What you appear to see here is that as first actually something which has spent almost 10 seconds, is main loop of some TKinter and this is actually now due to the fact that it took me some time to actually measure or close te graphical window appearing there. Normally we are not actually that interested at how long time the graphical window appears there so what we want to do is to disable the printing out or plotting out the figure before continuing with the profile. So, let’s exit the profiler and edit briefly our main file here , and comment out the plotting from this one.

Skip to 3 minutes and 48 seconds It’s a relatively common that there are some parts of the program that you are not interested in measuring, for example related to input and output which you might want to disable. We can now continue and run the program again under the cProfile just same as before and now the graphical window doesn’t appear and our previous profile got overwritten. If we now start again the interactive browser here, strip the path names there, sort according to time and print out the ten most time-consuming functions, we can see that now it’s actually the computational part that shows up here. Function kernel in the module mandel.py is definitely the most time-consuming function.

Skip to 4 minutes and 43 seconds It has been called 40,000 times and in total it has spent about 0.9 seconds. The other time we can see here, cumulative time, measures also the time spent in the children of the function, so if we sort according to cumtime and print now ten most time-consuming functions in this respect we can see that the first there is the main program which then calls all others and for example the compute_mandel function here itself it takes very little time, 0.04 seconds, but because the kernel is called within that function the cumulative time includes also time spent in this kernel.

Skip to 5 minutes and 38 seconds cProfile provides easy way to find out which are really the most time-consuming parts of the program and where you should focus your optimization efforts in future.

Using cProfile

cProfile is powerful performance analysis tool included in the Python standard library.

In this video we walk through the main steps when investigating performance of Python programs with cProfile. As an example, we use a simple program that calculated Mandelbrot fractal.

Share this video:

This video is from the free online course:

Python in High Performance Computing

Partnership for Advanced Computing in Europe (PRACE)