Skip main navigation

Speeding up complex expressions with Numexpr

In this article we show how to speed up complex expressions with Numexpr.
© CC-BY-NC-SA 4.0 by CSC - IT Center for Science Ltd.

Complex expressions with large NumPy arrays present a bit of a catch-22 situation performance-wise.

On the one hand, using a one-liner for the expression is not a good idea due to high memory usage (unnecessary temporary arrays that are need for the evaluation).

On the other hand, evaluation of the expression with one operation at a time can lead into suboptimal performance. Effectively, one carries out multiple for loops in the NumPy C-code.

Numexpr package provides tools for fast evaluation of array expressions.

x = numpy.random.random((1000000, 1))
y = numpy.random.random((1000000, 1))

import numexpr
poly = numexpr.evaluate("((.25*x + .75)*x - 1.5)*x - 2")

The expression is enclosed in quotes and will be evaluated using a single
C-loop. Speed-ups in comparison to NumPy are typically between 0.95 and 4.
Performance improves normaly most with arrays that do not fit in CPU cache.

Supported operators and functions include e.g.:

  • +, -, *, /, **
  • sin, cos, tan
  • exp, log, sqrt

Thread support

By default, numexpr tries to use multiple threads, which can also speed up
the execution. The number of threads can be queried and set with:

numexpr.set_num_threads(n)

The number of threads can also be controlled by the environment variables
OMP_NUM_THREADS or NUMEXPR_NUM_THREADS.

© CC-BY-NC-SA 4.0 by CSC - IT Center for Science Ltd.
This article is from the free online

Python in High Performance Computing

Created by
FutureLearn - Learning For Life

Our purpose is to transform access to education.

We offer a diverse selection of courses from leading universities and cultural institutions from around the world. These are delivered one step at a time, and are accessible on mobile, tablet and desktop, so you can fit learning around your life.

We believe learning should be an enjoyable, social experience, so our courses offer the opportunity to discuss what you’re learning with others as you go, helping you make fresh discoveries and form new ideas.
You can unlock new opportunities with unlimited access to hundreds of online short courses for a year by subscribing to our Unlimited package. Build your knowledge with top universities and organisations.

Learn more about how FutureLearn is transforming access to education