6 Python Performance Tips

[this post was written by john paul mueller]

python is such a cool language because you can do so much with it in such a short time with so little code. not only that, it supports many tasks, such as multiprocessing, with ease.

python detractors sometimes claim python is slow. but it doesn’t have to be that way: try these six tips to speed up your python applications.

1. rely on an external package for critical code

python makes many programming tasks easy, but it may not always provide the best performance with time-critical tasks. using a c, c++, or machine language external package for time-critical tasks can improve application performance. these packages are platform-specific, which means that you need the appropriate package for the platform you’re using. in short, this solution gives up some application portability in exchange for performance that you can obtain only by programming directly to the underlying host. here are some packages you should consider adding to your performance arsenal:

• cython
• pyinlne
• pypy
• pyrex

the packages act in different ways. for example, pyrex makes it possible to extend python to do things like use c data types to make memory tasks more efficient or straightforward. pyinline lets you use c code directly in your python application. the inline code is compiled separately, but it keeps everything in one place while making use of the efficiencies that c can provide.

2. use keys for sorts

there is a lot of really old python sorting code out there that will cost you time in creating a custom sort and speed in actually performing the sort during runtime. the best way to sort items is to use keys and the default sort() method whenever possible. for example, consider the following code:

import operator
somelist = [(1, 5, 8), (6, 2, 4), (9, 7, 5)]
somelist.sort(key=operator.itemgetter(0))
somelist
#output = [(1, 5, 8), (6, 2, 4), (9, 7, 5)]
somelist.sort(key=operator.itemgetter(1))
somelist
#output = [(6, 2, 4), (1, 5, 8), (9, 7, 5)]
somelist.sort(key=operator.itemgetter(2))
somelist
#output = [(6, 2, 4), (9, 7, 5), (1, 5, 8)],

in each case the list is sorted according to the index you select as part of the key argument. this approach works just as well with strings as it does with numbers.

3. optimizing loops

every programming language emphasizes the need to optimize loops. when working with python, you can rely on a wealth of techniques for making loops run faster. however, one method developers often miss is to avoid the use of dots within a loop. for example, consider the following code:

lowerlist = ['this', 'is', 'lowercase']
upper = str.upper
upperlist = []
append = upperlist.append
for word in lowerlist:
    append(upper(word))
    print(upperlist)
    #output = ['this', 'is', 'lowercase']

every time you make a call to str.upper, python evaluates the method. however, if you place the evaluation in a variable, the value is already known and python can perform tasks faster. the point is to reduce the amount of work that python performs within loops because the interpreted nature of python can really slow it down in those instances.

( note: there are many ways to optimize loops; this is only one of them. for example, many programmers would say that list comprehension is the best way to achieve speed benefits in loops. the key is that optimizing loops is one of the better way to achieve higher application speed.)

4. use a newer version

anyone who searches python information online will find countless messages asking about moving from one version of python to another. in general, every version of python included optimizations that make it faster than the previous version. the limiting factor is whether your favorite libraries have also made the move to the newer version of python. rather than asking whether the move should be made, the key question is determine when a new version has sufficient support to make a move viable.

you need to verify that your code still runs. you need to use the new libraries you obtained to use with the new version of python and then check your application for breaking changes. only after you make the required corrections will you notice any difference.

however, if you just ensure your application runs with the new version, you could miss out on new features found in the update. once you make the move, profile your application under the new version, check for problem areas, and then update those areas to use new version features first. users will see a larger performance gain earlier in the upgrade process.

5. try multiple coding approaches

using precisely the same coding approach every time you create an application will almost certainly result in some situations where the application runs slower than it might. try a little experimentation as part of the profiling process. for example, when managing items in a dictionary, you can take the safe approach of determining whether the item already exists and update it or you can add the item directly and then handle the situation where the item doesn’t exist as an exception. consider this first coding example:

n = 16
mydict = {}
for i in range(0, n):
    char = 'abcd'[i%4]
    if char not in mydict:
        mydict[char] = 0
        mydict[char] += 1
        print(mydict)

this code will generally run faster when mydict is empty to start with. however, when mydict is usually filled (or at least mostly filled) with data, an alternative approach works better.

n = 16
mydict = {}
for i in range(0, n):
    char = 'abcd'[i%4]
    try:
        mydict[char] += 1
    except keyerror:
        mydict[char] = 1
    print(mydict)

the output of {'d': 4, 'c': 4, 'b': 4, 'a': 4} is the same in both cases. the only difference is how the output is obtained. thinking outside the box and creating new coding techniques can help you obtain faster results with your applications.

6. cross-compile your application

developers sometimes forget that computers don’t actually speak any of the languages used to create modern applications. computers speak machine code. in order to run the application, you use an application to convert the human readable code you use into something the computer can understand. there are times when writing an application in one language, such as python, and running it in another language, such as c++, makes sense from a performance perspective. it depends on what you want the application to do and the resources that the host system can provide.

one interesting cross-compiler, nuitka , converts your python code into c++ code. the result is that you can execute the application in native mode instead of relying on an interpreter. depending on the platform and task, you could see a significant performance increase.

( note: nuitka is currently in beta, so use it with care on production applications. in fact, it’s best used for experimentation right now. there is also some discussion as to whether cross-compilation is the best way to achieve better performance. developers have used cross-compilation for years to achieve specific goals, such as better application speed. just remember that every solution comes with trade-offs and you should consider them before using the solution in a production environment.)

when working with a cross-compiler, be sure it supports the version of python you work with. nuitka supports python 2.6, 2.7, 3.2, and 3.3. to make this solution work, you need both a python interpreter and a c++ compiler. nuitka supports a number of c++ compilers, including microsoft visual studio , mingw , and clang/llvm .

cross-compilation can bring some serious downsides. for example, when working with nuitka, you find that even a small program can consume major drive space because nuitka implements python functionality using a number of dynamic link libraries (dlls). so this solution may not work well if you’re dealing with a resource-constrained system.

bottom line

each of the six tips in this article can help you create faster python applications. but there are no silver bullets. none of the tips will work every time. some work better than others with specific versions of python—even the platform can make a difference. you need to profile your application to determine where it works slowly and then try the tips that appear to best address those issues.