Author: masnun

Many beginners seem to take the concept of decorators as a fairly advanced and complex topic. It’s advanced alright but it probably is much simpler than you think.

Decorators Explained

Let’s say, we have a function which returns a message. But we want to also return the time with the message. So what can we do? We can modify the function’s source code to add the time with the message. But what if we can’t or don’t want to modify the source code but still want to extend/transform the functionality?

In that case, we can wrap it within another function, something like this:

Here, greet was our original function, which only returns a message but no time with it. So we be clever and write a wrapper – time_wrapper. This wrapper function takes a function as it’s argument and returns the new_function instead. This new function, when invoked, can access the original function we passed, get the message out and then add the time to it.

The interesting bit is here – greet = time_wrapper(greet). We’re passing greet to time_wrapper. The time_wrapper function returns the new_function. So greet now points to the new_function. When we call greet, we actually call that function.

By definition, a Decorator is a callables which takes a callable and returns a callable. A callable can be a few things but let’s not worry about that right now. In most cases, a decorator just takes a function, wraps it and returns the wrapped function. The wrapped function can access a reference to our original function and call it as necessary. In our case time_wrapper is the decorator function which takes the greet function and returns the new_function.

The @ decorator syntax

But you might be wondering – “I see a lot of @ symbols while reading on decorators, how can there be a decorator without the @?”. Well, before PEP 0318, we used to write decorators like that. But soon the wise people of the Python community realized that it would be a good idea to have a nicer syntax for decorators. So we got the @. So how does the @ work?

So when we add a callable name prepended with a @ on top of a function, that function is passed to that callable. The return value from that callable becomes the new value of that function.

Writing our own decorators

Let’s say we want to write a decorator which will take a function and print the current time every time the function is executed. Let’s call our function timed. This function will accept a parameter fn which is the function we wrap. Since we need to return a function from the timed function, we need to define that function too.

In this example, the timed function takes the fn function and returns the wrapped function. So by definition it’s a decorator. Within the wrapped function, we’re first printing out the current time. And then we’re invoking the fn() function. After the decorator is applied, this wrapped function becomes the new fn. So when we call fn, we’re actually calling wrapped.

Let’s see example of this decorator:

With the @timed decorator applied to hello, this happens: hello = timed(hello), hello now points to the wrapped function returned by timed. Inside the for loop, every time we call, hello, it’s no longer the original hello function but the wrapped function. The wrapped function calls the copy of the original hello from it’s parent scope.

Two things you might have noticed – it is possible to nest functions and when we nest a function within a function, the inner function can access the parent scope too. You can learn more about the scope by reading on closure.

Decorator Parameters

Decorators can take parameters too. Like this:

When a decorator takes a parameter, it’s executed like:

As we can see, it gets a level deeper. Here sleeper has to take the parameter and return the actual decorator function which will transform our say_hello function.

In this case, sleeper(4) returns the decorator function. We pass say_hello to the decorator. The decorator wraps it inside the wrapped function and returns wrapped. So finally, say_hello is actually the wrapped function which gets fn and secs from the closure.

Chaining Decorators

We can chain multiple decorators. Like this:

The bottom most one gets executed first, then the returned function is passed to the decorator on top of that one. This way the chain of execution goes from bottom to top.

Using Classes as Decorators

In our previous examples, we have only focused on functions, but in Python, any callables can be used as decorator. That means we can uses Classes too. Let’s first see an example:

When we’re using the Sleeper decorator, we are getting the parameter 5 to the constructor. We are storing it in an instance variable. The constructor returns an object instance, when we call it, it gets the function and returns a decorated, wrapped function.

This is just like before, say_hello = Sleeper(5)(say_hello). The first call is the constructor. The second call is made to the __call__ magic method.

Decorating Class and Class Methods

We can decorate any callables, so here’s an example where we’re decorating a Class to forcefully convert the age argument to int.

We can decorate the methods as well. If you know Python’s OOP model well, you probably have already came across the @property decorator. Or the @classmethod and @staticmethod decorators. These decorate methods.

The concurrent.futures module is part of the standard library which provides a high level API for launching async tasks. We will discuss and go through code samples for the common usages of this module.

Executors

This module features the Executor class which is an abstract class and it can not be used directly. However it has two very useful concrete subclasses – ThreadPoolExecutor and ProcessPoolExecutor. As their names suggest, one uses multi threading and the other one uses multi-processing. In both case, we get a pool of threads or processes and we can submit tasks to this pool. The pool would assign tasks to the available resources (threads or processes) and schedule them to run.

ThreadPoolExecutor

Let’s first see some codes:

I hope the code is pretty self explanatory. We first construct a ThreadPoolExecutor with the number of threads we want in the pool. By default the number is 5 but we chose to use 3 just because we can ;-). Then we submitted a task to the thread pool executor which waits 5 seconds before returning the message it gets as it’s first argument. When we submit() a task, we get back a Future. As we can see in the docs, the Future object has a method – done() which tells us if the future has resolved, that is a value has been set for that particular future object. When a task finishes (returns a value or is interrupted by an exception), the thread pool executor sets the value to the future object.

In our example, the task doesn’t complete until 5 seconds, so the first call to done() will return False. We take a really short nap for 5 secs and then it’s done. We can get the result of the future by calling the result() method on it.

A good understanding of the Future object and knowing it’s methods would be really beneficial for understanding and doing async programming in Python. So I highly recommend taking the time to read through the docs.

ProcessPoolExecutor

The process pool executor has a very similar API. So let’s modify our previous example and use ProcessPool instead:

It works perfectly! But of course, we would want to use the ProcessPoolExecutor for CPU intensive tasks. The ThreadPoolExecutor is better suited for network operations or I/O.

While the API is similar, we must remember that the ProcessPoolExecutor uses the multiprocessing module and is not affected by the Global Interpreter Lock. However, we can not use any objects that is not picklable. So we need to carefully choose what we use/return inside the callable passed to process pool executor.

Executor.map()

Both executors have a common method – map(). Like the built in function, the map method allows multiple calls to a provided function, passing each of the items in an iterable to that function. Except, in this case, the functions are called concurrently. For multiprocessing, this iterable is broken into chunks and each of these chunks is passed to the function in separate processes. We can control the chunk size by passing a third parameter, chunk_size. By default the chunk size is 1.

Here’s the ThreadPoolExample from the official docs:

And the ProcessPoolExecutor example:

as_completed() & wait()

The concurrent.futures module has two functions for dealing with the futures returned by the executors. One is as_completed() and the other one is wait().

The as_completed() function takes an iterable of Future objects and starts yielding values as soon as the futures start resolving. The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result for the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.

Let’s see an example:

The wait() function would return a named tuple which contains two set – one set contains the futures which completed (either got result or exception) and the other set containing the ones which didn’t complete.

We can see an example here:

We can control the behavior of the wait function by defining when it should return. We can pass one of these values to the return_when param of the function: FIRST_COMPLETED, FIRST_EXCEPTION and ALL_COMPLETED. By default, it’s set to ALL_COMPLETED, so the wait function returns only when all futures complete. But using that parameter, we can choose to return when the first future completes or first exception encounters.