Async execution in Python using multiprocessing Pool

The “multiprocessing” module has a class Pool that is quite convenient if we want to do parallel processing. The API is simple and rather straightforward. I wanted to see if I can craft an example out of the official docs and here’s the code:

Let’s see whta we’re doing here – we have two functions, the main function “square” and the callback function “cb_func” which just prints out the result. We want to run the main function asynchronously and as soon as the result is available, we want the callback to execute, printing out the result.

What we’re doing inside the AsyncFactory is storing the function and callback function inside an instance. Then we create a worker pool. The worker pool by default uses the available CPUs. We can also pass values to the “processes” argument to determine the number of worker processes in the pool.

Then we repeatedly call the apply_async on the Pool object to pass the function with the arguments. Finally, we wait for the pool to close it’s workers and rest in peace. As soon as each worker returns a value, the callback would print it out. We have also added a print statement to check the process ID, passed value and sleep duration.

Isn’t it fun?


6 Comments Async execution in Python using multiprocessing Pool

  1. Pingback: برنامه نویسی موازی در پایتون – بخش سوم – پردازش کلان داده ها و سیستم های توزیع شده

  2. Paul hewlett

    It would be an improvement to make AsyncObject inherit from object:

    class AsyncFactory(object):

    and change the call method to __call__(self,…)

    then you don’t need to explicitly use call method:

    async_square(1)

     

     

    Reply
  3. Zaffy

    Hi Paul,

    Sorry for ignorance for the __call__ semantics. I am new to Python. Can you elaborate the following:

    and change the call method to __call__(self,…)

    Did you mean replace the 5 async_square.call(x) with a single __call__(self,…) method? And what should be the arguments to __call__?

    Thanks

    Zaffy

    Reply

Leave a Reply to Paul hewlett Cancel reply

Your email address will not be published. Required fields are marked *