Author: masnun

Python

Async execution in Python using multiprocessing Pool

Post author By masnun
Post date January 1, 2014
6 Comments on Async execution in Python using multiprocessing Pool

The “multiprocessing” module has a class Pool that is quite convenient if we want to do parallel processing. The API is simple and rather straightforward. I wanted to see if I can craft an example out of the official docs and here’s the code:

from multiprocessing import Pool
from time import sleep
from random import randint
import os


class AsyncFactory:
    def __init__(self, func, cb_func):
        self.func = func
        self.cb_func = cb_func
        self.pool = Pool()

    def call(self,*args, **kwargs):
        self.pool.apply_async(self.func, args, kwargs, self.cb_func)

    def wait(self):
        self.pool.close()
        self.pool.join()



def square(x):
    sleep_duration = randint(1,5)
    print "PID: %d \t Value: %d \t Sleep: %d" % (os.getpid(), x ,sleep_duration)
    sleep(sleep_duration)
    return x*x

def cb_func(x):
    print x

async_square = AsyncFactory(square, cb_func)

async_square.call(1)
async_square.call(2)
async_square.call(3)
async_square.call(4)
async_square.call(5)

async_square.wait()

from multiprocessing import Pool

from time import sleep

from random import randint

import os

class AsyncFactory:

def __init__(self, func, cb_func):

self.func = func

self.cb_func = cb_func

self.pool = Pool()

def call(self,*args, **kwargs):

self.pool.apply_async(self.func, args, kwargs, self.cb_func)

def wait(self):

self.pool.close()

self.pool.join()

def square(x):

sleep_duration = randint(1,5)

print "PID: %d \t Value: %d \t Sleep: %d" % (os.getpid(), x ,sleep_duration)

sleep(sleep_duration)

return x*x

def cb_func(x):

print x

async_square = AsyncFactory(square, cb_func)

async_square.call(1)

async_square.call(2)

async_square.call(3)

async_square.call(4)

async_square.call(5)

async_square.wait()

Let’s see whta we’re doing here – we have two functions, the main function “square” and the callback function “cb_func” which just prints out the result. We want to run the main function asynchronously and as soon as the result is available, we want the callback to execute, printing out the result.

What we’re doing inside the AsyncFactory is storing the function and callback function inside an instance. Then we create a worker pool. The worker pool by default uses the available CPUs. We can also pass values to the “processes” argument to determine the number of worker processes in the pool.

Then we repeatedly call the apply_async on the Pool object to pass the function with the arguments. Finally, we wait for the pool to close it’s workers and rest in peace. As soon as each worker returns a value, the callback would print it out. We have also added a print statement to check the process ID, passed value and sleep duration.

Isn’t it fun?

PHP

Generators and Coroutines in PHP

Post author By masnun
Post date December 27, 2013

If you’re a nerd and would love a very detailed post on Coroutines, here’s one for you – http://nikic.github.io/2012/12/22/Cooperative-multitasking-using-coroutines-in-PHP.html. If you decide to read the above mentioned article, I believe you can safely skip my article below, it doesn’t have anything extra, rather it tries to explain co-routines on a rush.

NOTE: The features of PHP described in this post were introduced on PHP 5.5.0. If you are using an older version, this is not going to work. Please upgrade your PHP version and enjoy the performance bonus with these awesome new additions! 🙂

Before we can start on Coroutines, let’s review our knowledge on Generators. Generators allow us to quickly create iterators which instead of populating a large dataset once, populate each item in turn, one after another. Since one item is produced/returned every turn, it is very memory efficient. In fact you can handle an unlimited stream of data with minimal memory usages using iterators. Let’s see a quick example:

<?php

function gen() {
    for($i=0; $i < 100; $i++) {
        yield $i;
    }
}

$x = gen();
foreach ($x as $value) {
    echo $value.PHP_EOL;   
}

<?php

function gen() {

for($i=0; $i < 100; $i++) {

yield $i;

}

$x = gen();

foreach ($x as $value) {

echo $value.PHP_EOL;

}

So what does it do? The gen() function returns an iterator of the “Generator” class. (If you look at the manual, Generator class implements the “Iterator” interface and thus we can use foreach on the class).

Internally, when we iterate over the generator object, it starts executing the gen() function. When it hits the yield statement, it pauses execution of the function and returns the value to the generator object (and thus back to our main loop). We then print the value. In the next iteration, the gen() function is resumed from where it was paused and executed until it hits the yield statement again.

Let’s see an example of generator which doesn’t use any loops inside and rather has multiple yield statements. This should help us in better understanding the use of yield in generators.

<?php

function gen() {
    yield 1;
    yield 2;
    yield 3;
    yield 4;
    yield 5;

}

$x = gen();
foreach($x as $val) {
    echo $val.PHP_EOL;
}

<?php

function gen() {

yield 1;

yield 2;

yield 3;

yield 4;

yield 5;

}

$x = gen();

foreach($x as $val) {

echo $val.PHP_EOL;

}

The above example should explain better how Generators are actually interruptible functions which can be paused and resumed with the yield statement to make iteration possible on the returned values. Now, as long as Generators are concerned, the function returns value. This is a one way communication. But what if we want to send back a value to the function? Coroutines come into play. Coroutines allow two way communication through the “send()” method on a “Generator” class. When we send a value back to the function, yield switches it’s role. Instead of returning a value back to the iteration process, it then receives the value we send into the function. A very plain example to describe this:

<?php

function gen() {
    echo "[1] Incoming value: ".yield.PHP_EOL;
    echo "[2] Incoming value: ".yield.PHP_EOL;
    echo "[3] Incoming value: ".yield.PHP_EOL;
    echo "[4] Incoming value: ".yield.PHP_EOL;
    echo "[5] Incoming value: ".yield.PHP_EOL;
}

$x = gen();
$x->send(5);
$x->send(4);
$x->send(3);
$x->send(2);
$x->send(1);

<?php

function gen() {

echo "[1] Incoming value: ".yield.PHP_EOL;

echo "[2] Incoming value: ".yield.PHP_EOL;

echo "[3] Incoming value: ".yield.PHP_EOL;

echo "[4] Incoming value: ".yield.PHP_EOL;

echo "[5] Incoming value: ".yield.PHP_EOL;

}

$x = gen();

$x->send(5);

$x->send(4);

$x->send(3);

$x->send(2);

$x->send(1);

What’s happening here? We’re sending a value back and the value is available to the “yield” statement inside the function.

We can return and receive values with the same yield statement. Let’s see how:

<?php

function gen() {
    echo "[1] Incoming value: ". (yield 1) . PHP_EOL;
    echo "[2] Incoming value: ". (yield 2) . PHP_EOL;
    echo "[3] Incoming value: ". (yield 3) . PHP_EOL;
    echo "[4] Incoming value: ". (yield 4) . PHP_EOL;
    echo "[5] Incoming value: ". (yield 5) . PHP_EOL;
       
}

$x = gen();
echo "Iterator value: ". $x->current(). PHP_EOL;
echo "Iterator value: ". $x->send(5). PHP_EOL;
echo "Iterator value: ". $x->send(4). PHP_EOL;
echo "Iterator value: ". $x->send(3). PHP_EOL;
echo "Iterator value: ". $x->send(2). PHP_EOL;
echo "Iterator value: ". $x->send(1). PHP_EOL;

<?php

function gen() {

echo "[1] Incoming value: ". (yield 1) . PHP_EOL;

echo "[2] Incoming value: ". (yield 2) . PHP_EOL;

echo "[3] Incoming value: ". (yield 3) . PHP_EOL;

echo "[4] Incoming value: ". (yield 4) . PHP_EOL;

echo "[5] Incoming value: ". (yield 5) . PHP_EOL;

}

$x = gen();

echo "Iterator value: ". $x->current(). PHP_EOL;

echo "Iterator value: ". $x->send(5). PHP_EOL;

echo "Iterator value: ". $x->send(4). PHP_EOL;

echo "Iterator value: ". $x->send(3). PHP_EOL;

echo "Iterator value: ". $x->send(2). PHP_EOL;

echo "Iterator value: ". $x->send(1). PHP_EOL;

Awesome no? Well, Coroutines are definitely complex and sometimes very hard to handle. If you don’t get the concepts at the first look or would love to understand more of the internal implementation, I recommend you read and understand the article I mentioned at the beginning. And do play with it. Have fun!

PHP

Multithreading in PHP: Doing it right!

Post author By masnun
Post date December 15, 2013
20 Comments on Multithreading in PHP: Doing it right!

Long ago I wrote a similar post on multi threading in php – http://www.masnun.com/2009/09/07/php-multi-threading.html – which allowed us to fork a new process and execute codes in parallel. While it often served the purpose right, it was not true multi-threading.

I have come across the awesome “pthreads” PECL extension for some time. Tonight, I planned to give it a shot. What sparked my curiosity? Well, let me quote them –

this is not a hack, we don’t use forking or any other such nonsense, what you create are honest to goodness posix threads that are completely compatible with PHP and safe … this is true multi-threading 🙂

True multi-threading in PHP, isn’t that worth wasting a night’s sleep to check out?

Installation

Since it’s a PECL extension, you can just do –

pecl install pthreads

1	pecl install pthreads

It requires ZTS aka thread safety enabled in PHP. So if your PHP is non-zts, please make sure to recompile with zts. Please consult appropriate docs on your platform on how to do that.

If you’re on OS X and use homebrew php, you can use the “–with-thread-safety” flag to reinstall php with ZTS enabled. Now install the extension and it should work.

Let’s write some codes

Of course, here we go!

<?php

/**
 * Author: Abu Ashraf Masnun
 * URL: http://masnun.me
 */

class WorkerThreads extends Thread
{
    private $workerId;

    public function __construct($id)
    {
        $this->workerId = $id;
    }

    public function run()
    {
        sleep(rand(0, 3));
        echo "Worker {$this->workerId} ran" . PHP_EOL;
    }
}

// Worker pool
$workers = [];

// Initialize and start the threads
foreach (range(0, 5) as $i) {
    $workers[$i] = new WorkerThreads($i);
    $workers[$i]->start();
}

// Let the threads come back
foreach (range(0, 5) as $i) {
    $workers[$i]->join();
}

<?php

/**

* Author: Abu Ashraf Masnun

* URL: http://masnun.me

class WorkerThreads extends Thread

{

private $workerId;

public function __construct($id)

{

$this->workerId = $id;

}

public function run()

{

sleep(rand(0, 3));

echo "Worker {$this->workerId} ran" . PHP_EOL;

}

// Worker pool

$workers = [];

// Initialize and start the threads

foreach (range(0, 5) as $i) {

$workers[$i] = new WorkerThreads($i);

$workers[$i]->start();

}

// Let the threads come back

foreach (range(0, 5) as $i) {

$workers[$i]->join();

}

The code is pretty straightforward. We extend the Thread class which has an abstract method – “run” which we must implement with our business logic. The codes inside this method actually get executed when the threads run. Then we create 5 instances and call the “start” method. After the threads have started, we again call the “join” method on each. This will ask our main thread to wait for the child threads to return.

As you can see, the main worker threads all output their ID, thus we can identify which one executed when. This could be helpful to determine if they were actually running in threads (and thus being async in nature) or ran one after one.

I got these outputs from several run:

 ~/Codes/php ⮀ php threads.php
Worker 3 ran
Worker 1 ran
Worker 4 ran
Worker 2 ran
Worker 0 ran
Worker 5 ran
 ~/Codes/php ⮀ php threads.php
Worker 4 ran
Worker 2 ran
Worker 3 ran
Worker 1 ran
Worker 5 ran
Worker 0 ran
 ~/Codes/php ⮀ php threads.php
Worker 4 ran
Worker 1 ran
Worker 3 ran
Worker 5 ran
Worker 0 ran
Worker 2 ran
 ~/Codes/php ⮀

~/Codes/php ⮀ php threads.php

Worker 3 ran

Worker 1 ran

Worker 4 ran

Worker 2 ran

Worker 0 ran

Worker 5 ran

~/Codes/php ⮀ php threads.php

Worker 4 ran

Worker 2 ran

Worker 3 ran

Worker 1 ran

Worker 5 ran

Worker 0 ran

~/Codes/php ⮀ php threads.php

Worker 4 ran

Worker 1 ran

Worker 3 ran

Worker 5 ran

Worker 0 ran

Worker 2 ran

~/Codes/php ⮀

Okay, so looks like it worked. No?

Here’s some helpful links:

Github Repo: https://github.com/krakjoe/pthreads

Manual: http://docs.php.net/manual/en/book.pthreads.php

Recent Posts

Recent Comments

Archives

Categories