Python: A quick introduction to the concurrent.futures module

The concurrent.futures module is part of the standard library which provides a high level API for launching async tasks. We will discuss and go through code samples for the common usages of this module.


This module features the Executor class which is an abstract class and it can not be used directly. However it has two very useful concrete subclasses – ThreadPoolExecutor and ProcessPoolExecutor. As their names suggest, one uses multi threading and the other one uses multi-processing. In both case, we get a pool of threads or processes and we can submit tasks to this pool. The pool would assign tasks to the available resources (threads or pools) and schedule them to run.


Let’s first see some codes:

I hope the code is pretty self explanatory. We first construct a ThreadPoolExecutor with the number of threads we want in the pool. By default the number is 5 but we chose to use 3 just because we can ;-). Then we submitted a task to the thread pool executor which waits 5 seconds before returning the message it gets as it’s first argument. When we submit() a task, we get back a Future. As we can see in the docs, the Future object has a method – done() which tells us if the future has resolved, that is a value has been set for that particular future object. When a task finishes (returns a value or is interrupted by an exception), the thread pool executor sets the value to the future object.

In our example, the task doesn’t complete until 5 seconds, so the first call to done() will return False. We take a really short nap for 5 secs and then it’s done. We can get the result of the future by calling the result() method on it.

A good understanding of the Future object and knowing it’s methods would be really beneficial for understanding and doing async programming in Python. So I highly recommend taking the time to read through the docs.


The process pool executor has a very similar API. So let’s modify our previous example and use ProcessPool instead:

It works perfectly! But of course, we would want to use the ProcessPoolExecutor for CPU intensive tasks. The ThreadPoolExecutor is better suited for network operations or I/O.

While the API is similar, we must remember that the ProcessPoolExecutor uses the multiprocessing module and is not affected by the Global Interpreter Lock. However, we can not use any objects that is not picklable. So we need to carefully choose what we use/return inside the callable passed to process pool executor.

Both executors have a common method – map(). Like the built in function, the map method allows multiple calls to a provided function, passing each of the items in an iterable to that function. Except, in this case, the functions are called concurrently. For multiprocessing, this iterable is broken into chunks and each of these chunks is passed to the function in separate processes. We can control the chunk size by passing a third parameter, chunk_size. By default the chunk size is 1.

Here’s the ThreadPoolExample from the official docs:

And the ProcessPoolExecutor example:

as_completed() & wait()

The concurrent.futures module has two functions for dealing with the futures returned by the executors. One is as_completed() and the other one is wait().

The as_completed() function takes an iterable of Future objects and starts yielding values as soon as the futures start resolving. The main difference between the aforementioned map method with as_completed is that map returns the results in the order in which we pass the iterables. That is the first result from the map method is the result for the first item. On the other hand, the first result from the as_completed function is from whichever future completed first.

Let’s see an example:

The wait() function would return a named tuple which contains two set – one set contains the futures which completed (either got result or exception) and the other set containing the ones which didn’t complete.

We can see an example here:

We can control the behavior of the wait function by defining when it should return. We can pass one of these values to the return_when param of the function: FIRST_COMPLETED, FIRST_EXCEPTION and ALL_COMPLETED. By default, it’s set to ALL_COMPLETED, so the wait function returns only when all futures complete. But using that parameter, we can choose to return when the first future completes or first exception encounters.

Parsing Upwork Job Feed to Monitor Clojure Jobs

I was checking Upwork to asses the job market for Clojure and it hit me – I can parse the Upwork Job Feed for Clojure and monitor it programmatically. So I fired up the REPL and started coding.

Before I began, I had to choose a Clojure library to parse RSS feeds. I went for So I added this dependency ([org.clojars.scsibug/feedparser-clj "0.4.0"]) to my project.clj:

Now we can start writing some codes. First, we would fetch the content of the RSS feed and parse it. The parse-feed function from the above mentioned library would do that for us.

Next, we need a function to extract the data we need. We will run this function (map) over the collection of items.

Here we’re simply getting the values of :title key and :uri key and putting them in another hashmap. We’re naming our key :url instead of their :uri

We can grab the collection of items in the :entries key of the feed variable we declared before. So here’s our main function:

We’re mapping the function we wrote over the entries and getting a collection of hashmaps. Then we’re using doseq to iterate over them and print the data out.

The final code looks like this:

Here, we have extracted only two fields and printed them out. We extracted the data into a new hashmap as an example. As a matter of fact, we could just print them out from the original feed variable. Then the code would have been shorter:

Web Scraping with Clojure

I have recently started learning Clojure and I must say I am totally hooked. Clojure is a sane Lisp on the JVM. So I can express myself better while being able to take advantage of the huge JVM eco system. This is just wonderful!

After finishing the popular book Clojure for the Brave and True, I wanted to try something out myself. I decided to try web scraping with clojure. In this post, I am going to walk you through a very simple web scraping task.

We’re going to scrape KAT (Kick Ass Torrents) for TV series. To keep things simple, we would scrape the first page of the TV section and print out the titles. The reason I like KAT is they serve the response gzipped – if your http client can’t handle their response, you probably want to switch.

We will use the following libraries for the task:

  • http-kit
  • Enlive

Most tutorials for Clojure would use Java’s built in URL class with Enlive’s html-resource function but in our case it would not work, because it can’t handle compressed responses well. So we will use http-kit instead.

To begin with, we would add these libraries to our project.clj file (assuming we’re using Leiningen).

Now we’re ready to start writing the codes. Let’s first :require our libraries.

First we have to fetch the HTML response. We can use http-kit’s get function to grab the HTML. This function would return a promise. So we would have to dereference it using deref or the shorthand syntax @. When the promise is resolved, we would get a hashmap which would have a :body key along with :status and few other keys related to the request. We can pass this HTML response to Enlive’s html-snippet function to get an iterable DOM like object from which we can select the elements using select function.

We are using the {:insecure? true} part to ignore issues with SSL. So far, we have a function get-dom which would give us a DOM like object on which we can do select. We will now write another function which will extract the titles from this DOM like object.

Each Torrent title (which is a link, aka anchor tag) has the CSS class cellMainLink so we can select a.cellMainLink to get all the title links. Each title link would have their text part in the :content key. Each text part in the :content key is a vector. So we would need to use first on it to grab the actual text. Here’s what I wrote:

I simply could not resist using comp to do some magic here. comp allows us to combine two functions to compose one which allowed us to first grab the content and then get the first element in our case.

Finally, we can run our functions like this:

Here’s the complete file:

The code is under 20 lines! 😀

Auto Update for your Frozen Python Applications using Esky

Let’s assume you have a desktop application built with Python. It could be a traditional GUI app built with PyQT/wxPython/Kivy or any other GUI framework. Or it could be a web server that serves a browser based HTML GUI for the user. Either way, you have “frozen” the app using cx_freeze, py2app/py2exe or pyinstaller and now you want to add “auto update” to the app, so when there’s a new version of the application is available, the app can download and install the update, automatically. For this particular task, I found esky to be a good viable option. In this article, I am going to demonstrate how we can use esky to deliver updates to our apps.

Quick Introduction to Esky

If we want to use Esky to deliver updates, we need to freeze the app first. But this time, we will ask Esky to freeze the app for us, using our freezer of choice. For example, if we used py2app before, we will still use py2app but instead of directly using it, we will pass it to Esky and Esky will use the py2app to freeze the app for us. This step is necessary so that Esky can inject the necessary parts to handle updates/patches and install them gracefully.

For the apps to locate and download the updates, we need to serve the updates from a location on the internet/local network. Esky produces a zip archive. We can directly put it on our webserver. The apps we freeze needs to know the URL of the webserver and must have access to it.

On the other hand, inside our app, we need to write some codes which will scan the URL of the above mentioned webserver, find any newer updates and install them. Esky provides nice APIs to do these.

So now that we know the steps to follow, let’s start.

Creating a setup file

If you have frozen an app before, you probably know what a setup file is and how to write one. Here’s a sample that uses py2app to freeze an app:

Now we can generate the frozen app using:

This should generate a zip archive in the dist directory.

Hosting the app

Collect the zip file from the dist directory and put it somewhere accessible on the internet. For local testing, you can probably use Python’s built in HTTP server to distribute it.

Finding, Downloading and Installing Updates

Now we will see the client side code that we need to write to locate and install the updates.

Here’s some codes taken from a PyQT app. The find_esky_update method is part of a QMainWindow class. It is called inside the onQApplicationStarted method. So it checks the update as soon as the application starts.

We first check if the app is frozen. If it’s not, then there’s no way we can install updates. sys.frozen will contain information about the app if it’s frozen. Otherwise it will not be available. So we first ensure that it is indeed a frozen app.

Then we create an Esky app instance by providing it the URL of our webserver (where the updates are available). We only pass the root URL (without the zip file name). The find_update() method on the Esky app will find newer update and return some information if a new update is available. Otherwise it will be falsy.

If an update is available, we ask our user if s/he wants to update. Here we used QMessageBox for that. If they agree, we call the auto_update method with a callback. We will see the callback soon. The auto_update downloads the update and installs it. The callback we pass – it gets called every time something happens during the process. It can be a good way to display download progress using this callback.

Let’s see our example code here:

As you can see from the code, the callback gets a dictionary which has a key status and if it is “downloading”, we also have the amount of data we have received so far and the total size. We can use this to calculate the progress and print it. We can also display a nice progress bar if we wish.

So basically, this is all we need to find and install updates.

Rolling a new update

We have learned to use Esky, we have seen how to add auto update to our app. Now it’s time to build a new update. That is easy, we go back to the file we defined earlier. We had version="0.1", inside the setup() function. We need to bump it. So let’s make it 0.2 and build it. We will get a new zip file (the file contains the version if you notice carefully). Drop it on the webserver (the URL where we put our app). Run an older copy of the app (which includes the update checking codes described above). It should ask you for an update 🙂

Please note, you need to call the find_esky_update() method for the prompt to trigger. As I mentioned above, I run it in onQApplicationStarted method for PyQt. You need to find the appropriate place to call it from in your application.

Further Reading

You can find a nice tutorial with step by step instructions and code samples here:

Why I love Django & Django REST Framework

If you’re reading this article, it is very likely you are a serious Python developer or at least dabble with the language. If you happen to be a web developer, you probably have already used or at least heard about Django. Django is a pretty awesome web framework. And in this post, I am going to highlight a few key features that I absolutely love about Django. Along with Django, I will also discuss why I love DRF, the very popular REST API development framework for Django. Lately, these two have become indispensable parts of my life as a web developer and I feel it’s very important that I credit them for their awesomeness and spread the words!

Why I love Django?

  • I can quickly generate a project skeleton using the django-admin command. There is management commands to generate apps too.
  • I absolutely love Django’s project structure. It is very well organized and meaningful. A project is composed of reusable apps and the apps can self contain the codes and resources they need/use. The idea of reusable apps is very handy. There are plenty of open source apps that you can plug into your project and extend them as necessary. This allows us to reuse codes and get things done faster.
  • Django’s way of handling database access makes perfect sense to me. I define models, then automatically generate migrations from those. The migrations are written in Python, so I can adjust or tweak if necessary, programmatically. The Model definitions can be introspected by other parts of the framework to generate Admin UIs or API end points automagically – this is a huge win as we will see.
  • Django has a very powerful DB routing mechanism that allows us to use multiple databases and programmatically control the access to them. We can totally control which database is read from, which database is written to, for each app and each model.
  • The URL routing portion also makes great sense. There’s a main route configuration per project. We can add app specific routes to this configuration by include-ing them. You can add namespaces to these included ones. The namespacing gets rids of naming conflict when importing routes from 3rd party apps.
  • For simple views, the function based views are pretty efficient. But there’s also class based views for more control and dealing with complex needs.
  • The Generic Class Views are awesome! You can define ListView, CreateView, UpdateView, DetailView or a DeleteView. You pass them the Model to use and some configuration options, they will handle your CRUD operations for you while you can focus your time and concentration on preparing the templates. These have allowed me to build custom CRUD views very very fast. The generic views are very customizable, so you can alter the database queries, pagination, form fields, values passed to the template. If you’re still writing your own business logic for CRUD views, it’s time you took generic class views for a ride. What I like most is that generic views are generated dynamically. It’s not like scaffolding or code generation. In the case of scaffolding, when you need to add an extra field, you need to modify your model, controller, template, form etc. In class based generic views, you just update your model and everything is updated dynamically.
  • Talking about CRUD views, often you won’t even need to build them. The Django Admin interface takes in your models and generates a beautiful (and powerful) admin interface from them. The admin interface is very customizable. You can add custom actions, filtering, templates and whats not! Once you master Django’s admin interface, you will probably never need to build CRUD views from scratch. Many of my projects use Django Admin with high client satisfaction. And remember what we said about reusable apps and open source packages? There are several packages which provide alterations or replacement for the built in admin interface. You can plug in one of those for a different taste!
  • Management commands are easy to write and comes in very handy for simple tasks. When you install apps, they can provide useful management commands too. One of my favourite management command is “dbshell” which allows me to drop into the database’s command line without remembering the login details.
  • Django Templates are pretty awesome too. The syntax is quite similar to Jinja2 (which inspired many templating languages across different platforms). You can extend the templates by writing your own tags and filters. And there’s of course a backend for direct Jinja2 support.
  • Django is quite mature and the eco system is thriving. There’s a django package for almost everything you would need. There are many 3rd party packages for extending the different parts of Django. There are blogs, forums, cms etc apps which you can plug into your current project to add those functionality. Python also has a vast libraries for numerous purposes. It’s easy to plug them in when we need.
  • Oh, and did I forget to mention the documentation? Django documentation is superb. Very well written and covers almost everything in the framework. I would say Django Docs is one of the best out there.

The above mentioned features are just a few that I can remember right now. Django has been very pleasant to develop in. I have always felt a productivity boost as well as mental satisfaction when working with Django. Every time I faced a challenge, I have found a very simple solutions to it.

Why I love Django REST Framework?

  • Browsable APIs on the web – this is one of the best things about DRF. DRF provides very nice web browsable views for your API end points. These views allow HTML forms to submit data to the APIs. At the same time, they also show sample JSON payload for those requests. You can submit a request through the forms/json input and get back a response. So this is pretty much self documentation of the APIs. I have worked with front end and mobile developers (who consumed my APIs) and they absolutely loved the web browsable interface. In most cases, I didn’t even need to provide any API docs to them, they managed to figure things out for themselves. And if you would still want Swagger, there are 3rd party extensions for that too.
  • The ModelViewSet is like those admin views for Django. You can quickly generate CRUD based REST APIs out of your models using this class.
  • There is APIView which provides get, post, put, delete etc methods representing the HTTP verbs. You can extend APIView and provide business logic for these methods to quickly craft your APIs. There are mixins and some preset class based views to generate these methods based on querysets.
  • But don’t be satisfied just yet, ViewSets take things one step further. When you use APIView, you need to create two routes (for example one for /users and one for /users/1) but with ViewSets allow us to focus more on the business logic while forgetting about the route management. ViewSet is a class based view that has methods like list, retrieve, create, update, destroy – instead of the http verbs. The ViewSet methods allows us to cover all the REST operations in just one class. We add the view set to a router and the router generates all the necessary underlying django routes, ready to be included in a urlconf. And as mentioned earlier, the ModelViewSet is a ViewSet that generates these methods from a Model (queryset)
  • Authentication is baked in the framework. It’s very easy to setup different authentications for your APIs. The permission management is also very simple. The framework integrates nicely with some third party packages to support authentication methods which are not supported by the framework by default.
  • The framework is very customizable. We can easily extend the functionality to suit our needs. It’s all very simple, quite like the simplicity of Django!
  • The documentation is excellent. Adequate examples and very clearly described.

The only thing I am not completely happy with DRF is creating nested routes (/user/1/comment/2) is still pretty difficult/cumbersome. But the way DRF adds features quickly, I hope this will be resolved soon, in some upcoming releases!

Helping These Projects Grow

Django and DRF both are open source projects and are the results of many hours of dedication from kind hearted OSS enthusiasts. Today, we can build and maintain awesome projects because Django and DRF exists. We should all consider giving something back to these projects, in return to the benefits they have brought us!

We can help these projects by contributing codes, helping with translations, answering questions, promoting the projects or making donations.

Here you can learn how to contribute to Django —

Django Software Foundation accepts donations here —

DRF Contribution guide here —

Refund Policy for Payoneer

Payoneer has recently started asking for a Refund Policy if you want to continue receiving money from clients using their platform. Here’s the refund policy I follow:

  • The entire amount will be refunded if I can’t manage schedules after accepting a project.
  • I accept payment by milestones, there will be no refund for milestones completed (as agreed by the client).
  • If I can not finish the project within deadline, the client can ask for refund of the current milestone and they can end the contract

Django: Limiting User Access to Views

In this post, we would like to see how we can limit user accesses to our Django views.

Login Required & Permission Required Decorators

If you have worked with Django, you probably have used the login_required decorator already. Adding the decorator to a view limits access only to the logged in users. If the user is not logged in, s/he is redirected to the default login page. Or we can pass a custom login url to the decorator for that purpose.

Let’s see an example:

There’s another nice decorator – permission_required which works in a similar fashion:

Awesome but let’s learn how do they work internally.

How do they work?

We saw the magic of the login_required and permission_required decorators. But we’re the men of science and we don’t like to believe in magic. So let’s unravel the mystery of these useful decorators.

Here’s the code for the login_required decorator:

By reading the code, we can see that the login_required decorator uses another decorator – user_passes_test which takes/uses a callable to determine whether the user should have access to this view. The callable must accept an user instance and return a boolean value. user_passes_test returns a decorator which is applied to our view.

If we see the source of permission_required, we would see something quite similar. It also uses the same user_passes_test decorator.

Building Our Own Decorators

Now that we know how to limit access to a view based on whether the logged in user passes a test, it’s quite simple for us to build our own decorators for various purposes. Let’s say we want to allow access only to those users who have verified their emails.

Now we can use the decorator to a view like:

Users who have verified their email addresses will be able to access this view. And if they didn’t, they will be redirected to the login view. Using the reason query string, we can display a nice message explaining what’s happening.

Please note, we have used two decorators on the same view. We can use multiple decorators like this to make sure the user passes all the tests we require them to.

হিস্টোগ্রাম নাকি বার চার্ট?

ডাটা ভিজুয়ালাইজেশনের সময় আমরা প্রায়শই হিস্টোগ্রাম ও বার চার্ট এর ব্যবহার দেখবো । বার চার্ট এবং হিস্টোগ্রাম দেখতে অনেকটা একই রকম । দুটোতেই আমরা লম্বা লম্বা বার আকিঁ । x-axis এ ইন্ডিপেন্ডেন্ট ভ্যারিয়েবল ও y-axis এ ডিপেন্ডেন্ট ভ্যারিয়েবল এর মান বসিয়ে দিয়ে আমরা সুন্দর করে হিস্টোগ্রাম ও বার চার্ট একেঁ ফেলি ।

দেখতে একই রকম হলেও দুটোর মধ্যে রয়েছে মৌলিক পার্থক্য । হিস্টোগ্রাম টা আমরা রেন্জ বা ডিস্ট্রিবিউশানের ক্ষেত্রে ব্যবহার করি । যেমন নিচের হিস্টোগ্রামটা দেখি:

এখানে আমরা দেখছি বিভিন্ন বয়সের মানুষ, যারা ছুটি কাটাতে যায় তাদের মধ্যে কতজন হোটেলে থাকে । আমরা খেয়াল করলে দেখবো, এখানে ২০ বছরের নিচে যারা, তাদের মধ্যে ৫ জন হোটেলে থাকে, ২১-৩০ এর মধ্যে ১৫ জন, ৩১-৪০ এর মধ্যে আছে ১০ জন এবং ৪১-৫০ বয়সীদের মধ্যে ৫ জন এর কম ।

লক্ষ্য করি, এখানে আমাদের ডাটা নিউমেরিক্যাল এবং আমরা রেন্জ নিয়ে কাজ করছি – বয়সের রেন্জ । বিভিন্ন বয়সের মানুষকে নির্দিষ্ট কিছু রেন্জে আমরা ডিস্ট্রিবিউট করেছি । এখানে ডাটার এই রেন্জে কোন গ্যাপ নেই । এবং হিস্টোগ্রামের পার্টগুলোকে বা বার গুলোকে অন্য কোন অর্ডারে রি-এ্যারেন্জ করার সুযোগ নেই ।

এবার দেখি একটি বার চার্ট:

এই চার্টে আমরা দেখছি বিভিন্ন ব্র্যান্ডের গাড়ি নির্মাতাদের জানুয়ারী মাসে বিক্রিত গাড়ির সংখ্যা । এখানে প্রধান লক্ষনীয় বিষয় হলো এইবার আমাদের ডাটা কিন্তু নিউমেরিক্যাল না, বরং ক্যাটেগরিক্যাল । বিভিন্ন ব্র্যান্ডের মধ্যে তুলনা করতে আমরা এই বার চার্টটি ব্যবহার করতে পারি । বার চার্টের ক্ষেত্রে বার গুলোকে রি-এ্যারেন্জ করলেও কোন সমস্যা হয় না । এবং বার গুলো একটা আরেকটার সাথে কানেক্টেড না, মাঝ খানে তাই গ্যাপ থাকে ।

TL;DR – হিস্টোগ্রাম একটা ভ্যারিয়েবলের ডিস্ট্রিবিউশান রিপ্রেজেন্ট করার জন্য ব্যবহার করা হয়, বার চার্ট বিভিন্ন ভ্যারিয়েবলের মধ্যে তুলনা করার জন্য ব্যবহার করা হয় ।

Embedding IPython in your application

If you work with Python regularly, you probably know about IPython already. IPython has web based notebooks, QT based GUI consoles and plain old simple Terminal based REPL which is simply fantastic. But that’s not all, we can also embed IPython in our applications too. And this can lead to a number of potential use cases.

Use Cases

A common use case could be to drop into a IPython shell for quick interactive debugging. This can come very handy during prototyping.

Let’s see an example:

When we run this code, we will get a nice IPython REPL where we can try out things. In our case, we haven’t done much except defining a variable named name. We can print it out.

I use workers/queues/caches at my day to day job. So I often need to check status of the workers or get the size of a queue or even queue a few workers. I also need to check a few records on Mongodb. An interactive prompt can be really helpful for these.

Now I can just do launch_workers("send_emails", 3) to launch 3 worker instances for the “send_emails” worker. Or get the number of buyers with more than 100 purhcases with the top_buyers() function.

Customizing The Prompt

When we embed IPython, it displays it’s common banner when starting.

We can easily disable that. To do so, we need to pass empty string to the banner1 parameter to the embed method.

Or we can further customize the 2nd banner or the exit message like this: