Category: Python

Categories
Python

Python: Learning Resources and Guidelines

The Python programming language is rapidly gaining popularity in all directions. Universities are teaching programming with Python. Jobs for the language is also on the rise. It has so many use cases – web development, data science, machine learning, command line tools, security research, desktop applications and what not.

Because of it’s massive popularity, mastery of the language has become a highly valuable skill in the job market these days. Many large enterprises are starting to adopt the language. Small startups are also betting their money on it. Google, Spotify, Instagram, NASA, Twilio, Disqus, Dropbox – all the cool brands are using it in their products / services.

The triumph in the academia is another major win for the programming language. This area has mostly been dominated by languages like C/C++/Java. Beginners have had the first taste of programming in those languages. However, I personally believe, Python is a much better choice in such cases. People who are just getting familiar with programming needs an easy and fun way to learn the basic concepts. C family or Java might require  much more efforts on their side. That is why the following chart makes me happy.

top us universities teaching python
Top US Universities are teaching more Python (Source)

Why is Python so popular?

That is an excellent question. What makes Python so special? Well, to begin with Python is a nice, easy to learn general purpose language. You can use it for many different purposes, it suits many use cases. You will find necessary libraries, frameworks, toolkits for Python which will make your work easier.  Python syntax is very clear and concise. It is  very readable and easy to comprehend. When you read some source code, it all makes sense so easily. The only thing that might confuse you in the beginning is the forced use of whitespace for indentation. This might be a little difficult to grasp in the beginning but once you get it, you will start appreciating the value of properly indented code. Your code will look so much better and cleaner. Don’t you want your code to look beautiful?

5th place on the tiobe index
Python has been in top 10 on the TIOBE Index for a while

 

Apart from the code being very nice and easy to read, Python codes can be very often shorter than other languages. The language is very expressive so you can actually do more while writing less. While reading Python code, you may often feel that you’re actually reading English. The ease of learning and the powerful syntax makes Python very fun to learn and use. We all love fun, no?

Python is dynamically and strongly typed. This will make things easier to handle. Don’t worry if you do not understand the concepts of dynamic or strong typing. You will learn more about them as you start learning the language.

Friendly Community

Probably the best thing about Python  is it’s community. The community has crafted a huge amount of well maintained resources that you can learn from. From free to read online books to free video tutorials on youtube – the freely available resources have you covered. Need to up your Python game? There are excellent paid books and courses which will help you level up your programming skills. There are places like StackOverflow where fellow Pythonistas will answer any questions you have. You can also hang around IRC or Slack rooms for help. The community is very friendly and will warmly welcome you. The good people there will also guide on your journey and help you make right choices from time to time. Got a question? Just ask!

Learning Resources

These are the resources I recommend for learning the language:

  • Automate The Boring Stuff with Python: Excellent book. Really really well written. The author will guide you through a series of interesting tasks which will also teach you various concepts of the language. It will be a very exciting journey for the beginners and an effective learning experience. This is the book I currently recommend to most beginners. You can read the book free online. The author is also creating free youtube videos based on the content.
  • Python 101: Another excellent book. This one is free too. The book will cover the basics of the language. The content is very well written and beginner friendly.
  • Official Tutorial: The official Python documentation is very user friendly. It contains it’s own getting started kind of tutorial. And of course freely available.
  • Python 202: Another good book from Mike Driscoll, the author of “Python101” mentioned above. Once you have learned the basics, this book will help you dive into some more advanced concepts and language features.
  • Pro Python: This is another recommended book for levelling up your Python skills.
  • Fluent Python: I can’t recommend this book enough. When it comes to learning advanced Python, this book is a must read. It is worth every penny!

Practice Makes Us Perfect

Reading books or watching videos are not enough. We need to practise what we learn. There’s a common belief that you need to work at least 10 thousands hours before you can become expert in an area. The same idea applies here too. Read the books, watch the tutorials but at the same time, write codes by hand, every time! You will learn far more by trying those codes yourself and making mistakes. Mistakes are okay – we all make them when we begin. Learning from the mistakes is far more important.

While learning to code, these websites can be of great help. They will help you practise what you have learned so far. Some of them will also get you started with problem solving and challenge your brain. I recommend you pick a site and continue with it.

  • Hackerrank: They have all sorts of problems to challenge you. From data structures to machine learning algorithms, they have you covered. They have excellent support for Python. They even have a dedicated track for Python where you can further shun your skills.
  • Codewars: They have a very diversified set of programming problems to solve. The problems are classified into difficulty levels, so you can work your way through the various levels.
  • Codecombat: Want to learn coding while playing games? Codecombat would allow you to program your character with your language of choice. You can even fight other players.
  • Project Euler: Solve different mathematical problems on this platform.

There are many other online judges for practising algorithmic problem solving. You can try those too. But some of them might not be very Python friendly.

How to get help?

It is very likely that when you are learning Python for the first time, you will be stuck and seeking help. Please remember, Google is your best friend. Searching in Google will often find you the solutions. If it does not, feel free to join StackOverflow and ask there. You can also join this Slack Community to get help from the people there. They are very friendly.

What’s Next?

Once you have had a solid grasp of Python, you probably want to dive into a certain industry. if you want to learn web development, start learning Django or Flask. If you are into Machine Learning or Data Science, start learning the relevant tools / frameworks.

Finally, always remember – learning one language will never be enough. You have to become a Polyglot Ninja!

 

Categories
Django Python

A Brief Introduction to Django Channels

There’s a new updated version of this article here: http://masnun.rocks/2016/09/25/introduction-to-django-channels/

Django has long been an excellent web framework. It has helped many developers and numerous businesses succeed over the years. But before the introduction of Channels, Django only supported the http protocol well. With the gradual evolution of the web technologies, standing here in 2016, supporting http only is simply not enough. Today, we are using websockets for real time communications, WebRTC is getting popular for real time collaboration or video calling, HTTP/2 is also being adapted by many. In the current state of the web, any modern web framework needs to be able to support more and more protocols. This is where Django Channels come into play. Channels aim at adding new capabilities to Django, including the support for modern web technologies like websockets or http2.

How does “Channels” work?

The idea behind Channels is quite simple. To understand the concept, let’s first walk through an example scenario, let’s see how Channels would process a request.

A http/websocket request hits the reverse proxy (ie, nginx). This step is not compulsory but we’re conscious developers and always make sure our requests first go through a hardened, battle proven reverse proxy before it hits our application server

Nginx passes the request to an application server. Since we’re dealing with multiple protocols now, instead of application server, let’s call it “Interface Server”. This interface server knows how to handle requests using different protocols. The interface server accepts the request and transforms into a message. It then passes the message on to a channel.

We have to write consumers which will listen on to specific channels. When new messages arrive on those channels, the consumers would process them and if needed, send a response back to a reply/response channel. The interface server listens on to these response channels and when we write back to these channels, the interface server reads the message and transmits it to the outside world (in this case, our user). The consumers are run in background worker processes. We can spawn as many workers as we like to scale up.

So as you can see, the concept is really simple – an interface server accepts requests and queues them as messages on channels. Consumers process these queues and write back responses on response channels. The interface server sends back the responses. Plain, simple yet effective!

There are channels which are already available for us. For example – http.request channel can be listened on if we want to handle incoming http messages. Or websocket.receive can be used to process incoming websocket messages. In reality, we would probably be less interested in handling http.request ourselves and rather let Django handle it. We would be more interested in adding our custom logic for websocket connections or other protocols. Besides the channels which are already available, we can also create our own custom channels for different purposes. Since the project works by passing messages to channels and handling them with background workers, we can actually use it for managing our background tasks too. For example, instead of generating thumbnails on the fly, we can pass the image information as a message to a channel and the worker does the thumbnailing in the background. By default Channels ship with a management command – runworker which can run background workers to listen to the channels. However, till now, there is no retry mechanism if the message delivery somehow fails. In this regard, Celery can be an excellent choice for writing / running / managing the background workers which would process these channels.

Daphne is now the de-facto interface server that works well with Channels. The channels and message passing work through a “channel layer” which support multiple backends. The popular ones are – In Memory, Redis, IPC. As you can guess, these backends and the channel layer is used to abstract away the process of maintaining different channels/queues and allowing workers to listen to those. In Memory backend maintains the channels in memory and is a good fit for local development. While a Redis cluster would be more suitable in a production environment for scaling up.

Let’s Build a WebSocket Echo Server

Enough talk. Let’s build a simple echo server. But before we can do that, we first have to install the package.

That should install Django (as it’s a dependency of channels) and channels along with the necessary packages. Start a Django project with django-admin and create an app.

Now add channels to the INSTALLED_APPS list in your settings.py. For local development, we are fine with the in memory channel layer, so we need to put these lines in settings.py to define the default channel layer:

In the above code, please note the ROUTING key. As the value of this key, we have to pass the path to our channel routing. In my case, I have an app named realtime and there’s a module named routing.py which has the channel routing.

In the channel routing list, we define our routes which looks very similar to Django’s url patterns. When we receive a message through a websocket connection, the message is passed on to the websocket.receive channel. So we defined a consumer to consume messages from that channel. We also defined a path to indicate that websocket connections to /chat/ should be handled by this particular route. If we omit the path, the clients can connect to any url on the host and we can catch them all! But if we define a path, it helps us namespace things and in another cause which we will see later in this article.

And here’s the consumers.py:

The consumer is very basic. It retrieves the text we received via websocket and replies back. Note that the websocket content is available on the content attribute of the message. And the reply_channel is the response channel here (the interface server is listening on to this channel). Whatever we send to this channel is passed back to the websocket connection.

We have defined our channel layer, created our consumer and mapped a route to it. Now we just need to launch the interface server and the background workers (which run the consumers). In local environment, we can just run – python manage.py runserver as usual. Channels will make sure the interface server and the workers are running in the background. (But this should not be used in production, in production we must use Daphne separately and launch the workers individually. See here).

Once our dev server starts up, let’s open up the web app. If you haven’t added any django views, no worries, you should still see the “It Worked!” welcome page of Django and that should be fine for now. We need to test our websocket and we are smart enough to do that from the dev console. Open up your Chrome Devtools (or Firefox | Safari | any other browser’s dev tools) and navigate to the JS console. Paste the following JS code:

If everything worked, you should get an alert with the message we sent. Since we defined a path, the websocket connection works only on /chat/. Try modifying the JS code and send a message to some other url to see how they don’t work. Also remove the path from our route and see how you can catch all websocket messages from all the websocket connections regardless of which url they were connected to. Cool, no?

Our websocket example was very short and we just tried to demonstrate how things work in general. But Django Channels provide some really cool features to work with websockets. It integrates with the Django Auth system and authenticates the websocket users for you. Using the Group concept, it is very easy to create group chats or live blogs or any sort of real time communication in groups. Love Django’s generic views? We have generic consumers to help you get started fast. The channels docs is quite nice, I suggest you read through the docs and try the concepts.

Using our own channels

We can create our own channels and add consumers to them. Then we can simply add some messages to those channels by using the channel name. Like this:

WSGI or ASGI?

Since Daphne and ASGI is still new, some people still prefer to handle their http requests via WSGI. In such cases, we can configure nginx to route the requests to different servers (wsgi / asgi) based on url, domain or upgrade header. In such cases, having the real time end points under particular namespace can help us easily configure nginx to send the requests under that namespace to Daphne while sending all others to wsgi.

Categories
Python

Building a Facebook Messenger Bot with Python

Facebook now has the Messenger Platform which allows us to build bots which can accept messages from users and respond to them. In this tutorial, we shall see how we can build a bot and add it to one of our pages so that the users can interact with the bot by sending messages to the page.

To get started, we have three requirements to fulfill:

  • We need a Facebook Page
  • We need a Facebook App
  • We need a webhook / callback URL to accept incoming messages

I am assuming you already have a Facebook Page. If you don’t, go ahead and create one. It’s very simple.

Creating and Configuring The Facebook App

(1) First, we create a generic facebook app. We need to provide the name, namespace, category, contact email. Simple and straightforward. This is how it looks for me:

Create a New FB App.

(2) Now we have to browse the “Add Product” section and add “Messenger”.

Add Messenger

(3) Generate access token for a Page you manage. A popup will open asking you for permissions. Grant the permission and you will soon see the access token for that page. Please take a note of this token. We shall use it later send messages to the users on behalf of the page.

Next, click the “Webhooks” section.

(4) Before we can setup a webhook, we need to setup an URL which is publicly accessible on the internet. The URL must have SSL (that is it needs to be https). To meet this requirement and set up a local dev environment, we setup a quick flask app on our local machine.

Install Flask from PyPi using pip:

Facebook will send a GET request to the callback URL we provide. The request will contain a custom secret we can add (while setting up the webhook) and a challenge code from Facebook. They expect us to output the challenge code to verify ourselves. To do so, we write a quick GET handler using Flask.

We run the local server using python server.py. The app will launch at port 5000 by default. Next we use ngrok to expose the server to the internet. ngrok is a fantastic tool and you should seriously give it a try for running and debugging webhooks/callback urls on your local machine.

With that command, we will get an address like https://ac433506.ngrok.io. Copy that url and paste it in the Webhook setup popup. Checkmark the events we’re interested in. I check them all. Then we input a secret, which our code doesn’t care about much. So just add anything you like. The popup now looks like this:

Click “Verify and Save”. If the verification succeeds, the popup will close and you will be back to the previous screen.

Select a Page again and click “Subscribe”. Now our app should be added to the page we selected. Please note, if we haven’t generated an access token for that page in the earlier step, the subscription will fail. So make sure we have an access token generated for that page.

Handling Messages

Now every time someone sends a message to the “Masnun” page, Facebook will make a POST request to our callback url. So we need to write a POST handler for that url. We also need respond back to the user using the Graph API. For that we would need to use the awesome requests module.

Here’s the code for accepting incoming messages and sending them a reply:

The code here accepts a message, retrieves the user id and the message content. It reverses the message and sends back to the user. For this we use the ACCESS_TOKEN we generated before hand. The incoming request must be responded with a status code 200 to acknowledge the message. Otherwise Facebook will try the message a few more times and then disable the webhook. So sending a http status code 200 is important. We just output “ok” to do so.

You can now send a message to your page and see if it responds correctly. Check out Flask’s and ngrok’s logs to debug any issues you might face.

You can download the sample code from here: https://github.com/masnun/fb-bot