Categories
Python

Python: Iterators

If you have written some Python code and used the for loop, you have already used iterators behind the scene but you probably didn’t know about it. Iterators are objects that we can iterate over one by one. They are practically everywhere in a Python codebase. Understanding the concepts of iterators and how they work can help us write better, more efficient code from time to time. In this post, we will discuss iterators and other related concepts.

How does iteration work?

Before we can dive into iterators, we first need to understand how iteration works in Python. When we do the for loop, how does Python fetch one item at a time? How does this process work?

There are two functions that come into play – iter and next. The iter function gets an iterator from an object. It actually calls the __iter__ special method on the object to get the iterator. So if an object wants to allow iteration, it has to implement the __iter__ method. Once it gets the iterator object, it continues to call next on the iterator. The next function in turn calls the __next__ method on the iterator object. Let’s see a quick example:

>>> l = [1, 2, 3]

>>> i = iter(l)

>>> type(l)
<class 'list'>

>>> type(i)
<class 'list_iterator'>

>>> next(i)
1

>>> next(i)
2

>>> next(i)
3

>>> next(i)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration


>>>

Let’s see. We first create a list named l with 3 elements. We then call iter() on it. The type of l is list but look at the type of i – it’s list_iterator – interesting! Now we keep calling next on i and it keeps giving us the values we saw in the list, one by one, until there’s a StopIteration exception.

Here the list is an iterable because we can get an iterator from it to iterate over the list. The list_iterator object we got is an iterator, it’s an object that we can actually iterate over. When we loop over a list, this is what happens:

l = [1, 2, 3]

iterator = iter(l)

while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        break

Makes sens? The for loop actually gets the iterator and keeps looping over until a StopIteration exception is encountered.

Iterator

The iterator is an object which implements __next__ method so we can call next on it repeatedly to get the items. Let’s write an iterator that keeps us giving us the next integer, without ever stopping. Let’s name it InfiniteIterator.

class InfiniteIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        self.__int += 1
        return self.__int

If we keep calling next on it, we will keep getting the integers, starting from one.

>>> inf_iter = InfiniteIterator()
>>> next(inf_iter)
1
>>> next(inf_iter)
2
>>> next(inf_iter)
3
>>> next(inf_iter)
4
>>>

Iterable

What if we wanted to create an InfiniteNumbers iterable? It would be such that when we use the for loop on it, it never stops. It keeps producing the next integer in each loop. What would we do? Well, we have an InfiniteIterator. All we need is to define an __iter__ method that returns a new instance of InfiniteIterator.

class InfiniteNumbers:
    def __iter__(self):
        return InfiniteIterator()


infinite_numbers = InfiniteNumbers()

for x in infinite_numbers:
    print(x)

    if x > 99:
        break

If you remove the break statement and the if block, you will notice, it keeps running – like forever.

Using StopIteration

Instead of breaking out from our code ourselves, we could use the StopIteration exception in our iterator so it stops after giving us the 100 numbers.

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


class InfiniteNumbers:
    def __iter__(self):
        return HundredIterator()


one_hundred = InfiniteNumbers()

for x in one_hundred:
    print(x)

Iterators must also implement __iter__

We saw that the __next__ method does it’s work just fine. But we also need to implement the __iter__ method on an iterator (just like we did in iterable). Why is this required? Let me quote from the official docs:

Iterators are required to have an__iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted.

If we tried to use the for loop over our iterator, it would fail:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

We will get the following exception:

Traceback (most recent call last):
  File "iter.py", line 15, in <module>
    for x in one_hundred:
TypeError: 'HundredIterator' object is not iterable

That kind of makes sense because we saw that the for loop runs the iter function on an object to get an iterator from it. Then calls next on the iterator. That’s the problem, we don’t have an __iter__ method. The official documentation suggests that every iterator should be a proper iterable too. That is, it should implement the __iter__ method and just return an instance of itself. Let’s do that:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.__int > 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

Now the code works fine 🙂

The Iterator Protocol

The iterator protocol defines the special methods that an object must implement to allow iteration. We can summarize the protocol in this way:

  • Any object that can be iterated over needs to implement the __iter__ method which should return an iterator object. Any object that returns an iterator is an iterable.
  • An iterator must implement the __next__ method which returns the next item when called. When all items are exhausted (read retrieved), it must raise the StopIteration exception.
  • An iterator must also implement the __iter__ method to behave like an iterable.

Why do we need Iterables?

In our last example, we saw that it’s possible for an object to implement a __next__ method and an __iter__ method that returns self. In this way, an iterator behaves just like an iterable alright. Then why do we need Iterables? Why can’t we just keep using Iterators which refer to itself?

Let’s get back to our HundredIterator example. Once you have iterated over the items once, try to iterate again. What happens? No numbers are output on the screen. Why? Well, because the iterator objects store “state”. Once it has reached StopIteration, it has reached the end line. It’s now exhausted. Every time you call iter on it, it returns the same instace (self) which has nothing more to output.

This is why Iterables are useful. You can just return a fresh instance of an iterator every time the iterable is looped over. This is actually what many built in types like list does.

Why is Iterators so important?

Iterators allow us to consume data each item at a time. Just imagine, if there’s a one GB file and we tried to load it all in memory, it would require huge memory. But what if we implemented an iterator that reads the file one line at a time? We could then just store that one line in memory and do necessary processing before moving on to the next item. This allow us to write really efficient programs 🙂

This all seems very confusing

If you find the concepts very confusing and hard to grasp, don’t worry. Give it a few tries, write the codes by hand and see the output. Tinker with the examples. Inspect the code, try to see what happens when you modify part of it. All things become easier when you practise more and more. Try writing your own iterables and iterators – perhaps try to clone the built in containers’ functionalities? May be write your own list implementation? Don’t worry, it will come to you in time.

 

Categories
Python

Django REST Framework: JSON Web Tokens (JWT)

(This post is a part of a tutorial series on Building REST APIs in Django)

Our last post was about Authentication and Permissions and we covered the available methods of authentication in Django REST Framework. In that post, we learned how to use the built in Token based authentication in DRF. In this post, we will learn more about JSON Web Tokens aka JWT and we will see if JWT can be a better authentication mechanism for securing our REST APIs.

Understanding JSON Web Tokens (JWTs)

We have actually written a detailed blog post about JSON Web Tokens earlier. In case you have missed it, you probably should read it first. We have also described how to use JWT with Flask – reading that one might also help better understand how things work. And of course, we will briefly cover the idea of JWT in this post as well.

If we want to put it simply – you take some data in JSON format, you hash it with a secret and you get a string that you use as a token. You (your web app actually) pass this token to the user when s/he logs in. The user takes the token and on subsequent requests, passes it back in the “Authorization” header. The web app now takes this token back, “decodes” it back to the original JSON payload. It can now read the stored data (identity of the user, token expiry and other data which was embedded in the JSON). While decoding, the same secret is used, so third party attackers can’t just forge a JWT. We would want our token to be small in size, so the JSON payload is usually intentionally kept small. And of course, it should not contain any sensitive information like user password.

JWT vs DRF’s Token Based Authentication

So in our last blog post, we saw Django REST Framework includes a token based authentication system which can generate a token for the user. That works fine, right? Why would we want to switch to JSON Web Tokens instead of that?

Let’s first see how DRF generates the tokens:

    def generate_key(self):
        return binascii.hexlify(os.urandom(20)).decode()

It’s just random. The token generated can not be anyway related to the user that it belongs to. So how does it associate a token with an user? It stores the token and a reference to the user in a table in database. Here comes the first point – while using DRF’s token based auth, we need to query database on every request (unless of course we have cached that token which). But what if we have multiple application servers? Now we need all our application servers to connect to the same database or same cache server. How will that scale when the project gets really really big? What if we want to provide single sign on across multiple services? We will need to maintain a central auth service where other services request to verify a token. Can JWT simplify these for us?

JWT is just an encoded (read – hashed / signed) JSON data. As long as any webservice has access to the secret used in signing the data, it can also decode and read the embedded data. It doesn’t need any database calls. You can generate the token from one service and other services can read and verify it just fine. It’s more efficient and simply scales better.

JWT in Django REST Framework

DRF does not directly support JWTs out of the box. But there’s an excellent package that adds support for it. Let’s see how easily we can integrate JWT in our REST APIs.

Install and Configure

Let’s first install the package using pip –

pip install djangorestframework-jwt

That should install the package. Now we need to add rest_framework_jwt.authentication.JSONWebTokenAuthentication to the default authentication classes in REST Framework settings.

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    )
}

We added it to the top of the list. Next, we just have to add it’s built in view to our urlpatterns.

from rest_framework_jwt.views import obtain_jwt_token

urlpatterns = router.urls + [
    url(r'^jwt-auth/', obtain_jwt_token),
]
Obtain a Token

The obtain_jwt_token view will check the user credentials and provide a JWT if everything goes alright. Let’s try it.

$ curl --request POST \
  --url http://localhost:8000/api/jwt-auth/ \
  --header 'content-type: application/json' \
  --data '{"username": "test_user", "password": "awesomepwd"}'

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0"}

Awesome, everything worked just fine. We have got our token too. What do we do next? We use this token to access a secured resource.

Using the obtained JWT

We need to pass the token in the form of JWT <token> as the value of the Authorization header. Here’s a sample curl request:

$ curl -H "Content-Type: application/json" -H "Authorization: JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0" -X GET  http://localhost:8000/api/subscribers/

[{"id":1,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":2,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":3,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":4,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"}]

So our token worked fine! Cool!

Where to go next?

Now that you have seen how simple and easy it is to add JSON Web Token based authentication to Django REST Framework, you probably should dive deeper into the package documentation. Specially these topics might be interesting –

  • Refresh Tokens: If you enable JWT token refreshing, you can exchange your current token with a new, fresh one before the existing one expires. The new token will of course have a renewed expiry time set.
  • Verify Token: If you just share the secret, all services can verify the user on their own. However, in modern micro service based architecture, you may want to provide an API end point that other services can use to verify a JWT they received from the user. This can be useful for those scenarios.
  • And of course look at the settings options available and see how you can customize the token generation process.

In the future, we shall try to cover more about Django, Django REST Framework and Python in general. If you liked the content, please subscribe to the mailing list so we can notify you when we post new contents.

Categories
Kotlin

Kotlin: An Introduction

In Google I/O 2017, Kotlin was announced as an officially supported language for android development. The news created a massive buzz among the developers community. Google is a big name and so is Android. So that level of attention was only natural. But many of those who have followed the language since it’s inception, didn’t need to wait this long to fall in love with the language. Kotlin is a brilliant language. It’s statically typed yet quite productive. The syntax does not feel rigid or restrictive, rather it feels rather expressive and enjoyable. Writing code in Kotlin is really fun. You don’t have to trust my word for that, just give it a try.

History of Kotlin

In case you didn’t know, Kotlin is named after the Kotlin Island near Saint Petersburg, Russia. You can read more about the Island in Wikipedia. So another programming language named after an Island, but why? Well, Kotlin is developed by those awesome guys at Jetbrains  who build our favorite IDEs. Apparently Kotlin was developed in their Saint Petersburg office and the developers named it after the Island.

Jetbrains was using Java to build their IDEs but why did they decide to create a language of their own? The answer is probably obvious to you by now. Every new language is created out of frustration from the existing ones. We already saw a similar story in our post about Golang. The same thing happened here too. Jetbrains lead Dmitry Jemerov mentioned they were looking for features not offered by Java or most other languages. They liked Scala but the compile time was a bugger. So they came up with Kotlin and thanks to them that they did!

Work on Kotlin started in 2010 and Jetbrains announced the language publicly in July 2011. They made the project open source under the Apache 2 License in 2012. They continued developing the language by accepting and reviewing user’s feedback. After a lot of improvements and iteration, the language reached version 1.0 in February 2016, the first officially stable release with commitment towards backward compatibility. That was just a year go. Kotlin became quickly popular and developers seemed to love it. Many large companies started adopting it even before it became officially supported on Android. But the official support didn’t take long either, in Google I/O 2017, first class support came for Kotlin on Android.

Write once, Run Everywhere

Do you recognize the words – “Write once, run anywhere / everywhere”? If you’re a Java developer, you might recognize them. Kotlin takes things a little bit further than that. Kotlin started off as a language based on the JVM. You could run Kotlin anywhere you could run Java. But that’s not all, Kotlin code now can be transpiled into Javascript too! You already know it works on Android. The guys are also working on a native version based on LLVM. Once that happens, you can truly write code once and run them anywhere and everywhere. Of course you can’t fully port all codes from one platform to another, for example if you’re using JavaFX on the JVM, those will not work on the browser. But the common business logic can be ported without much efforts in most cases.

Kotlin is fully interoperable with Java and can take advantage of the JVM. So the plethora of existing Java libraries and packages are at your disposal. This opens up enormous possibilities and a large eco system to the enterprising Kotlin developer.

Most Loved Kotlin Features

So what makes Kotlin great? Why do people love it? Here are some of the features people love in the language:

  • Full interoperability with Java (and the JVM)
  • Multi platform – JVM, Android, Browser and Native (coming soon)
  • The syntax is quite similar to other popular programming languages. Expressive and productive.
  • Type inference is another cool feature. You can just write val number = 23 and it an identify that it’s an Int. You can of course declare the type if you wish 🙂
  • The when block can cut the need of complex if/else block or switch statements and makes things simpler.
  • Libraries like Anko not only makes android development faster, it also makes if more enjoyable. Also the android extensions for Kotlin makes many pain points easier to handle.
  • Functions are fun, literally 😉 fun meaningOfLife() = 42 – see? fun – it is! 😀
  • Simple one line functions paired with string interpolation is pretty useful – fun fullName() = "${first} ${last}"
  • The idea of data class is to provide easy POJOs in a very short, concise manner.
  • Operator overloading is quite simple.
  • You can add extension functions to classes, even built in ones, easily extending their functionality.
  • Kotlin provides null safety by forcing you to check nullable types for null first. No more NullPointerException for you! 😉
  • Lambdas are super fun. Expressive, powerful and of course productive.
  • Easy to use and useful ranges.
  • Easy to add custom get and set methods to public fields. Pair that with simple functions. You have got properties without much boilerplate code.
  • The default arguments and named arguments are pretty cool
  • You can use == for checking equality, no need for equals calls.
  • The is operator provides automatic smart casts. When you write if (param is String) { // code }, the param is cast into a string for you in the block. So no more instance checks.
  • Super expressive maps, filters with lambdas
  • Ability to create custom DSLs
  • Coroutines
  • Awesome tooling support. IntelliJ Idea lets you convert Java code to Kotlin in just a click. You know their IDE is the best, right? But there’s also support for other popular IDEs like Eclipse.

Learning Kotlin

You can also visit us on the Facebook, we have a group named – Kotlin Ninja where some Kotlin enthusiasts learn and share together 🙂