Category: Python

Python

Deploying A Flask based REST API to AWS Lambda (Serverless) using Zappa

Post author By masnun
Post date November 24, 2017

I have heard about AWS Lambda and all the cool things happening in the serverless world. I have also deployed Go functions using the Apex framework for serverless deployment. But recently I have started working on some Python projects again and decided to see how well the Python community is adapting to the serverless era. Not to my surprise, the Python community is doing great as usual. I quickly found an awesome framework named Zappa which makes deploying Python code to AWS Lambda very easy. Python is already natively supported on the AWS Lambda platform. But with the native support you need to configure the API Gateway, S3 bucket and other stuff on your own. Thanks to Zappa, these things are now automated to our convenience. We can easily deploy WSGI apps as well. That means, we can now take our Flask / Django / API Star apps and deploy them to AWS Lambda – with ease and simplicity. In this blog post, I will quickly walk through how to deploy a flask based rest api to the serverless cloud. If you want to learn more about API integration you can look for Jitterbit.

Setup Flask App

Before we can get started, we need to create a simple flask app. Here’s a quick rest api (that doesn’t do much):

from flask import Flask, jsonify

app = Flask(__name__)


@app.route('/')
def hello_world():
    return jsonify({"message": "Hello World!"})

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/')

def hello_world():

return jsonify({"message": "Hello World!"})

Let’s save the above code in app.py. We can now install Flask using pip:

pip install flask

1	pip install flask

And then run the code:

FLASK_APP=app.py flask run

1	FLASK_APP=app.py flask run

This should run our app and we should be able to visit http://127.0.0.1:5000/ to see the output.

Setting Up AWS

Please make sure you have an account for AWS where you have added your credit card and completed the sign up process. AWS Lambda has 1 million free requests per month which is promised to be always free (not for the first 12 months or anything). When you add your card, you shall also be eligible for a free tier of S3 for 12 months. So you won’t be charged for trying out a sample app deployment. So don’t worry about adding a card. In fact, adding a card is a requirement for getting the free tier.

Once you have your AWS account setup, click on your name (top right) and click “My Security Credentials”. From there, choose the “Access Keys” section and generate a new pair of key and secret. Store them in your ~/.aws/credentials file. AWS Cli (and Zappa) will use these to connect to AWS services and perform required actions. The file should look like this:

[default]
aws_access_key_id=[...]
aws_secret_access_key=[...]

[masnun]
aws_access_key_id=[...]
aws_secret_access_key=[...]

[default]

aws_access_key_id=[...]

aws_secret_access_key=[...]

[masnun]

aws_access_key_id=[...]

aws_secret_access_key=[...]

I have created two profiles here. Named profiles are useful if you have more than one accounts/projects/environments to work with. After adding the credentials, add the region information in ~/.aws/config:

[default]
region=us-west-2
output=json

[profile masnun]
region=us-east-2
output=text

[default]

region=us-west-2

output=json

[profile masnun]

region=us-east-2

output=text

This will mostly help with choosing the default region for your app. Once you have these AWS settings configured, you can get started with Zappa.

Install and Configure Zappa

First install Zappa:

pip install zappa

1	pip install zappa

Now cd into the project directory (where our flask app is). Then run:

zappa init

1	zappa init

Zappa should guide you through the settings it needs. It should also detect the Flask app and auto complete the app (app.app ) for you. Once the wizard finishes, you’re ready to deploy your API.

zappa deploy dev

1	zappa deploy dev

This should now deploy the app to the dev stage. You can configure different stages for your app in Zappa settings. Once you make some code changes, you can update the app:

zappa update

1	zappa update

Both of these commands should print out the url for the app. In my case, the url is: https://1gc1f80kb5.execute-api.us-east-2.amazonaws.com/dev 🙂

What’s Next?

Congratulations, you just deployed a Flask rest api to AWS Lambda using Zappa. You can make the url shorter by pointing a domain to it from your AWS console. To know more about Zappa and all the great things it can do, please check out Zappa on Github.

Tags AWS, Cloud, Flask, Lambda, REST API, Serverless

Python

Python: Iterators

Post author By masnun
Post date May 31, 2017

If you have written some Python code and used the for loop, you have already used iterators behind the scene but you probably didn’t know about it. Iterators are objects that we can iterate over one by one. They are practically everywhere in a Python codebase. Understanding the concepts of iterators and how they work can help us write better, more efficient code from time to time. In this post, we will discuss iterators and other related concepts.

How does iteration work?

Before we can dive into iterators, we first need to understand how iteration works in Python. When we do the for loop, how does Python fetch one item at a time? How does this process work?

There are two functions that come into play – iter and next. The iter function gets an iterator from an object. It actually calls the __iter__ special method on the object to get the iterator. So if an object wants to allow iteration, it has to implement the __iter__ method. Once it gets the iterator object, it continues to call next on the iterator. The next function in turn calls the __next__ method on the iterator object. Let’s see a quick example:

&gt;&gt;&gt; l = [1, 2, 3]

&gt;&gt;&gt; i = iter(l)

&gt;&gt;&gt; type(l)
&lt;class 'list'&gt;

&gt;&gt;&gt; type(i)
&lt;class 'list_iterator'&gt;

&gt;&gt;&gt; next(i)
1

&gt;&gt;&gt; next(i)
2

&gt;&gt;&gt; next(i)
3

&gt;&gt;&gt; next(i)
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
StopIteration


&gt;&gt;&gt;

>>> l = [1, 2, 3]

>>> i = iter(l)

>>> type(l)

>>> type(i)

>>> next(i)

Traceback (most recent call last):

File "<stdin>", line 1, in <module>

StopIteration

>>>

Let’s see. We first create a list named l with 3 elements. We then call iter() on it. The type of l is list but look at the type of i – it’s list_iterator – interesting! Now we keep calling next on i and it keeps giving us the values we saw in the list, one by one, until there’s a StopIteration exception.

Here the list is an iterable because we can get an iterator from it to iterate over the list. The list_iterator object we got is an iterator, it’s an object that we can actually iterate over. When we loop over a list, this is what happens:

l = [1, 2, 3]

iterator = iter(l)

while True:
    try:
        item = next(iterator)
        print(item)
    except StopIteration:
        break

l = [1, 2, 3]

iterator = iter(l)

while True:

try:

item = next(iterator)

print(item)

except StopIteration:

break

Makes sens? The for loop actually gets the iterator and keeps looping over until a StopIteration exception is encountered.

Iterator

The iterator is an object which implements __next__ method so we can call next on it repeatedly to get the items. Let’s write an iterator that keeps us giving us the next integer, without ever stopping. Let’s name it InfiniteIterator.

class InfiniteIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        self.__int += 1
        return self.__int

class InfiniteIterator:

def __init__(self):

self.__int = 0

def __next__(self):

self.__int += 1

return self.__int

If we keep calling next on it, we will keep getting the integers, starting from one.

&gt;&gt;&gt; inf_iter = InfiniteIterator()
&gt;&gt;&gt; next(inf_iter)
1
&gt;&gt;&gt; next(inf_iter)
2
&gt;&gt;&gt; next(inf_iter)
3
&gt;&gt;&gt; next(inf_iter)
4
&gt;&gt;&gt;

>>> inf_iter = InfiniteIterator()

>>> next(inf_iter)

>>>

Iterable

What if we wanted to create an InfiniteNumbers iterable? It would be such that when we use the for loop on it, it never stops. It keeps producing the next integer in each loop. What would we do? Well, we have an InfiniteIterator. All we need is to define an __iter__ method that returns a new instance of InfiniteIterator.

class InfiniteNumbers:
    def __iter__(self):
        return InfiniteIterator()


infinite_numbers = InfiniteNumbers()

for x in infinite_numbers:
    print(x)

    if x &gt; 99:
        break

class InfiniteNumbers:

def __iter__(self):

return InfiniteIterator()

infinite_numbers = InfiniteNumbers()

for x in infinite_numbers:

print(x)

if x > 99:

break

If you remove the break statement and the if block, you will notice, it keeps running – like forever.

Using StopIteration

Instead of breaking out from our code ourselves, we could use the StopIteration exception in our iterator so it stops after giving us the 100 numbers.

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int &gt; 99:
            raise StopIteration

        self.__int += 1
        return self.__int


class InfiniteNumbers:
    def __iter__(self):
        return HundredIterator()


one_hundred = InfiniteNumbers()

for x in one_hundred:
    print(x)

class HundredIterator:

def __init__(self):

self.__int = 0

def __next__(self):

if self.__int > 99:

raise StopIteration

self.__int += 1

return self.__int

class InfiniteNumbers:

def __iter__(self):

return HundredIterator()

one_hundred = InfiniteNumbers()

for x in one_hundred:

print(x)

Iterators must also implement iter

We saw that the __next__ method does it’s work just fine. But we also need to implement the __iter__ method on an iterator (just like we did in iterable). Why is this required? Let me quote from the official docs:

Iterators are required to have an__iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted.

If we tried to use the for loop over our iterator, it would fail:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __next__(self):
        if self.__int &gt; 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

class HundredIterator:

def __init__(self):

self.__int = 0

def __next__(self):

if self.__int > 99:

raise StopIteration

self.__int += 1

return self.__int

one_hundred = HundredIterator()

for x in one_hundred:

print(x)

We will get the following exception:

Traceback (most recent call last):
  File "iter.py", line 15, in &lt;module&gt;
    for x in one_hundred:
TypeError: 'HundredIterator' object is not iterable

Traceback (most recent call last):

File "iter.py", line 15, in <module>

for x in one_hundred:

TypeError: 'HundredIterator' object is not iterable

That kind of makes sense because we saw that the for loop runs the iter function on an object to get an iterator from it. Then calls next on the iterator. That’s the problem, we don’t have an __iter__ method. The official documentation suggests that every iterator should be a proper iterable too. That is, it should implement the __iter__ method and just return an instance of itself. Let’s do that:

class HundredIterator:
    def __init__(self):
        self.__int = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.__int &gt; 99:
            raise StopIteration

        self.__int += 1
        return self.__int


one_hundred = HundredIterator()

for x in one_hundred:
    print(x)

class HundredIterator:

def __init__(self):

self.__int = 0

def __iter__(self):

return self

def __next__(self):

if self.__int > 99:

raise StopIteration

self.__int += 1

return self.__int

one_hundred = HundredIterator()

for x in one_hundred:

print(x)

Now the code works fine 🙂

The Iterator Protocol

The iterator protocol defines the special methods that an object must implement to allow iteration. We can summarize the protocol in this way:

Any object that can be iterated over needs to implement the __iter__ method which should return an iterator object. Any object that returns an iterator is an iterable.
An iterator must implement the __next__ method which returns the next item when called. When all items are exhausted (read retrieved), it must raise the StopIteration exception.
An iterator must also implement the __iter__ method to behave like an iterable.

Why do we need Iterables?

In our last example, we saw that it’s possible for an object to implement a __next__ method and an __iter__ method that returns self. In this way, an iterator behaves just like an iterable alright. Then why do we need Iterables? Why can’t we just keep using Iterators which refer to itself?

Let’s get back to our HundredIterator example. Once you have iterated over the items once, try to iterate again. What happens? No numbers are output on the screen. Why? Well, because the iterator objects store “state”. Once it has reached StopIteration, it has reached the end line. It’s now exhausted. Every time you call iter on it, it returns the same instace (self) which has nothing more to output.

This is why Iterables are useful. You can just return a fresh instance of an iterator every time the iterable is looped over. This is actually what many built in types like list does.

Why is Iterators so important?

Iterators allow us to consume data each item at a time. Just imagine, if there’s a one GB file and we tried to load it all in memory, it would require huge memory. But what if we implemented an iterator that reads the file one line at a time? We could then just store that one line in memory and do necessary processing before moving on to the next item. This allow us to write really efficient programs 🙂

This all seems very confusing

If you find the concepts very confusing and hard to grasp, don’t worry. Give it a few tries, write the codes by hand and see the output. Tinker with the examples. Inspect the code, try to see what happens when you modify part of it. All things become easier when you practise more and more. Try writing your own iterables and iterators – perhaps try to clone the built in containers’ functionalities? May be write your own list implementation? Don’t worry, it will come to you in time.

Python

Django REST Framework: JSON Web Tokens (JWT)

Post author By masnun
Post date May 28, 2017

(This post is a part of a tutorial series on Building REST APIs in Django)

Our last post was about Authentication and Permissions and we covered the available methods of authentication in Django REST Framework. In that post, we learned how to use the built in Token based authentication in DRF. In this post, we will learn more about JSON Web Tokens aka JWT and we will see if JWT can be a better authentication mechanism for securing our REST APIs.

Understanding JSON Web Tokens (JWTs)

We have actually written a detailed blog post about JSON Web Tokens earlier. In case you have missed it, you probably should read it first. We have also described how to use JWT with Flask – reading that one might also help better understand how things work. And of course, we will briefly cover the idea of JWT in this post as well.

If we want to put it simply – you take some data in JSON format, you hash it with a secret and you get a string that you use as a token. You (your web app actually) pass this token to the user when s/he logs in. The user takes the token and on subsequent requests, passes it back in the “Authorization” header. The web app now takes this token back, “decodes” it back to the original JSON payload. It can now read the stored data (identity of the user, token expiry and other data which was embedded in the JSON). While decoding, the same secret is used, so third party attackers can’t just forge a JWT. We would want our token to be small in size, so the JSON payload is usually intentionally kept small. And of course, it should not contain any sensitive information like user password.

JWT vs DRF’s Token Based Authentication

So in our last blog post, we saw Django REST Framework includes a token based authentication system which can generate a token for the user. That works fine, right? Why would we want to switch to JSON Web Tokens instead of that?

Let’s first see how DRF generates the tokens:

    def generate_key(self):
        return binascii.hexlify(os.urandom(20)).decode()

1 2	def generate_key(self): return binascii.hexlify(os.urandom(20)).decode()

It’s just random. The token generated can not be anyway related to the user that it belongs to. So how does it associate a token with an user? It stores the token and a reference to the user in a table in database. Here comes the first point – while using DRF’s token based auth, we need to query database on every request (unless of course we have cached that token which). But what if we have multiple application servers? Now we need all our application servers to connect to the same database or same cache server. How will that scale when the project gets really really big? What if we want to provide single sign on across multiple services? We will need to maintain a central auth service where other services request to verify a token. Can JWT simplify these for us?

JWT is just an encoded (read – hashed / signed) JSON data. As long as any webservice has access to the secret used in signing the data, it can also decode and read the embedded data. It doesn’t need any database calls. You can generate the token from one service and other services can read and verify it just fine. It’s more efficient and simply scales better.

JWT in Django REST Framework

DRF does not directly support JWTs out of the box. But there’s an excellent package that adds support for it. Let’s see how easily we can integrate JWT in our REST APIs.

Install and Configure

Let’s first install the package using pip –

pip install djangorestframework-jwt

1	pip install djangorestframework-jwt

That should install the package. Now we need to add rest_framework_jwt.authentication.JSONWebTokenAuthentication to the default authentication classes in REST Framework settings.

REST_FRAMEWORK = {
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework_jwt.authentication.JSONWebTokenAuthentication',
        'rest_framework.authentication.BasicAuthentication',
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication',
    )
}

REST_FRAMEWORK = {

'DEFAULT_AUTHENTICATION_CLASSES': (

'rest_framework_jwt.authentication.JSONWebTokenAuthentication',

'rest_framework.authentication.BasicAuthentication',

'rest_framework.authentication.SessionAuthentication',

'rest_framework.authentication.TokenAuthentication',

)

}

We added it to the top of the list. Next, we just have to add it’s built in view to our urlpatterns.

from rest_framework_jwt.views import obtain_jwt_token

urlpatterns = router.urls + [
    url(r'^jwt-auth/', obtain_jwt_token),
]

from rest_framework_jwt.views import obtain_jwt_token

urlpatterns = router.urls + [

url(r'^jwt-auth/', obtain_jwt_token),

]

Obtain a Token

The obtain_jwt_token view will check the user credentials and provide a JWT if everything goes alright. Let’s try it.

$ curl --request POST \
  --url http://localhost:8000/api/jwt-auth/ \
  --header 'content-type: application/json' \
  --data '{"username": "test_user", "password": "awesomepwd"}'

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0"}

$ curl --request POST \

--url http://localhost:8000/api/jwt-auth/ \

--header 'content-type: application/json' \

--data '{"username": "test_user", "password": "awesomepwd"}'

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0"}

Awesome, everything worked just fine. We have got our token too. What do we do next? We use this token to access a secured resource.

Using the obtained JWT

We need to pass the token in the form of JWT <token> as the value of the Authorization header. Here’s a sample curl request:

$ curl -H "Content-Type: application/json" -H "Authorization: JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0" -X GET  http://localhost:8000/api/subscribers/

[{"id":1,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":2,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":3,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":4,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"}]

$ curl -H "Content-Type: application/json" -H "Authorization: JWT eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ1c2VyX2lkIjoyLCJlbWFpbCI6IiIsInVzZXJuYW1lIjoidGVzdF91c2VyIiwiZXhwIjoxNDk1OTkyOTg2fQ.sWSzdiBNNcXDqhcdcjWKjwpPsVV7tCIie-uit_Yz7W0" -X GET http://localhost:8000/api/subscribers/

[{"id":1,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":2,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":3,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"},{"id":4,"name":"Abu Ashraf Masnun","age":29,"email":"masnun@polyglot.ninja"}]

So our token worked fine! Cool!

Where to go next?

Now that you have seen how simple and easy it is to add JSON Web Token based authentication to Django REST Framework, you probably should dive deeper into the package documentation. Specially these topics might be interesting –

Refresh Tokens: If you enable JWT token refreshing, you can exchange your current token with a new, fresh one before the existing one expires. The new token will of course have a renewed expiry time set.
Verify Token: If you just share the secret, all services can verify the user on their own. However, in modern micro service based architecture, you may want to provide an API end point that other services can use to verify a JWT they received from the user. This can be useful for those scenarios.
And of course look at the settings options available and see how you can customize the token generation process.

In the future, we shall try to cover more about Django, Django REST Framework and Python in general. If you liked the content, please subscribe to the mailing list so we can notify you when we post new contents.

Tags Django, Django REST Framework, JWT, REST API

Recent Posts

Recent Comments

Archives

Categories

Setup Flask App

Setting Up AWS

Install and Configure Zappa

What’s Next?

How does iteration work?

Iterator

Iterable

Using StopIteration

Iterators must also implement __iter__

The Iterator Protocol

Why do we need Iterables?

Why is Iterators so important?

This all seems very confusing

(This post is a part of a tutorial series on Building REST APIs in Django)

Understanding JSON Web Tokens (JWTs)

JWT vs DRF’s Token Based Authentication

JWT in Django REST Framework

Install and Configure

Obtain a Token

Using the obtained JWT

Where to go next?

Iterators must also implement iter