Categories
Golang

Golang: Working with JSON

We previously wrote an introductory blog post on Golang describing why and how the language is gaining traction quickly. Go aka Golang is being used by many large scale systems for various purposes. One thing common in most complex systems is that we have to communicate with other systems / services. JSON is a very popular data interchange format for this kind of scenarios. In fact, it is so popular that you may even go ahead and call it the de-facto data interchange format on the internet.  If you have followed any of our REST API tutorials (Django / Flask), you might have also noticed they all output JSON. In this blog post, we will see the different ways we can work with JSON in Golang.

Creating JSON

We can use the encoding/json package to easily create JSON from our Go data structures. There’s a few things to consider while creating JSON in Go. One of them is the choice of data structure.

From Structs

Structs are very convenient to use and often come as cheaper compared to map or other complex structures. We can pass any instance of a struct to the json.Marshal function and get back the JSON as a slice of bytes ([]byte).

Output should be:

Things to note here:

  • The field name has to start with a capital letter – this is important and will be discussed soon
  • You can nest other structures, in this example our Emails field contain a list of strings
  • The json.Marshal function returns the JSON and error, don’t forget to handle the error. You may not often get errors from the Marshal function but in some cases where some types can not be converted into JSON, you will get errors. So look out for it.
  • The returned JSON is in the form of bytes list. So if you want to use it as string, you will need to convert it.

Choosing Field Names

If you have seen the output, the keys/fields in the created JSON structure all start with a capital letter (our struct fields were similar, so this is no surprise). If you have a curious and adventurous mind, you might have tried to go ahead and convert the struct fields into lower caps. But that doesn’t work, does it? Check it out – https://play.golang.org/p/93eDoFSjnW – because if a field / member name does not start with a capital letter, it is not exported. External code can not access it. This is why our name and age fields are not part of the generated JSON structure.

But don’t worry, there’s a simple way of “tagging” our struct fields so we can describe how to marshal our structs. Let’s modify our codes to look like this:

Next to each field, we have provided tags to describe how this field should be marshalled or unmarshalled. Now we can see you expected results here: https://play.golang.org/p/xlcjU1_VSE 🙂

If you don’t understand the tags, don’t worry, try reading this answer on StackOverflow – https://stackoverflow.com/questions/10858787/what-are-the-uses-for-tags-in-go. We will understand the concepts more when we use structs for various purposes (for example mapping to database tables, but for now, let’s not worry).

Omitting Empty Fields

What if some of the fields are empty? Let’s try that.

The output would be:

If a field is empty, we might want to omit that from our JSON. We can do so by using the omitempty flag in our json field tags.

Now if we check the output again:

Nice, no?

Skipping Fields

Let’s say in our struct, we need to keep the Age field. But we don’t want it to be a part of the produced JSON. We can use the - flag in the json tag for that particular field.

The output would be:

Even though the struct had the Age field set, it wasn’t included in the output. This comes very handy in some cases, for example when a User struct has a Password field that we don’t want to serialize into JSON.

Using Maps and Slices

So far we have used structs. We can also use maps and slices instead. Here’s a quick code example:

And using slices:

They both work as expected. But in most codebases I have come across, structs are more widely used.

 

Parsing JSON

We have so far seen how we can generate JSON from our go data. Now we will see the opposite. We will be parsing JSON into Go data structures.

Into Structs

We will first see how we can parse JSON data into structs. It’s quite similar to what we did earlier. We will be using the Unmarshal function which takes bytes and pointer to any interface{} type. It reads through the JSON and stores the data in the struct we pass as the second parameter. Let’s see an example:

Here json_bytes hold the JSON we want to process. We already have a Person type with tagged fields. We just need to pass this json_bytes and a pointer to an instance of Person to the Unmarshal function. Please note the pointer is important. We have to pass a pointer otherwise the parser would not be able to write to the struct.

If the struct doesn’t have some fields which are present in the JSON, those will be silently ignored. In the same way, if the struct has fields which are not available in the JSON, they will be ignored too.

In the above example, the struct has a field named Address which the JSON doesn’t provide. On the other hand, the JSON has the Score key which the struct knows nothing about. In this case, masnun.Address will be empty string.

Into Maps / Slices

We have previously mentioned how structs are cheaper and more widely used than maps. But there’s these use cases where we can not be certain about the structure of the JSON data we want to parse. In such cases, maps can be very useful. Let’s see:

See? We have passed map[string]interface{} and received all the data in JSON. But please remember, the values to each key in the map will be of type interface{}. If we want to extract part of the data, for example, one of the emails and then use it as a string, I will have to manually convert it to a string.

For example this code will fail:

We will get an error:

That’s what I was trying to explain just above. The Emails key has a value of interface{} type. Let’s cast it to a list of interface{} first. Then we can take an element (which will be again interface{} type). We further cast it to a string.

You may be wondering why couldn’t we just get Emails as []string? Well, Go doesn’t know the types of the values in our JSON. So it uses interface{} . That is when it stores Emails, it stores it as a list of unknown types or a list of interface{}. That is why we first need to get it as a list and then we can take individual items and further convert them to the type we want.

Now it works fine 🙂

Streaming JSON Encoding and Decoding

The json package offers NewEncoder and NewDecoder functions which would get us Encoder and Decoder types. These types can work with other objects that support io.Reader and io.Writer interfaces to offer streaming support.

Streaming JSON from a File

We can open a JSON file using the os.Open function and stream it using the json.NewDecoder function. Here’s a quick example:

We opened a file which implements the io.Reader interface. So we can use it with our Decoder type. We created a decoder with the file reader and then called the Decode method on it. That’s all we needed 🙂

Streaming JSON into a File

Writing JSON is also very similar. We need to open a file in write mode, grab the io.Writer and pass it to json.NewEncoder – then we can pass our data to the Encode method to stream the json into the file.

 

Custom Marshal / Unmarshal

If we want to change how our own types are marshalled or unmarshalled, we can implement the json.Marshaler and json.Unmarshaler interfaces. It’s actually simple. We need to define the MarshalJSON and UnmarshalJSON methods on our structs and we’re done. Here’s an example from the official documentation:

Pretty nice, no?

Categories
Python

Python: Iterators

If you have written some Python code and used the for loop, you have already used iterators behind the scene but you probably didn’t know about it. Iterators are objects that we can iterate over one by one. They are practically everywhere in a Python codebase. Understanding the concepts of iterators and how they work can help us write better, more efficient code from time to time. In this post, we will discuss iterators and other related concepts.

How does iteration work?

Before we can dive into iterators, we first need to understand how iteration works in Python. When we do the for loop, how does Python fetch one item at a time? How does this process work?

There are two functions that come into play – iter and next. The iter function gets an iterator from an object. It actually calls the __iter__ special method on the object to get the iterator. So if an object wants to allow iteration, it has to implement the __iter__ method. Once it gets the iterator object, it continues to call next on the iterator. The next function in turn calls the __next__ method on the iterator object. Let’s see a quick example:

Let’s see. We first create a list named l with 3 elements. We then call iter() on it. The type of l is list but look at the type of i – it’s list_iterator – interesting! Now we keep calling next on i and it keeps giving us the values we saw in the list, one by one, until there’s a StopIteration exception.

Here the list is an iterable because we can get an iterator from it to iterate over the list. The list_iterator object we got is an iterator, it’s an object that we can actually iterate over. When we loop over a list, this is what happens:

Makes sens? The for loop actually gets the iterator and keeps looping over until a StopIteration exception is encountered.

Iterator

The iterator is an object which implements __next__ method so we can call next on it repeatedly to get the items. Let’s write an iterator that keeps us giving us the next integer, without ever stopping. Let’s name it InfiniteIterator.

If we keep calling next on it, we will keep getting the integers, starting from one.

Iterable

What if we wanted to create an InfiniteNumbers iterable? It would be such that when we use the for loop on it, it never stops. It keeps producing the next integer in each loop. What would we do? Well, we have an InfiniteIterator. All we need is to define an __iter__ method that returns a new instance of InfiniteIterator.

If you remove the break statement and the if block, you will notice, it keeps running – like forever.

Using StopIteration

Instead of breaking out from our code ourselves, we could use the StopIteration exception in our iterator so it stops after giving us the 100 numbers.

Iterators must also implement __iter__

We saw that the __next__ method does it’s work just fine. But we also need to implement the __iter__ method on an iterator (just like we did in iterable). Why is this required? Let me quote from the official docs:

Iterators are required to have an__iter__() method that returns the iterator object itself so every iterator is also iterable and may be used in most places where other iterables are accepted.

If we tried to use the for loop over our iterator, it would fail:

We will get the following exception:

That kind of makes sense because we saw that the for loop runs the iter function on an object to get an iterator from it. Then calls next on the iterator. That’s the problem, we don’t have an __iter__ method. The official documentation suggests that every iterator should be a proper iterable too. That is, it should implement the __iter__ method and just return an instance of itself. Let’s do that:

Now the code works fine 🙂

The Iterator Protocol

The iterator protocol defines the special methods that an object must implement to allow iteration. We can summarize the protocol in this way:

  • Any object that can be iterated over needs to implement the __iter__ method which should return an iterator object. Any object that returns an iterator is an iterable.
  • An iterator must implement the __next__ method which returns the next item when called. When all items are exhausted (read retrieved), it must raise the StopIteration exception.
  • An iterator must also implement the __iter__ method to behave like an iterable.

Why do we need Iterables?

In our last example, we saw that it’s possible for an object to implement a __next__ method and an __iter__ method that returns self. In this way, an iterator behaves just like an iterable alright. Then why do we need Iterables? Why can’t we just keep using Iterators which refer to itself?

Let’s get back to our HundredIterator example. Once you have iterated over the items once, try to iterate again. What happens? No numbers are output on the screen. Why? Well, because the iterator objects store “state”. Once it has reached StopIteration, it has reached the end line. It’s now exhausted. Every time you call iter on it, it returns the same instace (self) which has nothing more to output.

This is why Iterables are useful. You can just return a fresh instance of an iterator every time the iterable is looped over. This is actually what many built in types like list does.

Why is Iterators so important?

Iterators allow us to consume data each item at a time. Just imagine, if there’s a one GB file and we tried to load it all in memory, it would require huge memory. But what if we implemented an iterator that reads the file one line at a time? We could then just store that one line in memory and do necessary processing before moving on to the next item. This allow us to write really efficient programs 🙂

This all seems very confusing

If you find the concepts very confusing and hard to grasp, don’t worry. Give it a few tries, write the codes by hand and see the output. Tinker with the examples. Inspect the code, try to see what happens when you modify part of it. All things become easier when you practise more and more. Try writing your own iterables and iterators – perhaps try to clone the built in containers’ functionalities? May be write your own list implementation? Don’t worry, it will come to you in time.

 

Categories
Python

Django REST Framework: JSON Web Tokens (JWT)

(This post is a part of a tutorial series on Building REST APIs in Django)

Our last post was about Authentication and Permissions and we covered the available methods of authentication in Django REST Framework. In that post, we learned how to use the built in Token based authentication in DRF. In this post, we will learn more about JSON Web Tokens aka JWT and we will see if JWT can be a better authentication mechanism for securing our REST APIs.

Understanding JSON Web Tokens (JWTs)

We have actually written a detailed blog post about JSON Web Tokens earlier. In case you have missed it, you probably should read it first. We have also described how to use JWT with Flask – reading that one might also help better understand how things work. And of course, we will briefly cover the idea of JWT in this post as well.

If we want to put it simply – you take some data in JSON format, you hash it with a secret and you get a string that you use as a token. You (your web app actually) pass this token to the user when s/he logs in. The user takes the token and on subsequent requests, passes it back in the “Authorization” header. The web app now takes this token back, “decodes” it back to the original JSON payload. It can now read the stored data (identity of the user, token expiry and other data which was embedded in the JSON). While decoding, the same secret is used, so third party attackers can’t just forge a JWT. We would want our token to be small in size, so the JSON payload is usually intentionally kept small. And of course, it should not contain any sensitive information like user password.

JWT vs DRF’s Token Based Authentication

So in our last blog post, we saw Django REST Framework includes a token based authentication system which can generate a token for the user. That works fine, right? Why would we want to switch to JSON Web Tokens instead of that?

Let’s first see how DRF generates the tokens:

It’s just random. The token generated can not be anyway related to the user that it belongs to. So how does it associate a token with an user? It stores the token and a reference to the user in a table in database. Here comes the first point – while using DRF’s token based auth, we need to query database on every request (unless of course we have cached that token which). But what if we have multiple application servers? Now we need all our application servers to connect to the same database or same cache server. How will that scale when the project gets really really big? What if we want to provide single sign on across multiple services? We will need to maintain a central auth service where other services request to verify a token. Can JWT simplify these for us?

JWT is just an encoded (read – hashed / signed) JSON data. As long as any webservice has access to the secret used in signing the data, it can also decode and read the embedded data. It doesn’t need any database calls. You can generate the token from one service and other services can read and verify it just fine. It’s more efficient and simply scales better.

JWT in Django REST Framework

DRF does not directly support JWTs out of the box. But there’s an excellent package that adds support for it. Let’s see how easily we can integrate JWT in our REST APIs.

Install and Configure

Let’s first install the package using pip –

That should install the package. Now we need to add rest_framework_jwt.authentication.JSONWebTokenAuthentication to the default authentication classes in REST Framework settings.

We added it to the top of the list. Next, we just have to add it’s built in view to our urlpatterns.

Obtain a Token

The obtain_jwt_token view will check the user credentials and provide a JWT if everything goes alright. Let’s try it.

Awesome, everything worked just fine. We have got our token too. What do we do next? We use this token to access a secured resource.

Using the obtained JWT

We need to pass the token in the form of JWT <token> as the value of the Authorization header. Here’s a sample curl request:

So our token worked fine! Cool!

Where to go next?

Now that you have seen how simple and easy it is to add JSON Web Token based authentication to Django REST Framework, you probably should dive deeper into the package documentation. Specially these topics might be interesting –

  • Refresh Tokens: If you enable JWT token refreshing, you can exchange your current token with a new, fresh one before the existing one expires. The new token will of course have a renewed expiry time set.
  • Verify Token: If you just share the secret, all services can verify the user on their own. However, in modern micro service based architecture, you may want to provide an API end point that other services can use to verify a JWT they received from the user. This can be useful for those scenarios.
  • And of course look at the settings options available and see how you can customize the token generation process.

In the future, we shall try to cover more about Django, Django REST Framework and Python in general. If you liked the content, please subscribe to the mailing list so we can notify you when we post new contents.