Categories
General Python

Auto incrementing IDs for MongoDB

If you’re familiar with relational databases like MySQL or PostgreSQL, you’re probably also familiar with auto incrementing IDs. You select a primary key for a table and make it auto incrementing. Every row you insert afterwards, each of them gets a new ID, automatically incremented from the last one. We don’t have to keep track of what number comes next or ensure the atomic nature of this operation (what happens if two different client wants to insert a new row at the very same time? do they both get the same id?). This can be very useful where sequential, numeric IDs are essential. For example, let’s say we’re building a url shortener. We can base62 encode the ID of the url id to quickly generate a short slug for that long url.

Fast forward to MongoDB, the popular NoSQL database doesn’t have any equivalent to sequential IDs. It’s true that you can insert anything unique as the required _id field of a mongodb document, so you can take things to your hand and try to insert unique ids yourselves. But you have to ensure the uniqueness and atomicity of the operation.

A very popular work around to this is to create a separate mongodb collection. Then maintain documents with a numeric value to keep track of your auto incrementing IDs. Now, every time we want to insert a new document that needs a unique ID, we come back to this collection, use the $inc operator to atomically increment this number and then use the incremented number as the unique id for our new document.

Let me give an example, say we have an messages collection. Each new message needs a new, sequential ID. We create a new collection named sequences. Each document in this sequences collection will hold the last used ID for a collection. So, for tracking the unique ID in the messages collection, we create a new document in the sequences collection like this:

Next, we will write a function that can give us the next sequential ID for a collection by it’s name. The code is in Python, using PyMongo library.

If we need the next auto incrementing ID for the messages collection, we can call it like this:

Find and Modify – Deprecated

If you have searched on Google, you might have come across many StackOverflow answers as well as individual blog posts which refer to findAndModify() call (find_and_modify in Pymongo). This was the way to do things. But it’s deprecated now, so please use the new find_one_and_update function now.

(How) Does this scale?

We would only call the get_sequence function before inserting a new mongo document. The function uses the $inc operator which is atomic in nature. Mongo guarantees this. So even if 100s of different clients trying to increment the value for the same document, they will be all applied one after one. So each value they get will be unique, new IDs.

I personally haven’t been able to test this strategy at a larger scale but according to people on StackOverflow and other forums, people have scaled this to thousands and millions of users. So I guess it’s pretty safe.

Categories
Javascript

REST API with KoaJS and MongoDB (Part – 3)

In Part -1 of this series, we saw how we can get started with KoaJS and in Part – 2 we built CRUD endpoints with MongoDB. In this part, we’re going to work with authentication. We will be using JSON Web Tokens aka JWT for the auth part. We have written detailed pieces on JWT before. You can read Understanding JWT to check out the basics and read our tutorial on JWT with Flask or JWT with Django to see how other frameworks like Flask uses JWT.

JWT with KoaJS

To implement JSON Web Tokens with KoaJS, we would be using two packages – koa-jwt and jsonwebtoken. The second package (jsonwebtoken) provides useful helper functions to generate and verify JWTs. Where as koa-jwt provides an easy to use middleware that we can use with KoaJS.

Let’s go ahead and install these packages:

That should install the dependencies and save them in our package.json.

Securing Routes with JWT

We have the required packages installed. So we can now start securing our routes with JWT. We can just require the koa-jwt package directly and use it. But we want to customize some aspects. For that we would create our own module named jwt.js and put the custom stuff in there.

Now in our index.js file, we would add the middleware to the app.

If we try to visit http://localhost:3000/, we will get a plain text error message saying “Authentication Error”. While the message is clear and concise, we want to output JSON, not a plain text error message. For that, we will write a custom middleware.

The code for this middleware is pretty simple. It invokes the next middleware and if it catches an error, it checks if it’s 401, if so, it sets a nice detailed JSON as the output. If you’re familiar with how middlewares work in express / koa, this should make sense. If it doesn’t make sense, don’t worry, you will get it over time.

Now we need to export this function from our jwt.js module. Let’s change the exports a little bit.

Now we’re exporting two functions, which, when called will return the specific middlewares. We also need to change our imports in index.js –

Please note the order of the middleware we used. The error handler must come before the JWT middleware itself, so it can call next() and check for the 401 error.

If we try to browse the API now, we should get a nice JSON like this:

Secured Routes and Router

We used the middleware directly on the koa app. That means all our routes are now secure. All the routes would now check for the Authorization header value and try to verify it’s value as a JSON Web Token. That’s good but there’s a slight problem. If we can’t access any of the routes without a token, which route do we access to get the token in the first place? And what token do we use for that? Yeah, we need to have at least one route which is not secured with JWT which will accept login details and issue the JWTs to the users. Besides, there could be other API end points which we can keep open to everyone, we don’t need authentication on those routes. How do we achieve that?

Luckily, Koa allows us to use multiple routers and each router can have their own set of middlewares. We will keep our current router open and add the routes to obtain the JWT. We will create a separate route which will use the middleware and be secured. We will call this one the “secured router” and the routes would be “secured routes”.

We modified our existing codes. We now have two routers and we added them both to the app. Let’s now move our old CRUD routes to the secured router and apply the JWT middleware to just the secured router.

We removed the previously setup JWT middleware from the app and used it on securedRouter instead. Remember, the JWT middleware must be setup before we setup the routes themselves. Ordering of middleware matters.

If we try to visit “http://localhost:3000/”, we will no longer get the auth error, rather will see “not found” (we didn’t define any routes for the root url). However, if we try to visit “http://localhost:3000/people”, we will get the authentication error again. Exactly what we wanted.

Issuing JWTs

We now need to create the route to issue JWTs to our users. We will be accepting their login (username and password) and if they’re valid, we will issue them JWTs which they can use to further access our APIs.

The koa-jwt package no longer supports issuing tokens. We have to use the jsonwebtoken package for that instead. Personally, I like to create a helper function in my custom jwt.js module like this:

Then we can write a new route on our public router like this:

We have hardcoded the username and password here. In production environments, we would store the details in a database and we would hash the password. No one in their right mind should store password in plain text.

In this view, we are accepting a JSON payload and checking the username and password. And then if the details match, we are issuing the token. To test if it’s working, we can make a curl request and checkout the response:

If it worked, we will get a JSON back with a token value containing the JWT.

Using the JWT

We made a request and got the following response:

That is our token. Now we can start using it in the Authorization header. The format should be like:Authorization: Bearer <Token> . We can make a request to our secured “/people” resource using curl with this header:

We will now get back the list of people we have stored in our mongodb.

Categories
Javascript

REST API with KoaJS and MongoDB (Part – 2)

In our last post about REST API with KoaJS and MongoDB, we got started with KoaJS and learned to create simple views. We also saw how we can access the query strings and incoming JSON payloads. In this tutorial, we are going to go ahead and implement the RESTful routes and CRUD operations with MongoDB.

In case you’re new to REST API development, you might also want to check out the REST API Concepts and our REST API Tutorials with Flask and Django REST Framework.

Installing MongoDB and NodeJS Driver

I am assuming you have installed MongoDB already on your system. You can install MongoDB using their official installer on Windows or Homebrew on OS X. On Linux systems, you can use the package manager that ships with your distro.

If you don’t have MongoDB installed locally, don’t worry, you can also use a free third party mongo hosting service like mLab.

Once we have MongoDB setup and available, we’re going to install the NodeJS driver for MongoDB next.

Connecting to MongoDB

We will create a separate file named mongo.js and put the following codes in it:

Our module exports just one function which takes the app object as the only parameter. Once called, the function connects to our mongodb instance and once connected, sets the people property on our app instance. The people property would actually be a reference to the people collection on our database. So whenever we will have access to the app instance, we will be just using the app.people property to access the collection from within our app. If the connection fails, we will have the error message printed on our terminal.

We have used promises instead of callback. Which makes the code a bit cleaner. Now in our index.js file, we will call the exported function like this:

That should import the function and invoke it. Assuming everything worked fine, you should see the message saying database connection established when you run the app next time.

Please Note: We didn’t create the mongodb database or the collection ourselves. MongoDB is smart enough to figure out that we used the names of non existing database / collection and create them for us. If anything with that name exists already, just uses them.

Inserting Records Manually

Before we can start writing our actual code, let’s connect to our mongo database and insert some entries manually so we can play with those data. You can use the command line tool or a mongodb GUI to do so. I will use the command line tool.

I inserted a document with my name and email address in the people collection of the polyglot_ninja db.

Implementing The Routes

Now we will go ahead and implement the routes needed for our REST API.

Please note: Too keep the actual code short, we will skip – validation, error handling and sending proper http status codes. But these are very important in real life and must be dealt with proper care. I repeat – these things are skipped intentionally in this tutorial but should never be skipped in a production app.

GET /people (List All)

This is going to be our root element for the people api. When someone makes a GET request to /people, we should send them a list of documents we have. Let’s do that.

Now if we run our app and visit the url, we shall see the document we created manually listed there.

POST /people (Create New)

Since we already have the body parser middleware installed, we can now easily accept JSON requests. We will assume that the user sends us properly valid data (in real life you must validate and sanitize) and we will directly insert the incoming JSON into our mongo collection.

You can POST JSON to the /people endpoint to try it.

Now go back to the all people list and see if your new requests are appearing there. If everything worked, they should be there 🙂

GET /people/:id (Get One)

To query by mongo IDs we can’t just use the string representation of the ID but we need to convert it to an ObjectID object first. So we will import ObjectID in our index.js file first:

The rest of the code will be simple and straightforward:

PUT /people/:id (Update One)

We usually use PUT when we want to replace the entire document. For single field updates, we prefer PATCH. In the following code example, we have used PUT but the code is also valid for a PATCH request since mongo’s updateOne can update as many fields as you wish. It can update just one field or the entire document. So it would work for both PUT and PATCH methods.

Here’s the code:

The updateOne method requires a query as a matching criteria to find the target document. If it finds the document, it will update the fields passed in an object (dictionary) in the second argument.

Delete /people/:id (Delete One)

Deleting one is very simple. The deleteOne method works just like the updateOne method we saw earlier. It takes the query to match a document and deletes it.

What’s Next?

In this tutorial, we saw how we can implement RESTful routes and use MongoDB CRUD operations. We have finally created a very basic REST APIs. But we didn’t validate or sanitize incoming data. We also didn’t use proper http status codes. Please go through different resources on the internet or our earlier REST API tutorials to learn more about those.

In our future tutorials, we shall be covering authentication, serving static files and file uploads.