Go aka Golang is a very promising programming language with a lot of potential. It’s very performant, easy to grasp and maintain, productive and backed by Google. In our earlier posts, we have tried to provide guidelines to learn Go and later we saw how to work with JSON in Go. In this blog post, we’re going to see how we can make http requests using Go. We shall make use of the net/http package in Go which provides all the stuff we need to make http requests or create new http servers. That is, this package would help you do all things “http”. To check / verify that we made correct requests, we would be using httpbin which is a nice service to test our http client requests.
A Simple HTTP Request
Let’s make a very simple GET request and see how we can read the response. We would be sending a simple HTTP GET request to https://httpbin.org/get and read the response. For that we can just import the net/http package and use the http.Get function call. Let’s see an example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
package main import ( "net/http" "log" "io/ioutil" ) func main() { MakeRequest() } func MakeRequest() { resp, err := http.Get("https://httpbin.org/get") if err != nil { log.Fatalln(err) } body, err := ioutil.ReadAll(resp.Body) if err != nil { log.Fatalln(err) } log.Println(string(body)) } |
We have created a separate MakeRequest function and called it from our main function. So going ahead, we will just see the changes inside this function and won’t need to think about the entire program. Inside this function, we have passed the url to http.Get and received two values – the response object and any errors that might have happened during the operation. We did a check to see if there were any errors. If there weren’t any errors, err would be nil. Please note that this err would be reported only if there was an issue connecting to the server and getting a response back. However, it would not be concerned about what http status code the server sent. For example, if the server sends a http 500 (which is internal server error), you will get that status code and error message on the resp object, not on err.
Next, we read the resp.Body which implements the io.ReadCloser interface and we can use ioutil.ReadAll to fully read the response. This function also returns two values – a byte slice ([]byte) and err. Again, we check for any potential errors in reading the response body. If there were no errors, we print out the body. Please note the string(body) part. Here, we’re converting the byte slice to a string. If we don’t do it, log.Println would print out representation of the byte slice, a list of all the bytes in that slice, individually. But we want a string representation. So we go ahead and make the conversion.
We would see the printed output is a JSON string. You will notice the httpbin service outputs JSON messages. So in the next example, we would see how we can send and read JSON messages.
JSON Requests and Responses
Now let’s send a JSON message. How do we do that? If you’re coming from Python / Node / Ruby, you might be used to passing a dictionary like structure to your favorite requests library and just mention it should be sent as JSON. Your library does the conversion for you and sends the request with required headers. In Go, however, things are more explicit, which is a good thing in fact. You will know what you’re doing, how you’re doing it. If the JSON related functionality is new to you, please do check our blog post – Golang: Working with JSON.
In Go, we would first convert our data structure to a byte slice containing the JSON representation of the data. Then we pass it to the request with the proper content type. Let’s see a code example:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
func MakeRequest() { message := map[string]interface{}{ "hello": "world", "life": 42, "embedded": map[string]string{ "yes": "of course!", }, } bytesRepresentation, err := json.Marshal(message) if err != nil { log.Fatalln(err) } resp, err := http.Post("https://httpbin.org/post", "application/json", bytes.NewBuffer(bytesRepresentation)) if err != nil { log.Fatalln(err) } var result map[string]interface{} json.NewDecoder(resp.Body).Decode(&result) log.Println(result) log.Println(result["data"]) } |
We first created message which is a map containing a string value, an integer value and another embedded map. Then we json.Marshal it to get the []byte out of it. We also check for any errors that might happen during the marshalling. Next, we make a POST request using the http.Post function. We pass the url, our content type, which is JSON and then we create and pass a new bytes.Buffer object from the bytes representation. Why do we need to create a buffer here? The http.Post function expects an implementation of io.Reader – which is a brilliant design, anything that implements an io.Reader can be passed here. So we could even read this part from disk or network or any custom readers we want to implement. In our case, we can just create a bytes buffer which implements the io.Reader interface. We send the request and check for errors.
Next we declare another result variable (which is also a map type) to store the results returned from the request. We could read the full body first (like previous example) and then do json.Unmarshal on it. However, since the resp.Body is an io.Reader, we can just pass it to json.NewDecoder and then call Decode on it. Remember, we have to pass a pointer to our map, so we passed &result instead of just result. The Decode function returns an error too. But we assumed it would not matter and didn’t check. But best practice would have been to handle it as well. We logged the result and result["data"]. The httpbin service sends different information about the request as the response. You can see those in the result map. If you want to see the data you sent, they will be in the data key of the result map.
Posting Form
In our last example, we have submitted JSON payload. What if we wanted to submit form values? We have the handy http.PostForm function for that. This function takes the url and url.Values from net/url package. The url.Values is a custom type which is actually map[string][]string internally. That is – it’s a map which contains string keys and against each key, there can be multiple string values ([]string). In a form request, you can actually submit multiple values against one field name. That’s the reason it’s a slice of string, instead of just a key to value mapping.
Here’s an example code snippet:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
func MakeRequest() { formData := url.Values{ "name": {"masnun"}, } resp, err := http.PostForm("https://httpbin.org/post", formData) if err != nil { log.Fatalln(err) } var result map[string]interface{} json.NewDecoder(resp.Body).Decode(&result) log.Println(result["form"]) } |
We would be reading the form key from the result map to retrieve our form values. We have seen how we can easily send form values using the net/http package. Next we would like to send a file along with typical form fields. For that we would also need to learn how to customize http requests on our own.
Custom Clients / Requests
The http.Get, http.Post or http.PostForm calls we have seen so far uses a default client already created for us. But now we are going to see how we can initialize our own Client instances and use them to make our own Requests. Let’s first see how we can create our own clients and requests to do the same requests we have made before. A quick example follows:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
func MakeRequest() { client := http.Client{} request, err := http.NewRequest("GET", "https://httpbin.org/get", nil) if err != nil { log.Fatalln(err) } resp, err := client.Do(request) if err != nil { log.Fatalln(err) } var result map[string]interface{} json.NewDecoder(resp.Body).Decode(&result) log.Println(result) } |
As you can see, we just take a new instance of http.Client and then create a new request by calling http.NewRequest function. It takes the http method, url and the request body. In our case, it’s a plain GET request, so we pass nil for the body. We then call the Do method on the client and parse the response body. So that’s it – create a client, create a request and then let the client Do the request. Interestingly the client also has convenient methods like Get, Post, PostForm – so we can directly use them. That’s what http.Get, http.Post, http.PostForm and other root level functions actually do. They call these methods on the DefaultClient which is already created beforehand. In effect, we could just do:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
func MakeRequest() { client := http.Client{} resp, err := client.Get("https://httpbin.org/get") if err != nil { log.Fatalln(err) } var result map[string]interface{} json.NewDecoder(resp.Body).Decode(&result) log.Println(result) } |
And it would work similarly. Now you might be wondering – why not just use the DefaultClient, why create our own? What is the benefit?
Customizing the Client
If we look at the definition of the http.Client structure, it has these fields:
|
1 2 3 4 5 6 |
type Client struct { Transport RoundTripper CheckRedirect func(req *Request, via []*Request) error Jar CookieJar Timeout time.Duration } |
If we want, we can set our own transport implementation, we can control how the redirection is handled, pass a cookie jar to save cookies and pass them to the next request or simply set a timeout. The timeout part is often very significant in making http requests. The DefaultClient does not set a timeout by default. So if a malicious service wants, it can start blocking your requests (and your goroutines) indefinitely, causing havoc in your application. Customizing the client gives us more control over how the requests are sent.
File Upload
For uploading files while sending a http request, we need to use the mime/multipart package with the net/http package. We will first see the code example and then walk through it to understand what we’re doing. The code might seem a lot (it includes a lot of error handling) and complex. But please bear with me, once you go through the code and understand what’s happening, it will seem so simpler 🙂
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 |
func MakeRequest() { // Open the file file, err := os.Open("name.txt") if err != nil { log.Fatalln(err) } // Close the file later defer file.Close() // Buffer to store our request body as bytes var requestBody bytes.Buffer // Create a multipart writer multiPartWriter := multipart.NewWriter(&requestBody) // Initialize the file field fileWriter, err := multiPartWriter.CreateFormFile("file_field", "name.txt") if err != nil { log.Fatalln(err) } // Copy the actual file content to the field field's writer _, err = io.Copy(fileWriter, file) if err != nil { log.Fatalln(err) } // Populate other fields fieldWriter, err := multiPartWriter.CreateFormField("normal_field") if err != nil { log.Fatalln(err) } _, err = fieldWriter.Write([]byte("Value")) if err != nil { log.Fatalln(err) } // We completed adding the file and the fields, let's close the multipart writer // So it writes the ending boundary multiPartWriter.Close() // By now our original request body should have been populated, so let's just use it with our custom request req, err := http.NewRequest("POST", "https://httpbin.org/post", &requestBody) if err != nil { log.Fatalln(err) } // We need to set the content type from the writer, it includes necessary boundary as well req.Header.Set("Content-Type", multiPartWriter.FormDataContentType()) // Do the request client := &http.Client{} response, err := client.Do(req) if err != nil { log.Fatalln(err) } var result map[string]interface{} json.NewDecoder(response.Body).Decode(&result) log.Println(result) } |
So what are we doing here?
- First we are opening the file we want to upload. In our case, I have created a file named “name.txt” that just contains my name.
- We create a
bytes.Bufferto hold the request body we will be passing with ourhttp.Requestlater on. - We create a
multipart.Writerobject and pass a pointer to ourbytes.Bufferobject so the multipart writer can write necessary bytes to it. - The multipart writer has convenient methods to create a form file or a form field. It gives us back a writer to which we can write our file content or the field values. We create a file field and copy our file contents to it. Then we create a normal field and write “Value” to it.
- Once we have written our file and normal form field, we call the
Closemethod on the multipart writer object. Closing it writes the final, ending boundary to the underlyingbytes.Bufferobject we passed to it. This is necessary, otherwise the request body may remain incomplete. - We create a new post request like we saw before. We passed the
bytes.Bufferwe created as the request body. The body now contains the multi part form data written with the help of themime/multipartpackage. - We send the request as before. But we set the content type by calling
multiPartWriter.FormDataContentType()– which ensures the correct content type and boundary being set. - We decode the response from httpbin and check the output.
If everything goes well, we will see the form field and the file name in the response we received from httpbin. The concept here is simple. We are sending a http request with a custom body. We could construct the request body ourselves but we just took the help of the mime/multipart package to construct it in a relatively easier fashion.
Always Close The Response Body
Here’s a lesson I learned the hard way. When we make a http request, we get a response and an error back. We may feel lazy and decide not to check for errors or close the response body (just like in the examples above). And from the laziness comes disaster. If we do not close the response body, the connection may remain open and cause resource leak. But if the error is not nil that is in case of an error, the response can be nil. So we can’t just do a defer resp.Body.Close() in this case. We have to properly check error and then close the response body.
|
1 2 3 4 5 6 |
client := http.DefaultClient resp, err := client.Do(req) if err != nil { return nil, err } defer resp.Body.Close() |
Always Use a Timeout
Try to use your own http client and set a timeout. Not setting a timeout can block the connection and the goroutine and thus cause havoc. So do something like this:
|
1 2 3 4 5 |
timeout := time.Duration(5 * time.Second) client := http.Client{ Timeout: timeout, } client.Get(url) |