A common operation when it comes to working with command line applications is the encoding and decoding of data into concrete Go types. In computer science, this is known as marshalling and unmarshalling, and allows us to effectively work with the data inside of our code.

In the last lesson on working with HTTP, we received data back from the HTTP bin service in the form of JSON. Here I have the same code that we looked at before, which was when we were performing the git request with the URL values of name, John, name, Sally, and age 20, as well as a request header of x user ID foobar123. If I go ahead and run this code again, as we saw before, you can see we receive back the following data, which is encoded in JSON.

If you're unaware, JSON, or JavaScript Object Notation, is perhaps the most common form of data interchange formats that is found when it comes to software development. At least when working with the web. JSON is actually a pretty simple format. If you want to know more about how JSON works, then I recommend reading the Wikipedia page on it. But in a nutshell, it allows you to use key value pairs, where the key is defined as a string, and the value can either be a string, a number, a boolean, another object which in turn contains key value pairs, or an array which can contain individual values such as strings, or it can contain objects themselves. Additionally, you can also define nullable values as well.

To show how encoding and decoding data works, I'm going to go ahead and create a new project called JSON data, and use the following go init command:

dreamsofcode.ashio/JSON data

and create our main.go file as we saw before. Then as for the actual data we want, let's go ahead and define a data string that we can use to represent our JSON response, which we can set to be as follows. We'll set this to be a name of Alice and age of 20. This will be the JSON string that we're going to use in this video.

Now we want to be able to turn this JSON object into a concrete type, which we can define as follows. We'll call this a person, and we'll define it as a struct. Then we want to give it two fields of name, which will be a string, and age, which will be an integer. To do so, we're going to use the encoding/json package of the standard library, which provides a number of different functions and types used for decoding and encoding data, as well as validating JSON as well.

When it comes to marshalling and unmarshalling data in Go, the individual properties that you want to map a field to need to begin with an uppercase letter. This is because the fields need to be exported in order for reflection to be able to pick them up. However, this comes with a bit of a caveat, as you'll notice that the names of the properties inside of our person struct no longer correlate with the names of the field, as they have now different casing. Fortunately, Go provides a solution to this in order to allow you to define which fields match to which properties. This is done using struct field tags, which you can actually read more about in the documentation.

Here you can see that there's a field with the value type of int, which is being given the struct field tag of json, mapping to the field called my name. Therefore, let's go ahead and add some struct field tags into our person struct. To begin, we can use the back tick to mark the actual struct field tag itself, then define the struct tag for json. This tells Go which encoding type we want the struct fields to apply to, in our case just the JSON encoding. Then we can define the actual field name, which in this case is going to be name.

Let's go ahead and repeat this for the age property as well. With that, we now have our struct field defined. Let's go ahead and convert our JSON string or JSON data into our person type. To do so, we need to use the encoding/json package of the standard library, which as we saw provides a lot of functionality for working with JSON data. Then we want to use, then in order to convert our JSON string into a person struct, we can use the unmarshal function of the JSON package, which takes a byte slice of our data and a second property of any. This any is what we want to convert the JSON type into. We'll take a look at what that means shortly. Additionally, this function also returns an error value in case something goes wrong.

First things first, let's go ahead and pass in our JSON data, which we need to convert into a byte slice as follows:

jsonData := []byte(`{"name": "Alice", "age": 20}`)

Next, we then want to pass in an instance of our person in order for it to convert into. To do so, we first need to define an instance using the following line:

var person Person

Then we can pass this instance of our person to the actual value, but we need to do so by passing it in as a pointer, rather than copying the value over, which means we need to prefix it with an ampersand.

Before we run this code, let's go ahead and capture the error value and do some light error handling. Again, log.Fatal just in case something goes wrong, we can see what actually happens:

log.Fatal("failed to unmarshal JSON data into person:", err)

And we'll print out the error just so that we have the ability to debug in case something goes wrong. Next, let's go ahead and actually print out our person's struct. First, printing out the name, which in this case is going to be person.name, and printing out the age, which is going to be the person.age property.

Now, if I go ahead and open up a new window and use the go run command, you can see that we're printing out the various properties from our JSON data encoding. Pretty cool.

As well as being able to unmarshal data into a concrete Go type, you can also do the inverse using the JSON package as well. To show what I mean, let me go ahead and quickly remove this code and create a new instance of our person struct called Bob is going to be a person with a name of Bob and give them an age of 72. Bit of an old boy. Now, let's go ahead and turn our person of Bob into some JSON data.

To do this, we can use the JSON package again. But this time, rather than using the unmarshal function, let's go ahead and use the marshal function, which takes in any type. In this case, we can go ahead and pass in Bob. This time, you'll notice we're not passing it by reference or by pointer, as we don't need the values of Bob to actually change. The response to this is a slice of bytes, which we can call as data, and an error in case something goes wrong. In this case, I'm going to go ahead and ignore the error because we know what error handling looks like.

And let's go ahead and then print out the data, casting it to a string first. Now, if I go ahead and run this code using the following command, you can see that Bob has been converted into a JSON string. Pretty cool.

As well as simple data types, such as strings and integers, you can also use JSON encoding with more complex data types as well, such as a slice of data. In this case, a slice of string, which we can define as skills, which if we go ahead and define, we can do as follows:

skills := []string{"C++", "Go", "Java"}

And if we go ahead and run this, you can see our slice has been converted into a JSON array, or with even sub-objects as well. For example, let's go ahead and create a new type called animal, which again is going to have a name, we'll call this name, we'll call it a type, I guess. And we'll also give this an age.

Then we can define a pet as animal, including json:"pet". Then in our case, let's say Bob has a cute cat. Type, cat, name, muffins, muffin, and age of four.

Oh, age needs to be an integer. Let's go ahead and set this. Now if I go ahead and run this code again, you can see that Bob's pet has also been encoded as well.

To make this easier, I'm going to pipe this into the jq command, which gives us some nicer output. However, what if we have a person that doesn't actually have a pet? Let's say Alice again, who happens to not like pets and doesn't actually have one. For example, if we go ahead and define this, let's go ahead and then marshal Alice into her own type as well.

Now, if we go ahead and run this code, you can see that, as you can see, Alice has no skills and no pets. However, these are kind of clunking up the actual JSON output that we're receiving. Instead, it would be a lot nicer if we could just emit these fields from being empty.

Fortunately, Go actually provides us the ability to do that by adding in the omitempty property as follows. This property will prevent the JSON package from encoding any fields that conform to the empty value in Go. In Go, empty values are what the default is if you don't specify an individual value. So for a number, this would be zero. A string, this would be an empty string. And for a slice, it's going to be the nil value as well.

With the omitempty property defined, let's now go ahead and run this code to see what's changed. As you can see, our skills are no longer being produced, but even though our pet doesn't exist, this actually is. This is because when it comes to sub-objects, it's not actually an empty value. As you can see, we're still printing out both the name, the type, and the age.

Therefore, to resolve this, we would also need to add the omitempty value here as well, which would then cause it to no longer exist. Unfortunately, however, when it comes to a nested struct, the empty value doesn't exist unless you use a pointer. Therefore, if we wanted to exclude the pet from being encoded when it's empty, we would need to turn this into a pointer value instead of a concrete struct.

Which now, if I go ahead and run the code again, you can see no longer prints out our pet field. As well as using both the marshal and unmarshal functions to turn the data into byte arrays, or byte slices, we can also use the decoder and encoder types from the JSON package in order to work with both IO writers and IO readers.

For example, rather than marshalling this data into a byte array and printing it out, we can instead go ahead and use the json.NewEncoder type, passing in the os.Stdout as the writer function. Then we can just go ahead and use the Encode function, passing in Alice as follows:

json.NewEncoder(os.Stdout).Encode(Alice)

This function does return an error, but again, we're going to ignore it for this stage. Now, if we go ahead and run this code again, you can see it works as it did before. However, this time we didn't need to use any byte array in order to store the data, as it was written directly to the IO.writer we passed in, in this case stdout. This makes encoding and decoding data very useful when it comes to data operations in Go.

If we head back on over to the HTTP bin project that we looked at in the last lesson, you'll remember that we were receiving data back from the HTTP request that we were posting up. As you may remember, we were using the io.Copy function in order to copy the response body, which was an IO.reader, into the os.Stdout. However, if we want to actually perform some computation on that data or use it within our code, we can instead use it with the json.Decoder type in order to convert it into a concrete Go type.

To show what this looks like, let's go ahead and create a new struct called result and create some fields to capture properties we want to obtain from the actual response. If I take a quick look at the response again, we can see we have some args, some data, some json, some origin, and a URL. Let's go ahead and set the URL. Let's go ahead and create a field for the URL and the origin as well. And we'll define our JSON struct tags as follows:

type Result struct {
    URL    string `json:"url"`
    Origin string `json:"origin"`
}

Then with our result struct defined, let's go ahead and unmarshal the response data to it. We can do so by first importing the encoding/json package and then using it to create a new decoder, passing in our response body. Then we can just decode it into our any type, which we'll define as results, which is going to be a result. Let's pass that in as follows. And we'll scroll down so we can see it a little better. Again, I'm just going to ignore the error handling at this point, but in a production setting, you would want to make sure to handle this properly.

Now with our results, let's go ahead and print out the origin value as follows:

fmt.Println(results.Origin)

Now you can see we're getting back the IP address from where our request is originating from. And we can also print out the result.URL as follows. This allows us to take the response that we receive from our HTTP requests and we can use it within our code.

We'll take a look at this more later on in the last lesson of the video, where we're going to be interfacing with an actual API in order to receive response and use it in our computations. The last thing I think worth showing is actually sending up some data in this HTTP request. To do so, let's go ahead and create a new type called user, which will have a username of string, which will give the JSON struct of username and a level, I guess.

Let's say it's an online role-playing game of integer. And we'll set it to be there. Actually, let's call this a player. It's a little bit easier to understand. Then let's create a new player of Gandalf, which will be the player with a username of Gandalf. And we'll have a level of 99.

88. That'll do, I guess. Then in order to send up our Gandalf player through the HTTP request, let's convert it into a byte array by using the JSON.Marshal function, passing in Gandalf. Again, we'll ignore the error for the meantime. And we can then use the bytes.Buffer, new buffer, passing in our data slice as follows:

data := bytes.NewBuffer(jsonData)

Now, if I go ahead and run this code, you should see that the JSON field is now populated. However, you can see it's now still a form. This is because we've not corrected the content type that we want to use. In this case, rather than being x-www-form-urlencoded, we want to call it application/json, which tells the server that we're sending up JSON data.

Now, if we go ahead and run this again, this time we should see the JSON field be populated with the values that we've marshaled from our player struct. That gives a brief overview of how we can encode JSON data when it comes to the Go standard library.

As well as JSON, there are a number of other different encoding types that you can use when it comes to Go, such as YAML and CSV. Go itself provides a few of these encoding types within the actual standard library, such as the CSV package. However, others, such as YAML, are provided in third-party packages or extensions, such as the go-package.in/yaml/v3, which adds YAML support to the Go library.

In any case, whichever encoding that you want to use, I would recommend checking out the documentation for the package that you want to use, especially as they can provide different flags when it comes to the struct tags. In any case, as you can tell, when it comes to encoding data, having both the encoding and decoding functions allowing you to use I/O readers and I/O writers makes it really easy when it comes to data manipulation with Go, especially when it comes to working with files, such as reading the data in or writing to it.