Writing an API Wrapper in Golang
This article explores the process of writing an API wrapper in Golang and a few different programming steps to get there.
Join the DZone community and get the full member experience.
Join For FreeI had a really time-limited effort to do to prove how to write a command line wrapper for an open API a customer is developing.
The target REST API is the jquants-api, as presented in a previous article.
I chose to implement the wrapper in Golang, which proved to be extremely fast and pleasant to do. The task was eventually done in a short evening, and the resulting Golang wrapper with core features has been uploaded on GitHub.
This is the short story on the process to write the API and the few different programming steps to get there.
Goals
So first, let’s list the programming tasks that we will have to deal with:
- Create a test, and supporting code, checking we can save the username and password in an edn file compatible with the jquants-api-jvm format
- Write another test and supporting code to retrieve the refresh token
- Write another test and supporting code to retrieve the ID token
- Write another test and supporting code using the ID token to retrieve daily values
- Publish our wrapper to GitHub
- Use our Go library in another program
Start by Writing a Test Case, Preparing and Saving the Login Struct to Access the API
We always talk about writing code using TDD — now’s the day to do it. Check that we have code to enter and save the username and password in an edn file compatible with the jquants-api-jvm format.
In a helper_test.go file, let’s write the skeleton test for a PrepareLogin function.
package jquants_api_go
import (
"fmt"
"os"
"testing"
)
func TestPrepareLogin(t *testing.T) {
PrepareLogin(os.Getenv("USERNAME"), os.Getenv("PASSWORD"))
}
Here, we pick up the USERNAME and PASSWORD from the environment, using os.GetEnv
.
We will write the prepare function in a helper.go
file. It will:
- Get the username and password as parameters
- Instantiate a Login struct
- Marshal this as an EDN file content
func PrepareLogin(username string, password string) {
var user = Login{username, password}
encoded, _ := edn.Marshal(&user)
writeConfigFile("login.edn", encoded)
}
Our Login struct will first simply be:
type Login struct {
UserName string `edn:"mailaddress"`
Password string `edn:"password"`
}
And the call to edn.Marshal
will create a byte[] array content that we can write to file, and so writeConfigFile
will simply call os.WriteFile
with the array returned from the EDN marshaling.
func writeConfigFile(file string, content []byte) {
os.WriteFile(getConfigFile(file), content, 0664)
}
To be able to use the EDN library, we will need to add it to the go.mod
file with:
require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3
Before running the test, be sure to enter your jquants API’s credential:
export USERNAME="youremail@you.com"
export PASSWORD="yourpassword"
And at this stage, you should be able to run go test
in the project folder, and see the following output:
PASS
ok github.com/hellonico/jquants-api-go 1.012s
You should also see that the content of the login.edn
file is properly filled:
cat ~/.config/jquants/login.edn
{:mailaddress "youremail@you.com" :password "yourpassword"}
Use the Login to Send an HTTP Request to the jQuants API and Retrieve the RefreshToken
The second function to be tested is TestRefreshToken
, which sends a HTTP post request with the username and password and retrieve the refresh token as an answer of the API call. We update the helper_test.go
file with a new test case:
func TestRefreshToken(t *testing.T) {
token, _ := GetRefreshToken()
fmt.Printf("%s\n", token)
}
The GetRefreshToken
func will:
- Load user stored in file previously and prepare it as JSON data
- Prepare the HTTP request with the URL and the JSON formatted user as body content
- Send the HTTP request
- The API will returns data that will store in a RefreshToken struct
- And let’s store that refresh token as an EDN file
The supporting GetUser
will now load the file content that was written in the step before. We already have the Login
struct, and will then just use edn.Unmarshall()
with the content from the file.
func GetUser() Login {
s, _ := os.ReadFile(getConfigFile("login.edn"))
var user Login
edn.Unmarshal(s, &user)
return user
}
Note, that, while we want to read/write our Login struct to a file in EDN format, we also want to marshal the struct to JSON when sending the HTTP request.
So the metadata on our Login struct needs to be slightly updated:
type Login struct {
UserName string `edn:"mailaddress" json:"mailaddress"`
Password string `edn:"password" json:"password"`
}
We also need a new struct to read the token returned by the API, and we also want to store it as EDN, just like we are doing for the Login
struct:
type RefreshToken struct {
RefreshToken string `edn:"refreshToken" json:"refreshToken"`
}
And now, we have all the bricks to write the GetRefreshToken
function:
func GetRefreshToken() (RefreshToken, error) {
// load user stored in file previously and prepare it as json data
var user = GetUser()
data, err := json.Marshal(user)
// prepare the http request, with the url, and the json formatted user as body content
url := fmt.Sprintf("%s/token/auth_user", BASE_URL)
req, err := http.NewRequest(http.MethodPost, url, bytes.NewReader(data))
// send the request
client := http.Client{}
res, err := client.Do(req)
// the API will returns data that will store in a RefreshToken struct
var rt RefreshToken
json.NewDecoder(res.Body).Decode(&rt)
// and let's store that refresh token as an EDN file
encoded, err := edn.Marshal(&rt)
writeConfigFile(REFRESH_TOKEN_FILE, encoded)
return rt, err
}
Running go test
is a little bit more verbose, because we print the refreshToken to the standard output, but the tests should be passing!
{eyJjdHkiOiJKV1QiLC...}
PASS
ok github.com/hellonico/jquants-api-go 3.231s
Get the ID Token
From the Refresh Token, you can retrieve the IdToken which is the token then used to send requests to the jquants API. This is has almost the same flow as GetRefreshToken
, and to support it we mostly introduce a new struct IdToken
with the necessary metadata to marshal to/from edn/json.
type IdToken struct {
IdToken string `edn:"idToken" json:"idToken"`
}
And the rest of the code this time is:
func GetIdToken() (IdToken, error) {
var token = ReadRefreshToken()
url := fmt.Sprintf("%s/token/auth_refresh?refreshtoken=%s", BASE_URL, token.RefreshToken)
req, err := http.NewRequest(http.MethodPost, url, nil)
client := http.Client{}
res, err := client.Do(req)
var rt IdToken
json.NewDecoder(res.Body).Decode(&rt)
encoded, err := edn.Marshal(&rt)
writeConfigFile(ID_TOKEN_FILE, encoded)
return rt, err
}
Get Daily Quotes
We come to the core of the wrapper code, where we use the IdToken, and request daily quote out of the jquants HTTP API via a HTTP GET request.
The code flow to retrieve the daily quotes is:
- As before, read ID token from the EDN file
- Prepare the target URL with parameters code and dates parameters
- Send the HTTP request using the idToken as a HTTP header
- Parse the result as a daily quotes struct, which is a slice of Quote structs
The test case simply checks on non-nul value returned and prints the quotes for now.
func TestDaily(t *testing.T) {
var quotes = Daily("86970", "", "20220929", "20221003")
if quotes.DailyQuotes == nil {
t.Failed()
}
for _, quote := range quotes.DailyQuotes {
fmt.Printf("%s,%f\n", quote.Date, quote.Close)
}
}
Supporting code for the func Daily
is shown below:
func Daily(code string, date string, from string, to string) DailyQuotes {
// read id token
idtoken := ReadIdToken()
// prepare url with parameters
baseUrl := fmt.Sprintf("%s/prices/daily_quotes?code=%s", BASE_URL, code)
var url string
if from != "" && to != "" {
url = fmt.Sprintf("%s&from=%s&to=%s", baseUrl, from, to)
} else {
url = fmt.Sprintf("%s&date=%s", baseUrl, date)
}
// send the HTTP request using the idToken
res := sendRequest(url, idtoken.IdToken)
// parse the result as daily quotes
var quotes DailyQuotes
err_ := json.NewDecoder(res.Body).Decode("es)
Check(err_)
return quotes
}
Now we need to fill in a few blanks:
- The sendRequest needs a bit more details
- The parsing of DailyQuotes is actually not so straightforward
So, first let’s get the sendRequest func out of the way. It sets a header using http.Header
, and note that you can add as many headers as you want there. Then it sends the HTTP GET request and returns the response as-is.
func sendRequest(url string, idToken string) *http.Response {
req, _ := http.NewRequest(http.MethodGet, url, nil)
req.Header = http.Header{
"Authorization": {"Bearer " + idToken},
}
client := http.Client{}
res, _ := client.Do(req)
return res
}
Now to the parsing of the daily quotes. If you use Goland as your editor, you’ll notice that if you copy-paste a JSON content into your Go file, the editor will ask to convert the JSON to go code directly!
Pretty neat.
type Quote struct {
Code string `json:"Code"`
Close float64 `json:"Close"`
Date JSONTime `json:"Date"`
AdjustmentHigh float64 `json:"AdjustmentHigh"`
Volume float64 `json:"Volume"`
TurnoverValue float64 `json:"TurnoverValue"`
AdjustmentClose float64 `json:"AdjustmentClose"`
AdjustmentLow float64 `json:"AdjustmentLow"`
Low float64 `json:"Low"`
High float64 `json:"High"`
Open float64 `json:"Open"`
AdjustmentOpen float64 `json:"AdjustmentOpen"`
AdjustmentFactor float64 `json:"AdjustmentFactor"`
AdjustmentVolume float64 `json:"AdjustmentVolume"`
}
type DailyQuotes struct {
DailyQuotes []Quote `json:"daily_quotes"`
}
While the defaults are very good, we need to do a bit more tweaking to unmarshal Dates properly. What follows comes from the following post on how to marshal/unmarshal JSON dates.
The JSONTime type will store its internal date as a 64bits integer, and we add the functions to JSONTime to marshall/unmarshall JSONTime. As shown, the time value coming from the JSON content can be either a string or an integer.
type JSONTime int64
// String converts the unix timestamp into a string
func (t JSONTime) String() string {
tm := t.Time()
return fmt.Sprintf("\"%s\"", tm.Format("2006-01-02"))
}
// Time returns a `time.Time` representation of this value.
func (t JSONTime) Time() time.Time {
return time.Unix(int64(t), 0)
}
// UnmarshalJSON will unmarshal both string and int JSON values
func (t *JSONTime) UnmarshalJSON(buf []byte) error {
s := bytes.Trim(buf, `"`)
aa, _ := time.Parse("20060102", string(s))
*t = JSONTime(aa.Unix())
return nil
}
The test case written at first now should pass with go test
.
"2022-09-29",1952.000000
"2022-09-30",1952.500000
"2022-10-03",1946.000000
PASS
ok github.com/hellonico/jquants-api-go 1.883s
Our helper is now ready and we can adding some CI to it.
CircleCI Configuration
The configuration is character to character close to the official CircleCI doc on testing with Golang.
We will just update the Docker image to 1.17
.
version: 2.1
jobs:
build:
working_directory: ~/repo
docker:
- image: cimg/go:1.17.9
steps:
- checkout
- restore_cache:
keys:
- go-mod-v4-{{ checksum "go.sum" }}
- run:
name: Install Dependencies
command: go get ./...
- save_cache:
key: go-mod-v4-{{ checksum "go.sum" }}
paths:
- "/go/pkg/mod"
- run: go test -v
Now we are ready to set up the project on CircleCI:
The required parameters USERNAME and PASSWORD in our helper_test.go can be set up directly from the Environment Variables settings of the CircleCI project:
Any commit on the main branch will trigger the CircleCI build (or you can manually trigger it of course) and if you’re all good, you should see the success steps:
Our wrapper is well-tested. Let’s start publishing it.
Publishing the Library on GitHub
Providing our go.mod file has the content below:
module github.com/hellonico/jquants-api-go
go 1.17
require olympos.io/encoding/edn v0.0.0-20201019073823-d3554ca0b0a3
The best way to publish the code is to use git tags. So let’s create a git tag and push it to GitHub with:
git tag v0.6.0
git push --tags
Now, a separate project can depend on our library by using it in their go.mod
.
require github.com/hellonico/jquants-api-go v0.6.0
Using the Library from an External Program
Our simplistic program will parse parameters using the flag module, and then call the different functions just like it was done in the test cases for our wrapper.
package main
import (
"flag"
"fmt"
jquants "github.com/hellonico/jquants-api-go"
)
func main() {
code := flag.String("code", "86970", "Company Code")
date := flag.String("date", "20220930", "Date of the quote")
from := flag.String("from", "", "Start Date for date range")
to := flag.String("to", "", "End Date for date range")
refreshToken := flag.Bool("refresh", false, "refresh RefreshToken")
refreshId := flag.Bool("id", false, "refresh IdToken")
flag.Parse()
if *refreshToken {
jquants.GetRefreshToken()
}
if *refreshId {
jquants.GetIdToken()
}
var quotes = jquants.Daily(*code, *date, *from, *to)
fmt.Printf("[%d] Daily Quotes for %s \n", len(quotes.DailyQuotes), *code)
for _, quote := range quotes.DailyQuotes {
fmt.Printf("%s,%f\n", quote.Date, quote.Close)
}
}
We can create our CLI using go build
.
go build
And the run it with the wanted parameters here:
- Refreshing the ID token
- Refreshing the refresh token
- Getting daily values for entity with code 86970 between 20221005 and 20221010
./jquants-example --id --refresh --from=20221005 --to=20221010 --code=86970
Code: 86970 and Date: 20220930 [From: 20221005 To: 20221010]
[3] Daily Quotes for 86970
"2022-10-05",2016.500000
"2022-10-06",2029.000000
"2022-10-07",1992.500000
Nice work. We will leave it to the user to write the remaining statements
and listedInfo
that are part of the JQuants API but not yet implemented in this wrapper.
Published at DZone with permission of Nicolas Modrzyk. See the original article here.
Opinions expressed by DZone contributors are their own.
Comments