perjantai 2. kesäkuuta 2017

Trying out Graphql with Golang and Vue.js

I've made normal Rest APIs for many years now. For some time now, there has been interest for Facebooks GraphQL especially on the front-end side. I too have been curious about it but haven't had the change to play with it. So, I made a small project to test it and to test Vue.js too which also is new for me. I have used Angular a bit and more React but Vue.js was something totally new.

GraphQL with Go

There actually is a ready project for using GraphQL so it's quite simple to start using it on the backend side.

I couldn't think of any really good topic to make so I wrote some Star Wars characters into a file and used those.

var personType = graphql.NewObject(
  graphql.ObjectConfig{
    Name:   "person",
    Fields: fields,
  },
)

There is only one type and it's definition is simple enough.

var queryType = graphql.NewObject(graphql.ObjectConfig{
  Name: "RootQuery",
  Fields: graphql.Fields{
    "person": &graphql.Field{
      Type: personType,
      Args: args,
      Resolve: func(p graphql.ResolveParams) (interface{}, error) {
        return filterPerson(data, p.Args), nil
      },
    },
    "personList": &graphql.Field{
      Type: graphql.NewList(personType),
      Resolve: func(p graphql.ResolveParams) (interface{}, error) {
        return data, nil
      },
    },
  },
})

Now, the query type includes two fields to make queries. Person field filters the data with a query parameter from the request. PersonList field just returns the whole data or list of Star Wars characters.

schema, _ = graphql.NewSchema(
  graphql.SchemaConfig{
    Query: queryType,
  },
)

Schema defines the capabilities of the server with queryType.

func main() {
  err := importJSONDataFromFile(jsonDataFile)
  if err != nil {
    fmt.Printf("Error: %s\n", err.Error())
    return
  }

  http.HandleFunc("/graphql", func(w http.ResponseWriter, r *http.Request) {
    result := executeQuery(r.URL.Query()["query"][0], schema)
    w.Header().Add("Access-Control-Allow-Origin", "*")
    json.NewEncoder(w).Encode(result)
  })

  fs := http.FileServer(http.Dir("frontend/dist"))
  http.Handle("/", fs)

  fmt.Println("Now server is running on port 8000")
  http.ListenAndServe(":8000", nil)
}

The main function of the server does four things. First, it gets the Star Wars character json and makes the personType "fields" and queryType "data" available. Second, it sets listening endpoint to /graphql for queries. Third, it defines a file server for static content for root "/" endpoint. And for last, it starts the server.

At this point, qraphql queries already work. If your start the server, and query it for example with:

curl http://localhost:8000/graphql?query={person(name:%22Darth%22){id,name,surname}}

You get an answer:

{
  data: {
    person: {
      id: "1",
      name: "Darth",
      surname: "Vader"
    }
  }
}

Vue.js

To really test what the difference is between using ordinary Rest APIs and GraphQL I set up a small front-end to fetch and show the data.


I first set up the vue.js project under front-end directory, in production it would be delivered from there. In development it can be run separetily with hot reloading and the works.

cd frontend
npm install -g vue-cli
vue init webpack graphql-vue-example
mv graphql-vue-example/* .
rmdir graphql-vue-example
npm install
npm run dev

Now, this sets up an initial project, it needs some work to get all things like we want them. Almost everything that needs to be changed, is at "frontend/src/components/Persons.vue". It contains the html template, vue javascript for the page and scoped styles.

<template>
  <div class="personclass">
    <h1>{{ msg }}</h1>
    <input id="personName" type="text" value="Darth" v-model="personName"></input>
    <button class="btn btn-primary" v-on:click="getPerson()">Get Star wars person</button>
    <h2>One fetched person</h2>
    <div>{{ person.person.id }} {{ person.person.name }} {{ person.person.surname }}</div>
    <h2>List of all</h2>
    <div v-for="value in persons.personList">
      {{ value.id }} {{ value.name }} {{ value.surname }}
    </div>
  </div>
</template>

If you write a name of a character to the input field and then click the button, it goes and fetches the character in question. Of course, only if there is a match. There also is the whole list, which is fetched immediately when the page is rendered.

<script>
export default {
  name: 'personclass',
  data () {
    return {
      msg: 'Star Wars persons',
      person: {person: {id: '', name: '', surname: ''}},
      persons: ''
    }
  },
  mounted () {
    this.getPersons()
  },
  methods: {
    getPerson () {
      console.log(this.personName)
      this.$http.get('http://localhost:8000/graphql?query=' + 
          encodeURIComponent('{person(name:"' + this.personName + '"){id,name,surname}}'))
          .then(response => {
        this.person = JSON.parse(response.body).data
      }, response => {
        console.log(response)
      })
    },
    getPersons () {
      this.$http.get('http://localhost:8000/graphql?query=' + 
          encodeURIComponent('{personList{id,name,surname}}')).then(response => {
        this.persons = JSON.parse(response.body).data
      }, response => {
        console.log(response)
      })
    }
  }
}
</script>

The Vue javascript part is simple enough. There are two methods, getPersons gets all the Star Wars characters and it's called when the page is mounted. Method getPerson is called from the html page when the button is clicked and it returns only the character which is being searched.


The whole example project in github: https://github.com/jelinden/graphql-vue-example

References

https://facebook.github.io/react/blog/2015/05/01/graphql-introduction.html
https://github.com/graphql-go/graphql
https://vuejs.org/


lauantai 29. huhtikuuta 2017

Cache on the background with golang, Part 2

Part one can be found here.

Expiring cache


First, we're still using the var cache cmap.ConcurrentMap
Second, we need a new field to store, the expire information. So, let's save a struct as a value for the cache item.
type CacheItem struct {
    Key    string
    Value  []byte
    Expire time.Time
}
Adding an item to cache is almost the same, let's just take the expiring into account and also the set maximum size of the cache.
func AddItemToCache(key string, value []byte, expire time.Duration) {
    if cache.Count() <= cacheSize {
        cache.Set(key, CacheItem{Key: key, Value: value, Expire: time.Now().Add(expire)})
    } else {
        panic("cacheSize reached")
    }
}
Getting an item from the cache is trivial. Again, let's take the expired items into account by removing them.
func GetItemFromCache(key string) *CacheItem {
    if cache.Has(key) {
        if tmp, ok := cache.Get(key); ok {
            item := tmp.(CacheItem)
            if item.Expire.After(time.Now()) {
                return &item
            }
            removeItem(item.Key)
        }
    }
    return nil
}

Usages of an expiring cache

You can fill the cache with a scheduler on the background, or by filling the cache when you get a nil from a get. You can also take this further by fetching the almost expired item on the background and still giving the item from the cache. This way there is not so much cache misses when there is less traffic.
cacheItem := GetItemFromCache(weatherURLs[1])
if cacheItem != nil {
    w.Write(cacheItem.Value)
} else {
    if weather := fetchWeather(weatherURLs[1]); weather != nil {
        j, _ := json.Marshal(&weather.Query.Results.Channel.Item)
        AddItemToCache(weatherURLs[1], j, time.Minute)
        w.Write(j)
    } else {
        json.NewEncoder(w).Encode("not found")
    }
}

Removing stale items

In addition to removing stale items when bumping into them while getting an item from cache, we can remove stale items by running
func removeStaleCacheItems() {
    for _, item := range cache.Items() {
        cItem := item.(CacheItem)
        if time.Now().After(cItem.Expire) {
            removeItem(cItem.Key)
        }
    }
}
once in a minute for example. We loop all items and check if the current time is after expiration time. This is started as a background job in the init function.
func init() {
    go doEvery(time.Minute, removeStaleCacheItems)
}

func doEvery(d time.Duration, f func()) {
    for range time.Tick(d) {
        f()
    }
}

Testing

Testing is otherwise about the same, but we have the new expiring test. Easy enough, just give a little time for an added item and then sleep a bit more than that. When getting the first added item after expiring, you get nil back as supposed to.
func TestExpire(t *testing.T) {
    assert := assert.New(t)
    AddItemToCache("item", []byte("value"), time.Second)
    assert.Equal(string(GetItemFromCache("item").Value), "value", "should be equal")
    time.Sleep(time.Second * 2)
    assert.True(GetItemFromCache("item") == nil, "should be equal")
}

The whole source code can be found from github.
 

maanantai 11. heinäkuuta 2016

Monitor your applications and servers with Telegraf, InfluxDB and Grafana

Grafana with graphs from cpu, memory, mongodb

Telegraf

I stumbled upon a nice go project Telegraf which saves metrics from various sources to InfluxDB among others. I already had InfluxDB running so I gave it a shot.

There are two servers, Babylon, the internet facing and an other one named Black Pearl. Latter one is the Kubernetes master for a Raspberry Pi cluster. Installing Telegraf is easy enough.

Installation instructions from packages: https://github.com/influxdata/telegraf

I want to install this on docker images too so I did it like this:
sudo mkdir -p /work
cd /work
sudo curl -LO https://dl.influxdata.com/telegraf/releases/telegraf-1.0.0-beta2_linux_armhf.tar.gz
sudo tar xvzf telegraf-1.0.0-beta2_linux_armhf.tar.gz
cd telegraf
sudo cp etc/logrotate.d/* /etc/logrotate.d/
sudo mkdir -p /var/log/telegraf
sudo cp usr/bin/telegraf /usr/bin
sudo bash -c "telegraf -sample-config -input-filter cpu:mem:net:swap:mongodb -output-filter influxdb > 
/etc/telegraf/telegraf.conf"
sudo mkdir -p /etc/telegraf
sudo bash -c "nohup telegraf > /var/log/telegraf/telegraf.log 2>1 &"

On the raspberry pi, I also had to change the config line a bit,
cpu:mem:net:swap:mongodb -> cpu:mem:net:swap
since there is no mongodb on that instance.

Grafana

Grafana is a tool for making graph dashboards. Installing Grafana is easy. Adding a datasource is done straight from the UI, no need for config handling.
Adding a graph to a dashboard is easy too.
The main tab adding the graph is Metrics. You add the query there.

With Telegraf it's easy to get all metrics to a database and from there on using Grafana is a breeze.

Adding InfluxDB datasource to Grafana: http://docs.grafana.org/datasources/influxdb/
Telegraf: https://github.com/influxdata/telegraf
InfluxDB: https://influxdata.com/time-series-platform/influxdb/

sunnuntai 19. kesäkuuta 2016

Cache on the background with golang

Caching based on requests

Usually caches on the web are made so that the cache is filled with requests coming in from the browser. Positive about this is that you cache the most used requests and not the ones which are not used. The downside is that in case of errors or timeouts, you really have to think what to do. And what about expiration and the size of the cache?

Caching on the background

If you know what to fetch and you know the requests what to get before hand, you can easily schedule the cache fetching. Although I'm not using a popular cache like LRU here, it's certainly possible.

A simple example

This is a simple example for a cache. You can get, add and remove items from the cache.
package main

import (
    "github.com/streamrail/concurrent-map"
)

var cache cmap.ConcurrentMap

func init() {
    cache = cmap.New()
}

func AddItemToCache(key string, value string) {
    cache.Set(key, value)
}

func GetItemFromCache(key string) string {
    if cache.Has(key) {
        if tmp, ok := cache.Get(key); ok {
            return tmp.(string)
        }
    }
    return ""
}

func removeItem(key string) {
    cache.Remove(key)
}

Let’s get weather information on the background.
func fetchWeather() *Weather {
    var w Weather
    resp, err := client.Get(weatherURL)
    if err != nil {
        fmt.Println(err)
        return nil
    }
    defer resp.Body.Close()
    buf, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        fmt.Println(err)
        return nil
    }
    if err = json.Unmarshal(buf, &w); err != nil {
        fmt.Println(err)
    }
    return &w
}

Marshal it to json string
func weatherToCache() {
    fmt.Println("fetching weather")
    if w := fetchWeather(); w != nil {
        j, err := json.Marshal(&w.Query.Results.Channel.Item)
        if err != nil {
            fmt.Println(err)
        } else {
            AddItemToCache(weatherURL, string(j))
        }
    }
}

and schedule fetching it
func weatherSchedule() {
    for range time.Tick(time.Hour * 1) {
        weatherToCache()
    }
}

In main function We use iris to get the cached weather information after we put the weather scheduling to run on the background.
func main() {
    weatherToCache()
    go weatherSchedule()
    iris.Get("/weather", func(c *iris.Context) {
        c.Text(200, GetItemFromCache(weatherURL))
    })
    iris.Listen(":8080")
}

Used or mentioned dependencies
iris-go.com
Concurrent map
LRU cache

See the whole project at https://github.com/jelinden/go-background-cache-example

sunnuntai 22. toukokuuta 2016

Testing with Golang

Coming from a Java background, testing with Go has been a struggle from the beginning for me. For simple unit testing, there are simple enough guides for doing it, but what about the more complex cases? What if I use a library like Echo, should I test what a middleware function returns? How do I mock data needed by the function?

And there’s of course the simplicity of Go, everything is so small and nice, should I even test the smallest functions? I’ll share some of my experiences and thoughts about testing in Go.




Unit testing

Here's a function which drops everything that is not a letter or a number:
func validate(param string) string {
 r, err := regexp.Compile("[a-zåäöA-ZÅÄÖ0-9]+")
 if err != nil {
  fmt.Fatal(err)
 }
 return string(r.ReplaceAll([]byte(param), []byte(""))[:])
}

Testing this is easy enough:
func TestValidateSuccess(t *testing.T) {
    value := validate("foo bar ! 1 2[]()><")
    if value != "foobar12" {
        t.Errorf("Validate result was not 'foobar'")
    }
}

One thing to notice is that if you make all functions to return values, it's really easy to test. If you modify values without returning them, testing is a bit harder job to do (At least for me).

There are extensions to testing package also. For example testify (https://github.com/stretchr/testify).
With it you can easily do assertions with a nice syntax. It also supports mocking.
func TestValidateSuccess(t *testing.T) {
    assert := assert.New(t)
    assert.Equal(validate("foo bar ! 1 2[]()><"), "foobar12", "should be equal")
}

That was a test for a sunny day, how about testing for panics?
Say I have a book shelf which can hold certain amount of books.
type shelf struct {
 books []book
}

type book struct {
 name string
}

var myShelf = shelf{books: make([]book, 5)}

Ok, I now have a shelf which has room for five books. I also need a function to add books to the shelf:
func addBookToShelf(book book, i int) {
 myShelf.books[i] = book
}

These are the books I want to put on the shelf, you might see the problem already.
var conanDoyle = []string{
 "A Study in Scarlet",
 "The Sign of Four",
 "The Adventures of Sherlock Holmes",
 "The Memoirs of Sherlock Holmes",
 "The Hound of the Baskervilles",
 "The Return of Sherlock Holmes",
 "The Valley of Fear",
}

The loop to add the books with addBookToShelf function:
for i, b := range conanDoyle {
 addBookToShelf(book{name: b}, i)
}

This panics a runtime error for index out of range. How can I test this?

By searching the panic for example
func TestAddBooksToShelf(t *testing.T) {
    defer func() {
        if r := recover(); r != nil {
            t.Errorf("handleBooks() panicked, shouldn't have")
        }
    }()
    handleBooks()
}

This is how you can catch panics in a test and not in production, it is also useful in cases you really want to panic.

Smoke or sanity testing

This is usually done right before going to production. It might be all your other tests which are run when you're making for example your Docker image, but if you have lot's of slow tests, this is then usually not the case.

It might be a shell script with a set of curl requests to check that the most important 
endpoints are answering. Or it might be a test like this, you could easily add more checking:
func TestGetUrls(t *testing.T) {
 for i := 0; i < 5; i++ {
  go makeHttpGet(t, "http://localhost:2000/endpoint1")
  go makeHttpGet(t, "http://localhost:2000/endpoint2")
  go makeHttpGet(t, "http://localhost:2000/endpoint3")
 }
}

func makeHttpGet(t *testing.T, url string) {
 client := &http.Client{}
 req, _ := http.NewRequest("GET", url, nil)
 res, _ := client.Do(req)
 data, err := ioutil.ReadAll(res.Body)
 if err != nil {
  t.Error(err)
 }
 res.Body.Close()

  body := string(data)
 if res.StatusCode != http.StatusOK {
  t.Fatalf("Status code was wrong: %v\n %v\n %v\n", res.StatusCode, url, body)
 }
}

Load testing

Load testing is usually related to a website you made and you want to
see how fast it delivers pages.

There are lot's of ready made made tools, you don't have to make them your self.

https://github.com/lubia/sniper
https://github.com/tsenart/vegeta
https://github.com/wg/wrk
http://gatling.io/

And you can profile your program easily enough too to see where the time is going.

http://dave.cheney.net/2013/07/07/introducing-profile-super-simple-profiling-for-go-programs

Race detecting

You can also detect race conditions in Go, there is a tool for that. Consider following code:
var test = make(map[int]int)

func main() {
 var i = 0
 for i < 10000 {
  go setTest(i, i)
  fmt.Println(test[i])
  i++
 }
}

func setTest(name int, val int) {
 test[name] = val
}

You can test it with: go run -race loop.go

And you get a fatal error: concurrent map writes. This is really useful if you run your program asynchronously with goroutines.


Conclusion

I hope I had known more about testing with Go from the beginning. It's so easy to just start writing working code with Go that testing doesn't seem to be relevant. Until you make enough code to notice that everything you write doesn't actually work the way you wanted it to. Refactoring and fixing code is such a bliss when you have the tests to back it up.

Oh, I have to add this too to think about, unit tests are useful, but they are not enough.



lauantai 19. maaliskuuta 2016

Saving page loads to InfluxDB through NATS and showing it as a graph with c3.js

I wanted to try out updating graphs in real time, and at the same time try something new. I’ve heard good things about Nats and InfluxDB so these were my preferred choises. I already have a web site www.uutispuro.fi, which I'm also using.

Stack used

  • c3.js - made on top of d3.js, a javascript library for graphs
  • Nats - A fast message broker
  • InfluxDB - Time series database

Installing

First installing Nats and InfluxDB were really easy. Download Nats binary and run it. For InfluxDB there are apt-get installation instructions at https://influxdata.com/downloads.

The code

To get the data to the database, I first needed to send the messages for each page load to Nats. The client is easy to use. For data to be saved I chose the ip address of the server so that I can make own graphs for each of them in the future. URI is the path which was asked, this can be used to show the pages which are read the most. User agent is also an interesting field of information. Time is used for the time based query from the database.

uri := "\"uri\":\"" + c.Request().URI() + "\""
userAgent := "\"userAgent\":\"" + c.Request().Header().Get("user-agent") + "\""
localIP := "\"localIP\":\"" + nats.LocalIP + "\""
time := "\"time\":\"" + time.Now().UTC().String() + "\""
nats.NatsConn.Publish("click", []byte("{"+time+", "+uri+", "+userAgent+", "+localIP+"}"))
if err := next.Handle(c); err != nil {
  c.Error(err)
}

And then in plotter project, I need to listen to Nats.

func GetClicks() {
  NewInflux()
  nc, err := nats.Connect("nats://192.168.0.5:4222", 
    nats.MaxReconnects(60), 
    nats.ReconnectWait(2*time.Second))
  if err != nil {
      fmt.Println(err)
    return
  }
  defer nc.Close()
  ch := make(chan *nats.Msg, 128)
  _, err = nc.ChanSubscribe("click", ch)
  if err != nil {
    fmt.Println(err)
    return
  }

  for {
    msg := <-ch
    var data map[string]interface{}
    if err := json.Unmarshal(msg.Data, &data); err != nil {
      panic(err)
    }
    Save(data)
  }
}

And save the data to the database.

func Save(click map[string]interface{}) {
  bp, err := client.NewBatchPoints(client.BatchPointsConfig{
    Database:  influx.DbName,
    Precision: "s",
  })
  if err != nil {
    log.Println(err)
  }

  stamp, _ := time.Parse("2006-01-02 15:04:05 -0700 MST", click["time"].(string))
  pt, err := client.NewPoint("click", map[string]string{}, click, stamp)
  if err != nil {
    log.Println(err)
  }
  bp.AddPoint(pt)
  err = influx.Conn.Write(bp)
  if err != nil {
    log.Println(err)
  }
}

Now to the exiting part. We have the data in the database and it's easy to fetch it from there.

func GetList() map[string]int64 {
  q := client.NewQuery("select count(localIP) from click 
      where time > now() - 1h 
      group by time(1m) fill(0)",
      "plotter", "s")
  var fields = make(map[string]int64)
  response, err := influx.Conn.Query(q)
  if err != nil || response.Error() != nil {
    fmt.Println(err)
  } else {
    for _, result := range response.Results {
      for _, value := range result.Series[0].Values {
        id := strconv.FormatInt(convertType(value[0]), 10)
        val := convertType(value[1])
        fields[id] = val
      }
    }
  }
  return fields
}

What do we do with the data? We'll send it with websockets to the browser every second.

e.Get("/ws", standard.WrapHandler(websocket.Handler(func(ws *websocket.Conn) {
  for range time.Tick(time.Second) {
    clicks, err := json.Marshal(app.GetList())
      if err != nil {
        fmt.Println(err)
      }
      err = websocket.Message.Send(ws, string(clicks))
      if err != nil {
        fmt.Printf("client closed connection %s\n", err)
        break
      }
    }
})))

In the browser we take a websocket connection and listen.

window.onload = function() {
  writeState("document loaded");
  if ('WebSocket' in window) {
    var connection = new WebSocket('ws://uutispuro.fi:7000/ws');
    connection.onopen = function() {
      writeState("connection open");
    }
    connection.onclose = function() {
      writeState("connection closed");
    }
    connection.onerror = function(error) {
      writeState("error: " + error);
    }
    connection.onmessage = function(e) {
      var msg = JSON.parse(e.data);
      writeState("got a message");
      connection.send("ack");
      arr = ['per minute'];
      Object.keys(msg).map(function(key, i) {
        if (i == 0) {
          arr.push(0);
        } else {
          arr.push(msg[key]);
        }
      });
      chart.load({
        columns: [
          arr
        ]
      });
    }
  } else {
    writeState("You're browser does not support websockets.");
  }
}

And draw the graph with the data sent from the server.

var chart = c3.generate({
  data: {
    columns: [
      ['per minute', 0],
    ],
    type: 'bar'
  },
  bar: {
    width: {
      ratio: 1.0
    }
  },
  axis: {
    x: {
      label: 'minutes',
      inverted: true
    },
    y: {
      label: 'page loads'
    }
  }
});

Links

sunnuntai 6. maaliskuuta 2016

Simple Go balancer for Kubernetes Raspberry Pi cluster

I made a really simple load balancer/proxy in front of Kubernetes to share traffic between Raspberry Pi's. There are many of these available and a lot better ones too, but I wanted to try out myself what is required to make a load balancer. Previous post was about setting up the cluster.

The load balancer has only two methods, one for socket.io and one for everything else.

http.HandleFunc("/socket.io/", websocketProxy)
http.HandleFunc("/", proxy)

Optimally there would only be one method to handle both, but Socket.io with websocket upgrades needs something more. Also I would have liked to log more about Socket.io traffic, but don't know how (Is it even possible?).

Sharing traffic between addresses is done randomly

func random(min, max int) int {
  return rand.Intn(max-min) + min
}
and usage:
urls[random(0, 3)]

It's in use now, but far from ready. First of all, I'd like to get the addresses from Kubernetes automatically. For now, it's relying on an environment variable:

export BALANCED_URLS=http://[192.168.0.21]:1300,http://[192.168.0.22]:1300,http://[192.168.0.23]:1300

Secondly if and when one of the endpoints is down, balancer should not try to use that endpoint at all. I also don't have any idea how much traffic it will handle, that would be nice to know :)

Github: https://github.com/jelinden/go-loadbalancer