How to Use Netlify to Deploy a Free Go Web Application

Friday, March 6, 2020

Netlify is an web host that specializes in hosting static files. That makes it ideal for hosting a developer blog, a brochure site, or even just a one-off joke. It even has built-in support for Hugo. But Netlify also has various solutions for dynamic hosting, and their “Functions” service turns out to be a very easy way to host a Go web application, often for free. In this post, I will walk through a demo repo I’ve made that shows how to do this.

Netlify’s documentation is pretty good, so I definitely recommend to consult them for more in-depth information, but here I would like to walk through a basic scenario for why you might want a Go backend service and how to set it up. You can see the final version of all of the code on Github and a live demo on Netlify.

Let’s say you have a static HTML webpage, but you would like to have a dynamically populated news box on the page. For example, I have made webpages that were themselves static but pulled in related links from a sister site using JavaScript. This is a good way to have a long cache time for your page without sacrificing the ability to include some other content. I have used services for this before like Yahoo Query Language—which was deprecated and stopped working—and rss2json.com—which worked fine itself but got cut off by an anti-GDPR EU block of the feed I was requesting. Ideally, we could host this ourselves but without all the hassle entailed by deploying a scraper that periodically dumps the links into a database.

For the sake of example, let’s use Hacker News’s RSS feed, which is at https://news.ycombinator.com/rss. Unfortunately, this RSS feed does not have an Access-Control-Allow-Origin: * header, so it is not possible to fetch it via JavaScript and display it from another website due to cross-origin resource sharing rules. In addition, RSS is a form of XML, and handling XML is a bit verbose, so it would be nice to have a JSON Feed of the content instead of RSS. Our goal then is to have a backend fetch the RSS feed we need and return a JSON Feed from the same origin as our website.

Go makes it easy to write a service to turn a URL with XML in it into JSON, and as luck would have it, I have already written a module that uses the standard library http.Handler interface which will do this for us.

So, let’s write a quick Go application that uses the feed2json module to create an HTTP server:

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"

    "github.com/carlmjohnson/feed2json"
)

func main() {
    port := flag.Int("port", -1, "specify a port")
    flag.Parse()
    http.Handle("/api/feed", feed2json.Handler(
        feed2json.StaticURLInjector("https://news.ycombinator.com/rss"), nil, nil, nil))
    log.Fatal(http.ListenAndServe(":"+*port, nil))
}

If we run this from the command line with go run myfile.go -port 8000, the Go standard flag package will parse -port 8000 for us and start a server that we call open at http://localhost:800/api/feed.

When we look at the docs for Netlify, we find our first challenge is that they use an AWS Lambda, rather than a normal HTTP service, so we’ll either need to rewrite our service to use Lambda or find an adapter. It is my belief that it’s better to avoid tightly coupling your architecture to a particular cloud provider in ways that make it difficult to ever migrate off of a platform, so we don’t want to write more AWS-specific code than necessary. Fortunately, adapters between AWS Lambda and the Go standard http package already exist. They take the JSON objects that AWS Lambda functions consume and emit and adapt them into normal Go http.Handler calls. Gateway (which I forked from Apex) is easy to use. They also make it easy to try out our code in local development without spinning up an AWS Lambda simulator of some kind.

package main

import (
    "flag"
    "fmt"
    "log"
    "net/http"

    "github.com/apex/gateway"
    "github.com/carlmjohnson/feed2json"
)

func main() {
    port := flag.Int("port", -1, "specify a port to use http rather than AWS Lambda")
    flag.Parse()
    listener := gateway.ListenAndServe
    portStr := "n/a"
    if *port != -1 {
        portStr = fmt.Sprintf(":%d", *port)
        listener = http.ListenAndServe
        http.Handle("/", http.FileServer(http.Dir("./public")))
    }

    http.Handle("/api/feed", feed2json.Handler(
        feed2json.StaticURLInjector("https://news.ycombinator.com/rss"), nil, nil, nil))
    log.Fatal(listener(portStr, nil))
}

With this version of the code, if we run go run main.go -port 8000, the server will start on http://localhost:8000, but if we omit a port, it will start in AWS Lambda mode. It also runs a file server in HTTP mode, so we can preview the static files that we’re working on in our browser as we’re developing.

Next, let’s add a static webpage with some basic JavaScript to display the feed:

<!DOCTYPE html>
<html>
<head>
  <title>Demo document</title>
</head>
<body>
  <h1>Demo of feed2json</h1>
  <ul id="contents"></ul>
  <script type="module">
    let options = {
      weekday: "short",
      year: "numeric",
      month: "short",
      day: "numeric",
      hour: "numeric",
      dayPeriod: "short"
    };

    const toDate = new Intl.DateTimeFormat("default", options);

    (async () => {
      let data = await fetch("/api/feed").then(rsp => rsp.json());
      let listEls = [];
      for (let item of data.items) {
        let row = document.createElement("li");
        let link = document.createElement("a");
        link.href = item.url;
        link.innerText = item.title;
        row.append(link);
        let d = new Date(item.date_published);
        row.innerHTML += " " + toDate.format(d);
        listEls.push(row);
      }
      let el = document.getElementById("contents");
      el.append(...listEls);
    })();
  </script>
</body>
</html>

Once we have the code working in development, it’s time to tell Netlify how to use it. (For simplicity, I am going to assume we already have an account with our repository connected to Netlify.) We can tell Netlify how to use our static files and Go function by creating a simple build command and a netlify.toml config file.

We can add a one line build.sh that tells Go to install our binaries in the “functions” sub-directory.

GOBIN=$(pwd)/functions go install ./...

Next our netlify.toml file specifies that Netlify should use that script to build our functions and look from them in the functions subdirectory.

[build]
  command = "./build.sh"
  functions = "functions"
  publish = "public"

[build.environment]
  GO_IMPORT_PATH = "github.com/carlmjohnson/netlify-go-function-demo"
  GO111MODULE = "on"

Since we’re using Go modules for dependency management in this project, we also need to tell Netlify to set GO111MODULE to on and give Netlify an import path with GO_IMPORT_PATH to consider as the starting point when building Go files. As of the time of publication, Go 1.14 has been released, but Netlify defaults to Go 1.12. We don’t need to change Go versions for this project, but if we did, we could add a .go-version file into our project root to specify our desired Go version.

The next hurdle to consider is that Netlify makes all of its functions available at /.netlify/functions/binary-name. This isn’t the worst thing—we could just re-write our JavaScript to use their URL conventions—but again, it would be nice to avoid unnecessary tight coupling to a vendor if possible.

Netlify has a simple solution for us, which is their URL rewriting option. (Of course, we could have just used URL rewriting to solve the CORS issue with Hacker News and parsed the XML in JavaScript, but that wouldn’t be very fun.) We can just add this to the end of our netlify.toml:

[[redirects]]
  from = "/api/*"
  to = "/.netlify/functions/gateway/:splat"
  status = 200

Now, any request to /api/* will be sent to our binary, which is called “gateway”. status = 200 means that this should be done as a server-side rewrite, so there won’t be any client redirects.

Netlify gives us 125,000 free function invocations per month. That’s enough for approximately one request every three minutes. As a final service enhancement, it would be nice to ensure we don’t use more function calls than intended by caching our responses. Fortunately, Netlify respects the standard Cache-Control headers. By adding, w.Header().Set("Cache-Control", "public, max-age=300"), we can tell Netlify’s CDN to only make a request to our function once every 5 minutes (300 seconds) and share that same response among all users of our site.

Again, here are the Github code repo and a live demo on Netlify.

Netlify is definitely not the only way to host a Go service (to give the most obvious example, you could just deploy to AWS Lambda yourself), but it’s one of the easiest. With their built-in Github and Gitlab integrations, you can just point Netlify at a repo and it will automatically deploy on push to master and create deploy previews for Pull Requests for you. If you just want to host something quickly without spending a few days on initial setup of environmental variable secret sharing, Content Delivery Network caching, continuous delivery, etc., etc., it’s hard to beat Netlify.

The Ethically-Trained Programmer