gosl-basics/startup-scripts

Installing as FastCGI with systemd

A brief introduction to FastCGI vs. pure standalone server

As explained on the main README, one of the possibilities of this codebase is to run it as a FastCGI process, side-by-side with your web server (I'm using nginxbut of course you can use whatever you wish).

There are a few reasons for doing so, and because they’re not obvious, let me address them before showing.

Typically, Go server applications will run on their own port, as standalone services; this is the most usual form of programming them, and, thanks to the availability of amazing frameworks (extending and/or replacing the Go default one, which is frankly quite good), it’s usually the recommended way of developing servers in Go.

Because, however, such applications will expose a public port that requires punching a hole through firewalls — both on the server as well as on any clients wishing to connect to the service — the most usual method of addressing this is to place a reverse proxy in front of the Go application. This is basically a special configuration of a regular web server (most will also work that way) where all incoming traffic to a specific virtual host (or to a subdirectory of an existing virtual host) is transparently redirected to the Go server application. The major advantage of this approach is that it allows an extra layer of protection — you can add whatever web firewall rules you wish on your existing webserver, such as blocking ranges of IP addresses, for example; or filter out invalid/unallowed calls, using regular expressions; or tie the Go application into a chain of existing security layers. This is not only quite easy to do, but, as a bonus, you can also do something to improve performance: caching. While you can certainly develop your own caching mechanisms in Go, assuming that your application will return HTML, XML, JSON, or any other popular text-based file format (or, alternatively, popular binary formats, such as pictures or videos), the enveloping web server can deal with that content as if it originated from, say, the PHP subsytem (just to give one possibility, of course), and use whatever caching mechanisms are already available. In other words: you can simply ‘drop and forget’ your Go application into your existing toolchain, and let the other components of the Web ecosystem do their work, treating content from Go as content emanating from any other typical web source (from static files to dynamically-generated web pages or images/videos), and acting upon it accordingly. Last but not least, you even get uniform logging that way — even if your Go application uses its own, internal logging mechanism, the enveloping web server will also capture some error logs from the Go application and write them on its own logs, for further processing.

With so many advantages, it’s no wonder that this approach is so popular!

Nevertheless, it has a handful of limitations (or, rather, gotchas).

The first, and perhaps the most obvious one, is that such a setup requires assigning one available port for that application, permanently. Again, this is not so problematic: using the before-mentioned approach, it’s more common to attach the Go server application to an internal address (i.e. a private address that will not be exposed to the Internet-at-large), which actually adds another layer of security: there is no connection from the outside world directly to an application running on a private, internal address, so the Go application is not exposed — only the reverse proxy will be able to contact it and provide a carefully monitored link from the outside world to your Go application.

But what about having two Go applications that are connected to the same port? Well, that will not work. Fortunately, most Go applications will have some way of assigning themselves to a specific port; and if they don’t, but you have access to its source code, you can, in a worst-case scenario, just hard-code a different port. In other words: aye, it might be annoying here and there, but this is an exception that will happen seldom.

Note that there are a few alternatives to using Internet ports, depending on your operating system; you might be able to use Unix sockets instead, and these have no limits, and modern web servers should be able to easily read/write to sockets as well as ports. From the perspective of most of your code, the difference is pretty much hidden under abstraction layers, so this is certainly a valid approach. And another alternative is to set up a different private IP address for the server, and bind to that address instead (Internet ports are defined per IP address, after all).

These approaches are available if you have full access (i.e. root) to the server where you’re running both your Go application as well as the reverse proxy; there might be some limitations or variations depending on what operating system you’re using (as well as what web server you’re using as a reverse proxy), but that’s the only issue.

The problems begin when you do not have full access to your server’s so-called ‘bare metal’, but merely to an abstract container running on top of it — there are a dozen different technologies that enable that, from so-called virtual machines (or virtual private servers), to jails, to shared environments (where each user has a login and a password to a separate area of the server, and access to a few commands, but cannot make any dramatic changes to the underlying system), to Docker containers, to cloud-based servers... these days, there are quite a lot of options for running a Go application, and each has its advantages, its limitations, and a different price tag.

On the main README, I mentioned using Dreamhost’s shared hosting. It’s dirt cheap, and while you don’t get root access, you get a pretty decent ability to tinker with your environment. While their product is mostly targeted to PHP programmers requiring a bit more than simply having some static HTML pages, it technically does not restrict it to PHP. I believe that they don’t offer you a C/C++ compiler, but you can certainly compile things on your own desktop and upload it to your shared environment, and those applications will certainly work (from the shell, that is). Even though Dreamhost does not provide a Go compiler system-wide, you can install it from the official sources, and run it locally. I have done that successfully — it’s far easier than you might think!

Dreamhost, however, is not fond of having people running standalone applications (on publicly-exposed ports), and imposes a certain amount of limitations — e.g. how much memory such applications are allowed to get, how much time they can run continuously, etc. This is to prevent people from launching their own IRC or BitTorrent nodes (or even a Tor node, or a cryptocurrency mining application), or any of the popular online multiplayer first-person-shooters out there which provide you with a download option for running your own server. While such usage is, in general, legitimate, the problem is that such applications may consume a lot of resources — thus preventing other users to get a fair share of CPU and memory, and who might just wish to get fast web hosting (which consumes considerably few resources).

Thus, to prevent abuse, but still to be able to offer some ability to run non-PHP applications on-demand, Dreamhost gives an alternative: use CGI (or, rather, FastCGI).

CGI is the most ancient web protocol that allowed a web server to contact an application (running on the same server), pass it some information (i.e. data in the POST variables), let the application process it, capture the application’s output, and present it to the user. It’s conceptually very simple — you can imagine it as a special connection, where the web server launches your application, and connects the standard input of your application to the web server’s standard output (and vice-versa). In practice, it’s a bit more complex than that, but that’s a very rough idea on how this works — and considering that it was invented a quarter of a century ago, you can imagine that it’s a very robust solution, having been debugged and audited for security flaws by millions of developers over the decades.

This works exceptionally well to conserve system resources because of the way the web works. Conceptually (in the days of HTTP/0.9...), the web protocol is stateless and therefore works in bursts: each connection fired from a client is independent, and requests a single atomic operation (i.e. GETting some data from a URL or POSTing some data). It’s up to the endpoints to figure out how to establish the concept of ‘state’ on top of the stateless protocol. The most popular alternatives are sessions or cookies; before those concepts were invented, it was popular to send a ‘session id’ as part of the URL (this has serious security issues and needed therefore to be replaced by something different), and that would be all.

Putting it in other words: the communication between the web server and a CGI application mimics the very nature of the HTTP protocol: a client sends a request to a web server, and expects a response, upon which it might be able to formulate further requests. Each of those requests is atomic (i.e. either things are successful — the request was either correctly received, processed, and the result returned back to the user), or it failed; there are not supposed to be any ‘intermediate states’ (potentially an infinity of them!) that require tracking of some kind.

(Disclaimer: I used to be a Dreamhost user, for many years, and if I give them as an example, it’s mostly because I know how they used to provide their shared hosting services, what limitations they placed to prevent abuse — ‘resource starvation’ — but also what nifty tools they allowed to run on their shared services.)

While this has many advantages — there is no need for either the web server or the CGI application to keep an elaborate transaction track of what communications have been received between both — it has an obvious drawback: every time you get a new request, your application has to be launched again, and that will be true for all new requests. This is, sadly, unavoidable.

Now, if your application requires a complex setup — imagine that, before being able process requests, it has to flush a database, or go through all records in it... — it might also have a long launching time, and this will happen every time your application gets a new request.

Early CGI programmes were very simple Perl or shell scripts, so they didn’t have a lot of overhead to deal with when launching. Even when things became substantially more complicated, the idea was that you could delegate the bulk of the work to other applications instead. Consider the simple scenario of converting a video online: you could simply use your Perl script to get the contents of the variables submitted by a form, including the attached video, and then pass it for a different application for batch processing in the background. It would be up to the background application to deal with all initialisation procedures.