Dev blog 27/07/2018

Last month I promised you a dev blog about jobs and how they work, unfortunately that won't happen. However, that's because jobs are the last major thing remaining before we can launch v3, and jobs are about 80% done.

This dev blog will be more of a rant than anything, as well as including some information about v3 at the end if you're not interested in our struggles.

So what better thing is there to talk about other than how much of a pain the previous week has been?

v3 infrastructure

Our new server infrastructure is quite advanced, and with each layer of complexity comes twice as much suffering. Why do I say this you ask? Well, then you clearly haven't had the joy of working with a Kubernetes cluster yet. I will try to explain.

We opted to use a piece of software called Rancher. Rancher acts as a controlpanel for your Kubernetes clusters, and makes life somewhat easier as you don't have to store and write a whole bunch of yaml files, which mind you, are the literal spawn of satan. I don't know if you tried to set up a Minecraft server as a kid, but those yaml files for any permission plugin always had my 13 year old self ripping my hair out. Of course, I've since understood where my 13 year old self went wrong, and so has every IDE ever. DON'T USE TABS IN YAML!

Enough about yaml, let's talk about the real issues. We started setting up Kubernetes last Saturday, and we've had to overcome a whole different array of hurdles along our way. This was neither of our first times working with Kubernetes, but it has some ways of making you very depressed no matter how many times you use it.

The first problem we ran into was the fact we have a Gitlab CI runner running in our cluster (a cluster is a collection of nodes(a node is a server)), this runner kept eating our CPU to the point where the usage was at 90%+ for the runner alone. You can imagine what happened next. We did not realize the cause of the issue at the time but we quickly learnt that whenever we ran a pipeline some anomalies started happening. First we noticed our web application being insanely slow, followed by it crashing completely. We thought okay, maybe we configured something wrong, but no, we pushed another few commits and a second pipeline started running simultaneously, but this time on the master node. Our entire cluster crashed. We thought that okay, maybe we just need to increase the RAM of our servers, so we did. Upped their RAM from 2GB each to 4GB each. We now had two nodes running etcd and being flagged as a worker, and one node running controlplane, etcd and being flagged as a worker.

Can you guess what happened next? Yeah, this didn't work either. Same problem. We thought that maybe we'll be fine if we don't run the worker on the master node, so we upped the node count to 5 so we could run two masters and three workers, all members of etcd. This still did not work. We moved the gitlab CI runner out of our cluster and it felt like all our problems were gone.

But they weren't. The website was still running insanely slow for a load balanced website running on multiple nodes. We quickly realized that all assets were loading really slowly. The cause of this: a connection to the website went like this: Client -> CloudFlare -> Load Balancer -> Internal k8s NGinx ingress -> Website. That's at least one step too many. The solution: Move all assets to the cloud so that the DOMContentLoaded time would be reduced significantly. Did this work? Sort of. Now we had a new issue! CloudFlare, despite being such a great service decided that it would not purge the cache of a single file for the life of me. I pasted the entire URL to the file as it told me to, with HTTPS and everything. After about an hour of waiting and trying again I tried to replace HTTPS with HTTP in the url as a last attempt. And lo and behold! It worked! Why? I don't know, beats me.

Is that all the problems? Noooooo, of course not! The website was still slower than we would have liked and we pinpointed the persistent storage disks as an issue. Increased the size, now they worked fine. Next up was our PostgreSQL cluster being slower than we would have liked. Solution? Swap from Rancher's inbuilt NGinx ingress to consul. Problem? The helm chart (helm is a repository of automated kubernetes pod/service configurations) that we were using for our PostgreSQL setup does not support consul, and the official helm chart for PostgreSQL didn't work with Rancher. One CLA signed and a pull request later and we now have a nonuple postgres setup running.

The website is now at an acceptable speed with automatic recovery using healthchecks for both liveness and readiness. The database is persistent, is backed up every hour onto a DigitalOcean spaces object storage repository and also has redundancy. Redis has redundancy as well. We can literally shut down one node and and the application will failover and continue working without downtime.

Highly available

All of this considered and it basically means that unless our highly available load balancer dies, we should in theory never have downtime. Of course, if DigitalOcean has network trouble or two nodes go down at once we will have an issue. If a database corruption ever occurs we will have a slight downtime while importing a backup, and some data loss but not an amount we can't deal with.

This are not all of the issues we ran into during the last week, but telling you about any more of them would probably bore you. You're probably not interested in our server infrastructure anyway.

When is v3, for real this time...

So when is v3 actually coming out? You already have the server's set up and everything!? Good question, we're aiming to finish jobs and a few other small things and then launch a beta for content creators to test and make themselves familiar with early the coming week.

The beta is strictly limited to content creators as we still don't want a whole lot of traffic going through the site, and if a serious bug is found it's limited to people who would be otherwise affected by such a bug.

Content curators will get access to v3 to familiarize themselves with the new UI and features before the beta officially starts.

When the public official release will come depends on how well the beta goes. I am going on a holiday starting the 8th and won't be back until the 14th, so if the beta goes badly and I don't get sufficient time to launch v3 it won't happen until after I'm back.

As usual, keep replies and discussions civil.

If you want to keep up with development you may join our discord and keep an eye on the #commits channel.