Maybe Kubernetes will save us

What was going on?

API outages caused the service to be unavailable for long periods. Another issue was that there was little to no observability, meaning I was in the dark while fighting errors and outages. During the week this was going on, I was on break for health reasons, so my input on fixing it was very limited. However, I did draw up a plan for a new way Florality will do our architecture. Before implementing these fixes, I think it may be best that I explain what our old infrastructure was, what our thought process behind it was, and why we had to switch over to Kubernetes.

Our old infrastructure was four virtual machines running Docker; each had one shared SSH key used to log in to the server and automatically update the Docker image for deployments. We chose this method at the beginning for Florality because it let us deploy new features fast and keep updates rolling off the production line at cheetah speed. Two of the virtual machines were dedicated to Web, and the other two were dedicated to API, load-balanced through HAProxy, served to you by Nginx Proxy Manager and Cloudflare.

Unsurprisingly, the method we chose was slow and ineffective and ultimately caused API outages, since our server was running out of resources and the services were competing for them. This caused the APIs to run out of memory and die, but since they weren't managed by anything, they didn't clear memory and just started up again. No, instead, they held onto their old memory and just rebooted the Docker instance.

Combine these issues with no observability, and the only way I could find this out was by manually looking through the logs of the actual virtual machine they were running on and the logs of the Proxmox hypervisor to see why they were crashing, but never what exact module was causing the crash. We also ran into issues with cron jobs because regular Docker Compose servers didn't handle cron jobs well.

What happens when all these issues combine?

When all these issues combine, it creates a shit show so spectacular that it crashes the API and causes production to be out for multiple hours at a time. The lack of observability means it doesn't tell me until it's already too late and affecting users. I tried putting a Band-Aid on it by rebooting the API servers every hour, and that worked for a bit, but then sometimes the main VPS server would run out of CPU or RAM and crash everything.

At that point, I knew we needed to change our architecture to make everything easier to manage. We needed better observability and the ability to automatically provision what we needed...

...We needed Kubernetes.

A wise shaman once told me that we should use Kubernetes for Florality, or I would regret it. Seems like he was right. But I knew that I couldn't rely on the old Kubernetes architecture that we used back in prehistoric times. I had to bring it up to speed; I also had to do some spring cleaning in the codebase to make it easier to find bugs and errors.

The first thing I did was not write a single line of code; instead, I moved files into different folders and updated imports. If anyone is familiar with Shopify, I was turning our existing monolith mono-repo into a modular monolith monorepo. Of course, I didn't move and update every import by hand; that would have taken too long, and time is not my friend. So I wrote some scripts that I could run with Bun to help me move files into the appropriate folders and places.

Once I organized the codebase better, I was ready to add more observability. So I planned 6 Linear Epics and some sub-issues to better understand the scope and stay on track. I started adding Prometheus so I could track the P50's, P95's, and P99's for the different modules that the code was now separated into. Once prometheus was added I could work on the Grafana config to actually see the data that was being stored within Prometheus. Once that was done our first part of Observability was complete. Now it was time for phase 2 Sentry and Posthog.

We use Sentry and Posthog for two different reasons. One is to track telemetry to see what areas's of the UI we would need to improve based on how many users actually uses a feature and what other features don't need our attention as much. This is the role of Posthog. Sentry on the other hand play a more important role within the observability chain. Sentry allows me to get information on why a error happened and the stack trace of the error within Production. Everything, from high latency to a full on site crash is logged and alerted straight back to me with the stack trace allowing me to have an easier time to diagnose the issue and fix it.

Okay, so now that we have finished the trifecta of healthy living. It was time to setup Kubernetes. Yatin the shaman had previously left us some remnants of a Kubernetes delployment, and I happen to already have Kubernetes VM's on Proxmox thanks to Iris. So using the existing Flux and Kustomize architechture I decided to update Iris' deployment architecture along side Florality's. First we had to figure out a way to build the docker images for both the Webapp and the API since flux wont do that for us. Luckily we already had that figured out, I modified our Jenkins file to instead of Deploying to 4 VM's using ssh it would just hand off to Flux.

Now at this point I was running into some issues. I was testing the deployment and it wouldn't load since the API couldn't find some tables. I realized that I forgot to add a way to migrate the database on deployment, so I added it to the Jenkins pipeline. Now that the API was launching and after a few more hours of setting up Network rules within Kubernetes and setting up a web interface for viewing Kubernetes at a glance. It was ready for deployment. So I deployed by changing the HAproxy configuration.

What comes next

This section is regarding potential changes to architechture. What I would like to do next is setup PlanetScale and migrate our database over to PlanetScale. This will allow me to not think too much about maintaining the database and will allow me to focus more on the rewrite and fixing bugs within Florality. I would also like to move our file store off of the main server and off of Minio and into Cloudflare R2 since that will provide us with access to Cloudflare's powerful CDN network allowing for highspeed file serving capabilities. Finally I would like to work more on the mobile app but with the current state of the codebase I cannot promise much at the moment. First I want to make sure our architecture is good and stable before focusing on yet another part of the codebase I will need to watch over and setup the exact same thing I was explaining here.

This is where a fundraiser may come in in the future. In-order to continue development and scale to the place I want this project to be at I may need to raise some capital. With the goal to get help with creating the mobile app so its out by late 2026, early 2027. Please know that I'm trying the best I can but I am a solo developer at the moment with alot of other responsibilities. I love the Florality project and the community built around it and I hope that I can improve the project so that it is the most reliable accessibility tool for you to use.

TL;DR

Florality’s recent API outages were cause by an infrastructure setup that the project has outgrown. There were 4 docker containers on 4 different virtual machine fighting for resources, all while lack of proper monitoring made problems difficult to diagnose before they hit actual users. To improve reliability on Florality we reorganized the code into a modular monolith structure. Implemented Prometheus, Grafana, Sentry and Posthog for error tracking and monitoring. Finally we moved deployments over to a kubernetes-based system using Jenkins, Flux and Kustomize. This allows us to have a better deployment workflow, automated recovery, and support for scheduled jobs.

Next, we may migrate our database to PlanetScale and File storage to R2 cloudflare storage, once the platform is stable, attention can return to mobile app, supported by potential future fundraising and also outside help with development.

Have a great rest of your day - JSL