24 Hour Status Report

What a RUSH

What a RUSH

The last 24 hours have been fraught with stress and amazing feedback from people all over.


The feedback, positive vibes and commentary have been very helpful for getting through some hiccups the last 24 hours brought to bear.

Server Crash

Within 4 hours of going live we suffered a server crash. One of our services had a RAM spike that crashed the server. Our primary admin was napping when the outage occurred. They rectified the problem immediately upon waking up.

We are keeping a tight eye on the server and are investigating mitigation strategies for the future.

Transparency Report

In the interest of being good internet citizens we’ve published a transparency report and enabled public access to our munin logs. You can find the full report here (link).

The report outlines our infrastructure, where we have our code and a few other details. We will be keeping it up to date as we deploy services, update infrastructure and more.

No Secrets

Self Hosted Chat Progress

We deployed Matrix and XMPP as our primary chat systems (with bridging between them even!). We’ve been hard at work the last 24 hours to improve self-hosting of both using our project.

We’re far from done. The updates over the last 24 hours were meant to help our admin team manage our offical accounts more easily while setting a good foundation for others.


KemoNine managed to get a self-hosted XMPP browser client deployed today. The rough code and notes are in our Gitea instance here (link). The setup uses converse.js (link) served via Caddy. We will be cleaning up the code and details soon. If you’re used to working with a command line on Linux the deployment should be pretty straight forward.

Please note : BOSH is a bit tricky and may need some additional parameters in prosody.cfg.lua. The main prosody website has docs that can help. Also watch out for CORS issues if you deploy converse.js or BOSH on a sub-domain.


KemoNine managed to get a self-hosted riot.im (link) instance deployed today.. The code is very rough and difficult to follow. We recommend skipping this setup for the time being. The code is here (link) for the curious. This should be a good foundation for the future.

Next Steps

At this point we feel the fundamental infrastructure and services are online and stable enough for us to get back to code and documenation.

Our primary goals for the near term are

  • Adoption of a Code of Conduct
    • For now : Don’t 💩 in the pool. Consider yourself warned.
    • We are investigating different CoC’s adopted by large projects and will be making an announcement once we have a final CoC drafed and published on our Gitea instance.
    • You can track our progress here (link)
  • Cleaning up the primary documentation
    • The fork from GitLab to our own infrastructure has caused a bit of a mess and we intend to clean it up
    • Add additional documentation on how to get started with a Lollipop Cloud deployment.
  • Letting KemoNine recharge a little. 3 months later and they deserve a mild break 😉