Hardware Failure and Upgrades

Disk shuffles where it matters

Another week, more updates. We hope you enjoy!

Ensuring Non Public Infrastructure is 100%

Over the last few days, we have been hard at work ensuring our non public infrastructure is running the best it can. Specifically, our storage.

We run a NAS for our build infrastructure, our backups, and some other team member services. Over the last month or so, it was clear it needed an upgrade and some optimizations. Our original NAS was deployed on an Orange Pi Zero (512mb RAM) and an Orange Pi NAS board (mSATA -> USB adapter board). During the last month or so the board was running at full RAM and swap utilization. Our backups, sync processes, and Docker Hub mirroring were causing significant resource utilization on the board with intermittent hiccups. This wasn’t a “panic mode!” problem, but certainly concerning and important to address.

Filesystem Failure

We had intended to start small by migrating the NAS filesystem to BTRFS, but ultimately decided to upgrade the whole Lollipop NAS over the last few days. During the BTRFS migration, we suffered a full filesystem failure that forced us to restore from backups. Since we were restoring from backups, we felt it best to perform a full upgrade as part of the restore process.

Our NAS is now an O-Droid HC2 with an MSATA -> SATA disk adapter and BTRFS on the main storage. The upgrade/migration is complete and running exceptionally well. Our RAM usage is better and disk operations are significantly faster.

O-Droid HC2

On top of that, the O-Droid HC2 is the board we recommend for Lollipop deployments that need a lot of disk performance (NAS for example). Our NAS is the same set up we recommend to our users, and we are beyond happy with the results. It can handle multiple concurrent borg backups, rclones, and deduplication all at the same time. for such a small SBC, this is amazing.

The O-Droid HC2 continues to impress us and remain our primary recommendation for those in need of a lot of disk performance.

Armbian Build Lollipop

The Lollipop that builds our Armbian images also received a disk upgrade. Originally, we were using a Sandisk USB disk for our Armbian builds for storage. The disk has a habit of getting very warm during heavy, sustained use (more than 60 minutes of non-stop use), causing a very large drop in performance. It was also too small to allow us to keep build artifacts and other build caching on the Lollipop. The build cache can speed up Armbian builds by 10x or more. Between the write performance being slow, our inability to keep a build cache, and the write performance being slow, we opted to upgrade the Armbian Build Lollipop.

As of this writing, the Armbian Build Lollipop has a Samsung SSD with BTRFS as the filesystem. In our tests, we are seeing about a 10x improvement in build speeds as well as improved storage use. We can now keep our build cache between builds, further speeding things up. To say we are seeing a major improvement is understatement. We are very happy with the results.

For the curious: we have a 250GB Samsung 960 EVO mSATA disk connected to our NAS and a Samsung 850 EVO connected to our Armbian build server.

Our Commitment to You, Our Users

We are very focused on this project and truly believe it can be a 10+ year project for us. These upgrades and hardware changes facilitate the longevity of the Lollipop Cloud. We do the hard work for you. If it wasn’t for this blog post and our posts on the Activity Pub network (Mastodon/Pleroma), the changes would likely have gone unnoticed. We are focused on ensuring failures, upgrades, and hardware shuffles have minimal impact on our users and preferably go unnoticed.

We hope you didn’t notice these changes until we called attention to them, but we called attention to them because we strongly believe in transparency.

We look forward to the next week and the future. We will be back with more updates.

Thank you for your interest, questions, and support!