[manjaro-general] Moving the forum to Bytemark: two weeks later

[Jonathon Fernyhough]

Just under two weeks ago we moved the Discourse server from [Digital
Ocean](https://www.digitalocean.com/) to
[Bytemark](https://www.bytemark.co.uk/r/manjaro). I thought I should
write up a little post to give a bit of background and a bit more
information about where we are now.

## TL;DR

Server performing better. Discourse performing better. Bytemark pretty cool.

## Enter Discourse

A couple of years back one of the team members, Roland, contacted
Digital Ocean who  provided a generous starting credit which, at the
time, would have provided for several years of forum activity. The
reasons behind Discourse were covered at the time of the switch so I'll
not go into them here.

However, over the following years the small droplet simply wasn't able
to cope with the Discourse requirements - Discourse is designed to scale
to support hundreds of thousands of active users, and with that design
choice comes "stepping" of hardware requirements. That is, performance
will remain consistent for a range of users, then drop off sharply until
more resource is made available to support the next "step" of users.

For the past few months there had been many concerns raised over poor
forum performance, timeouts, and gateway errors ("500 errors").
Essentially, this was because interest in the forum had grown to the
point where we needed to "step up" the available resources.

For us, this meant stepping the server up from a 2GB instance to a 4GB
instance which doubled the server's monthly cost, though it did reduce
the number of 500 errors people were hitting. However, possibly due to
"noisy neighbours" (or the fact we had been running with high disk I/O
for the past year so _we_ were the noisy neighbour...), latency of the
site remained quite high.

## Enter Bytemark

I was made aware of Bytemark several years ago. I work with several
people who know the founders personally, along with other members of
their team. I also kept seeing Bytemark logos on several
high-profile/big name OSS project sites and services, and they provide
mirrors for a good number of Linux distros. It seemed pretty obvious to
me to approach them and ask whether they could help our project too.

After only a couple of emails over a couple of days we had a shiny new
VPS matching the DO droplet, and Phil migrated Discourse over. It wasn't
plain sailing as almost immediately we were hit by a [Discourse
bug](https://forum.manjaro.org/t/forum-500-errors-and-non-loading-topics/30322/)
which basically brought the site to a grinding halt... not a great start. :)

Once this had been corrected, and the server had received a few tweaks
to make Redis happy, not only were we seeing almost no 500 errors (even
when the Yandex spider hits the site) but page latencies were also
vastly reduced.

## An illustration

Linked below is a (poorly-illustrated) view of the last couple of months
of Discourse stats. The top chart shows the total number of views per
day, the bottom chart shows the number of 500 errors per day.

With the 2GB DO droplet, the server registered 500 errors whenever page
views went above about 50,000. When a web spider hit the site (see
spikes) the forum went offline as Discourse restarted parts of its
stack. Much frustration for users.

With the 4GB DO droplet, the number of 500 errors was much lower, but
the web spidering only took the views above 50,000 on three occasions.
The site latency effectively prevented any larger number of views.

With the move to Bytemark, the nominal page view count has increased to
50,000 per day with no corresponding 500 errors, and as others have
commented, the overall responsiveness of the forum is much better than
before. It has also been able to cope with several intensive web
spiderings (spikes at 70k) without so much as a squeak (except one bump
which I think was a full container wipe and rebuild after noticing a
[Docker
issue](https://meta.discourse.org/t/docker-disk-usage-issue-c-sept-1-2017/69194)).

https://forum.manjaro.org/uploads/default/original/3X/f/0/f0eecc6f3d244a242fb526ce3599654fbb39b5f1.png

## Conclusion

The forum is in a much better place than it was two months ago. Not only
is hosting secure for the foreseeable future, but it's performing
better, and we get to work with a pretty awesome company that does a lot
for OSS projects.

I'm looking forward to see what else Bytemark will let us accomplish. :)

## About Bytemark

> Bytemark was founded with a simple mission: reliable, UK hosting.
> Co-founders Matthew Bloch & Peter Taphouse, both engineers by nature
> built the business from the ground up.
>
> Today, they lead a team of 33 staff who operate Bytemark’s own data
> centre in York, monitor its 10Gbps national network and deliver 24/7
> support to clients of all sizes. Brands hosted on Bytemark’s network
> include the Royal College of Art, data.gov.uk and DVLA Auctions.


Originally posted on the forum:
https://forum.manjaro.org/t/moving-the-forum-to-bytemark-two-weeks-later/31508

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.manjaro.org/pipermail/manjaro-general/attachments/20170920/195b47b4/attachment.sig>