RE: Announcing the @gridcoin.science Witness - STEM on STEEM

No worries; your skepticism is valid.

High availability is necessary because of a few things we noticed:

If the memory-mapped files are lost (usually to a reboot because they perform best when stored on a ramdisk), the entire blockchain will have to be replayed, which takes hours. Witnesses are expected to be reliable (i.e., not down for hours and missing blocks), so another instance is needed to take over.
Best practices for backup Steem witnesses currently dictate that there can only be one active daemon for a witness because if there were two (with the same private signing key), they could both try to stake a block. That is somehow problematic, but the reason doesn't appear to be documented, so we'll need to research this further.
Existing ways to detect failed primary daemons rely on checking for incremented missed blocks, which means a failure must have already happened before it's noticed.

If we can come up with an active/active setup, witness failover would not be necessary, and perhaps it could almost eliminate missed blocks.

Yes, the daemon needs many gibibytes of RAM, and that requirement is going to keep growing unless there's an overhaul of what gets loaded into RAM. For now, we must scale vertically.

The use of zram is a mitigation that buys us quite a bit more time before we eventually run out of RAM. Hopefully a solution comes up before we need to upgrade physical RAM.