This post will be a go-to reference for myself, and a resource for anyone interested in setting up an api node. It's very technical, so if you don't belong to one of those two groups you can skip it.

---

Introduction

My current setup is close to the default hivemind setup provided in the example jussi config. After talks with @gtg one of the nodes offers a few more apis for performance reasons, as the fat node is very slow. There surely are not-optimal or even misconfigurations, optimization will be an ongoing process. See the end of the post for more info, make sure to read and understand everything before you start. I will update this post with any important changes.

---

Hardware

First, we need hardware. The setup consists of 3 nodes, for which I selected the following specs:

• hivemind
  32GB RAM
  2x240GB SSD RAID0
• fat
  64GB RAM
  2x480GB SSD RAID0
• accounthistory
  64GB RAM
  2x512GB NVMe RAID0 (64GB SWAP)

All are set up with a clean install of Ubuntu 18.04

---

Setup

Common steps

Set up each server to log in securely and with a dedicated user. The user on all machines will be called "hive" during this process. Individual needs may differ so I don't go into details here and only provide the steps necessary to proceed.

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install -y screen git nginx certbot python-certbot-nginx

Step 1: make everything sync

hivemind node

install software

cd ~
sudo apt-get install -y python3 python3-pip postgresql postgresql-contrib docker.io
git clone https://gitlab.syncad.com/hive/hivemind.git

cd hivemind
sudo pip3 install -e .[test]

setup database

sudo su postgres
createdb hive

Create db user hive and grant access to database
createuser --interactive
psql
GRANT ALL PRIVILEGES ON DATABASE hive TO hive;
\q
exit

optimize postgres
sudo nano /etc/postgresql/10/main/postgresql.conf
use https://pgtune.leopard.in.ua/ to find the optimal settings for your machine. I used the following:

# DB Version: 10
# OS Type: linux
# DB Type: web
# Total Memory (RAM): 32 GB
# Data Storage: ssd

max_connections = 200
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.7
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 20971kB
min_wal_size = 1GB
max_wal_size = 4GB

sudo service postgresql restart

The irredeemables list is a blacklist containing mass spammers mostly. It's recommended to use it if you serve browser based interfaces, because the amount of comments by these accounts creates a lot of traffic and is a burden on browsers. It's defined in /home/hive/hivemind/hive/conf.py under --muted-accounts-url. You can change it there, or add the environment variable MUTED_ACCOUNTS_URL in both scripts if you do not want to use the default. I offer an empty version if you don't want to filter the results.

Create sync script
nano sync.sh
Insert the following (the STEEMD_URL is temporary until your own fat node has synced, update and restart it in a few days to speed up the hivemind sync):

#!/bin/bash
export DATABASE_URL=postgresql://hive:pass@localhost:5432/hive
export STEEMD_URL='{"default": "https://fat.pharesim.me"}'
export HTTP_SERVER_PORT=28091
hive sync

chmod +x sync.sh
screen -S hivesync

./sync.sh
Use Ctrl-a d to detach screen, screen -r hivesync to reattach

The whole sync process takes about a week. Don't forget changing the STEEMD_URL when your fat node is finished. Unlike the steemd replays, you can interrupt this sync at any time and it picks up where you stopped.

The sync is finished when you see single blocks coming in. Keep it running, and set up the server:

cp sync.sh hivemind.sh
nano hivemind.sh

Change sync in the end for server

screen -S hivemind

./hivemind.sh
Use Ctrl-a d to detach screen, screen -r hivemind to reattach

steemd nodes

Both the fat and the accounthistory node will run an instance of steemd, these are the steps to prepare them:

sudo apt-get install -y autoconf automake cmake g++ git libbz2-dev libsnappy-dev libssl-dev libtool make pkg-config python3-jinja2 libboost-chrono-dev libboost-context-dev libboost-coroutine-dev libboost-date-time-dev libboost-filesystem-dev libboost-iostreams-dev libboost-locale-dev libboost-program-options-dev libboost-serialization-dev libboost-signals-dev libboost-system-dev libboost-test-dev libboost-thread-dev doxygen libncurses5-dev libreadline-dev perl ntp

cd
git clone https://github.com/openhive-network/hive
cd hive
git checkout v0.23.0
git submodule update --init --recursive
mkdir build
cd build

The build options differ for the two nodes

fat

cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=OFF -DCLEAR_VOTES=OFF -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=ON -DSTEEM_STATIC_BUILD=ON ..

accounthistory

cmake -DCMAKE_BUILD_TYPE=Release -DLOW_MEMORY_NODE=ON -DCLEAR_VOTES=ON -DSKIP_BY_TX_ID=OFF -DBUILD_STEEM_TESTNET=OFF -DENABLE_MIRA=OFF -DSTEEM_STATIC_BUILD=ON ..

Again on both:

make -j$(nproc) steemd
cd
mkdir bin
cp /home/hive/hive/build/programs/steemd bin/v0.23.0
mkdir .steemd
nano .steemd/config.ini

And again, the configs differ for the two nodes

fat

log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes

plugin = webserver p2p json_rpc witness account_by_key reputation market_history
plugin = database_api account_by_key_api network_broadcast_api reputation_api
plugin = market_history_api condenser_api block_api rc_api

history-disable-pruning = 0
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200

shared-file-dir = "blockchain"
shared-file-size = 360G

shared-file-full-threshold = 0
shared-file-scale-rate = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760

p2p-seed-node = anyx.io:2001 gtg.steem.house:2001 seed.jesta.us:2001

rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 0

webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32

enable-stale-production = 0
required-participation = 33
witness-skip-enforce-bandwidth = 1

Not sure about the shared_file_size here, it's rocksdb? Better safe than sorry...

accounthistory

log-appender = {"appender":"stderr","stream":"std_error"} {"appender":"p2p","file":"logs/p2p/p2p.log"}
log-logger = {"name":"default","level":"info","appender":"stderr"} {"name":"p2p","level":"warn","appender":"p2p"}
backtrace = yes

plugin = webserver p2p json_rpc witness
plugin = rc market_history_account_history_rocksdb transaction_status account_by_key
plugin = database_api condenser_api market_history_api account_history_api transaction_status_api account_by_key_api
plugin = block_api network_broadcast_api rc_api

history-disable-pruning = 1
account-history-rocksdb-path = "blockchain/account-history-rocksdb-storage"
block-data-export-file = NONE
block-log-info-print-interval-seconds = 86400
block-log-info-print-irreversible = 1
block-log-info-print-file = ILOG
sps-remove-threshold = 200

shared-file-dir = "/run/hive"
shared-file-size = 120G

shared-file-full-threshold = 9500
shared-file-scale-rate = 1000
flush-state-interval = 0
follow-max-feed-size = 500
follow-start-feeds = 0
market-history-bucket-size = [15,60,300,3600,86400]
market-history-buckets-per-size = 5760

p2p-seed-node = anyx.io:2001 seed.jesta.us:2001

rc-skip-reject-not-enough-rc = 0
rc-compute-historical-rc = 0
statsd-batchsize = 1
tags-start-promoted = 0
tags-skip-startup-update = 0
transaction-status-block-depth = 64000
transaction-status-track-after-block = 42000000

webserver-http-endpoint = 0.0.0.0:28091
webserver-ws-endpoint = 0.0.0.0:28090
webserver-thread-pool-size = 32

The fat node also needs a database.cfg
nano .steemd/database.cfg

These settings are for 32GB of RAM. Adapt global.shared_cache.capacity, global.write_buffer_manager.write_buffer_size and global.object_count accordingly

{
  "global": {
    "shared_cache": {
      "capacity": "21474836480"
    },
    "write_buffer_manager": {
      "write_buffer_size": "4294967296"
    },
    "object_count": 250000,
    "statistics": false
  },
  "base": {
    "optimize_level_style_compaction": true,
    "increase_parallelism": true,
    "block_based_table_options": {
      "block_size": 8192,
      "cache_index_and_filter_blocks": true,
      "bloom_filter_policy": {
        "bits_per_key": 10,
        "use_block_based_builder": false
      }
    }
  }
}

As well as increased file limits (hive is the username, adapt to your memory again)
sudo nano /etc/security/limits.conf
Insert near the end of the file

hive soft  nofile 262140
hive hard nofile 262140

sudo nano /etc/sysctl.conf
Insert near the end of the file

fs.file-max = 2097152

You need to log in to the server again for these to take effect.

The accounthistory requires a change of the size of /run
mount -o remount ,size=120G /run
and a directory /run/hive

sudo mkdir /run/hive
sudo chown hive:hive /run/hive

Then continue on both servers.
Download block_log.index and block_log

rsync -avh --progress --append rsync://files.privex.io/hive/block_log.index .steemd/blockchain/block_log.index
rsync -avh --progress --append rsync://files.privex.io/hive/block_log .steemd/blockchain/block_log

Go for a walk or have dinner.

Start up steemd and replay blockchain
screen -S hive

echo    75 | sudo tee /proc/sys/vm/dirty_background_ratio
echo  1000 | sudo tee /proc/sys/vm/dirty_expire_centisecs
echo    80 | sudo tee /proc/sys/vm/dirty_ratio
echo 30000 | sudo tee /proc/sys/vm/dirty_writeback_centisecs

~/bin/v0.23.0 --replay
Use Ctrl-d a to detach from screen, and screen -r hive to reattach.

The sync process takes a bit more than 2 days on the fat node, less on accounthistory. Do not interrupt them, or you will have to start over. If you are syncing a hivemind node, don't forget to switch to the fat node when that's finished.

Step 2: webserver + routing

all nodes

All requests will be proxied by nginx, so we need this on all machines. We will install SSL certificates, so all communication is encrypted and all nodes can be called individually.

sudo nano /etc/nginx/sites-enabled/hive
The config is the same for each node, only change the server_name

upstream hivesrvs {
# Dirty Hack. Causes nginx to retry node
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   server 127.0.0.1:28091;
   keepalive 10;
}

server {
    server_name hivemind/fat/acchist.you.tld;
    root /var/www/html/;

    location ~ ^(/|/ws) {
        proxy_pass http://hivesrvs;
        proxy_set_header Connection "";
        include snippets/rpc.conf;
    }

Add the rpc.conf to each server
sudo nano /etc/nginx/snippets/rpc.conf
Insert

access_log off;

proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_connect_timeout 10;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

keepalive_timeout 65;
keepalive_requests 100000;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
proxy_ssl_verify off;

Let certbot configure the domains for automatic redirect to https
sudo certbot --nginx

hivemind node

We need an additional file for nginx, for the general entry point

sudo cp /etc/nginx/sites-enabled/hive /etc/nginx/sites-enabled/api
sudo nano /etc/nginx/sites-enabled/api

Change both occurances of hivesrvs to jussisrv, the ports from 28091 to 9000, and the server_name to api., or whatever you want your node to be accessible on.

jussi

cd ~
git clone https://gitlab.syncad.com/hive/jussi.git

Create build script
nano build.sh
and insert

#!/bin/bash
cd /home/hive/jussi
sudo docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" .

chmod +x build.sh

Create run script
nano jussi.sh
and insert

#!/bin/bash
cd /home/hive/jussi
sudo docker run -itp 9000:8080 --log-opt max-size=50m "$USER/jussi:$(git rev-parse --abbrev-ref HEAD)"

chmod +x jussi.sh

screen -S jussi

cd ~/jussi
nano DEV_config.json

Currently, my config looks like this:

{
    "limits": { "accounts_blacklist": [ "accounttoblock" ] },
    "upstreams": [
      {
        "name": "steemd",
        "translate_to_appbase": true,
        "urls": [["steemd", "https://fat1.pharesim.me" ]],
        "ttls": [["steemd", 3]],
        "timeouts": [["steemd",3]]
      },
      {
        "name": "appbase",
        "urls": [
          ["appbase", "https://fat1.pharesim.me"],

          ["appbase.account_history_api", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_account_history", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_ops_in_block", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_transaction", "https://acchist1.pharesim.me"],

          ["appbase.condenser_api.get_followers", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_following", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_follow_count", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_trending", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_hot", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_promoted", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_created", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_blog", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_feed", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_comments", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_reblogged_by", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_replies_by_last_update", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_trending_tags", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_discussions_by_author_before_date", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_post_discussions_by_payout", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_comment_discussions_by_payout", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_blog", "http://localhost:28091"],
          ["appbase.condenser_api.get_blog_entries", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_account_votes", "https://hivemind.pharesim.me"],
          ["appbase.condenser_api.get_state", "https://hivemind.pharesim.me"],

          ["appbase.condenser_api.get_state.params=['witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['/witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['/~witnesses']", "https://acchist1.pharesim.me"],
          ["appbase.condenser_api.get_state.params=['~witnesses']", "https://acchist1.pharesim.me"],

          ["appbase.follow_api", "https://hivemind.pharesim.me"],
          ["appbase.tags_api", "https://hivemind.pharesim.me"],

          ["appbase.market_history_api", "https://acchist1.pharesim.me"],
          ["appbase.transaction_status_api", "https://acchist1.pharesim.me"],
          ["appbase.account_by_key_api", "https://acchist1.pharesim.me"],
          ["appbase.block_api", "https://acchist1.pharesim.me"],
          ["appbase.network_broadcast_api", "https://acchist1.pharesim.me"],
          ["appbase.rc_api", "https://acchist1.pharesim.me"]
        ],
        "ttls": [
          ["appbase", 3],
          ["appbase.login_api",-1],
          ["appbase.network_broadcast_api", -1],
          ["appbase.follow_api", 10],
          ["appbase.market_history_api", 1],
          ["appbase.condenser_api", 3],
          ["appbase.condenser_api.get_block", -2],
          ["appbase.condenser_api.get_block_header", -2],
          ["appbase.condenser_api.get_content", 1],
          ["appbase.condenser_api.get_state", 1],
          ["appbase.condenser_api.get_state.params=['/trending']", 30],
          ["appbase.condenser_api.get_state.params=['trending']", 30],
          ["appbase.condenser_api.get_state.params=['/hot']", 30],
          ["appbase.condenser_api.get_state.params=['/welcome']", 30],
          ["appbase.condenser_api.get_state.params=['/promoted']", 30],
          ["appbase.condenser_api.get_state.params=['/created']", 10],
          ["appbase.condenser_api.get_dynamic_global_properties", 3]
        ],
        "timeouts": [
          ["appbase", 3],
          ["appbase.network_broadcast_api",0],
          ["appbase.chain_api.push_block", 0],
          ["appbase.chain_api.push_transaction", 0],
          ["appbase.condenser_api.broadcast_block", 0],
          ["appbase.condenser_api.broadcast_transaction", 0],
          ["appbase.condenser_api.broadcast_transaction_synchronous", 0],
          ["appbase.condenser_api.get_account_history", 20],
          ["appbase.condenser_api.get_account_votes", 20],
          ["appbase.condenser_api.get_ops_in_block.params=[2889020,false]", 20],
          ["appbase.account_history_api.get_account_history", 20],
          ["appbase.account_history_api.get_ops_in_block.params={\"block_num\":2889020,\"only_virtual\":false}", 20]
        ]
      },
      {
        "name": "hive",
        "urls": [["hive", "http://localhost:28091"]],
        "ttls": [["hive", -1]],
        "timeouts": [["hive", 30]]
      },
    {
      "name": "bridge",
      "translate_to_appbase": false,
      "urls": [["bridge","http://localhost:28091"]],
      "ttls": [["bridge",-1]],
      "timeouts": [["bridge",30]]
    }
    ]
  }

cd
./build.sh
This takes a while, when finished
./run.sh
Use Ctrl-a d to detach screen, screen -r jussi to reattach
If you update your config, run build.sh outside of screen, then attach to screen and restart the run script. (There may be faster ways to update the docker with a new config, but I'm new to this).

Something about steemd apis

As you might have realized, there are some duplicates in the apis fat and accounthistory provide. That's because of what I mentioned above, fat is slow. I did not investigate which aren't needed on fat now to still work for hivemind, so I didn't change anything in that default configuration. I also just took the list for apis on accounthistory from @gtg without questioning. There is unnecessary redundancy for sure, instructions may change in the future to improve on this. Your perfect setup may differ completely, depending on which apis you need to serve (most).

Finishing words

That's it. After everything is synced, you should have a working public node ready to serve requests! If this guide has helped you and/or you want to support my work on Hive infrastructure, education, onboarding and retention, please help me secure my witness spot!

Stats

Current output of df -h on the three servers (April 14):

hivemind

/dev/md2        407G  272G  115G  71% /

fat

/dev/md2        815G  369G  406G  48% /

accounthistory

tmpfs           120G   58G   63G  48% /run
/dev/md2        874G  531G  299G  65% /