Hivemind / HAF how to save and restore the database for development purposes

Hello,

mindjourney generation: "soviet era poster of workers putting bees in a server"

As I've been working on hivemind again, my workflow has required a lot of "test x thing, revert the database to previous state, change the code then test again"

so I figured I'd share my workflow.

Why not resync the normal way

It would be best to do a proper removal of the database and then do a fresh sync every time, but we can't do that because a full sync from 0 to 5 000 000 blocks takes about half an hour. And we can't afford that whenever we want to test something 😅

What we want

We want to get a dump of the database with all the mocks except the ones I intend to work on synced up to the block where our mocks would otherwise kick-in. For me since I work on communities, that's util 4 997 990.

Then we want our workflow to be:

• delete database
• Restore our database dump to 4 997 990
• add our modified mocks
• execute the hivemind indexer to go through the remaining blocks and test our updated logic.

Installation

Follow the setup from:
https://peakd.com/hivemind/@howo/haf-powered-hivemind-dev-environment-setup-guide-according-to-my-findings-it-may-or-may-not-be-the-best-way

Sync HAF to 5m blocks then when you populate mocks, here's a handy script to remove the file you want from the mocking dirs:

it's super crude but it gets the job done

#!/bin/bash

# Move the mocked ops somewhere else
mv /home/howo/projects/hivemind/mock_data/block_data/community_op /home/howo/projects/hivemind

# Set the environment variables
export MOCK_BLOCK_DATA_PATH=/home/howo/projects/hivemind/mock_data/block_data
export MOCK_VOPS_DATA_PATH=/home/howo/projects/hivemind/mock_data/vops_data
export PYTHONPATH=$PYTHONPATH:/home/howo/projects/hivemind

# Execute the mocker script
python3 /home/howo/projects/hivemind/hive/indexer/mocking/populate_haf_with_mocked_data.py --database-url postgresql://postgres:root@localhost:5432/haf_block_log

# Move the directory back to its original location
mv /home/howo/projects/hivemind/community_op /home/howo/projects/hivemind/mock_data/block_data

Now you can sync hivemind we will do things differently so that it stops at a specific block:

./cli.py --database-url postgresql://postgres:@localhost:5432/haf_block_log --test-max-block=4997990

when that's done (it'll take a little while). We can finally dump our database:

pg_dump -j 10 -Fd -f dump.dump -h localhost -p 5432 -U postgres -d haf_block_log

And at this point we have our base setup. so you can always come back to this "save state".

Adding the remaining mocks and syncing to 5m

Similar thing to the other script except we just add the specific mock file we were missing, make sure to do that in a different shell or reset the environment variables

# notice how it's now only the community_op directory
export MOCK_BLOCK_DATA_PATH=/home/howo/projects/hivemind/mock_data/block_data/community_op
export PYTHONPATH=$PYTHONPATH:/home/howo/projects/hivemind
# Execute the mocker script
python3 /home/howo/projects/hivemind/hive/indexer/mocking/populate_haf_with_mocked_data.py --database-url postgresql://postgres:root@localhost:5432/haf_block_log

and now we can sync hivemind all the way to 5m blocks:

./cli.py --database-url postgresql://postgres:@localhost:5432/haf_block_log --test-max-block=5000011 --community-start-block 4997991

Notice that I added community-start-block this may not be needed depending on your workflow.

Restoring the database

So finally you ran your test, and want to revert to the previous version I tried some versions of --clean to try to get the restore to work but never got as good as a result as if I just destroyed and recreated the entire database, here's the script I use:

before you execute it, make sure you copy /etc/postgresql/14/main/pg_hba.conf and /etc/postgresql/14/main/postgresql.conf somewhre (I put them at /) so that you don't have to edit things manually

# destroy and recreate the db
sudo pg_dropcluster 14 main --stop
sudo pg_createcluster 14 main
sudo systemctl restart postgresql.service
# remove password, see the other guide as to why
sudo -i -u postgres psql -c 'alter role postgres password null;'

# remove config files and copy paste the good config files
sudo rm /etc/postgresql/14/main/pg_hba.conf
sudo rm /etc/postgresql/14/main/postgresql.conf
sudo cp /pg_hba.conf /etc/postgresql/14/main/pg_hba.conf
sudo cp /postgresql.conf /etc/postgresql/14/main/postgresql.conf
sudo chmod 777 /etc/postgresql/14/main/postgresql.conf
sudo chmod 777 /etc/postgresql/14/main/pg_hba.conf
sudo systemctl restart postgresql.service

# setup HAF
cd /home/howo/projects/hivemind/haf/scripts
sudo chown -R postgres:postgres /home/howo/projects/hivemind/haf/scripts/haf_database_store
sudo chmod -R 700 /home/howo/projects/hivemind/haf/scripts/haf_database_store
sudo ./setup_postgres.sh
sudo ./setup_db.sh --haf-db-admin=postgres

# restore the database
pg_restore --section=pre-data  --disable-triggers -h localhost -p 5432 -U postgres -d haf_block_log /home/martin/projects/hivemind/dump.dump
pg_restore -j 20 --section=data --disable-triggers -h localhost -p 5432 -U postgres -d haf_block_log /home/martin/projects/hivemind/dump.dump
pg_restore -h localhost -p 5432 -U postgres --section=post-data --disable-triggers --clean --if-exists -d haf_block_log /home/martin/projects/hivemind/dump.dump

And now you're ready to go again, sync the extra mocks and sync hivemind back to 5m :)