How to setup and run the EOSIO Genesis Snapshot Generator

These instructions and acompanying video provide detailed steps for creating a genesis snapshot used to bootstrap an EOS.IO blockchain that will reflect the contributions of stakeholder in the EOS tokensale.

Update 15th Feb 2018
We've just heard from the EOSIO development team that there may not be a 'genesis block' as such with the new EOS BIOS process so don't invest too much time with the process we documented here until we all get the update. The GitHub repository is also being updated with new code so the procedure in this post uses now used the 0.2.1 Testnet Release of the snapshot generator.

We are using the software provided by the EOS.IO development team at https://github.com/EOSIO/genesis and expanding on the high-level installation guide, so any developer or enthusiast can create a snapshot, genesis file and so validate the tools and process.

As we get closer to a community blockchain launch, it’s important for as many people as possible to validate this code, it’s output and ideally develop alternate implementations so we have certainty in the starting stake of the EOS community in any EOS.IO based blockchain.

At a very high level, the snapshot generator queries the Ethereum EOS tokensale contracts to determine all contributions and balances and then outputs this to a comma seprated file snapshot.csv along with a snapshot meta-data file snapshot.json which contains some high-level stats and checksums.

These snapshot files, can then be used to create an EOS.IO compatible genesis file using the 'Genesis Block Generator' at https://eosio.github.io/genesis/

The EOS.IO development team have told us to expect a significant update to the Genesis Snapshot Generator and accompanying documentation at some point. We'll update the post and video when it's available.

Let’s get started…

---

How to setup and run the EOSIO Genesis Snapshot Generator

Prepare a fresh VM, physical or cloud server with the following specifications:

Ubuntu 16.04 (LTS) Server, 4 CPU cores , 8GB memory and at least 120Gb of SSD storage.

Update unbutu with the latest repository package lists
sudo apt-get update

Install screen

Let's install screen which allows us to keep our processes running with output even if you get a dropped connection.

sudo apt-get install screen

screen –S eos

If you've never used screen before, check out this quick tutorial https://www.mattcutts.com/blog/a-quick-tutorial-on-screen/

Exit the eos screen by typing Ctrl-a then d.

Install parity.io and full sync the ethereum blockchain

bash <(curl https://get.parity.io -Lk) -r stable

Run parity in it’s own ‘screen’ so we can monitor it easily

screen –S parityio

parity --no-warp

It’s very important to use the --no-warp option with parity to be able to create accurate snapshot data

warping will skip over historical data that the snapshot requires, tricking web3 into thinking the chain is synced when it is not.

The current ethereum full blockchain is over 50GB in size and will likely take a few days to sync!

Again, exit the screen by typing Ctrl-a then d, leaving parity to sync.

EOS.IO Genesis Snapshot Build Process

Clone the EOS.IO genesis GitHub respository

git clone https://github.com/EOSIO/genesis
cd genesis
git checkout 0.2.1

Install MySQL

cd
sudo apt-get install mysql-server

When prompted be sure to enter a root password for the mysql root account

Test that the mysql service is running...

mysqladmin -u root -p version

Create the mysql eos_snapshot database and load the schema

Log into mysql command line
mysql -u root -p

Create the eos_snapshot database
CREATE DATABASE eos_snapshot;
exit

Create the eos_snapshot database schema by importing the EOS.IO developed schema definitions
mysql -u root -p eos_snapshot < ./genesis/bin/schema.sql

Check that the mysql schema import was successful

Login.
mysql -u root -p eos_snapshot
and
SHOW TABLES;

You should see the following output:

+------------------------+
| Tables_in_eos_snapshot |
+------------------------+
| buys                   |
| claims                 |
| keys                   |
| reclaimables           |
| registrations          |
| snapshot               |
| state                  |
| transfers              |
| wallets                |
+------------------------+
9 rows in set (0.00 sec)

exit the mysql client.

Install Node.js and NPM

Ensure your ubuntu installation can access the correct Node version repositories, this will enable us to install Node.js v6.x
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -

Install nodejs (JavaScript Runtime)
sudo apt-get install -y nodejs

Change to the EOS.IO genesis root directory
cd genesis

Install the 'build-essentials' so we can compile the scrypt 'cryptographic hash' npm dependancy
sudo apt-get install build-essential

Now install/compile the required npm dependencies for the genesis snapshot software that are referenced in package.json (npm is a package manager for JavaScript)

sudo npm install

Set your snapshot generator defaults by creating a config.js

cp config.default.js config.js

edit the config file using your favourite editor
nano config.js

Remember to change the eth_node_path to your home folder / parity defaults.

module.exports = {

  period: 1,
  include_b1: true,
  cache: false,
  fallback: false,
  overwrite_snapshot: false,
  recalculate_wallets: false,
  author: "info@eosphere.io",

  //ETH node
  eth_node_type: 'ipc',
  eth_node_path: '/home/rdold/.local/share/io.parity.ethereum/jsonrpc.ipc',

  //redis
  redis_host: "localhost",
  redis_port: 6379,

  mysql_db: "eos_snapshot",
  mysql_user: "root",
  mysql_pass: "eos",
  mysql_host: "localhost",
  mysql_port: 3306

}

This snapshot config file will create a snapshot for Period's 0 and 1, without the experimental fallback support.

Run the snapshot

From the genesis directory run
nodejs snapshot.js

:~/genesis$ nodejs snapshot.js
prompt: Config file detected, load it?:  (true) 
.--------------------------------------------------------------------------------.
|                                    Settings                                    |
|--------------------------------------------------------------------------------|
|              period | 1                                                        |
|          include_b1 | true                                                     |
|               cache | false                                                    |
|            fallback | false                                                    |
|  overwrite_snapshot | false                                                    |
| recalculate_wallets | false                                                    |
|              author | info@eosphere.io                                         |
|       eth_node_type | ipc                                                      |
|       eth_node_path | /home/rdold/.local/share/io.parity.ethereum/jsonrpc.ipc  |
|          redis_host | localhost                                                |
|          redis_port | 6379                                                     |
|            mysql_db | eos_snapshot                                             |
|          mysql_user | root                                                     |
|          mysql_pass | 42                                                       |
|          mysql_host | localhost                                                |
|          mysql_port | 3306                                                     |
'--------------------------------------------------------------------------------'
Starting in 5 seconds.
sequelize deprecated String based operators are now deprecated. Please use Symbol based operators for better security, read more at http://docs.sequelizejs.com/manual/tutorial/querying.html#operators node_modules/sequelize/lib/sequelize.js:242:13
MySQL: Connected
Web3: Connected
Web3 is Still Syncing (At Block #5041978). trying again in 30 seconds
Web3: Synced
Truncating tables, starting clean
---------------------
INFO: Period Map needs to be Updated - not synced, syncing to Period  (Today is Period 230)
Generating new row(s) for Period Block Map (Up to Period 1)
This will take a little while
---------------------
Added period 0 to the Period Map :)
{ begin: 3904416, end: 3957867 }
Added period 1 to the Period Map :)
{ begin: 3957868, end: 3962664 }
Syncing Contract State between block #3904416 & 3962664
Syncing Contracts between block #3904416 and #3962664, this may take a while.
.-----------------------.
|  3904416 ~> 3958215   |
|-----------------------|
|     Transfers | 3648  |
|          Buys | 19967 |
|        Claims | 2917  |
| Registrations | 6537  |
|  Reclaimables | 0     |
'-----------------------'
Started: 29.86 seconds ago, Average: 55.50 ms/pass
finished
.-----------------------.
| Complete: 3904416 ~> 3962664 |
|-----------------------|
|     Transfers | 14127 |
|          Buys | 22442 |
|        Claims | 8044  |
| Registrations | 7981  |
|  Reclaimables | 5     |
'-----------------------'
Started: 40.30 seconds ago, Average: 69.12 ms/pass
Syncing Wallets
...
...
...
....
{ period_map: 
  [ { begin: 3904416, end: 3957867 },
    { begin: 3957868, end: 3962664 } ],
 block_begin: 3904416,
 block_end: 3962664,
 sync_contracts: 
  { buys: 22442,
    claims: 8044,
    registrations: 7981,
    transfers: 14127,
    reclaimables: 5 },
 total: 15992,
 tests: 
  { total_supply: true,
    daily_buys: true,
    negative_balances: true,
    validation_balance: true } }
.------------------------.
|      Tests Passed      |
|------------------------|
| total supply       | ✓ |
| daily buys         | ✓ |
| negative balances  | ✓ |
| validation balance | ✓ |
'------------------------'
Fallback: Skipping Fallback
Snapshot Table Synced
5229 Records Saved to CSV
Generating snapshot.json
Generating DB Table Checksums
Snapshot meta written to snapshot.json
Snapshot for Period #1 Completed.

Genesis

If all has run well, you'll have a 'snapshot.csv' that identifies all the EOS ERC-20 stakeholder addresses along with their registered EOS public key and snapshot EOS balance.

You can find these files in $HOME/genesis/data/ with a folder for each period snapshot you constructed. To see the Period 1 snapshot files we just created :
ls $HOME/genesis/data/1

We'll leave actually creating a genesis block file for another tutorial, but if your keen head over to https://eosio.github.io/genesis/ and use the files you just created as input.

As always, leave any feedback or suggestions/requests for other walkthroughs in the comments.

---

EOSphere are passionate group of EOS enthusiasts and data centre professionals setup to help in the global community launch of an EOS.IO blockchain later this year. Based in Australia, EOSphere are initially focused on bootstrapping the Australian EOS community by hosting a series of meetups in each state capital.

Follow us on steemit @eosphere and on twitter @ https://twitter.com/eosphere_io