Life as an EOS block producer (BP) can be a roller coaster ride. One minute you are in the top ten, the next minute you aren’t even in the top 21. No matter what happens, a block producer needs to always make sure their nodes are up, functioning, and ready to produce. At Cypherglass we knew from the very beginning that we would need an enterprise class monitoring tool to insure that our gear is always operating at peak performance. There are some great external tools out there, like the Cryptolions EOS
Network Monitor, that provide the EOS community a view of block producer endpoints. These tools are good for what they do, but there is no tool built for a block producer to monitor their own gear internally. This week we are going to take a look at Cypherglass Windshield. Windshield is the only monitoring tool on the planet that is built specifically for EOS Block Producers to monitor and maintain their block producer infrastructure. You can see our live demo version in action here
At Cypherglass we currently run an array of twelve block producer nodes. One primary and one backup block producer node, three full nodes, four proxy nodes, an API node, a public node, and a blockchain backup node that holds a continuous FIFO backup of the full blockchain. Each type of node services the EOS community in a different way, and they are all important.
Let’s take a tour through Windshield and start by looking at our Nodes Dashboard view. This is the initial Windshield view that allows a BP to see at a glance what is going on with all of their nodes.
As you can see, the headings are pretty straightforward. The Account name is what the BP decides to call that node, in our case we made them as self descriptive as possible. Address can be either external or internal, and either a name or IP address. Type of node is currently limited to Block Producer, Full Node and External Block producer nodes. As Windshield evolves we anticipate expanding the stratification to include more specific node types. Next we see the Last Produced Block column. (Yes unfortunately for us this was six days ago when we plummeted from number six to number 27 in a matter of seconds. Can you say Bitfinex? (Neither can we right now) Continuing along from left to right we can see the Last Prd At column which represent the time of the last produced block, and then we have the current Vote Rank of the BP node. The Status shows the response time from that particular node, and the Head Block represents the latest block produced.
hey also really like getting woken up in the middle of the night. That why Windshield has an array of alerting and alarm functionality. In fact, just the other night I was awakened by Windshield when we experienced a bug on our nodes that run the history.api. Windshield quickly told us that block production was unaffected, which was a relief. We’ve still never missed a block to this day. One we determined block production was fine, we looked to Windshield to tell us which nodes were affected. Since Windshield is fully integrated with email, SMS and Slack you can wake bolt upright and scared from a variety of sources. Additionally we also have audio alerts from the dashboard view.
Here is a list of the alerts Windshield provides:
Block creation
Receive alerts if your Block Producer node has not produced blocks in a specified period of time
Forked/unsynchronized node
- Windshield will let you know if your node has become forked or unsynchronized
Handshake response
- Windshield constantly checks to insure that your node is up and running.
Full Node to Block Producer node synchronization
- If your Full Nodes aren’t talking to your Block Producer node Windshield will send you an alert
BP vote movement
- When your node shifts up or down in position among Block Producers Windshield will tell you
Availability of one third of block producers
- If the number of Block Producers available drops below fourteen, Windshield will send you an alert
If you’d like more information feel free to check out our open source code or a working demo. As always we love it when people hit us up on our Telegram channel so feel free to swing on by. We are constantly enhancing the functionality of Windshield so stay tuned for future updates.