A Hacky Guide to Hive (part 1.5)

Black Box

6 months ago, I attempted writing a guide for programming with Hive.
I tried looking at Hive as a black box.
I still like that concept, but after reading it all again, I realized that I've jumped too far ahead.
I'll try taking shorter strides this time.
I also changed the title and theme a little.

I am trying to explain the exact same things as in the first guide, just with a different approach and more elaborate. By the end, the examples I posted recently, should make more sense.

I'll base the guide around the publicly available nodes, and I'll further explain, why that doesn't matter, later, but in this part we need to look at how you would install your own Hive node.

Hive Nodes

Hive is a software with open source code.
So... you can download the code,
compile it and then run a node. (the rhyme!)

block_log

The Hive blockchain itself gets stored as series of events.
The events are all segmented in 3-second intervals, which are called blocks.
But in the end, all events end up in one long file called block_log.
It's almost like a long tape recording.

Hived

Compiling the code for a Hive node will leave you with a hived binary.
That's the Hive daemon.
You then need to synchronize your node with the network.
This means, your Hive daemon needs to go through all blocks and replay all state changes that happened up until now.
In the tape analogy, you could load a block_log into you hived tape recorder and fast forward the tape, going over all past events, called indexing.
The Hive daemon watches all events and creates databases in the background.

tables

There are different modules you can activate for your node, called plugins.
What's important here: for the most part, the API calls represent get methods for underlying database tables.

The blockchain itself only documents the changes.
For example: The balance of an account is not directly accessible on the blockchain.
While the blockchain contains all transfers to and from an account, the final balance needs to be calculated;
The data needs to be compiled first, for quick access.

Custom Client

I'll try to write a hacky guide on how to build your own custom client for Hive.
It will enable you to build slim, reliable, customized services for Hive and operate them on a budget.

This is approach isn't the best for all applications; If you want to do historical stuff, analyzing large amounts of data, statistics and such things, HiveSQL is better.

If you don't know, where else to start, this could be a good bet.
However, ironically, I got questions about this stuff myself.

Questions

To some extent, Hive's still a black box to me.
I know how to use some of it, but I don't understand how some of it really works.
Sometimes, all I can hope for is that one of the gods of Hive takes pity and lets themselves down from the Olymp to answer to a mortal.

• Wouldn't witness nodes need tables for things like account balance? How do they know, if someone isn't stealing?

• block_api.get_block_range seems to access a table where the block number is a key. Why is there no public method, which starts from the last entry? At least in SQL databases that's a request with linear speed. (And I could get to the head block in one step). It probably has something to do with block reversibility or rocksDB...

The good part is, that it doesn't really matter; I don't have to fully understand all of it.
If there was a fundamental flaw in consensus finding, serialization, signing and other such things, it would be too lucrative to exploit it - someone would have found it and we'd know.

Notes

• Since I will be using Python requests, you can use the exact same code to access your own node, locally, or the public nodes, via https.
• I will not attempt to sign and broadcast anything for users. I will build a small html + JS based client as example, implementing keychain, because as the developer portal states:

By utilizing Authenticating services, you can eliminate or give more confidence to user, so they are assured their keys are safe. They can securely interact with your application, website or service.

• I have medical issues with my hand, so I asked @manclar to help me with the images for these posts, and paid him for it.
• This is a work in progress and subject to change.
  I appreciate all feedback, but please try to not comment 'That's not right' without further explanation. At least provide a link or something.

_{tagging:
@chrislybear, @ana-maria, @arc7icwolf, @slobberchops
If you want to be included/excluded from this list of subscribers, please let me know.}