Below is a list of some of the Hive-related programming issues worked on by the BlockTrades team during the past work period. We’ve been pretty busy during the last period (so busy that I had trouble finding time to make this report, but one benefit of everyone taking off for the holidays is I finally got some free time). Because of the amount of time that has passed since the last report and the amount of work that has been done, I’ll just cover some of the highlights here.

Hive Application Framework: framework for building robust and scalable Hive apps

Our recent work has focused on optimizations, primarily of HAF and HAF apps.

As mentioned in my last report, we noticed that while the sql_serializer indexed most of the blocks into the HAF database very quickly, its performance was considerably slower once it dropped out of “massive sync” mode (where many blocks are processed simultaneously) into “single block” live sync mode. We observed that the sql_serializer was dropping out of massive sync mode earlier than it could theoretically, which resulted in hundreds of thousands or even a million or more blocks being processed in this slower single block mode, depending on how old the block_log being used to initialized the hived node.

After a few design revisions based on the new algorithm described in my previous post and a lot of testing and bug fixes (we resolved the last bug yesterday and finished verifying the fix today), the sql_serializer now stays in massive sync mode until the node is within one minute of syncing to the headblock, then it builds the database indexes and foreign keys needed for live sync mode, then it transitions to live sync mode to process the remaining blocks to the head block, then syncs block by block as new blocks are created.

With the current blockchain at just over 60M blocks, reaching headblock while filling a HAF database only takes around 10 hours now on our fast systems, which is pretty fantastic, in my opinion. By way of a minor comparison, reindexing an account history hived node (the need of which is obsoleted by HAF) took over 17 hours back when the blockchain was much smaller.

The above benchmark presumes starting with a block_log file that is within a million blocks or so of the current headblock, because block delivery times between p2p nodes begins to dominate as the time bottleneck if starting with a very old block_log file (or no initial block_log at all). As a side note, I hope we will find some time soon to look at speeding up normal P2P sync time. While it is fairly fast now, I think there may still be room for improvement.

Balance tracker user interface (HAF app)

Our two new Hive devs have created an API and a web interface for the balance_tracker (an example HAF app) using the PostgREST web server as an experiment in creating a HAF app that only requires SQL and Javascript coding. For more details on this app, see my previous post.

So far this process has gone well, and we’re currently collecting benchmarks and seeing how far we can optimize the app. If you want to check it out, you can visit: https://gitlab.syncad.com/hive/balance_tracker (checkout the develop branch).

Optimization HAF account history app (hafah)

We continued to experiment with various ways of optimizing the HAF-based account history app that is designed to replace account history hived nodes.

In its current form, it is a somewhat unique HAF app, because it doesn’t create any additional database tables of its own now, instead it just relies on the default database tables created by hived’s sql_serializer plugin. We were able to achieve this by changing the sql_serializer to directly generate the impacted_accounts table that was previously computed by the hafah app. As discussed in previous posts, this change eliminated entirely the need to index a hafah app and only minimally impacted the time that the sql_serializer takes to fill a HAF database.

But reuse of HAF’s tables vs the app creating its own tables has posed some unique challenges as far as optimization goes, and we’re looking at how to best optimize the queries of the app when it is only returning irreversible data (we expect we can get much better performance in this case, because the app only needs to look at data in the irreversible tables, instead of having to operate over both the irreversible and reversible tables).

Yet more HAF example apps

We created some more example HAF apps to demonstrate various types of HAF apps:

• a fork-impervious app: haf_new_account_colloector is an app that only processes blocks as they become irreversible, so it serves up API results very fast and the data it returns is guaranteed to not be changed by a fork, but the data it returns is delayed relative to the head block.
• a standard app: haf_memo_scanner is an app that process blocks before they become irreversible, providing the most recent data, but some of the data can potentially be reversed if the blockchain experiences a micro-fork.
• haf_account_creation_fee_follower and haf_account_creation_fee_follower_threads: these are two versions of the same app, but the second version uses multiple threads for higher performance (at the expense of some additional code complexity).

During this process we also found and fixed a few more bugs in HAF itself.

Hived (blockchain node software) work

Rationalization of resource credit (RC) costs

We’re continuing to work on improving the calculations associated with resource credits (the manna costs to an account when the account broadcasts a transaction such as a post, a vote, or a coin transfer). Right now that means we are creating tools to effectively measure and record these costs. I expect we will complete this work in the first half of January.

Optimization of get_block API calls

We’re also looking to optimizing one of the most expensive API calls still processed by hived itself: the API calls for returning data about blocks. We’ve made some progress in this area (reduction in CPU usage and some reduction in latency), but we’re still investigating if there’s more room for improvement.

What’s next?

• Clean up docs for HAF
• Create scripts to simplify setup of a HAF-based server
• Finish conversion of hivemind to a HAF-based app
• Add support for filtering of operations by sql_serializer to allow for smaller HAF server databases
• Complete work on resource credit rationalization
• Finish optimization of hafah when operating in “irreversible” mode and collect benchmarks for the two different operating modes (irreversible vs reversible mode)
• Finish up optimization of get_block API calls
• Integrate tests and more benchmarking into CI system for various repos