Recently, I’ve had many good opportunities to write an update, because there has been a lot going on.
But every time I asked myself if I should spend the little time I had writing posts or doing something more productive for the Steem platform, I usually chose the latter. On the other hand, I also know that the words “trust me, I know what I’m doing” are not enough for everyone.
So, here is another witness update.
And another promise to write more frequently.
_{(I’m better at solving actual problems than at writing, and I’m glad that the reverse is not true.)}

^{Full HD video rendered exclusively for my witness updates}

So what has happened since my last update?

We have release dates for SMTs

January 15, 2019 - SMT development complete and live on the TestNet.
March 24, 2019 - SMT fully released and live on the Steem Blockchain.
That’s a major improvement for the Steem platform.
It is going to be much more complex than recent Hardforks, with a lot moving parts that can be sources of potential issues.

Tracking SMT development progress

There’s an open tracking system for SMT development available: https://github.com/orgs/steemit/projects/14
Which is a very valuable source of knowledge about SMTs.

An unexpected fork

Before the HF20 scheduled date, on Sept 17, the chain unexpectedly forked when already running 0.20.0 nodes produced blocks that did not match the consensus of nodes still running version 19 of steemd.
Thanks to the hard work of Steemit Inc. developers and witnesses (shoutout to @abit, @riverhead, and @roelandp), the network was again up and running.
Here’s where the ability to replay the whole blockchain from scratch in less than 4 hours and multiple high-end servers at your service pays off. Especially when you need to do that multiple times.
I started producing again at block 26038153 (hell of a block!), soon after we got back to a fully operational status.
There were no transactions going on between 12:47:00 and 19:56:51 UTC.
I'm sorry for that.

HF20

Hardfork 20 happened at block 26256743.
Hardfork itself was a success, unfortunately one of the components, the Resource Credits algorithm caused a lot of trouble, although we had expected some issues and the need for some time to reach the equilibrium.
Resource pools were set to 90% of max equilibrium, and we expected that this will allow users to continue using the blockchain and gradually transition into the RC system, unfortunately it was not enough for basic usability for the common users.
Adjustments were made, and after many iterations (currently we are running v0.20.5), we finally achieved stability.
Lesson learned.
We want to minimize the risk that such a foobar will happen again.

Bugs, issues, improvements

There are still many known and unknown issues out there and improvements to make to clean our way towards HF21.

Questions from the community

I try to react to various concerns voiced by the members of the community on an ongoing basis. If you have any doubts about a topic that has not been described on Steem or discussed on the public #witness channel, you can contact me directly on steem.chat, as Gandalf, but please be patient if don’t respond for a longer time.

Here is a bunch of questions that @rishi556 asked in his Question For Witnesses. Some of them are related to recent events, so I’ve decided to include answers in this update as well.

1. Why did you decide to become a witness?

Out of curiosity. Running your own set of nodes is a great opportunity to learn about the inner workings of the platform. Also, given my field of expertise, I hoped that I could be of value to the Steem platform.

2. Have you made any contributions that have been accepted into the STEEM codebase? If so, what was it?

Yes. Contributions mean not only making pull requests but also finding issues.
My contributions, however, are usually far from changes at the conceptual level. In most cases, they are related to security, reliability, and performance.
After a quick look at GitHub, I can tell you that I’ve made 12 PRs to the core Steem codebase with some minor security/stability fixes, docs; resource usage, seed updates, but also dozens of issues (38 closed) that were solved and therefore helped improve Steem.

3. HF20 started out with a ton of bugs. What do you plan on doing in order to make sure that we don't suffer a repeat of that?

There was already a discussion on Sunday on the #witness channel at steem.chat:

Here’s a simple algorithm posted by @ats-david (aka @ats-witness):

• Is there a problem in need of a fix? If no, then no hard fork is necessary. If yes...
• Does proposed change solve the specific problem? If no, reject it. If yes...
• Has the new code been audited? If no, reject it. If yes...
• Has the new code been tested? If no, reject it. If yes...
• And changes have been made, audit and test it again.
• If everything looks good and you like the proposed change, then accept the hard fork.

That should be a basic universal standard for witnesses.

The worst thing is that I did all the steps.

I had some doubts and uncertainty, and of course you can always dig deeper (certainly deep enough to notice the difference between seconds and hours!) and work harder to have less uncertainty

So I guess I need to improve this uncertainty to a level that is acceptable

Also, I made one mistake:
We received a good question before the scheduled HF20 date:
"What will we do, if RC does not work as expected?"
RC was the most risky part but also a non-consensus one, so it was highly underestimated.

The answer was "we will switch to the bandwidth algorithm".
Later on, we decided that it was not the best idea, but the question should have been reevaluated at that time.
Of course, with the wisdom of hindsight, it's always super easy to make a perfect plan to avoid all the problems...
... that occurred in the past. Unfortunately, the RC plugin was very difficult to test in the artificial testnet environment.

TL;DR version:

I need to improve this uncertainty to a level that is acceptable

Oh, and how we can do that?
@timcliff wrote a very good post about Hardfork approval standards
Is that the ultimate solution that will make sure that we won’t hit severe issues like we did last time?
No, it is not. But it can lower the risk significantly.

4. What have you done to promote the STEEM blockchain outside of the blockchain itself?

I've released promotional videos (including: Flourishing Steem, Steem Power Inside, Steem Pressure, SteemSTEM promo, more on the way)
I’m promoting and lobbying for the Steem platform, and STEEM (including exchanges, service providers, and high quality content creators), during various meetups and talks at every latitude, longitude, and altitude I visit.

5. Communities have become a very big thing in the STEEM ecosystem recently. Have you done anything to promote communities and help them out?

I'm delegating SP to Utopian (almost 50% of my all SP) to help incentivize development work and to members of the SteemSTEM community, supporting i-Talent competition ("Competition Both Performed And Judged By The Artists"), and actively (manually) curating local non-English communities, mostly #Polish and #Japanese (shout-out to all the people involved in manual content discovery and curation).

6. Do you understand enough of the STEEM codebase and C++ to be able to tell what changes are being made when they are committed?

Yes, of course.

7. Have you developed anything for STEEM? It doesn't need to be software, a community that utilizes STEEM counts too.

I keep the Steem.chat up and running.
I write the Steem Pressure series, I really hope to get back to it soon (I wasn't very active lately, because there was a lot of work regarding HF20).
I also consult and help with a lot of various development initiatives related to Steem.

8. Communication was a major problem for STEEMIT INC. Many people want the public to be able to read what’s
happening in the private witness channel that STEEMIT INC has created. What do you think about that?

Not all communications should be public, some are not meant to be public.
Requesting public access to private channels makes no sense.

Obviously, public communication channels are as important as private ones, and I would very much like the #witness channel on steem.chat to be such a public place for conversations. It is available to the public in the broad sense.

Maintaining good communication is always problematic, and that problem is not limited to Steemit Inc. As you can see, I also have a problem communicating my efforts as a witness in a timely manner.
In one of my previous posts, I introduced myself as a witness witnessing witnesses ;-) As someone who wants to connect all these complex issues and make them more easily understandable to the general public. Unfortunately, I underestimated the amount of effort required to do that effectively.

During the recent problems with the platform I did everything to get my infrastructure (and therefore the platform) up and running again, and I didn’t have time to communicate everything I did via the chat. That would have slowed me down to a considerable degree without bringing any benefits. (Time shared witnesses wouldn’t have helped much at that point anyway).

Still, I understand those who were frustrated with the fact that there was not enough communication.
We usually think “someone will do that”.
There is no “someone” here, it’s ultimately the job of each and every one of us.

In other words, there is always room for improvement when it comes to communication.

9. STEEMIT INC has done a major part of the development for the STEEM blockchain. Who should be doing more of the development for the blockchain, them or the witnesses? Or is it a shared responsibility where the witnesses do one part and STEEMIT INC does the rest?

Steemit Inc. has its product roadmap and enough resources to run this development. I’m fine with that. Witnesses are not blockchain developers, although they could obviously do that too. I will gladly support other development efforts, whether run by witnesses or by companies interested in the platform or by organizations or communities, as long as they make sense.

10. Why do you deserve to become the top witness?

I’m doing my best to help the Steem platform.
Whether this is enough to deserve votes depends only on those who vote for me.

Thanks for asking all these questions and for being interested in these issues.

For the record

API endpoints

Of course, all my public nodes are running v0.20.5. Obviously, there are no longer non-appbase nodes running, but I still can see some old-style requests at my API endpoint. Please check your software and make sure it is up to date.

State of Steem

Have you noticed? State size is smaller than before. Also, full node runs RockDB based account_history plugin.

PS

SteemFest³ is coming.
7-11 November, Krakow, Poland.

---

^{If you believe I can be of value to Steem, please vote for me (gtg) as a witness on Steemit's Witnesses List or set (gtg) as a proxy that will vote for witnesses for you.
Your vote does matter!
You can contact me directly on steem.chat, as Gandalf}

Steem On