Over the last couple of days I've been going back and forth with some of Steem's technical experts and full node witnesses @themarkymark, @anyx & @bobinson exploring the question of whether full RPC steem nodes can be run on relatively inexpensive (~$2k capital cost) high end desktop systems (HEDT) rather than renting high end servers costing $10-15k per month.

While some may wonder why I am focussing on this issue arising from Ned's bombshell announcement, given the many concerns that arise (delay in SMTs etc), there is a very good reason.

Ned's post has revealed the very concerning reality that our whole wonderful Steem ecosystem is highly dependent on Steemit Inc running full RPC nodes which provide the APIs that all the amazing Steem frontends and apps (@steempeak, @steemmonsters, @utopian-io, @steemworld, @partiko, @esteem, @actifit, @steemhunt etc) rely upon.

Steemit Inc is spending around $2M per year on outsourced (rented) infrastructure costs, of which full nodes are a big component. This is simply unsustainable in a low Steem price environment. These fiat costs are covered by Steemit Inc selling Steem and this pushes the price of Steem down further.

It is a vicious cycle that can only be solved by dramatically cutting infrastructure costs by properly decentralising Steem.

Only a few witnesses (including @themarkymark & @anyx) are running full RPC nodes to supplement Steemit Inc. This is because currently it is very expensive to run a full node as they currently require massive amounts of RAM (256Gb+) only available on expensive server grade computers to run efficiently due to the huge size of the Steem blockchain. This is a crucial failure of decentralisation which must be resolved if Steem is to thrive.

shown that it is possible to cut costs substantially by purchasing (rather then renting) a very high end 512Gb Xeon Gold server for around $12k, and run a full RPC node on it. This is a vast improvement over spending the same amount PER MONTH as Steemit Inc has been doing.@anyx has

However it is not enough for full decentralisation.
$12k upfront plus a couple hundred $ monthly for hosting is a big commitment that many witnesses, especially those outside the top 20, will not be willing or able to make.

To fully decentralise and remove central points of failure and control, all crucial elements of a decentralised network need to be able to be run on consumer grade equipment.

This is why in the Proof of Work (PoW) blockchain world, true believers in decentralisation support GPU based mining and oppose ASICs. If the equipment needed to participate fully in a decentralised blockchain network is not readily accessible and within the financial means of large numbers of people, then it will naturally centralise and become vulnerable.

In addition to the cost and accessibility issues, there are some real technical advantages of HEDT machines for blockchain applications. HEDT machines have much faster single core performance than server CPUs and single core performance is crucial to Steem nodes, particularly in the time consuming replay function, performed every time there is an outage or upgrade.

The tables below show the much higher single thread performance of HEDT CPUs costing $250-$550 like the:

• Intel Core i7-7740X (max RAM 128Gb, Optane ready)
• i7-6800K (max RAM 128Gb)
• i9-9900K (max RAM 64Gb, Optane ready)
• i5-8400 (max RAM 64Gb, Optane ready)

compared to server CPUs such as XEON Gold 6126 or 6130 costing more than $1900.

AMD Ryzen Threadripper HEDT CPUs are also an option, with the advantage that they support ECC memory, but the disadvantage of lower single thread performance vs Intel HEDT CPUs.

There are two ways that HEDT class machines may be able to be used to provide full nodes:

1. 
   It is crucial however that tests are done on consumer grade Optane, as server grade is 3x the price for marginal performance increase.By supplementing RAM with Intel Optane memory. @bobinson is doing testing on this.

2. By "sharding" full nodes into multiple HEDT class machines, splitting the APIs between them to provide redundancy for all but the most lightly used APIs. This sharding could be done within the one witness's setup and/or across multiple cooperating witnesses.
   Currently consensus data (which must be on every machine) is 45Gb and the total for all APIs is 202Gb. Thus each HEDT class machine would have around 80Gb available for running APIs.

• Partial redundancy (of the most critical or highly used APIs) could be achieved with 3 HEDT class machines (~$6k)
• Full redundancy (2 machines running each API) could be achieved with 5 HEDT class machines (~$10k). Even taking into account the somewhat lower reliability of HEDT class machines, the overall per witness reliability and performance of such a setup would be higher than a single Xeon Gold server (as well as still being cheaper).

But it is important not to get too focussed on reliability levels.
There is always a cost / reliability tradeoff. However, the whole point of decentralisation is to have a very large number of (implicitly lower reliability) nodes in the system so a few can go down without problems to the overall system.

While no individual witness wants to have an unreliable node, the system as a whole is designed so another witness steps in if one of the main witnesses servers go down. The consequence is just a decrease in the earnings of that witness, not an outage of the Steem blockchain.

It is simply not necessary to have the same levels of reliability and individual witness level redundancy in a properly decentralised system as it is in a centralised system.
These days all computers are very reliable and the difference between consumer grade and enterprise grade reliability is huge in dollar terms but very small in actual percentage terms.

Decentralised systems do not need enterprise grade reliability at the individual node level, because decentralisation provides much higher reliability at the system level.

I am pleased to announce that I am joining the @Utopian-io team as CFO where I will be applying my financial, technical legal and decentralisation expertise to take Utopian and Steem to an even higher level.

I look forward to working with all concerned parties on Steem, to resolve these and other challenges facing us and move forward to the bright future that I saw at SteemFest.

And don't forget to:

Signup for the Crypto Class Action against Facebook & Google's Crypto Ad Ban

https://gateway.ipfs.io/ipfs/Qmd9Lme3FP2CMxtRbuQWCuiVdpWNp6EdKa5BkQ4gqRZ4mz