It might be the state undo logic slowing down blockchain processing in a sequential manner (this computation is probably skipped for replay+validate). But I doubt there is a way to disable it to check that, short of modifying the code for the test.
Probably we should modify the code dealing with checkpoints to skip undo logic up to the checkpoint. This would allow us to confirm if it is the bottleneck, and it would also give us a speedup when checkpoints are set if it turns out to be the bottleneck.
It should be easy to test - just cut out two lines with
sessionindatabase::apply_block_extended(I'm actually assuming that out of order blocks won't reach that routine during sync, but if they do, it would be a source of slowdown).I'd be surprised if undo sessions were the problem. They are relatively slow and worthy of optimization, but in relation to simple transactions, mostly custom_jsons, so their performance is significant when there is many of them, like during block production, reapplication of pending or in extreme stress tests with
colony+queen. During sync we only have one session per block.Yes, your assumption is correct, blocks are strictly processed in order during sync, the P2P code ensures this. If it's easy to test, let me know what you find out: I guess work with @gandalf so that the test is performed on the same machine as previous measurements.