Okay. I'm glad that the problem was identified and that the blockchain came up as quickly as it could. For me, that was 11 hours. Don't know what it meant to others, but it could have been less.
Realistically, I might have lost two to three STEEM from it being down. No big deal.
Maybe it meant more to others. I don't want to downplay it's significance for anyone else.
I guess the main thing that sits wrong is the inevitability of it all. Yes, the blockchain is highly complex. Yes, an error in coding is bound to happen. Yes, it would take way too long to test everything that could possibly go wrong, even if you could anticipate everything that could go wrong.
But what is it about these incidences (this is the second blockchain stoppage since I've been here—don't know if there were any or not before January 1, 2018), that is meant to instill confidence that they won't happen again? The fact that the community came together and resolved the issue? I'm glad it happened and things weren't prolonged. I'd be much happier if it never happened.
But that apparently can't be. So, when the next unlikely, unique, shouldn't happen anomaly takes place, and then the next, and the next, how often do you think this can go on before you lose the confidence of the users?
And maybe it's not so a big a deal now, with 50,000-60,000 daily users that are largely scraping out earnings through their social media efforts, but what happens when it becomes SMTs and after that, entire industries built on the STEEM blockchain, as in a worldwide digital economy?
As I've commented elsewhere, I hope the inevitability of this, sooner than later, becomes totally avoidable. That the impossible to foresee or catch or prevent now becomes absolutely possible, if not automatic. That means resources and money I suppose, and technology and knowledge that doesn't yet exist. Maybe that will be much later than sooner and we'll just all have to cross our fingers that another stoppage won't happen for a while longer.
It's great we've got people who can troubleshoot and resolve things. I'm glad people can work together. It's got to be frustrating for all of those involved since it can largely be a thankless job when something like this happens. I'm just hoping the commitment is there to avoid the likelihood of an all day (in my case) downtime, and if there are alternatives for allowing people to work, cache it somewhere or whatever the proper terminology might be, and then get it going on the blockchain when it's up and running again would be great. I don't know what the solution is. I just know there needs to be something better than it's impossible or it's inevitable and great work guys when it's restored.
I hope I've been able to communicate what I'm feeling and trying to say. I'm not trying to ruffle feathers or point fingers. I'm just trying to express a point of view in hopes that it will be heard and that someone will understand it. If anything could be done about it now or in the new future, that would be ultimate, but right now I'd settle for someone getting what I'm saying and being okay with the fact that I said it.
Imagine somebody just invented the steam engine (see what I did there?) and people are worried it might derail or explode. It's due to arrive at the station at 3pm and at 4pm there's no sign of it.
If you're just a traveler on the platform you're going to worry that any number of terrible things have happened.
This post is the train finally arriving at the station, and the conductor announcing that somebody pulled on the emergency brake, which did what it was designed to do and brought the train to a screeching halt.
Most of the delay was the investigation by the engineers as they made sure they'd thoroughly investigated the cause of the problem, and double checked the timetable to make sure that restarting after the delay wouldn't put the engine on a collision course with any other trains.
Hey, @mattclarke.
I have no problem with everything that occurred the moment the bug/bugs came to play. I'm good with all that.
I'm more concerned with what happened to cause the emergency brake (ie the bugs). I'm understanding that it's impossible to account for every instance of why the brake might be pulled. Right now there aren't that many trains out there, at least not running the same length of track, nor are that many people waiting for the train, so the inconveniences might not be so bad.
Soon, however, we're hoping for many more people to be at the station waiting for the train to arrive (ie velocity), and we're anticipating that there will be many more trains providing service on that stretch of track (ie SMTs, entire industries). We're also understanding there will be other track built and trains on them (competitors) that will take away customers if we can't provide on time service all the time.
The one that's going to be able to get the most people from and to their destinations on time is going to be the one which stands.
As we said in the post, we have gained a lot of knowledge from this incident and know a lot of tools we want to develop to ensure this won't happen again. While the process might be unpleasant for you, and for that we apologize, this process of dealing with the unknowns and developing solutions is precisely how problems are solved for good. Building blockchains is still a new thing. Building blockchains that power real applications is something only we do. And building such a blockchain and updating it frequently is yet another thing only we do. We have no one to look to for guidance, no established model for anything. If someone tells you they can develop bug free code in such an environment, they're just a liar. What we can promise is that we will never release code that puts your important information at risk, and we have demonstrated that over and over again. This event was a consequence of that commitment as the blockchain stopped producing blocks specifically to safeguard what matters most. Those who come after us will be able to benefit from our solutions, but we do not have the same privilege. I am sorry we live in a world where we have to be the one's blazing the trail on this, but that's also why we are so committed to what we do. We've been doing this for over 2 years and our engineers and witnesses still respond at all hours and put in countless hours to fix the problems that arise. If you don't think that it is commitment, I don't know what is. And yes, for the people who do the real heavy lifting (i.e. not me) it is totally thankless.
Hey, @andrarchy.
I'm glad to hear all of this. It is all pioneering on your part, and the analogy you gave me the last time of building the airplane as it's flying has stuck with me. It does describe what you're doing, and attempting to do. Something that is basically impossible to do (at least building a commercial jet in flight would be).
I'm glad to hear about the dedication and expertise employed after the fact. I've seen it in action. I know it exists. I'm glad to hear about the tools that you all want to develop to prevent such things from happening.
I guess since it's nearly ten years since Bitcoin rolled out I figured there might be a little more on blockchain technology out there, along with project management, product testing, etc. I thought there would be more to it as far as common practices go. If not specifically for the blockchain, at least adapting from other industries.
I know the STEEM blockchain is unique in many aspects from other blockchains that aren't designed to do what it does and will do. I can see the uncharted territory there, but I was expecting there would be something else to draw on, rather than it basically being put it together as you go. That's more than just building the blockchain. That's also needing to build all the procedures and protocols surrounding how it gets built. I figured there would be a lot of that in place by now.
I've never heard of anyone saying they can create bug free code. Developers are pretty united on that front, even ones who think they can do it better. So, wasn't trying to say that at all.
Of course we use established project management techniques.