Github has an angry pink unicorn.
Twitter has the famous fail whale.
Steemit has... nothing.
Not even a professional status page. :(
Don't forget, if Steemit.com goes down again in the future, try using busy.org or chainbb.com. Though the blockchain is distributed, the servers ("full RPC nodes") which interact with the blockchain and provide an API endpoint for websites to use are not well distributed.
@sneak said it's a DDoS attack. A DDoS is a Distributed Denial-of-Service attack. Via Wikipedia:
In computing, a denial-of-service attack (DoS attack) is a cyber-attack where the perpetrator seeks to make a machine or network resource unavailable to its intended users by temporarily or indefinitely disrupting services of a host connected to the Internet. Denial of service is typically accomplished by flooding the targeted machine or resource with superfluous requests in an attempt to overload systems and prevent some or all legitimate requests from being fulfilled.[1]In a distributed denial-of-service attack (DDoS attack), the incoming traffic flooding the victim originates from many different sources. This effectively makes it impossible to stop the attack simply by blocking a single source.
A DoS or DDoS attack is analogous to a group of people crowding the entry door or gate to a shop or business, and not letting legitimate parties enter into the shop or business, disrupting normal operations.
These are certainly no fun. There are services that can help mitigate them such as Cloudflare. Also, during an outage like this, it should be fairly simple to update the DNS for steemit.com to point to a static page on a global CDN. That way people at least know Steemit, Inc is aware of the issue and working on it.
It seems we're back up now, and the IPs for Steemit.com have changed.
During the outage:
steemit.com. 59 IN A 54.164.45.228
steemit.com. 59 IN A 54.165.162.8
steemit.com. 59 IN A 34.200.95.128
steemit.com. 59 IN A 34.200.97.71
steemit.com. 59 IN A 34.196.252.62
steemit.com. 59 IN A 34.202.98.91
steemit.com. 59 IN A 52.73.91.40
steemit.com. 59 IN A 54.164.250.243
After the outage:
steemit.com. 59 IN A 52.204.202.47
steemit.com. 59 IN A 52.20.108.115
steemit.com. 59 IN A 34.229.26.131
steemit.com. 59 IN A 34.236.106.200
steemit.com. 59 IN A 34.231.145.134
steemit.com. 59 IN A 34.231.139.15
steemit.com. 59 IN A 34.230.175.154
steemit.com. 59 IN A 50.16.114.93
How long will it be before these IPs are targeted for a DDoS also? I'm a bit frustrated because a 9+ hour outage is a big deal. Having no status page is a big deal. Responses like this from the official Steemit twitter account are less than professional. No updates from @steemitdev and only one update from @sneak (who many may not know is a Steemit dev) 8 hours into the problems.
So yeah, I'm venting here a bit. For over a year, I've been a huge fan of Steemit and defender of Steemit, Inc. I've been called a fanboy (among many other more colorful things) for supporting them. I've never powered down my account because I believe in the future value of this platform and having run my own software as a service for over 10 years now, I understand how hard this stuff is.
I hope the Steemit team responds well to this issue and explains what happened and what steps they took to prevent it from happening again. I also hope they outline their plan for better communication and status updates in the future.
And, most of all, I want our own mascot of failure.
What animal should we use? Leave your ideas in the comments.
Luke Stokes is a father, husband, business owner, programmer, and voluntaryist who wants to help create a world we all want to live in. Visit UnderstandingBlockchainFreedom.com
The Steemit Shark!
"Uh-Oh"
"Steems like we are experiencing technical difficulties."
Love this. Here's my first take on it ;)
https://steemit.com/steemit/@voronoi/steemit-com-shark-website-error-mascot-proposal
Love it!!
I like it! Let's make it happen.
If I had the technical ware withal to do so, I would. I'd be happy to throw some SBD to whoever does make it happen though :)
Yes, we need a "Mascot of Failure." Since we're all about the ocean environment, why not build on that? A deflated whale? A whale with birds and stars circling its head, like it just bumped into a wall? Perhaps even make it a community challenge-- we have lots of talented artists and graphic designers in the community.
We also need a formal status page (something like "steemstatus.org") that lives completely independently of the main site and can become the default load when we have something like this DDoS attack happens. Stick it behind every Cloudflare fence you can muster so it has 100% uptime.
I'm a BIG fan of Steemit and sing its praises all over, but we have-- or are developing-- a problem. We have waaaay too many "developers" and "apps" and not nearly enough designers/artists, human factors and usability people involved in this project.
Steemit will never grow beyond niche status unless some serious front-end work is done on the user experience. What just happened might be OK for some hard core users with some tech knowledge, but let us not forget that the next Steemit user will be just a little less technically inclined, and the 10 millionth next user even less so... if it isn't intuitively "point and tap" they are going to give us 17 seconds and then be gone.
Seriously, I love this place... but someone(s) needs to understand that "scalability" isn't just about transactions per second, it's also about creating a user experience that makes all those transactions necessary.
Meanwhile, a big YES to a "Fail Mascot!"
I love the fail pages on Reddit!
Back when I was working as Support for @lukestokes' FoxyCart, I remember being tasked with sending personal emails to customers that were affected by errors and outages. In it, we included a detailed explanation of what went wrong, and we even sifted through the database to find the exact transactions that were affected. Granted, that's what's needed for an e-Commerce company, but the way it's handled (especially behind the scenes) is just so meticulous. Some of my work was off-hours to cater to non-US customers, but Luke and his co-founder Brett would go online mere minutes after the error and would get on it as soon as they can.
They would send out notifications and inform the customers that the company is aware of the error and are working hard to solve it. Attacks and outages cannot be avoided, so the best course of action is to keep the lines of communications open to reassure customers that they're not being forgotten. I don't work for FoxyCart anymore, so I'm not patting my own back. But, the experience during outages has truly become a benchmark of mine in terms of how to properly handle those kinds of situations. Sure, there were frustrated customers (that can't be avoided), but they remained as customers because of the compassion towards their plight.
I just wanted to put this out there as a testimonial that Luke knows what he's talking about with regard to situations like this. I spent hours yesterday thinking that it's a problem on my end. The lack of communication from Steemit Inc has been a real sore spot for me for quite some time now.
The least that could be done is to have that sort of status notification to tell people that the situation is being addressed. I really like @nanzo-scoop's suggestion for a mascot.
How about a Lobster? As in Steemed Lobster.
A Steemit badger?! Honey badgers are acceptable too...
I think Bitcoin owns the honey badger meme, but I do like this one the best so far. The flip side is, badgers never fail. They are always up and running.
Hahah yes! Well this picture is a typical badger, they're more beautiful than a honey badger but waaaaay less badass (still badass though).
So my next entry would be fish related, so we could consider something muscular and scary like the tattoo that Grandpa has in the Simpsons - Hellfish! Pretty relevant and a cool name, might need some tweaks to avoid any copyright.
Failing that, some mean looking shark? Hammerhead...hitting his head on a nail. Wait, no, hitting the nail on the head is a good thing. Trying to hit and MISSING a nail!?
Hah! I like it!
Yeah, I agree, something ocean related makes the most sense. A shark chasing away all the minnows, whales, and dolphins could work nicely.
Poor steemy.
Save Our Steem?
I'm still getting the hang of all this blockchain lingo .. I do agree the a failure mascot would be awesome lol good point ! Maybe A pirate ?? Since their are whales, minnows and dolphins hmmm idk a pirate attack would seem to fit since we're in the ocean ? Lol. I'm just spitballing here but il shut up before I go deeper haha
Did you see the report that it was a self inflicted DDoS?
Quote Removed for lack of proper source information
I am copying this from third party sources and I don't know where it came from originally, @sneak has denied it in a comment to a post by @sircork HERE which he flagged into the grey.
For most of us, a Disruption of Service is a minor inconvenience. But unlike other Social Media sites, there are people for whom this is their entire livelyhood, so a 9 hour+ service interruption with no good explanation is a pretty big deal.
I do hope we see a transparent explanation of what happened at some point. But human nature being what it is, I don't expect we will get it.
Dang, that's quite a claim. If it's a lie, it deserves to be flagged for spreading FUD. If it's the truth, then sneak should explain himself.
Being a .NET stack geek that is between jobs, this is the 2nd outage I've seen in the last few weeks. It is embarrassing to get into "distributed" computing / blockchain and then see it taken down like this. It can also be tough to get the non-geek social media masses to jump here for value when it looks rather fragile.
Wish I could help more. Once I land a regular paying gig, I might look into getting into the Witness program and/or writing some utilities. Looks like I have a few ideas where that might start.
You are right. And with that taken into account, I have removed the quote from your post chain since I cannot properly source it.
But either way, I do hope we get a valid explanation at some point.
Thank you, I'm glad you did. Too many people are too quick to spread information from random sources without proof or evidence. That has to stop.
Just when I thought Steemit is changing some font style and colors that must be taking a whole day. Wow user interactions, what a useful feature! Forgive me venting here. 😊
There must be at least some decent notification that reminds us our money is safe in the box, otherwise people might regret not giving their upvotes to us hungry writers instead. Awwww.
Please, validate the information you share before spreading FUD.
I couldn't care less about sharing FUD when food is what I need. 😊
@lukestokes
The information gleaned from a discord chat stream, claimed to be quoted from an unnamed steemit dev. I should have made that more clear, perhaps, but since it was so detailed, it seemed plausible enough. I really didn't give it a whole lot of thought. It seems a lot of people are giving it a lot of thought though, and it all seems rather overblown to me.
@trumpman writes posts about a whole bunch of weird and wonderful creatures if you need some inspiration... I thought the hydrothermal worm on his post on 7 Horrifying Micromonsters that will haunt your dreams would work nicely - after all, these site failures are a nightmare to most of us.
Wow! Thanks for the mention my friend :D
Love your work, so just doing my bit to share it with more people :) Besides, I think the idea was pretty appropriate given the subject matter :D I am curious to see what your mascot idea would be...
Dunno didn't even read the post , I was about to sleep and then got a notification on my phone about your comment. I will check it out tomorrow and try to come up with something good :D but now I am just so sleeeepppyyyyyyyyyyyy
Steemit should take a damn break from 'developing' and hire a fully dedicated team to focus on the damn website. Give it a face lift, improve it, listen to what the users are asking. The website is the entry door to Steem, if the door doesn't open properly, what does it tell the world about the platform? Any respectable business know the value of their website and are constantly and vigilantly paying attention to it, Steemit Inc must do the same. Look at Microsoft or Amazon for inspiration, or dare I say it... Facebook.
I agree that the in future, steem will grow huge.
Keeping that in mind I suggest we should use a "Giraffe".
Because we all believe that steem will go up like a giraffe's neck in the near future. @lukestokes
Singing Steemit Squid:
[source]
an evil fisherman, like Achab? :D
The unicorn is beautiful! I vote for a unicorn!
Nope. Need some imagination
I offer this to all a familiar woman, but do you know why? because it is NOT AVAILABLE, as is the page.
It should be a pissed off Koala Bear, like this:
source
Whoa, that's friggin scary!
I was looking for a cute Koala pic and saw this. If that's not a fail mascot, I don't know what is! lol
This may give me nightmares. Dang.
Much more memorable than the FailWhale! :)
I agree that the information during the downtime was pretty bad. I found most of the info by browsing the Steem blockchain on chainbb, since there was little official information available.
A fail mascot would be pretty cool!
I agree. Some info would have been nice! thanks for the links above.
So so sad.
Something like a bunch of small fishes in a big ocean overflooding. With the message: "Too less water here", "please hold on while we refill the ocean."
Good article! It is indeed the way everybody must have felt today when refreshing the steemit page and see the 503 error code. Unprofessional, but this kind of stuff gets easily fixed by a waiting image. Maybe something like a bird or wahle should be suitable since they are a mark of freedom and same time they are cute.
Great post @lukestokes ... I totally agree with you... We should use a big bear...as animal
Congratulations @lukestokes! You have completed some achievement on Steemit and have been rewarded with new badge(s) :
Award for the number of comments received
Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here
If you no longer want to receive notifications, reply to this comment with the word
STOP
The mascot shoult be a whale's vomit. Okay not really.
Omani fishermen stumble upon 80kg of ‘whale vomit’ worth $2.5 million
http://www.hindustantimes.com/world-news/omani-fishermen-stumble-upon-80kg-of-whale-vomit-worth-2-5-million/story-cTNKd26745Cnxmk5gmuESL.html
WUT.
That's crazy.
Ambergris! Super big deal in the high end perfume and scent essence industry. $20,000-25,000 a pound is not an unusual price.
How about this one?
@lukestokes voicing some frustration!?
Uh oh! Things are bad folks.
Edit: Welcome to the dark side Luke! lol
Hahah. I can put up with quite a bit, but 10 hours of downtime with no error page? That's too much. Sneak says a CDN introduces unacceptable risk. I think that's a bit shortsighted/paranoid. Anyway, hopefully Steemit will get stronger because of this.
Yep, reeks of amateur hour.
Sleeping panda for me LOL
Luke the Mascot is too easy. It has to be a dead, decaying fish belly up in the water.
The twitter response from steemit was amazing. Just because a system is decentralized, doesn't mean the customer support shouldn't be professional. Years ago I had a helpdesk group working under me. They would have been fired for a response like that . But then it's an anonymous account, basically. How does a blockchain go down for 8 hours?
The blockchain didn't, but the steemit.com front-end interface did. Busy.org and Chainbb.com were still working and the blockchain itself was fine.
Please this. I know Richard Branson has nothing to do with Steemit but to me, it just works...
I don't... what the... what am I even looking at here???
Holla! I hope the answer on twitter was only due the high stress level from one individuum. I try to be objective and think of issues i have to handle with customers. I'd never offended a single customer as i remember, even in the case the customer made a mistake and was impatient. I always try to deescalate things to look for a solution as fast as possible, be calm and friendly even if i'm self quite unhappy with a situation.
Especially if we look at such a fantastic system working on the blockchain, each dev can be proud of and can always stand above such bad emotions. From the perspective of the core development i would say, it's simply too important and great to give such bad emotional states a chance to unleash to the public. The offending style with big letters makes absolutely no sense for me because the point Diego Pucci made was pure logical reaction, nothing to concern about in terms of offending someone. It looks like trolling by hackers to create FUD.
The 10 hours of downtime was bad, but that Twitter reaction on the official steemit inc account was unacceptable!
I keep seeing talk of trying to get big time million dollar investors into steemit... 10 hours downtime, no status updates etc and that Twitter reaction. I think I will know their answer 😞
Oooh THE TERMS... you and Capt. Crypto would be GRAND to be in the same room together! LOL Ps. He told me he was going to sign up for Steemit and I almost cried. I really did. I begged him not to... because the amount of time it really takes to be successful at this (which I am not)... well, I'd just truly never actually see his eyeballs looking directly at me. #facepalm
Yeah, @corinnestokes often says she puts up with it because at least my hobby makes money for the family. Heheh. :)
yep and it looks like she's got herself a little hobby too! ;)
BAHAHAHAH best comment thread ever! I was cracking up! So funny! Great suggestions all around! The smashed tea kettle and train wreck might have been my faves, though maybe the shark, Hellfish, or dead whale might be more appropriate! ;)
I'm not sure I get it...
me either...