Comments on: The Meltdown That Brought Our Startup to Its Knees for 15 Hours

By: radfarrokh

radfarrokh — Mon, 20 Apr 2015 10:21:17 +0000

I read this story. You have a successful team. I congratulate you..کرکره برقی-راهبند اتوماتیک-لامپ کم مصرف

By: Josh

Josh — Wed, 08 Apr 2015 15:38:44 +0000

Are you guys still using statuspage.io?

By: Stéfano Willig

Stéfano Willig — Wed, 19 Nov 2014 15:26:54 +0000

Just at the moment. Passing through this situation.
Using your experience to manage situation.
THANKS AGAIN, ALEX!
ONE DAY, I WILL TO SEND YOU A GIFT!

By: Jerome Gentolia

Jerome Gentolia — Tue, 11 Mar 2014 15:18:59 +0000

Whoa! I almost died of a heart attack reading this. This must have been a stressful episode for you. Some people would have convulsed in fever and delirium. You on the other hand managed to keep yourselves composed.

Hopefully your transparency is enough to mollify customer’s ambivalent feelings. You did the right thing as a non-techie by helping answering the barrage of emails (it is a priority to communicate the problem almost immediately). Although it took several hours for you to respond, I think you’ve handled it deftly as soon as you’ve become aware of the problem.

Lots of lessons to learn here, I am sure every startup out there can learn something here. It’s all part of startup’s growing pains.

Thanks for sharing!

By: Vishal Nath

Vishal Nath — Mon, 10 Mar 2014 22:08:36 +0000

Hi Alex,
I totally understand what you must have gone through during that outage! Glad you guys managed to restore the systems properly. I must say transparency is a key to success! As a CIO of a SaaS company its my job to be on the ball every time something like this happens. My recommendation would be to have a couple of monitoring tools on your whole infrastructure. I use Pingdom and CopperEgg. Pingdom is great to keep you notified with all the domain outages and you can also set the same probes in CopperEgg so that acts as a backup to Pingdom. After all its a software and it can also go down so having redundancy at that level pays as well. Key is having a backup to everything you do 🙂

CopperEgg itself is a great monitoring tool as it not only monitors your domains but CPU, Memory, Disk space and Network activity as well. You can setup your thresholds so you get notified via email and push notifications on all the different things mentioned above. In this business model its paramount to have visibility in all this areas so you can see the potential issues before they happen and also plan for the future 🙂

I have so many different custom metrics hooked into CopperEgg which helps me to keep a virtual eye on things on daily basis and we all can sleep easily without worrying whether the systems are up or not.

Would love to have a chat to you at some point and give more ideas on things we do 🙂

By: Alois Reitbauer

Alois Reitbauer — Mon, 10 Mar 2014 15:30:47 +0000

Alex,

although your post is kind of tragic, it is a great inspiration. We are about to launch an Operations Monitoring as a Service company. Our goal is to help startups to avoid these situations without the need to build a big 24/7 operations team. You just make the perfect business case for what we are doing. May I use your post as a reference in my upcoming “why we do it” blog series?

Hope this never happens to you guys again. You really do awesome stuff.

By: Alin Tuhuţ

Alin Tuhuţ — Tue, 04 Mar 2014 08:30:05 +0000

Good to hear you’ve recovered gracefully!
For your monitoring stack you should also check http://www.monitive.com It’s a reliable startup too, built by a good friend of mine. You will get SMS notifications and even phone calls when something goes wrong.
Good luck with Groove! You’re really doing a great job!

By: branedy

branedy — Mon, 03 Mar 2014 19:23:53 +0000

Do yourself a favor, startup a small instance and install Nagios, AWS
monitoring is not reliable, or consistent. and get a sms server for
Nagios to send to. Then set it up in a different zone from your other
systems, so an AWS zone outage doesn’t get you.

By: Eric Dosal

Eric Dosal — Sun, 02 Mar 2014 01:15:41 +0000

Alex – I’ve been following your blog over the past few months and this one is probably my favorite. Not only did I find it insightful but extremely educational. Great work.

By: harishchouhan

harishchouhan — Fri, 28 Feb 2014 14:41:14 +0000

I had faced the issue with one of my services a while back. Even since than had used http://www.panopta.com for creating different rules to notify team members.

By: Ramli John

Ramli John — Fri, 28 Feb 2014 13:44:19 +0000

Trial by fire! Awesome you and the GrooveHQ team got through it, Alex.