Advanced Access Outage
May 21st, 2007 by Advanced AccessAt this time, Advanced Access is having significant power issues at our offices and all services are currently down. 100% of our resources are dedicated to solving this problem with many various solutions being implemented at this time. There has been no data loss or corruption of your services.
We are expecting that all services will be shut off for the rest of the day. We do anticipate all services to be running tomorrow, Tuesday May 22. We sincerely apologize for the downtime and are doing our best to get you up and running as quickly as possible.
Thank you for your business and support. We will continue to post updates here as we receive them and are able to post them from sources outside the office.
No corruption of services? Thats the understatement.
Up and running by tomorrow WOW, THANKS!!!!! my site…an entire day, MIA.
Darla
May 21st, 2007
Gee, you would think that you would have power backup systems in place. I have not had an email since 12:54 pm and now will not until perhaps tomorrow??????
I am glad it is the time of the year to reevaluate my webhost…..before I pay again.
Ken
May 21st, 2007
got to pay the electric bill…
Steve O
May 21st, 2007
One of the first questions we usually receive is about your data and we want it to be assured that everything is not harmed.
We do have many backup systems in place but because of the power issue they were not able to work. We have an offsite hosting building ready to be moved which was scheduled for Sunday but due to this unexpected problem we moved it to today.
I hope that things will be up later tonight but I don’t have a status update. I will definitely post again when I have something concrete.
Kristina
May 21st, 2007
I am discouraged by delay on 4.0, my site vanishing from yahoo, my site and AA site being down today, and nobody answering the phones. I am not feeling any love.
Evan
May 21st, 2007
The customer service rep I spoke to said his understanding was that it was not just an electrical issue. What is it then?
Dan Connolly
May 21st, 2007
Very disappointing.
Glenn
May 21st, 2007
Hi Dan, it was the main power source supplying power to our servers. I assure you that is the core issue. That is a little different than “electrical” - meaning we did not lose all power to the building.
Evan, I apologize for these issues sincerely. We have several folks still manning the phones, and I was manning them personally both as the main switchboard and fielding specific calls. Our lines are so loaded with incoming calls that perhaps you were unable to get through for a while. The calls have eased up a bit if you would like to speak with someone on the phone. However I can assist you here as well with any questions you may have.
Anna
May 21st, 2007
Hard to believe in this day and age that something like this could happen to more than 30,000 sites all at once.
Phoenix
May 21st, 2007
What credit is AA offering to its 40,000 customers?
Roberta Murphy
May 21st, 2007
Wonder how many e-mails and leads we’re going to lose this time around.
Let’s see…Yahoo penalty…Google penalty…continuing e-mail problems…Quickhomes…4.0 promises with no production…
Why are any of us still here? A mis-guided sense of loyalty?
Gene
May 21st, 2007
Sorry to tell you this Roberta, but even a server that promises 99.9% up time is allowed something like 525.6 minutes a year down time and still within their claim.
Read more here:
www DOT ibsteam DOT net
Phoenix
May 21st, 2007
You have impacted customers greatly, and you quote stats. Obviously you and your team have not planed any “BASIC” disaster plan. My next e-mail will be to
Kristina Morrision, CEO, the so-called Industry Visionary. oh that is if here e-mail works, I will aslo being cc’ing The Associated press and other national media outlets. Thanks for the substandard support of your clients.
Al
May 21st, 2007
Hello everyone… lots of comments and I certainly am not surprised. I will try and check in here as much as possible this evening.
We have added a new post here http://www.blaawg.com/2007/05/21/advanced-access-outage-update/ which I think will answer some of your questions as far as backups, etc, and it’s a general update.
Our own email and internal systems are currently down, it’s true. I have seen on other forums (as Matt mentioned) talk of downtime statistics. I wanted to be sure and note that this is actual downtime, and we have never had severe downtime such as this. Yes we have other programs that are being worked on, but they have not be “down” for 2 months, etc (as I have read elsewhere). They have been very much up and running; some needing a facelift with regard to certain areas of functionality, but up nonetheless. Not the same as down (sounds silly?). Yes we have had server replacements or reboots, but not systemwide prolonged outages. Perhaps my clarification is misguided, but it is simply a clarification for the sake of itself. I expect that some may take issue with that… it’s ok, but please know I am just trying to make sure we have facts coming to you from a technological standpoint.
Al - how can I serve you better here? This is my only platform from which to serve you currently and I will be happy to do whatever I can. If you’d like I’d be happy to speak with you on the phone or via private email. Just say the word. While I can’t access systems now, I have the ability to take notes and will assist you further when things are live. I wish there was more I could do.
This does happen with hosting and technology, no we are not the only ones… but it doesn’t make it any better or any easier for our clients. And we get that, we truly do.
Our entire IT and Software team as well as myself, our General Manager, and many others are going to be up all night working on this for you. As soon as we have systems on I’ll be testing as well as some other folks on call.
Maybe this is a lot of info, I just wanted to tell you as much as I can.
With my sincerest thanks to you all,
Anna
Anna
May 21st, 2007
Al >>-hate to tell ya this but the Kristina posting here is one and the same Kristina Morrison CEO-good luck. My letter will be sent direct to Dominion Enterprises.
Matt >> thats a bunch of crap!so AA could put my site to sleep for what would amount to 21.9 days and have no accountability?
Roberta >> yeah by my calculations AA takes in each month from hosting alone-not counting QH or any other add-ons; approx. $1278400.00 certainly with that revenue, one would think they would have a solid back up plan in place.
Gene >> yup.
Anna >> thanks. whatever they are paying you…its not enough.
Darla
May 21st, 2007
I just found this site that has some great info on what a UPS is and what it does, etc. It’s in plain language too. http://www.jetcafe.org/~npc/doc/ups-faq.html#02
As you can read there the UPS is what is supposed to work for you should power go… our UPS had an issue… so our backup power could not be generated. Hope that helps those seeking techie info a bit more.
When the UPS kicks in, we get actual generators and hook up the servers to that. However with the UPS not working as it should, we had to act fast to keep everyone’s data safe. We had to make some very fast decisions.
Anyway, just techie info for those who seek it.
Anna
May 21st, 2007
Although I just found the issue posted on AA’s Activerain blog, I’m surprised that they didn’t bother to post anything until 12:41 PM. My site was down at daybreak and they alreday knew then that they had a problem.
Quote: “This entry was posted on Monday, May 21st, 2007 at 12:41 pm”
Now AA is busy moving the operation over to their backup location. Sounds too convenient to me.
Quote: “We have an offsite hosting building ready to be moved which was scheduled for Sunday but due to this unexpected problem we moved it to today.”
(Darla, it wasn’t me who posted the ‘Gene’ post above but my thoughts are the same)
Gene C.
May 21st, 2007
Quote from Anna “When the UPS kicks in, we get actual generators and hook up the servers to that”
Duh! Can you explain that a little better, please? Are you saying that actual generators are not already in place with an automatic transfer panel flipping the switch to go from house current to generator current? I worked on Kohler generators for many years installing and maintaining backup generators in hospitals, State Police headquarters, etc. and everything was automatic when the power went out. Can you tell all of us just what type of UPS AA had in place, such as the UPS size and length of runtime it had before you needed to ‘get actual generators”? Was it sufficient to run your entire operation and for how long?
Gene C.
May 21st, 2007
Gene C - Our office was/is entirely down as well which is why there was no post until noon. We were not able to get to a connection until that time.
Explanations on how our UPS server and generators work is not available at this time. We hope to have more information as we gather it.
Advanced Access
May 21st, 2007
(Darla, it wasn’t me who posted the ‘Gene’ post above but my thoughts are the same)
No Gene; it was me.
Gene Hahn, Realtor, ABR, e-PRO
Bullhead City, Arizona
Gene
May 21st, 2007
Devastating day without WEBsite– terrible lack of communication, and just feel lucky I stumbled upon this site for an update…Think we all should receive some compensation, or contract extensions. Will it really be up by the 22nd??
Melinda Dutton
May 21st, 2007
As a former manager for service technology on Server Systems, and later an IT Director for a dot comm, I have a couple of replies: First, 99.9% (known as 3 nines is not the accepted network reliability for major server manufacturers. At the company I worked for, we were held to 5 nines, which doesn’t allow you hundreds of minutes per year.
Second, obviously the UPS was a single point of failure, which is unacceptable for any web server farm.
Third, locating backup sites in the same geographic location, especially with both in proximity of the San Andreas fault line, is almost as absurd as having only one “Main” UPS.
I feel compassion for the UPS support department, but whoever is planning their backup and disaster recovery plans should be working for the local computer store, not a provider of commercial websites to 30,000 clients.
Gil Meacham
May 22nd, 2007
Anna, sorry you are having difficulties and will be up all night, really sorry the site I point everyone to will be down all night.
Teckie 101 - every mission critical business application such as your hosting center should be power protected. Simply put every credible provider of hosting service has redundant power supplies to their servers, major power supply is backed up by an Uninterupted Power Supply (UPS) system which is technically a battery, these tend to last only about 4 hours and is intended to hold over until the generators build into the system automatically kick in and take over providing a seemless transition of power and uninterupted service.
While true that most hosting sites have technical problems, that is why they build in redundancy…so customers never know about the behind the curtain stuff.
Tom B
May 22nd, 2007
PS you might want to Call Ron Rose in to consult - he is the CIO for Priceline.com and can give you a read-out on you contingency planning very quickly
Tom B
May 22nd, 2007
Well at 4.50pm (CT) on day 2, site & email still down.
Any projected time for system recovery?
Graham
May 22nd, 2007
If I lose search engine placement on Google because of this, what is AA’s plan to remedy the situation. My site is the top site for most all things real estate in Franklin TN homes, real estate, relocation, Realtors, etc. Is AA going to do free search engine placement to get my site back? The Google spiders last hit my site on 5/19. Looks like it is time for them to come back on their weekly visit. My site has been #1 forever. My site’s email is now BANNED from my ISP….COMCAST for SPAM. I use Intellicards from AA. I do not SPAM. I NEED HELP ON MY SITE!!!
Brandon Patrick
May 22nd, 2007