While last week was a really good week for the markets - Dow up 3%, Nasdaq up over 5% - it was a really lousy week for StockCharts.com. As most of you know, the site had problems creating charts on Thursday and again on Friday and then we were also down much of the day on Sunday.
Soon after the market opened on Thursday, we suddenly stopped getting data from both of our data feed servers. It was an extremely unusual and unexpected situation. Why would two separate computers suddenly stop communicating at the same time? Another frustrating part of the puzzle was that as soon as we took the web site offline, the problem went away. It would only happen when real people were using the website to create real charts while the real market was open.
Most of Thursday was spent trying the "standard" approaches to finding and fixing such a problem. Unfortunately, this problem was resistant to those techniques and on Friday, when the market opened again, the problem resurfaced. Friday was spent trying more and more "drastic" measures and eventually, around 3PM Eastern, one of those techniques worked. The site was up for the final hour of trading on Friday and, knock on wood, it appears to be back to its old normal self again as I write this on Monday morning.
"No good deed goes unpunished" as the saying goes and this was no exception. Just as we started to recover from our marathon troubleshooting sessions, something else unusual happened. On Sunday morning, for the first time in our six and a half year history, we had a long power outage in our data center. Now, you are probably familiar with the heavy UPS battery devices that you can buy to protect your computer in case of a power failure. We have something similar here - except that our UPS can keep our 40+ servers running for about 3 hours (yes, it is REALLY heavy too!). Unfortunately, this power outage lasted much longer than that and it knocked us off the air. Once the power came back on, we were able to get the site up and running within an hour, but it wasn't until Monday morning that everything was fully functional again.
Again, we really do understand how frustrating and difficult it is for our users when our site isn't working correctly. We have worked very hard to get things back up quickly while keeping people as informed as possible about this situations. Going forward, we will be conducting a "post-mortem" review of our procedures to find ways to minimize the chances of these kind of problems occurring again in the future. We will be looking into things such as secondary data vendors, large scale power generators, scheduled maintenance periods and more. In addition, we have already added one week of free service to all active members' accounts as our way of saying "Thanks for hanging in there with us." Finally, we sincerely apologize for whatever inconvenience these outages have caused.
Oh, and all of the above "stuff" also explains why this newsletter is coming out later than usual. And we apologize for that also.
OK, so back to the markets and the market commentary...
