Yesterday (4/16/14) shortly before 1PM we began experiencing problems with our server. We noticed problems connecting to several websites and began investigating what appeared to be dropped packets and sluggish response times. By about 1:30PM the server began being intermittently unreachable and it went offline completely by 3PM. At that point all of our client websites (and ours as well) became unreachable.

While it is within the realm of normal to have occassional hiccups that may make a server unreachable for a few short moments, it became apparent there was a much more serious issue at play as minutes turned to into hours. Overall our server was unreachable for slightly over 22 hours. To be clear, we were not the only provider that was impacted. While we don't have exact numbers it is believed that tens of thousands of websites were brought down in this outage. Here in Vermont both VTDigger.com and VTLiving.com were offline for similar amounts of time.

We use dedicated servers that we source from HostGator - a leading reseller of hosting services [http://en.wikipedia.org/wiki/HostGator]. We have been doing business with HostGator for the better part of a decade both as Edgeworks Creative and with the businesses we had prior to forming Edgeworks Creative. We have had excellent service up until this last year after the company was sold to Endurance International Group [http://en.wikipedia.org/wiki/Endurance_International_Group]. After the sale to EIG our servers were moved from what had been a highly reliable data center in Dallas, TX to their current home in Provo, UT. While this data center is supposed to be state of the art, it did not prevent the outage that impacted our servers and your website.

We take reliability very seriously because we know that both your organization and our business depend on it. We've made significant investments in the last two years upgrading the equipment we use to host websites and are deeply disappointed that we experienced this lengthy downtime.

As a result of this outage we have been reviewing our current hosting infrastructure and setup with the intent of making changes to protect ourselves against this type of issue in the future.

In the short term we are adding redundant DNS to our hosting setup so those clients who host their email with GMail (Google Apps for Business) or other third-party email systems will not lose their email in the event our primary servers are cut off from the internet. We expect to have this implemented no later than this coming Tuesday as we evaluate options and find the one that best fits our needs and budget.

Within the coming weeks we will be switching hosting providers away from HostGator (or any other hosting company owned by EIG). We are investigating our options and will notify customers when their websites will be moved. The failure to keep those of us impacted by the outage fully informed of the problem (shutting off support chat, phone lines, help forums and support ticketing) leaves us with a very bad feeling about the direction this company seems headed. Full disclosure of the problem has still not been provided, but it seems likely that the problem could have been avoided through best practices for rolling out firmware updates at the data center. 22 hour outages are not acceptable to us. We deserve better and you do too.

In addition to moving away from HostGator, we are looking very closely at setting up fully redundant servers in a second location. Combined with the secondary DNS we would be able to fail over to backup systems with minimal downtime. The technical and financial commitment to accomplishing this may mean that this would be available as a paid option for those whose websites are absolutely mission-critical.

Hosting has never been a profit center for our business. In fact, we only recently broke even on our hosting expenses vs. hosting income. Moving to a fully redundant fail-over setup will be expensive and will put our hosting back in the red for years to come, but we want our clients to know that we are doing everything we can to provide the kind of service you deserve.

If there is anything we can do or if you have any questions whatsoever please do get in touch with us. You can call, email or connect with us on social media. We regularly post updates to our Facebook Page [https://www.facebook.com/edgeworkscreative] and recommend that you "like" us in order to connect with us there. We use Facebook and Twitter [https://twitter.com/edgeworksvt] as a primary means of communication during both anticipated maintenance windows and unexpected outages.

We deeply appreciate our clients and are commited to continuing to improve our services.