We encountered a server failure late yesterday and are working to resolve the issue. We are doing our best to get things back online and in a normal operating mode. While our new Amazon Web Services environment is supposed to be a much better reliable solution, they provided us with this message this morning:
Your volume experienced a failure due to multiple component failures and we were unable to recover it. Although EBS volumes are designed for reliability, including being backed by multiple physical drives, we are still exposed to durability risks when multiple concurrent component failures occur before we are able to restore redundancy.
Obviously not something we wanted to hear, but we do understand computers and hardware will still fail. Please accept our apology for the downtime and just know we are working as fast as we can to get things restored.
We will continue to add updates to this post.
9/30/20 – Update
We have been restoring older recordings in batches and it will continue to run until we’ve restored everything we can. All new recordings are working as normal. If you have an older file, just give it a little time and it should update soon.
10/6/20 – Update
Amazon has been worthless in helping us. They actually tried to blame us for their hardware failing. We have been able to restore all uploaded recordings for the past year which include Ad-Free Ticket shows. We’re still restoring Ticket and other recordings for the month of September. We’re trying to prevent overloading the server that processes things so it’s a slow process. As for recordings beyond that point, we have the raw data for those for the past year but we’ll be restoring on a per request basis. If you are missing old recordings you need that happened between September 17, 2019 to present, shoot us an email email@example.com and we’ll work on it. Be sure to include your account email, recording date needed and show name if you have it.