We're investigating an outage that is affecting users on multiple servers. We'll post more info as it becomes available.
Edit @10:43am - This issue is affecting users on s4, s7, s11, and s15.
Edit @11:05am - We've identified the issue as a hardware issue and are working to restore access as quickly as possible.
Outage affecting multiple servers
We are still working with the datacenter to get the server back online as soon as possible.
Want free Ad Removal? Find out how here.
12:39pm update - We're working with the datacenter as they are replacing the affected hardware.
2:12pm update - The hardware has been replaced. We're working to bring the machine back online and should have a further update soon.
3:53pm update - We're still restoring software at this time. We're working as fast as possible to restore access to your communities.
4:27pm update - The server we use for the support board is assisting in rebuilding the software on the server affected with the outage. The support server experienced a brief period of inaccessibility due to network congestion during this process. That has now been resolved and were continuing our work on restoring access to communities on s4, s7, s11, and s15. Thanks again for your patience.
2:12pm update - The hardware has been replaced. We're working to bring the machine back online and should have a further update soon.
3:53pm update - We're still restoring software at this time. We're working as fast as possible to restore access to your communities.
4:27pm update - The server we use for the support board is assisting in rebuilding the software on the server affected with the outage. The support server experienced a brief period of inaccessibility due to network congestion during this process. That has now been resolved and were continuing our work on restoring access to communities on s4, s7, s11, and s15. Thanks again for your patience.
The hardware failure this morning corrupted the database on the server running communities on s4, s7, s11, and s15. At this point we have repaired the hardware and are working on restoring both the software and the database. Thanks to a variety of our backup protection measures we currently do not believe there will be data lost. However the process of rebuilding the server and database is unfortunately a slow one. It has forced us to keep communities offline while the work is being completed. We will continue to update you on the progress of this restoration, and again we thank you for your patience.
Want free Ad Removal? Find out how here.
The database is still rebuilding and Brandon is continuing to work to bring the server and communities back on line as soon as possible.
We are continuing our work to restore access to the communities affected by yesterdays outage. The outage itself was caused by an abrupt hardware failure which led to a corrupted database. The defective hardware has been replaced and we're working to fix the database.
We maintain multiple layers of backup systems in the event of issues like this. However in this case, our primary backup, which would have allowed us to restore access much more quickly, was also unusable.
As a result we've had to move to our secondary backup systems. While these are complete backups and will restore all data, they take much longer to bring back online.
We currently have 2 steps remaining in the restoration process. Based on performance in the second to last step we expect that step to finish this afternoon. Once that is done and the final step begins we'll be able to provide a clearer estimate for full access to be restored.
We understand that this is a difficult experience for your communities and we're doing everything we can to restore them as quickly as possible. Thank you again for your patience.
We have moved on to the final step. The initial completion estimate is the morning of Wednesday (Eastern Time). I will continue to provide updates throughout the evening as I am able to provide more accurate estimates.
The final step continues to run smoothly. As of now, @3:59am, the current completion estimate is around 10AM (Eastern Time).
We maintain multiple layers of backup systems in the event of issues like this. However in this case, our primary backup, which would have allowed us to restore access much more quickly, was also unusable.
As a result we've had to move to our secondary backup systems. While these are complete backups and will restore all data, they take much longer to bring back online.
We currently have 2 steps remaining in the restoration process. Based on performance in the second to last step we expect that step to finish this afternoon. Once that is done and the final step begins we'll be able to provide a clearer estimate for full access to be restored.
We understand that this is a difficult experience for your communities and we're doing everything we can to restore them as quickly as possible. Thank you again for your patience.
We have moved on to the final step. The initial completion estimate is the morning of Wednesday (Eastern Time). I will continue to provide updates throughout the evening as I am able to provide more accurate estimates.
The final step continues to run smoothly. As of now, @3:59am, the current completion estimate is around 10AM (Eastern Time).
| Server Status as of 6:50pm | |
| zIF/IF 4 | Online @ 8:59am |
| zIF/IF 7 | Online @ 9:03am |
| zIF/IF 11 | Online @ 8:56am |
| zIF/IF 15 | Online @ 9:10am |
| ZetaBoards 4 | Online @ 10:28am 10:30am ETA |
| ZetaBoards 7 | Online @ Noon 11:45am ETA |
| ZetaBoards 11 | Online @ 6:49pm 6:45pm ETA 6pm ETA |
| ZetaBoards 15 | Online @ 10:42am 10:45am ETA |
Want free Ad Removal? Find out how here.
All servers are now back online. Thanks for your patience while we worked to resolve this outage.
Please submit a Support Ticket if you encounter any errors on your forum.
Liam
Please submit a Support Ticket if you encounter any errors on your forum.
Liam
