The storage vendor have not been able to recover the second node with the spare hardware we have onsite. They have organised a shipment of spare parts, however this is coming from interstate and will arrive on Monday morning at the earliest.
Once the hardware arrives, we will perform basic verification and then work with the storage vendor engineers to develop a plan to introduce and further verify the new hardware.
In the event that the Storage Cluster recovers correctly, we expect to be able to start verifying data by Tuesday morning at the earliest. Our engineers are already working on extracting available data from volumes on the Storage Cluster.
In the mean time alternative infrastructure is ready to go for customers with backups available. Please contact our support team if you would like to be restored from the latest backup available.
Please note that after restoring a service from backups we will not be able to restore any data for the service from the Storage Cluster when it is back online.
When requesting a restore, please contact our support team at firstname.lastname@example.org
, so that our System Administrators can work on the restore as quickly as possible.
We have scheduled System Administrators to be available to work on this continually until this issue is resolved.
As we do not expect the situation to change significantly in the next 40 hours, our next status update is expected to occur no earlier than Monday 09:00 AEDT (Sydney time).
In the event there is a significant change we will update here accordingly.