Update - Our technicians have identified a method to successfully transfer and restore VMs from the SolidFire storage cluster onto the new infrastructure and are prioritising this method to bring services back online. They are working through the list as fast as possible.

Those VMs which are not able to be restored in this way will need to be restored from backups. If you have already requested a restore from backup, we will first attempt to recover your data from the SolidFire storage cluster as this has proven to be the best approach.

We have scheduled System Administrators to be available to work on this continually until this issue is resolved.

We continue to wait for replacement hardware to arrive from NetApp which we hope will assist in the recovery of data also. This will happen most likely on Tuesday.

Our next scheduled status update will likely occur no earlier than Monday 09:00 AEDT (Sydney time).
Nov 16, 21:43 AEDT

Update - NetApp have not been able to recover the second node with the spare hardware we have onsite. They have organised a shipment of spare parts, however this is coming from interstate and will arrive on Monday morning at the earliest.

Once the hardware arrives, we will perform basic verification and then work with NetApp engineers to develop a plan to introduce and further verify the new hardware.

In the event that the Solidfire Storage Cluster recovers correctly, we expect to be able to start verifying data by Tuesday morning at the earliest. Our engineers are already working on extracting available data from volumes on the Solidfire Storage Cluster.

In the mean time alternative infrastructure is ready to go for customers with backups available. Please contact our support team if you would like to be restored from the latest backup available.

Please note that after restoring a service from backups we will not be able to restore any data for the service from the Solidfire Storage Cluster when it is back online.

When requesting a restore, please contact our support team at support@crucial.com.au, so that our System Administrators can work on the restore as quickly as possible.

We have scheduled System Administrators to be available to work on this continually until this issue is resolved.

As we do not expect the situation to change significantly in the next 40 hours, our next status update is expected to occur no earlier than Monday 09:00 AEDT (Sydney time).

In the event there is a significant change we will update here accordingly.
Nov 16, 17:16 AEDT

Update - Netapp have not been able to recover the second node with on site spares. They have organised a shipment of spare parts but this is coming from interstate and will arrive Monday morning at the earliest.

We are also discussing other options with NetApp to see if any further action can be taken with the hardware we have on hand.

In the mean time alternative infrastructure is ready to go for customers with backup services. Please contact our support team if you would like to be restored from the latest backup available as soon as possible.

When requesting a restore, please contact our support team at support@crucial.com.au, so that our System Administrators can work on the restore as quickly as possible.

Please note that after restoring a service from backups we will not be able to restore any data for the service from the Solidfire Storage Cluster when it is back online.

We will post a status update at 17:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 13:11 AEDT

Update - We are continuing to work closely with NetApp, and have been carefully executing a recovery procedure. This is taking longer then anticipated because we are trying to recover as much data as possible.

We are still looking at options to source replacement hardware as we have exhausted our spares for a particular hardware component.

At this stage we are still not able to provide an ETA on Virtual Machine Recovery.

We will post a status update at 14:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 11:40 AEDT

Update - We are continuing to work closely with NetApp, and have been carefully executing a recovery procedure.

This is taking longer then anticipated because we are trying to recover as much data as possible.

At this stage we are still not able to provide an ETA on Virtual Machine Recovery.

We will post a status update at 11:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 09:37 AEDT

Update - NetApp Engineers have provided a recovery procedure which we are carefully executing.
We will provide further updates on this process and an ETA on Virtual Machine recovery at 9:30 AEDT (Sydney time) or when there is a significant change in the situation.
Nov 16, 07:49 AEDT

Update - NetApp Engineers are testing the recovery procedure in their U.S. test lab to find the safest way to recover the storage cluster with minimum data loss.

This process is taking longer than anticipated.

We expect further news at 07:00 AEDT (Sydney time) , and will update the virtual machine recovery ETA at that time.

We will post a status update at 07:00, or when there is a significant change in the situation.
Nov 16, 05:12 AEDT

Update - Recovery efforts are continuing.

We still anticipate that we'll be able to start bringing Virtual Machines online between 04:00 and 05:00 AEDT (Sydney time).

We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 16, 03:07 AEDT

Update - We have suffered a 2 node failure with our SolidFire storage cluster. Although the cluster is designed to handle a multi node failure, the automated failover and recovery process has not worked as expected and has required manual intervention which includes repairing the failed nodes.

We have been actively working with the storage vendor (NetApp) in order to restore the storage cluster. Progress has been slow and careful as the priority is to prevent as much data loss as possible.

So far 1 node has been repaired and reintroduced to the cluster and at this stage, we've been able to recover almost all volumes on the cluster. NetApp staff have been working with their Engineering Team attempting to bring the remaining volumes online.

In order to minimise the potential of any unrecoverable volumes, NetApp staff has requested that we do not bring active workload on to the storage cluster until their Senior Engineering Team gives the go ahead.

We anticipate that we'll be able to start bringing Virtual Machines online between 04:00 and 05:00 AEDT (Sydney time).

We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 15, 22:30 AEDT

Update - Recovery efforts are continuing.

We are continuing to work with the storage vendor to recover the cluster. Engineers are currently performing verification procedures. We are unable to provide an ETA at this stage.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 15, 19:44 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster. We have made some progress, bringing one of the failed storage members back online. Engineers are currently performing verification procedures. We are unable to provide an ETA at this stage.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change in the situation.
Nov 15, 18:36 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant
Nov 15, 17:31 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are still verifying a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 16:09 AEDT

Update - Recovery efforts are still continuing.

Progress has been made on the recovery of the cluster. We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are verifying a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 15:09 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are still executing an action plan provided by the vendor. We are still unable to provide an ETA on this process at this stage.

Alternate infrastructure has been provisioned and configuration is underway to allow us the option to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 14:00 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are still executing an action plan provided by the vendor. We are still unable to provide an ETA on this process at this stage.

Alternate infrastructure has been provisioned and configuration is underway to allow us to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 13:00 AEDT

Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are currently executing an action plan provided by the vendor. We are still unable to provide an ETA on this process.

Alternate infrastructure has been provisioned and configuration is underway to allow us to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 12:04 AEDT

Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 11:19 AEDT

Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority.
Nov 15, 10:06 AEDT

Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority.
Nov 15, 08:14 AEDT

Update - Recovery efforts continue.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We are continuing to work with the storage vendor to recover the cluster, progress has been made but great care is being taken to avoid any data-loss.
Nov 15, 06:20 AEDT

Update - Recovery efforts continue.

Alternate infrastructure is in the process of being stood up to recover customer VPS onto as a contingency plan.

We are also continuing to work with the storage vendor to recover the cluster.
Nov 15, 03:43 AEDT

Update - Yesterday afternoon approximately 18:00 Sydney time we experienced a hardware failure within the storage infrastructure that supports the Blaze platform.

Recovery efforts were started immediately and have been ongoing since.

Alternate storage and hypervisor infrastructure is being built as a contingency in parallel while we work to engage the storage vendor to recover the cluster.

Some virtual machines will require bare metal restores, others will be able to be moved to the new infrastructure.
Nov 15, 01:42 AEDT

Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 01:19 AEDT

Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 00:21 AEDT

Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 23:25 AEDT

Update - We are continuing to make progress on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 22:20 AEDT

Update - We are making progress on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 21:59 AEDT

Update - We are continuing to troubleshoot the storage platform. We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 21:56 AEDT

Update - We are continuing to troubleshoot the storage platform. We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 20:47 AEDT

Update - We are continuing to troubleshoot the storage platform. No restoration progress to report as of yet. However, we are slowly but surely working our way through a list of potential outage causes.
Nov 14, 19:29 AEDT

Update - Multiple personnel are on site and currently manually taking stock of the storage platform health. We are continuing to look for the root cause of the issue, but as of yet do not have any further progress to report.
Nov 14, 18:41 AEDT

Identified - We have identified a major outage affecting the underlying storage layer for the Blaze VPS platform.

An incident response team has been formed and multiple personnel are currently working the problem as quickly as possible.

We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 18:07 AEDT

Investigating - We are currently investigating an outage affecting multiple VPS hypervisor nodes.

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.
Nov 14, 18:04 AEDT