Update - Our technicians have identified a method to successfully transfer and restore VMs from the SolidFire storage cluster onto the new infrastructure and are prioritising this method to bring services back online. They are working through the list as fast as possible.

Those VMs which are not able to be restored in this way will need to be restored from backups. If you have already requested a restore from backup, we will first attempt to recover your data from the SolidFire storage cluster as this has proven to be the best approach.

We have scheduled System Administrators to be available to work on this continually until this issue is resolved.

We continue to wait for replacement hardware to arrive from NetApp which we hope will assist in the recovery of data also. This will happen most likely on Tuesday.

Our next scheduled status update will likely occur no earlier than Monday 09:00 AEDT (Sydney time).
Nov 16, 21:43 AEDT
Update - NetApp have not been able to recover the second node with the spare hardware we have onsite. They have organised a shipment of spare parts, however this is coming from interstate and will arrive on Monday morning at the earliest.

Once the hardware arrives, we will perform basic verification and then work with NetApp engineers to develop a plan to introduce and further verify the new hardware.

In the event that the Solidfire Storage Cluster recovers correctly, we expect to be able to start verifying data by Tuesday morning at the earliest. Our engineers are already working on extracting available data from volumes on the Solidfire Storage Cluster.

In the mean time alternative infrastructure is ready to go for customers with backups available. Please contact our support team if you would like to be restored from the latest backup available.

Please note that after restoring a service from backups we will not be able to restore any data for the service from the Solidfire Storage Cluster when it is back online.

When requesting a restore, please contact our support team at support@crucial.com.au, so that our System Administrators can work on the restore as quickly as possible.

We have scheduled System Administrators to be available to work on this continually until this issue is resolved.

As we do not expect the situation to change significantly in the next 40 hours, our next status update is expected to occur no earlier than Monday 09:00 AEDT (Sydney time).

In the event there is a significant change we will update here accordingly.
Nov 16, 17:16 AEDT
Update - Netapp have not been able to recover the second node with on site spares. They have organised a shipment of spare parts but this is coming from interstate and will arrive Monday morning at the earliest.

We are also discussing other options with NetApp to see if any further action can be taken with the hardware we have on hand.

In the mean time alternative infrastructure is ready to go for customers with backup services. Please contact our support team if you would like to be restored from the latest backup available as soon as possible.

When requesting a restore, please contact our support team at support@crucial.com.au, so that our System Administrators can work on the restore as quickly as possible.

Please note that after restoring a service from backups we will not be able to restore any data for the service from the Solidfire Storage Cluster when it is back online.

We will post a status update at 17:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 13:11 AEDT
Update - We are continuing to work closely with NetApp, and have been carefully executing a recovery procedure. This is taking longer then anticipated because we are trying to recover as much data as possible.

We are still looking at options to source replacement hardware as we have exhausted our spares for a particular hardware component.

At this stage we are still not able to provide an ETA on Virtual Machine Recovery.

We will post a status update at 14:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 11:40 AEDT
Update - We are continuing to work closely with NetApp, and have been carefully executing a recovery procedure.

This is taking longer then anticipated because we are trying to recover as much data as possible.

At this stage we are still not able to provide an ETA on Virtual Machine Recovery.

We will post a status update at 11:00 AEDT (Sydney Time), or when there is a significant change in the situation.
Nov 16, 09:37 AEDT
Update - NetApp Engineers have provided a recovery procedure which we are carefully executing.
We will provide further updates on this process and an ETA on Virtual Machine recovery at 9:30 AEDT (Sydney time) or when there is a significant change in the situation.
Nov 16, 07:49 AEDT
Update - NetApp Engineers are testing the recovery procedure in their U.S. test lab to find the safest way to recover the storage cluster with minimum data loss.

This process is taking longer than anticipated.

We expect further news at 07:00 AEDT (Sydney time) , and will update the virtual machine recovery ETA at that time.

We will post a status update at 07:00, or when there is a significant change in the situation.
Nov 16, 05:12 AEDT
Update - Recovery efforts are continuing.

We still anticipate that we'll be able to start bringing Virtual Machines online between 04:00 and 05:00 AEDT (Sydney time).

We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 16, 03:07 AEDT
Update - We have suffered a 2 node failure with our SolidFire storage cluster. Although the cluster is designed to handle a multi node failure, the automated failover and recovery process has not worked as expected and has required manual intervention which includes repairing the failed nodes.

We have been actively working with the storage vendor (NetApp) in order to restore the storage cluster. Progress has been slow and careful as the priority is to prevent as much data loss as possible.

So far 1 node has been repaired and reintroduced to the cluster and at this stage, we've been able to recover almost all volumes on the cluster. NetApp staff have been working with their Engineering Team attempting to bring the remaining volumes online.

In order to minimise the potential of any unrecoverable volumes, NetApp staff has requested that we do not bring active workload on to the storage cluster until their Senior Engineering Team gives the go ahead.

We anticipate that we'll be able to start bringing Virtual Machines online between 04:00 and 05:00 AEDT (Sydney time).

We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 15, 22:30 AEDT
Update - Recovery efforts are continuing.

We are continuing to work with the storage vendor to recover the cluster. Engineers are currently performing verification procedures. We are unable to provide an ETA at this stage.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here every 3 hours, or when there is a significant change in the situation.
Nov 15, 19:44 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster. We have made some progress, bringing one of the failed storage members back online. Engineers are currently performing verification procedures. We are unable to provide an ETA at this stage.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change in the situation.
Nov 15, 18:36 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are planning a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant
Nov 15, 17:31 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are still verifying a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 16:09 AEDT
Update - Recovery efforts are still continuing.

Progress has been made on the recovery of the cluster. We are continuing to work with the storage vendor to recover the cluster, and are executing an action plan provided by the vendor.

Alternate infrastructure has been provisioned and we are verifying a process to allow the option to recover customer services onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 15:09 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are still executing an action plan provided by the vendor. We are still unable to provide an ETA on this process at this stage.

Alternate infrastructure has been provisioned and configuration is underway to allow us the option to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 14:00 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are still executing an action plan provided by the vendor. We are still unable to provide an ETA on this process at this stage.

Alternate infrastructure has been provisioned and configuration is underway to allow us to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 13:00 AEDT
Update - Recovery efforts are still continuing.

We are continuing to work with the storage vendor to recover the cluster, and are currently executing an action plan provided by the vendor. We are still unable to provide an ETA on this process.

Alternate infrastructure has been provisioned and configuration is underway to allow us to recover customer VPS onto new hardware as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 12:04 AEDT
Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 11:19 AEDT
Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority.
Nov 15, 10:06 AEDT
Update - Recovery efforts continue.

We are continuing to work with the storage vendor to recover the cluster.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We have been and will continue to work on resolving this issue with the highest priority.
Nov 15, 08:14 AEDT
Update - Recovery efforts continue.

Alternate infrastructure is still in the process of being stood up to recover customer VPS onto as a contingency plan.

We are continuing to work with the storage vendor to recover the cluster, progress has been made but great care is being taken to avoid any data-loss.
Nov 15, 06:20 AEDT
Update - Recovery efforts continue.

Alternate infrastructure is in the process of being stood up to recover customer VPS onto as a contingency plan.

We are also continuing to work with the storage vendor to recover the cluster.
Nov 15, 03:43 AEDT
Update - Yesterday afternoon approximately 18:00 Sydney time we experienced a hardware failure within the storage infrastructure that supports the Blaze platform.

Recovery efforts were started immediately and have been ongoing since.

Alternate storage and hypervisor infrastructure is being built as a contingency in parallel while we work to engage the storage vendor to recover the cluster.

Some virtual machines will require bare metal restores, others will be able to be moved to the new infrastructure.
Nov 15, 01:42 AEDT
Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 01:19 AEDT
Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 15, 00:21 AEDT
Update - We are continuing to work on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 23:25 AEDT
Update - We are continuing to make progress on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 22:20 AEDT
Update - We are making progress on our recovery efforts for this issue. We will continue to post further status updates here within 60 minutes, or if there is a significant change on the issue.
Nov 14, 21:59 AEDT
Update - We are continuing to troubleshoot the storage platform. We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 21:56 AEDT
Update - We are continuing to troubleshoot the storage platform. We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 20:47 AEDT
Update - We are continuing to troubleshoot the storage platform. No restoration progress to report as of yet. However, we are slowly but surely working our way through a list of potential outage causes.
Nov 14, 19:29 AEDT
Update - Multiple personnel are on site and currently manually taking stock of the storage platform health. We are continuing to look for the root cause of the issue, but as of yet do not have any further progress to report.
Nov 14, 18:41 AEDT
Identified - We have identified a major outage affecting the underlying storage layer for the Blaze VPS platform.

An incident response team has been formed and multiple personnel are currently working the problem as quickly as possible.

We will endeavour to post a further status update within the next 30-60 minutes.
Nov 14, 18:07 AEDT
Investigating - We are currently investigating an outage affecting multiple VPS hypervisor nodes.

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.
Nov 14, 18:04 AEDT

About This Site

Welcome to the Crucial status page. This page keeps you up to date with any scheduled or unscheduled maintenance on our network.

Core Network ? Operational
Blaze Cloud Platform Major Outage
Virtual Server Platform ? Operational
Web Hosting Services ? Operational
Reseller Hosting Services ? Operational
Crucial Website ? Operational
Crucial Client Area ? Operational
Help Centre ? Operational
Phone Support Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Nov 17, 2019

No incidents reported today.

Nov 14, 2019
Resolved - This incident has been resolved.
Nov 14, 17:11 AEDT
Update - We are continuing to monitor for any further issues.
Nov 14, 17:10 AEDT
Monitoring - A fix has been implemented and we are monitoring the results.
Nov 14, 17:10 AEDT
Investigating - Be advised our monitoring has indicated that there is currently an issue with accessing Crucial Control (our billing, contact and security PIN management system).

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.
Nov 14, 17:00 AEDT
Completed - The scheduled maintenance has been completed.
Nov 14, 05:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 14, 02:00 AEDT
Scheduled - Our systems administrators will be performing scheduled maintenance of the SAN arrays which power our Shared Hosting and Reseller Hosting clouds.

This work will include an important firmware upgrade.

Customers may experience slower than usual service times for a period of no longer than 10 minutes. All servers will stay on-line for the duration of this maintenance.
Nov 13, 13:43 AEDT
Nov 13, 2019
Completed - The scheduled maintenance has been completed.
Nov 13, 00:00 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 12, 22:00 AEDT
Scheduled - Our systems administrators will be performing maintenance on the Shared Hosting Server vmsh01.performance-hosting.com.au on the night of Tuesday 12th November between 22:00 and 00:00 (AEST).

As part of this maintenance, vmsh01.performance-hosting.com.au will be updated and rebooted.

We expect less than 15 minutes of downtime during the maintenance window.

We will provide updates on our progress here during the maintenance as needed.
Nov 11, 09:57 AEDT
Nov 11, 2019

No incidents reported.

Nov 10, 2019

No incidents reported.

Nov 9, 2019

No incidents reported.

Nov 8, 2019
Resolved - This incident has been resolved.
Nov 8, 18:01 AEDT
Monitoring - These services have been restored, and we are currently monitoring the situation.
Nov 8, 16:19 AEDT
Identified - Be advised our internal monitoring has indicated that some of our Shared and Reseller Hosting is currently unavailable.

This may be impacting access to your website, email services, ftp services and access to the cPanel control panel.

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.
Nov 8, 16:13 AEDT
Resolved - This incident has been resolved.
Nov 8, 18:01 AEDT
Monitoring - We are currently conducting maintenance on our storage network which supports part of our shared hosting infrastructure.
No impacts to service are expected.
Nov 8, 14:03 AEDT
Resolved - As advised on the cPanel Known Issues Status Page, the bad MariaDB package has been removed from the repository. See https://forums.cpanel.net/threads/known-issues-status-page.644133/post-2710361

MariaDB have not yet published the fixed package, but the broken package should no longer be installed by a WHM update.

There will be no further updates to this status page - please continue to monitor the MariaDB release page if you are waiting on the newest package to become available: https://mariadb.com/kb/en/library/mariadb-10320-release-notes/
Nov 8, 13:44 AEDT
Monitoring - MariaDB have advised that a fix has been released, however there are currently no official packages available for it. You can find a link to the release notes below:
https://mariadb.com/kb/en/library/mariadb-10320-release-notes/

For any Self-Managed clients still experiencing issues, please review the MariaDB Jira task which provides workarounds/solutions to this issue: https://jira.mariadb.org/browse/MDEV-20987

Once an official package has been released for download, we will provide another status update.
Nov 7, 10:54 AEDT
Identified - There has been an upstream issue regarding MariaDB failing to start with InnoDB Failed Assertion errors.
cPanel has identified this as an issue with MariaDB (Case ID MDEV-20987).
This issue is affecting the following MariaDB versions:

10.1.42
10.2.28
10.3.19
10.4.9

We are taking proactive action in an attempt to mitigate the impact of this MariaDB update.

cPanel is working with MariaDB on a fix as a priority. As soon as a fix is released, we will deploy it to our Managed VPS fleet.

Updates will be provided as they become available.
Nov 6, 10:59 AEDT
Resolved - This incident has been resolved.
Nov 8, 13:25 AEDT
Monitoring - The affected hypervisor has been successfully rebooted and all guest systems are back online.

We will continue to closely monitor this server in the usual way.
Nov 8, 08:43 AEDT
Identified - The affected hypervisor is currently being rebooted and its status assessed. A further update will be provided when we have more information about service recovery.
Nov 8, 08:16 AEDT
Investigating - Be advised our internal monitoring has indicated that the VPS Hypervisor xnsrv017.oa.apdcsy1.crucial.com.au is currently down or unavailable.

As a result VPS services which reside on this hypervisor may not be accessible or connectable.

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.
Nov 8, 08:15 AEDT
Nov 7, 2019
Resolved - An issue with our primary upstream voice provider meant that customers were unable to successfully navigate our phone system menu. To mitigate the issue, we migrated inbound calls to our backup voice provider whilst our primary provider investigated the problem. However, during this time, only calls to our 1300 numbers were able to successfully be completed.

Once our primary provider resolved the issue, we re-enabled calls to all our inbound numbers, and customers are once again able to successfully navigate our phone system menu.
Nov 7, 12:34 AEDT
Update - During our investigations, we have identified that some of our direct numbers are not working. If you'd like to contact us by phone, please call 1300 884 839
Nov 7, 09:26 AEDT
Investigating - Our Technical Operations team are currently troubleshooting an issue which has caused our phone system to become unresponsive.

Please login to your Client Account to submit a support request if necessary.

https://support.crucial.com.au

We are working to resolve this with top priority and will provide an update in 60 minutes, or if there is a significant change in the situation.

You can subscribe to this incident to receive an SMS or Email notification when our phone support is operational.
Nov 7, 09:09 AEDT
Nov 6, 2019
Completed - The scheduled maintenance has been completed.
Nov 6, 00:01 AEDT
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 5, 22:00 AEDT
Scheduled - Our systems administrators will be performing maintenance on the Shared Hosting Server sh32-eqx3-syd.ha-node.net on the night of Tuesday 5th November between 22:00 and 00:00 (AEST).

As part of this maintenance, sh32-eqx3-syd.ha-node.net will be updated and rebooted.

We expect less than 15 minutes of downtime during the maintenance window.

We will provide updates on our progress here during the maintenance as needed.
Nov 4, 10:48 AEDT
Nov 5, 2019
Resolved - This incident has been resolved.
Nov 5, 12:09 AEDT
Update - Staff have returned to the building and phone support is available.
Nov 5, 12:06 AEDT
Monitoring - Due to a fire evacuation, our phone support is currently unavailable.

Please contact us through our Help Center to submit a support request if necessary.

https://support.crucial.com.au
Nov 5, 10:00 AEDT
Nov 4, 2019

No incidents reported.

Nov 3, 2019

No incidents reported.