The portal to access and manage Symantec.cloud (the antivirus and other services hosted by Symantec in order to reduce the local infrastructure required) is suffering an outage that has been going on all day today. The company is not being very disclosing of the incident directly to customers or through social media channel despite being quite disruptive to their customers. The most information is available at https://identity.symanteccloud.com/, though the certificate is invalid.
The site has read the following status before being updated to just the 02:00 entry:
00:00 UTC: Database restore is in progress and our engineers continue to work through the restoration process. Our engineers are working through this incident with the utmost priority. Please accept our sincere apologies for any inconvenience caused.
22:00 UTC: Database configuration steps have now been completed. Along with some additional checks, our engineers are now performing database restores.
20:45 UTC: Please be advised that our database engineers are now working on configuring database engine and services. We will update this thread once new information becomes available.
19:10 UTC: Planned restoration steps are progressing well. Operating system tasks are now complete and we are now working on database installation and configuration.
17:02 UTC: Our engineers continue to work on restoration plan. Server rebuild in now in process and while we do not have a firm ETA, we are working as fast as possible to bring services back online.
13:58 UTC – Please be advised that the ClientNet portal outage is ongoing. Engineers have identified an issue and are in the process of implementing a solution. Once this solution has been deployed they will be able to see if the situation is resolved or if there are further problems to address. For this reason we are unable to provide an ETA at this time. Please accept our apologies for this ongoing incident and be assured that all available resources are working to restore service as quickly as possible.
02:00 UTC: Please be advised that the restore of our Database is ongoing and our engineers continue to work through this process with the utmost urgency.
Please accept our sincere apologies for any inconvenience caused.
The outage has taken down the ability to log into the ClientNet portal and other customers report receiving spam that Symantec’s email cloud service should have prevented, indicating that the services are not receiving updates throughout this outage, though at least email is passing through.
From the posted progress, we can see that a server rebuild was required and a database restore is now underway. Hopefully they have switched from BackupExec to a better product since Symantec split that side of the house to Veritas.
Updated status:
UPDATE 09:56 UTC: Our Engineers have advised that the task of data replication is underway. Whilst we are still unable to provide a precise time for the restoration of portal access we do know that the replication process may take until after the end of the EMEA working day. Once this process has been completed we will need to carry out some quality tests and, if no further issues are found, it will then be possible to bring the portal back online. Please accept our apologies for the continuing impacts caused by this incident and be assured that our engineering are focusing on restoring service as quickly as possible.
UPDATE 07:15 UTC: Our Engineers have advised that database restoration is completed and we are now focusing on repopulating the data. Please be assured that our infrastructure team continues to investigate all possible options to bring the service to production as quickly as possible.
04:30 UTC: Our engineers have advised that the database restoration process is nearing completion with error checking and analysis to follow. Our infrastructure team continue to investigate all possible options to bring the service to production as quickly as possible. Please accept our profuse apologies for any inconvenience caused.
02:00 UTC: Please be advised that the restore of our Database is ongoing and our engineers continue to work through this process with the utmost urgency.
Original
Please be advised that we are experiencing an outage with the Symantec.Cloud ClientNet Portal. We are treating this incident with top priority and our teams are working diligently to fix the issue asap. Please accept our apologies for any inconvenience caused.
Updated 4-12, Additions:
UPDATE 21:45 UTC: We have received confirmation that Web Security service is in full restore state. Clientnet portal restore is still in progress. Partners who are unable to place orders are requested to contact our partner services team by emailing CLD_PARTNER_SERVICES@symantec.com
UPDATE 19:26 UTC: Please be advised that the restore of our Clientnet database is still ongoing as well as the restoration of our web services. Our engineers continue to worth through this process with the utmost urgency.
UPDATE 17:20 UTC: We have received reports that a very small subset of web security.cloud customers are still experiencing browsing issues. Our engineers are well aware of this problem and already working on performing some backend work to restore web service as soon as possible. In parallel, our portal and database team continues to work on restoring clientnet portal.
UPDATE 15:30 UTC: Please be advised that as of 14:47 UTC, Web security.Cloud service has been restored globally. Our engineers continue to work on restoring Clientnet portal and API services.
UPDATE 15:26 UTC: “Please be advised that we are experiencing an outage with the Symantec.Cloud ClientNet Portal as well as Web cloud security services. We are treating this incident with top priority and our teams are working diligently to fix the issue asap. Please accept our apologies for any inconveniences caused.”