Summary of Incident
Below is a detailed write up for the Website Pro Outage we experienced, the cause and what we are doing to prevent this from occurring in the future.
Early in July, a phishing website that was hosted on an express site was the source of several complaints that were filed against the websitepro.hosting domain. As a response to these complaints, a lock was placed on the domain at the registry and registrar levels on July 13, and no communication about the action was sent to our team. As a result, on July 27, the cached DNS entries at the top level domain .hosting expired and DNS requests for websitepro.hosting began failing.
What did we do about it?
We started by confirming with Google that our own name servers were operating correctly, and then moved on to work with our domain registrar. We worked with the support team at the domain registrar to confirm that the issue causing the outage was not in our system but instead with theirs, and a support ticket was issued. Over several days, multiple attempts at a fix were attempted, but no indication of a root cause of the issue was provided.
On day five of the outage, our domain registrar determined that the root cause of the outage was that a lock had been placed on our domain at the registry level. We promptly confirmed that the offending site was no longer being hosted, and submitted the paper work and proof that we had removed the offending site to get the lock lifted on our domain. The lock was lifted on our domain at the registrar level, and they communicated up to the registry level to have the lock removed.
Unfortunately, the .hosting domain had been sold to a different registry than what our registrar had on file, and the notification to lift the lock was sent to the previous owner of the .hosting domain. While following up on day seven of the outage, we were able to track down the the new owner of the .hosting domain and finally got the lock lifted. Once the lock was lifted, the outage was resolved when DNS completely propagated in about two hours.
What steps are being taken to ensure that something like this doesn’t happen again?
There are a number of items we are going to do to ensure that something like this doesn’t happen again.
Replace our domain registrar
It took our domain registrar five days to be able to tell us that the root cause of the outage was that a lock was placed on our domain. The level of support we received during the incident was not acceptable. Because of this, we will be migrating our business to a registrar with proven excellence in customer support and a transparent reporting and communication system.
Implement monitoring for locks on our domain
Currently there is no commercially available software to monitor whois records to see if there is a lock on the domain. If we detect a lock, it will give us an opportunity to bring the issue to resolution before our domain becomes inaccessible.
Perform an audit of our infrastructure to identify and create redundancy in systems with single points of failure
While traditional DNS redundancy doesn’t apply here because our nameservers remained functional for the duration of the incident, we are still going to go through our infrastructure to ensure that any systems with single points of failure have sufficient redundancy available to them.
|8/03/2022 3:00 PM CST||
DNS requests are being fulfilled again for websitepro.hosting. The workarounds we had in place have been reversed, and Website Pro is now back to being fully functional. Thank you for your patience during this challenging outage.
At this time, if you implemented any workarounds listed to your CNAME or A Record, there is no urgency to change them back to have your sites become fully functional. While we do not anticipate the changes to the CNAME and A records pointed to the IP address to change or affect sites anytime in the foreseeable future, it is best practice to change them back to the recommended settings at your leisure.
|8/03/2022 1:55 PM CST||The lock on our DNS has been lifted by our former domain provider and we are already seeing some sites come back up. We need to roll back some work arounds before Website Pro returns to full functionality, but we anticipate that we will be back to full functionality by 3pm CST.|
|8/03/2022 12:21PM CST||The lock on our domain has continued to affect our ability to bring Website Pro back to full functionality. We are working with our original domain vendor to get the lock removed and with our new domain vendor to see if there is anything that they can do to help bring websites back online. We will continue to update regularly|
|8/03/2022 9:30AM CST||
We have been informed that the DNS transfer has been completed and that Google is again operating as our DNS server. This means that as soon as caches expire, sites will start to come up again. We expect this to happen over the next few hours.
|8/02/2022 4:30PM CST||
We are continuing to work with the 3rd party around the clock and will continue to provide updates in terms of resolution ETA or additional workarounds as we have the information.
We appreciate your patience and are committed to having a resolution in place as soon as possible.
|7/29/2022 5:00PM CST||Our development team has implemented a workaround that allows access to the admin dashboard while we continue to work toward a full resolution. Please see below for instructions:
From Partner Center
From Business App
Click on “Website” from the products section on the left navigation pane and you will be able to access the WSP dashboard. Sites with a custom domain that does not contain “websitepro.hosting” will be able to then access the WordPress dashboard.