All systems operating normally. Our team will conduct an internal postmortem and determine steps to avoid this type of outage in the future.
Posted Nov 16, 2021 - 18:02 UTC
Update
We are continuing to monitor for any further issues.
Posted Nov 16, 2021 - 17:49 UTC
Monitoring
The UI and sync engine are both back online, and we are monitoring the status of all syncs
Posted Nov 16, 2021 - 17:02 UTC
Update
The Census UI is now available and we are monitoring its health. The team is working to bring the sync engine back online now - ETA 9:30 AM Pacific
Posted Nov 16, 2021 - 16:55 UTC
Update
We have ETA of 15 minutes to restore access to the Census UI, and 1 hour to re-enable the sync engine.
Posted Nov 16, 2021 - 16:46 UTC
Update
We have nearly finished the manual failover to the standby database and are planning to bring the Census application back online shortly.
Posted Nov 16, 2021 - 16:36 UTC
Update
We have identified a database failure as the root cause of this issue, with a corresponding error while failing over to the hot standby database. We have initiated a manual failover to the standby database; once that is complete, we will bring the Census UI and Sync Engine back online.
Posted Nov 16, 2021 - 15:42 UTC
Identified
We have identified the root cause of the issue and put the Census system into maintenance mode while we finalize a plan to bring the system back online safely.
Posted Nov 16, 2021 - 15:11 UTC
Investigating
We are currently investigating this issue.
Posted Nov 16, 2021 - 14:30 UTC
This incident affected: Census Sync Management UI and Sync Engine.