Census Full Outage
Incident Report for Census
All systems operating normally. Our team will conduct an internal postmortem and determine steps to avoid this type of outage in the future.
Posted Nov 16, 2021 - 18:02 UTC
We are continuing to monitor for any further issues.
Posted Nov 16, 2021 - 17:49 UTC
The UI and sync engine are both back online, and we are monitoring the status of all syncs
Posted Nov 16, 2021 - 17:02 UTC
The Census UI is now available and we are monitoring its health. The team is working to bring the sync engine back online now - ETA 9:30 AM Pacific
Posted Nov 16, 2021 - 16:55 UTC
We have ETA of 15 minutes to restore access to the Census UI, and 1 hour to re-enable the sync engine.
Posted Nov 16, 2021 - 16:46 UTC
We have nearly finished the manual failover to the standby database and are planning to bring the Census application back online shortly.
Posted Nov 16, 2021 - 16:36 UTC
We have identified a database failure as the root cause of this issue, with a corresponding error while failing over to the hot standby database. We have initiated a manual failover to the standby database; once that is complete, we will bring the Census UI and Sync Engine back online.
Posted Nov 16, 2021 - 15:42 UTC
We have identified the root cause of the issue and put the Census system into maintenance mode while we finalize a plan to bring the system back online safely.
Posted Nov 16, 2021 - 15:11 UTC
We are currently investigating this issue.
Posted Nov 16, 2021 - 14:30 UTC
This incident affected: Census Sync Management UI and Sync Engine.