Resolved -
This incident has been resolved. Syncs have been running normally as of approximately 1:00pm EST, and the web has been running at normal latencies as of 2:00pm EST.
The issue was caused by an indexing change on our side that impacted our ability to run syncs reliably. Recovery was delayed because we needed to rebuild the previous indexes. To avoid any risk of data loss, we chose a safer recovery approach that allowed the system to remain temporarily degraded while repairs were being completed.
To restore syncs as quickly as possible, we prioritized sync execution and applied targeted fixes to improve performance. During this time, access to the UI was temporarily down.
Jan 5, 20:24 UTC
Monitoring -
The web is now operating normally, along with Sync Runs. We are monitoring the system for overall health.
Jan 5, 19:05 UTC
Update -
Syncs are now operational. Users may be experiencing issues loading Sync Configuration and Sync Run details on the web.
Jan 5, 18:34 UTC
Update -
Syncs are currently not operational. Scheduled syncs should still be queuing up but they won't start running at this time. We will provide an update once workers come back up.
Jan 5, 16:47 UTC
Update -
We are still seeing degraded performance and in some cases failing syncs. We are continuing efforts to mitigate as the system repairs.
Jan 5, 15:59 UTC
Update -
We have identified and triaged the affected system and systems are coming back up now. Web is operational. We will see Sync delays as we scale back up.
Jan 5, 15:27 UTC
Identified -
Census is currently having issues running Syncs and rendering the Website. Engineers have identified the issue and are remediating.
Jan 5, 15:17 UTC