Increased latency for syncs and warehouse operations
Incident Report for Census
Resolved
This incident has been resolved.
Posted Jan 14, 2022 - 22:18 UTC
Update
We are continuing to monitor for any further issues.
Posted Jan 14, 2022 - 20:17 UTC
Monitoring
We have added sufficient capacity to drain our backlog of jobs and no new jobs or UI interactions should be delayed. Some existing jobs are delayed but on track to complete as usual - our team is monitoring those jobs and also working to identify the root causes.
Posted Jan 14, 2022 - 20:17 UTC
Update
We have identified the root cause of this issue and our team is adding additional compute resources to catch up on any delayed syncs.
Posted Jan 14, 2022 - 20:06 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Jan 14, 2022 - 19:22 UTC
Investigating
We have identified a networking issue that has caused a delay in processing jobs. This causes Census to:
* Process syncs more slowly than usual, or take longer to start syncs than usual
* "Hang" in the UI when fetching data or metadata from data warehouses or SaaS applications
Posted Jan 14, 2022 - 19:15 UTC
This incident affected: Census Sync Management UI and Sync Engine.