Increased latency for syncs and warehouse operations

Incident Report for Census

Resolved

This incident has been resolved.
Posted Jan 14, 2022 - 22:18 UTC

Update

We are continuing to monitor for any further issues.
Posted Jan 14, 2022 - 20:17 UTC

Monitoring

We have added sufficient capacity to drain our backlog of jobs and no new jobs or UI interactions should be delayed. Some existing jobs are delayed but on track to complete as usual - our team is monitoring those jobs and also working to identify the root causes.
Posted Jan 14, 2022 - 20:17 UTC

Update

We have identified the root cause of this issue and our team is adding additional compute resources to catch up on any delayed syncs.
Posted Jan 14, 2022 - 20:06 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Jan 14, 2022 - 19:22 UTC

Investigating

We have identified a networking issue that has caused a delay in processing jobs. This causes Census to:
* Process syncs more slowly than usual, or take longer to start syncs than usual
* "Hang" in the UI when fetching data or metadata from data warehouses or SaaS applications
Posted Jan 14, 2022 - 19:15 UTC
This incident affected: Census Sync Management UI and Sync Engine.