On 2023-02-09, the Census team identified an issue that was causing sync status emails to not be sent to a subset of customers. These emails report whether syncs succeed, fail, or other states based on customer configuration. We developed and deployed a fix for this issue at approximately 5:50 PM Pacific Time.
At approximately 6:10 PM Pacific Time, we were alerted to the fact that Census was sending far more of these emails than usual. After a quick investigation, we discovered that the fix that had been applied had caused a “backfill” of status emails, and that some Census customers were receiving a large number of status emails for syncs that were days or weeks old.
At approximately 6:20 PM Pacific, our devops team disabled email sending across the entire product to halt these unnecessary alerts. The product team worked to diagnose the root cause and also “undo” the backfill behavior. This fix was tested and deployed at approximately 9:15 PM Pacific Time, at which point email sending was reenabled for all customers.
Customer Impact - A subset of Census customers received no emails for some or all of their sync statuses. We are currently investigating the extent of this monitoring outage and we will reach out to affected customers. - A subset of Census customers received a large flood of “backfilled” status emails - No Census customers received any status emails for any syncs that succeeded or failed between 6:20 PM Pacific and 9:15 PM Pacific on 2023-02-09 - The actual scheduling and execution of syncs was not affected - only the monitoring emails - Other email systems (password resets, invitations, weekly workspace summaries) were not affected
Posted Feb 10, 2023 - 05:21 UTC
We have discovered the root cause and our team is working on a fix. We have temporarily paused all status emails as we address the root issue.
Posted Feb 10, 2023 - 02:34 UTC
We are investigating an issue that caused Census to send or resend historic sync status emails going back up to six weeks to a subset of customers
Posted Feb 10, 2023 - 02:28 UTC
This incident affected: Census Sync Management UI.