Slow Email Sends
Incident Report for Interseller
Postmortem

At around 9:30am EST on Wednesday, Nov 26, engineers were immediately alerted that email sending was not present in the logs and has not ran for over 15 minutes. Engineering then started investigating the root cause.

At 9:45am EST, Engineering found that the root cause was due to a AWS resource that was not working as intended. Engineering tried recreating the resource with no results.

Immediately afterwards, Engineering created a temporary solution that we spun up to remediate the problem. Engineering then notified AWS through support channels of the issue.

At 10:30am EST, Engineering was contacted by AWS that several resources were affected from a cascading problem relating to AWS Kinesis Streams which was also affecting AWS CloudWatch Events.

At 7:00pm EST, Engineering was observed recovery of AWS CloudWatch Events and turned off the temporary solution in place. At 10:00pm EST we saw full recovery.

Our hope is to have rely less off external utilities to run Interseller and eliminate complex dependencies likes these in the future.

Posted Nov 26, 2020 - 09:26 EST

Resolved
This incident has been resolved.
Posted Nov 26, 2020 - 00:41 EST
Monitoring
We are seeing events restore to normal and monitoring to ensure that email sending is back up to speed.
Posted Nov 25, 2020 - 20:12 EST
Update
Our temporary solution is still in place and we're still waiting on AWS recovery.
Posted Nov 25, 2020 - 16:20 EST
Update
Due to an outage with AWS, daily user digest emails are currently not being processed.
Posted Nov 25, 2020 - 11:11 EST
Identified
We are currently experiencing issues with AWS CloudWatch Events which is currently affecting our timely email sends. We are temporarily using an alternative to ensure emails send until AWS CloudWatch Events is back in operation again.
Posted Nov 25, 2020 - 09:58 EST
Investigating
We're investigating issues where the intended sending volume is not reaching its peak.
Posted Nov 25, 2020 - 09:44 EST
This incident affected: Email Sending & Syncing.