Android Push Notification Errors

Incident Report for Mixpanel

Resolved

This incident has been resolved.
Posted May 21, 2020 - 00:00 PDT

Monitoring

It has been determined that the error rate increase is directly related to invalid device tokens and a change in the way Firebase reports those (users who have uninstalled the app, disabled messaging or where otherwise generated incorrectly). We retried all tokens that returned an error but these ones would still not succeed as they were invalid to begin with but the error reporting for this had changed. We're passing the status to monitoring while we finalize the conversation with Firebase engineering.
Posted May 20, 2020 - 06:46 PDT

Update

We are continuing to work on a fix for this issue with our upstream provider.
Posted May 14, 2020 - 17:57 PDT

Update

We are continuing to work on a fix for this issue with our upstream provider and will continue to update here.
Posted May 11, 2020 - 17:18 PDT

Update

We continue to work on reviewing the elevated number of errors we are receiving from Firebase Cloud Messaging for Android push Notifications. Notifications that result in an error are being retried.
Posted May 10, 2020 - 04:48 PDT

Update

We are continuing to work on a fix for this issue.
Posted May 08, 2020 - 21:51 PDT

Identified

We've identified these errors are related to issues with our upstream provider. We're coordinating with them to determine a fix and will continue to update here.
Posted May 08, 2020 - 16:20 PDT

Investigating

We're currently experiencing higher than normal error rates sending Android push notifications. We appreciate your patience while our engineers work to resolve this issue. If you have any questions, please contact support.
Posted May 08, 2020 - 13:32 PDT