The API errors from Tuesday morning are fixed. Here is what happened and what we are changing. ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏  ͏

Resolved

Tuesday's API outage is over

All services have been running normally since 10:42 am UTC. Requests during the outage may have failed, and you may need to retry them.

What happened

1

9:14 am: Errors begin

A routine database change locked a busy table, and API calls began to fail.

2

9:26 am: We find the cause

Our alerts fired and the on-call engineer traced the errors to that change.

3

10:05 am: We roll it back

Traffic moved to a standby database while we undid the change.

4

10:42 am: Service is back

Error rates returned to normal and we kept watching for another hour.

What comes next. Database changes like this one will now run in smaller steps and outside busy hours. We will also add an alert that fires sooner when a table locks.

Read the full report

Status page

Live state of every service.

See current status

Need help?

Our support team has the details.

Open a ticket

Stackyard Cloud
500 Harbor Way, Oakland, CA 94607

You get this notice because your team uses Stackyard Cloud.

Unsubscribe

Made with Email Template Examples