We run a twenty-year-old art-submission platform with a four-person team. The dangerous problems never threw an error. A process hung for an hour. A job failed silently. An artist paid less than the piece minimum. We were reactive. We waited for something to break, or for an upset user to email us.
Fourteen checkers now watch all of our issues on a ten-minute-to-daily cadence. So now, we can pro-actively identify, triage, and solve issues instead of waiting for a customer support email. A human will show up when there is an issue only we can solve.
The failure classes the checkers notice are ones that directly affect users. We had an issue where we would expect a $35 payment from an artist, but they were charged $15. Sometimes, that is OK if they have a coupon or if there are heavy discounts provided by call administrators. However, these would often result in galleries calling us and asking why we are $20 short on what we are charging people, since that is directly $20 out of their pocket. These are issues where we can’t easily create a system that throws an error, either due to too many false-positives or because the problem doesn’t lend itself to pro-active alerting. We had to wait and sit there until somebody ran into the issue.
A low signal condition doesn’t deserve to wake a human every ten minutes. Instead, we routed each checker by cadence (10 minutes / hourly / daily), gave each a severity (low, medium, high), and let a human override it.
Our fourteen checkers span application health, suspect data, payment mismatches, and the other issues our support team keeps meeting. Now, our checkers will catch underpayments before the gallery even notices, so we can be proactive about reaching out.
If there are issues that are easily solved, such as a scheduled action stalling, the checker will automatically attempt a repair and mark the action as resolved once unstuck. The rest raise an alert over email with the correct context (issue, history, suggested repairs) so a human can easily see what must be resolved.