The night we sent 40,000 push notifications at 03:14

On Sunday 30 March, at 03:14 Central European Summer Time, 40,182 people were woken by a push notification telling them about a new dog at a shelter in a city many of them do not live in. Some of them got it four times. This is what happened.

Timeline

02:00 CET  clocks go forward; 02:00–03:00 does not exist
03:12      daily digest job fires (expected 08:00 CEST)
03:14      first batch delivered, 12,400 devices
03:19      retry loop begins re-sending batch 1
03:31      on-call paged by a founder who was awake
03:38      job killed, sending stopped
04:02      1,904 uninstalls recorded overnight
09:30      apology sent, in-app and by email

The three causes

No single one of these would have done it. All three together made the outcome inevitable.

• The scheduler ran on UTC while the cron expression had been written by someone thinking in local time. For most of the year the offset was constant and the job simply ran an hour off, which nobody noticed because the digest is not time-critical.
• The DST transition moved the offset again, landing the job in the small hours instead of merely being wrong by one hour.
• The send worker retried failed batches without an idempotency key. A partial failure at the APNs boundary re-sent the entire batch rather than the failures, up to four times.

We had a quiet-hours rule. It was implemented in the client, and the client cannot decline a notification that has already arrived.

What we changed

The quiet-hours check moved server-side, into the send path itself, where it evaluates each recipient's local time immediately before delivery and drops anything between 21:00 and 08:00 into the next morning's queue. Client-side suppression remains as a second line, but it is no longer the only one.

// send-worker: evaluated per recipient, not per batch
const local = Temporal.Instant.fromEpochMilliseconds(Date.now())
  .toZonedDateTimeISO(recipient.timezone ?? 'Europe/Budapest');

if (local.hour >= 21 || local.hour < 8) {
  return defer(recipient, nextMorning(local));
}

• Every scheduled job now declares its timezone explicitly and the deploy fails if the field is absent. There is no default.
• Send batches carry an idempotency key derived from (digest_id, device_id). A retry is now a no-op at the provider.
• A rate ceiling: any job attempting to send to more than 5,000 devices outside business hours halts and pages instead of proceeding.

The part that actually hurt

We lost 1,904 installs in one night, and a notification permission, once revoked, is close to permanent. Six weeks later that cohort had re-enabled at a rate of 4%.

The apology went out at 09:30 the same morning, in plain language, naming the mistake. The replies were kinder than we deserved. Several people said the honesty was why they kept the app, which is a nice thing to hear and not a reason to do it again.