Incident Postmortem on 20-10-2026 and 21-10-2026

Incident Summary

Schoolyear was unavailable twice for a short while, on 20 and 21 October 2026, for 13 and 19 minutes respectively. During those periods, students could not start their exams, supervisors could not open the exam dashboard, and assessment platforms that integrate with us received errors.

Both outages had the same cause. One integration sent a very large burst of requests in a short time, and two flaws in our rate limiting turned that into an outage: rejected requests still put load on the database, and we told the sender to retry after one second. The retries kept hitting the database until it could no longer keep up, and Schoolyear stopped answering requests altogether.

The changes described below have been live since 22 October 2026.

Leadup

On both days, one integration sent a burst of requests at the start of an exam that far exceeded our fair-use limits. The burst was far larger than the number of students taking the exam would explain.

Schoolyear limits how many requests a single school or platform can send in a short period. When a sender goes over that limit, we ask it to wait before trying again. Exam size was not the problem: we regularly run exams with a larger population than were involved here.

Fault

The rate limit itself worked. Two things around it did not:

• The rate limiter checked the database before deciding whether to turn a request away, so even rejected requests still put load on the database.
• We told the sender to retry after one second. It did, in large numbers.

This created a feedback loop. Retries added load, the load made requests slower, slower requests failed, and the failures produced more retries. The database eventually ran out of connections and the service stopped responding.

Impact

• 20 October: one institution could not start its exam for 13 minutes.
• 21 October: three institutions, up to 2,100 students in total, could not start their exam for 19 minutes.

Exams already in progress continued as normal. Depending on how an assessment platform responds when Schoolyear is unavailable, students on some platforms may have had their exam interrupted. No data was lost or exposed.

Timeline

On 20 October a restart was enough. On 21 October restarts failed, and service came back only after changing how our servers connect to the database.

(All times are in CEST)

20 October 2026

21 October 2026

Reflection

Our rate limiting protected against too many requests, but not against the cost of rejecting them. Rejected requests still reached the database, and a fixed one-second retry turned a single burst into a sustained load.

Traffic like this can come from anywhere, including someone deliberately trying to take the service down. The changes therefore act on the pattern, not on the sender. All of them are live: