Incident Summary
On 17 August 2026 at 12:10 CEST, Schoolyear experienced a partial service outage affecting API availability. During this period, incoming API requests experienced elevated latency and returned `5xx` server errors, impacting users attempting to initialize exam workspaces and supervisors accessing exam dashboards.
Importantly, overall traffic at the time was relatively low and well below our normal peak operating capacity. Concentrated startup bursts are normal and expected during exam start windows. The underlying root cause was identified as a gap in our pre-deployment load testing when PgBouncer was introduced several weeks prior. Those initial load tests primarily validated steady-state traffic volume and did not accurately model the specific, high-burst onboarding workflows seen in production. Consequently, a connection pool configuration that lacked pre-warmed connections went undetected and saturated prematurely during a routine startup wave.
By temporarily bypassing PgBouncer and reconnecting the API services directly to the database at 12:28 CEST, service availability and response times were fully restored. The total outage lasted 18 minutes.
Subsequent load testing with realistic, end-to-end workspace onboarding flows confirmed that maintaining pre-warmed application connection pools in conjunction with PgBouncer allows the system to smoothly absorb routine startup bursts, ensuring high resilience under concentrated traffic spikes.
Leadup:
Several weeks before the incident, PgBouncer was deployed to the production environment as a connection pooler between our application and database. Prior to rollout, our technical team load tested the deployment against high-throughput steady-state and isolated endpoint scenarios, and it was confirmed production-ready on that basis.
Fault:
The incident was rooted in pre-deployment load testing that didn't sufficiently reflect real-world production traffic patterns. Before PgBouncer was introduced several weeks before the incident, testing focused on high-throughput steady-state volume and isolated endpoint stress, not the realistic, composite lifecycle of an exam start. As a result, the connection pool configuration passed pre-release checks without pre-warmed backend connections, and this went undetected because it operated fine under steady traffic.
The gap surfaced when a standard concurrent wave of exam starts hit at 12:10 CEST: on-demand connection establishment introduced latency and connection contention. While waiting on cold handshakes, configuration queries held connections open longer than expected, so even at low total traffic volume the connection limit was consumed rapidly, causing queued requests, including lightweight signature verifications, to time out.
Impact:
- Exam Workspace Onboarding & Startup: Students attempting to initialize or start exam workspaces between 12:10 and 12:28 CEST experienced timeouts or temporary failure screens.
- API & Integrations: Third-party assessment platform integrations making API calls (such as scheduling exams, fetching exam settings, or signature verification) received `500` / `5xx` errors or delayed responses.
- Ongoing Exams: By design, active local exams running inside Schoolyear remain fully functional offline and were not interrupted. However, students using certain third-party assessment platforms that continuously poll API endpoints and lack fault-tolerant offline handling may have experienced unexpected session interruptions.
- Secondary Investigation Impact: During the technical investigation on an isolated test environment, an automated background job scheduler resulted in a silently-failing Schoolyear AVD deployment, making the impacted students unable to connect to their virtual exam workspace.
Timeline (CEST)
- 12:00 - System operating normally under low baseline traffic.
- 12:10 - A routine wave of exam start events begins, producing a concentrated burst of workspace configuration and signature verification requests.
- 12:10 - Database connections rapidly reach the configured pool ceiling due to pool configuration constraints. Queries begin timing out, and the API starts responding with `5xx` errors.
- 12:12 - Automated monitoring triggers alerts for elevated HTTP error rates and response latency. First responders initiate the incident response protocol.
- 12:18 - The response team identifies connection acquisition bottlenecks between the API layer and the connection pooler (PgBouncer).
- 12:25 - The team executes an operational change to route API database traffic directly to the primary database, bypassing PgBouncer.
- 12:28 - Direct database connectivity is established. Connection saturation clears immediately, API latencies normalize, and error rates return to zero.
- 12:45 - Systems monitored and confirmed stable across all endpoints; incident officially closed.
Reflection
This incident exposed a gap in our pre-deployment load testing for the PgBouncer deployment. Concentrated connection bursts during exam starts are expected, routine behavior. The failure was in how we modeled load. Our tests measured isolated endpoints and aggregate throughput, which masked the actual failure mode: multi-query startup bursts need pre-warmed connection pools, not just capacity headroom. Without pre-warmed connections, on-demand handshakes bottlenecked and exhausted the pool even at low overall traffic. This was confirmed when we rebuilt testing in an isolated environment to replicate full student onboarding journeys and concurrent startup spikes. Adding connection pre-warming on top of PgBouncer eliminated acquisition latency entirely; the system absorbed the same spike with no query delays or dropped requests.
Following the incident, we've rebuilt our load-testing suite around composite, end-to-end journeys instead of isolated endpoints, and rolled out connection pool pre-warming across all API instances. We've also optimized workspace configuration queries and caching to cut query execution time during onboarding. Two items are still in progress: fine-grained alerting on connection acquisition latency and pool utilization, and stricter isolation safeguards for test environments to prevent conflicts during production database investigations.