12 March 2026 · Field notes
The gateway’s p95 is not the upstream’s p95
People bring two latency numbers to a sitting and want them to kiss. One was taken at the gateway. The other was taken inside the application. Both may be honest. They still describe different clocks.
The gateway’s duration usually includes the time to accept the connection, apply whatever checks sit in the front door, wait on the upstream, and write the response back toward the client. The application’s duration often starts after the request has already been accepted and may stop before bytes leave the machine. A slow client on a large payload makes the gateway look ill while the application looks calm. A queued worker inside the application does the reverse.
When the log carries both an upstream response time and a total request time, we draw them as two lines on the same hours. The gap is the story. If the export has only one duration field, the briefing says so in the window sheet and we refuse to caption the line as “application latency.”
Percentiles make the disagreement louder. A p95 at the gateway can be a handful of heavy clients. A p95 inside the application can be a lock, a query, or a cold path. Averaging them into one “experience” number is how a room spends an hour blaming the wrong side of the proxy.
Before you export, write down which clock each field uses. If your gateway names the field request_time and the application names a field duration_ms, send the dictionary, not only the files. We would rather spend an hour on the dictionary than a day drawing a confident wrong overlay.