October 9, 2026
Day 18 — I Guessed 20%. The Slow Function Was 95% to 100% of the Request
Day 17 ended with a cluster where I could see that something was wrong (a stuck rollout) but not why. Day 18 is about the tool for that question: a trace. I took a small Express server, added OpenTelemetry, and printed every span to the console. No collector, no UI, no metrics. From today I also use a shorter format for the challenge, about 90 minutes per day, so this post is shorter too.
The setup
Three pieces in instrumentation.ts: the Node SDK, a ConsoleSpanExporter, and
the automatic instrumentations. I start the server with
tsx --import ./instrumentation.ts index.ts, so the setup file runs before the
server code.
I wrote two predictions before I wrote any code. How many spans would one
GET /hash print? I said 5. What share of the request time would a function that
waits about 200 ms take? I said 20%.
What a request looks like
Then I added slowLookup(), which waits 100, 200 or 300 ms inside its own span,
and sent five requests. Every /hash request printed 5 spans. Four came from the
automatic instrumentation (GET /hash, middleware - patched and two
request handler - /hash) and one is mine. So my number 5 matches the total, but
I had not counted my own span when I wrote it down, and I do not want to claim
more than that.
The parent ids give a tree. For the first request:
GET /hash (root)
middleware - patched
request handler - /hash
request handler - /hash
slowLookup
I did not pass any context to slowLookup. It became a child of the handler span
by itself.
The 20% was wrong
trace slowLookup ms GET /hash ms share
45263815 201.5 211.3 95.4%
7cbafc63 301.4 304.1 99.1%
cc4ff7d7 201.2 203.8 98.7%
00b8906d 300.3 301.1 99.7%
d731a5b3 201.2 205.1 98.1%
The share was between 95% and 100%. The rest of the request took 0.8 to 9.8 ms, so a function that waits 100 to 300 ms is nearly the whole request. My 20% had no reason behind it, I just did not think about how small the other part is. The first request had the largest gap (9.8 ms). I do not know why.
What a failure looks like
GET /fail throws an error inside my own failWork span. I catch it, call
span.recordException(err), set the status to error, send a 500, and end the
span. The failWork span in the output has status: { code: 2, message: 'something broke' }
and an exception event with the type, the message and a stack trace whose
first line points at the throw in index.ts.
The automatic HTTP span for the same request, GET /fail, also has
status: { code: 2 }, with no message and no events. It got its error status from
the 500 response (http.response.status_code: 500). So the detail of what broke
is only on the span where I recorded it. I did not run /fail without my own
recordException, so that last point is my reading of the two spans and not
something I tested.
What I'm leaving open
Five requests and one failing request, so no spread to speak of. I did not look
into the extra 0.8 to 9.8 ms. No collector or trace viewer, no metrics, no
sampling. instrumentation.ts also has a metrics exporter that I added and did not
use for anything.
What's next
Day 19 is CI/CD: a GitHub Actions matrix build and a small canary script. Full
code in the repo — day-18-observability-opentelemetry/.