💾 Archived View for capsule.adrianhesketh.com › 2016 › 12 › 03 › testing-slow-http-responses captured on 2024-02-05 at 09:44:37. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2021-11-30)
-=-=-=-=-=-=-
Last week, I had a task to test what happens to a system when it attempts to send HTTP requests out to 3rd party webhooks and those 3rd parties fail to respond.
This came about when a customer was responding very slowly (sometimes more than an hour) to HTTP webhook requests, eventually using up all of the threads (on all of the machines) assigned to sending out webhooks and creating a backlog of unsent notifications to other customers.
It turned out that the root cause was that there is no default timeout on the Apache Commons HTTP Client used to send out HTTP requests - it will happily wait forever for a response.
While we reworked the queuing logic to avoid one customer affecting another, we needed to put a timeout in place. Easy enough, and easy enough to test in isolation, but I wanted to test it _in-situ_ to make sure we'd got it right, so I used [0] to route traffic from the Web to my local machine for testing and put together a quick Web server in Go to simulate some of the scenarios.
https://github.com/a-h/slowloris
You can start it up with a few parameters to delay sending the first byte back to the HTTP client, then delay sending the rest of the response.
go run main.go -delayBeforeFirstByte 10s -delayBeforeLastByte 10s
The default behaviour of the Go Web Server is to buffer output into 4KB chunks, which hampers testing the behaviour of sending some data, then hanging, so I set the response to be flushed to the client after each write as per this Stackoverflow question:
http://stackoverflow.com/questions/19292113/not-buffered-http-responsewritter-in-golang
There's also a client I wrote for testing, which just writes out a full stop (period) when it receives a byte to visually confirm the HTTP server behaviour.
There's a denial-of-service attack of the same name [1] which is, basically the same kind of thing that happened to the webhook service my team are supporting, except in reverse.
Mob Programming and No Estimates at Agile in Leeds
Immutable Windows Infrastructure on AWS
