Member-only story
Towards Fault Tolerant Web Service Calls in Java
In the Beginning
Once upon a time, I needed to call a microservice that could return a lot of data. So much data, that the request would often timeout. The usual solution to this is to request smaller chunks of data, also called pages. A page request takes two parameters, often called limit and offset. Limit is the number of items to be returned. Offset is the number of items to be skipped before beginning to return items
This being programming, solving one problem creates others. In my case, two problems appeared. First, this being the internet, an occasional page request would fail. Second, the simplest means to fetch pages is to do submit requests sequentially; that is, one at a time. But for an IO bound application, this can take a long, long time, resulting in unhappy customers. The next simplest alternative to sequential requests is to fire off all page requests simultaneously. This is a bit like using Low Orbit Ion Canon on my own services; it’s easy to overwhelm services using that technique.
My solution to scheduling requests was to put them all into a queue, then allow a few at a time to be executed in parallel. When one of the requests is completed, the next pending request is pulled from the queue and run. This is repeated until the queue is empty.
