The Blog

This post is about processing a task asynchronously and update the client with live feedback. For instance, a build is triggered from the client which runs multiple tasks in the backend and updates the client as soon as every task is processed. Could be any task triggered from the client with is compute heavy or runs in non-deterministic time.

Let's implement a WebSocket server to maintain a stateful connection with the client. In this example, we run two instances each running a simple WebSocket server vertically scaled up by four workers per instance. Each worker gets a unique ID which is a combination of instance ID and worker ID. Let's call it server ID. When a client is connected, it sends some metadata about the connection back to the client. Let's call it socket meta which comprises of a UUID for the connection and server ID of worker itself. This information is stored in the client locally throughout the connection's lifetime. The client ID is appended to the connection object. Thus, a client/connection can be identified and interacted with by knowing its client ID.

Every worker creates a RabbitMQ queue which has a binding with a RabbitMQ exchange, say "utils" with routing key as server ID. Something like the below diagram.



With the basic setup in place, when a task is triggered from the client, along with the task details, socket meta is sent. The back-end receives and processes the task. When the result for the specific task is ready,it sends the result to be consumed by the client to "utils" exchange with routing key as server ID and include client ID in the body. The worker which is listening to the exchange via a queue, picks up the message and identifies the connection via client ID and updates the message to the client.

This model is for real-time feedback to the client only. Assume the scenario where the task takes n seconds to process and has been triggered from the client when it was in state X (say, connected to server-1 and worker-3) which has to be maintained throughout the task's lifetime. But assume the connection between client and worker is disconnected in n-5 seconds, which leaves the task dangling as the client is in state Y. To avoid this, enable session affinity (at instance and worker level) based on cookie (or) network metadata ensuring within a specified period of time, the client always connects to the same worker.