Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Message queues (e.g. SQS) are inappropriate for tracking long-running tasks/workflows. This is due to the operational requirements such as:

- Checking the status of a task (queued, pending, failed, cancelled, completed) - Cancelling a queued task (or pending task if the execution environment supports it) - Re-prioritizing queued tasks - Searching for tasks based off an attribute (e.g. tag)

You really do need a database for this.



I’m reminded of classical LRU cache implementation - double linked list and a hash map that points to the list elements.

It is a queue if we squint really hard, but it allows random access and reordering. Do we have durable structures of this kind?

I can’t imagine how to shoehorn this into Kafka or SQS.


Sounds like a Celery with SQLAlchemy backend.


Canvas, celery's orchestration system, leaves a lot to be desired




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: