Rabbitmq Queued Messages Keep Increasing

We have a Windows based Celery/RabbitMQ server that executes long-running python tasks out-of-process for our web application. What this does, for example, is take a CSV file and p

Solution 1:

Answering my own question;

Celery sends a result message back for every task in the calling code. This message is sent back via the same AMPQ queue. This is why the tasks were working, but the queue kept filling up. We were not handling these results, or even interested in them.

I added ignore_result=True to the celery task, so the task does not send result messages back into the queue. This was the main solution to the problem.

Furthermore, the configuration option CELERY_SEND_EVENTS=False was added to speed up celery. If set to TRUE, this option has Celery send events for external monitoring tools.

On top of that CELERY_TASK_RESULT_EXPIRES=3600 now makes sure that even if results are sent back, that they expire after one hour if not picked up/acknowledged.

Finally CELERY_RESULT_PERSISTENT was set to False, this configures celery to not store these result messages on disk. They will vanish when the server crashes, which is fine in our case, as we don't use them.

So in short; if you don't need feedback in your app about if and when the tasks are finished, use ignore_result=True on the celery task, so that no messages are sent back. If you do need that information, make sure you pick up and handle the results, so that the queue stops filling up.

Solution 2:

If you don't need the reliability then you can make your queues transient.

http://celery.readthedocs.org/en/latest/userguide/optimizing.html#optimizing-transient-queues

CELERY_DEFAULT_DELIVERY_MODE = 'transient'