Discussion about this post

User's avatar
Abhishek Pan's avatar

The SQS duplicates that show up most in practice come from the visibility timeout. It defaults to 30 seconds, so a moderation call that takes 45 seconds on a busy GPU gets redelivered and a second worker starts the same photo. Size the timeout off p99 processing time and keep the idempotency check as the backstop.

Pawel Plaszczak's avatar

Thanks! Fundamentals, but useful. I've worked with Kafka, less with RabbitMQ so it was helpful how you framed the difference in duplicate processing.

2 more comments...

Ready for more?