4 Comments
User's avatar
Abhishek Pan's avatar

The SQS duplicates that show up most in practice come from the visibility timeout. It defaults to 30 seconds, so a moderation call that takes 45 seconds on a busy GPU gets redelivered and a second worker starts the same photo. Size the timeout off p99 processing time and keep the idempotency check as the backstop.

Pawel Plaszczak's avatar

Thanks! Fundamentals, but useful. I've worked with Kafka, less with RabbitMQ so it was helpful how you framed the difference in duplicate processing.

Edgar Bahilo Rodríguez's avatar

A queue can tell you a message exists, but it can't answer what's the status of photo XXX?. Also I think that the diagrams should distinguish between accepted (202) and done. Classical system of record polling or webhook for completion.

As usual the article is marvelous but I think state management deserves another post, with checkpointing included.

Nice to see you writing articles about fundamentals again :)

Bernardo Gomes de Abreu's avatar

Good article !!

Thanks for sharing !!!