The SQS duplicates that show up most in practice come from the visibility timeout. It defaults to 30 seconds, so a moderation call that takes 45 seconds on a busy GPU gets redelivered and a second worker starts the same photo. Size the timeout off p99 processing time and keep the idempotency check as the backstop.
A queue can tell you a message exists, but it can't answer what's the status of photo XXX?. Also I think that the diagrams should distinguish between accepted (202) and done. Classical system of record polling or webhook for completion.
As usual the article is marvelous but I think state management deserves another post, with checkpointing included.
Nice to see you writing articles about fundamentals again :)
The SQS duplicates that show up most in practice come from the visibility timeout. It defaults to 30 seconds, so a moderation call that takes 45 seconds on a busy GPU gets redelivered and a second worker starts the same photo. Size the timeout off p99 processing time and keep the idempotency check as the backstop.
Thanks! Fundamentals, but useful. I've worked with Kafka, less with RabbitMQ so it was helpful how you framed the difference in duplicate processing.
A queue can tell you a message exists, but it can't answer what's the status of photo XXX?. Also I think that the diagrams should distinguish between accepted (202) and done. Classical system of record polling or webhook for completion.
As usual the article is marvelous but I think state management deserves another post, with checkpointing included.
Nice to see you writing articles about fundamentals again :)
Good article !!
Thanks for sharing !!!