I have been working on Amazon Simple Queue Service (SQS) since last week. This is the first time that I play with SQS and it disappoints me in several ways. Maybe I’m just not familiar enough with this product, but if someone could point out my misunderstandings of SQS, I would be absolutely appreciative instead of being ashamed for my ignorance.
Poll
Before last week, I naively assumed that, as the name of Message Queue indicates, SQS must have some kind of asynchronous API to notify client that new message has come. Unfortunately it doesn’t, which leaves me no choice but to start a thread to poll the queue, which is really clumsy since a short interval will cause unnecessary bandwidth and CPU waste whereas a long interval makes my code less responsive.
I guess this behaviour is due to SQS uses HTTP as its fundamental protocol, which doesn’t have a way to push data from server to client.
Latency
What really surprised me is that I found SQS is not as real-time as I thought. I wrote a test case for Sending and Receiving SQS message. Basically, it sends a message and then calls the receiving API to check if the message is successfully sent. Sometimes the green bulb of nunit shined happily. But most of the time, the receive operation got nothing and the assertion miserably failed, which means the message doesn’t appear in the queue immediately after sending is done and I need to poll the queue for a while to check if it’s really there.
The high latency also makes cleaning up the queue awkward. As SQS doesn’t have any API for emptying the queue, for each test case, a new queue with a random name is created in setup and destroyed in teardown, which prevents messages from the last test interfere future tests. This works, but creating a queue in SQS is really a time consuming operation. It makes running the test suite too slow to be practical.
Now what I do is polling the queue for new messages and deleting them in teardown. It seems to solve the problem and is way faster than creating and deleting queues but I’m not quite sure if the messages are really deleted, as occasionally my test suite still fails due to messages sent by the last execution of test.
To avoid polling, you could use SNS with HTTP Notifications, I’ve done this, and it works like a charm. I haven’t tested the latency though.
Thanks. I will have a look. But it seems that SNS is in beta.
Its been in beta for about 1,5 years, so I wouldn’t worry if I were you…
Hi ablmff,
If you are using java then there is a new class in amazon SDK, AmazonSQSAsyncClient which is worth looking at. This will allow you to make async calls to SQS. Also you can use Apache Camel to create a listener to the SQS. In this way you don’t have to create threads and start polling on your own. Camel has its own async processors so you can take further advantage of that to scale your app. I have a blog post on how to use camel and SQS in tandem.
http://weblog4j.com/2013/05/14/amazon-sqs-listening-to-sqs-using-apache-camel-the-spring-dsl-way/
As far as latency goes, it is inevitable if your main app is not deployed in amazon. To see true speed of SQS both your queue and app needs to be deployed in same amazon availability zone. I hope it helps.
Regards Niraj