Why is there a need for persistent queues?
Most of the JMS based queues like ActiveMQ etc store the data inside filesystem. These writes most of the time are not transactional. Hence even if the write succeeds there is still a minor chance that data writes might have failed. This is due to reason that writes are mostly done on buffer-cache on the operating system and OS flushes these writes later to the disk. This problem doesn't exists for most of the systems. However for critical systems this is a problem. Sometimes its not ok to have data losses at all.
However having DB based queues suffer from many disadvantages as compared to standard JMS based queues. I won't go into reasons as its well explained into many books. Crucial part is databases queries are poll based which reduces their capacity to scale as many instances of subscriber contend for the same resource.
To understand this phenomenon I created a persistent queue based on SQLLite and understand why it happens at code level and can we do something about it?
Full code is at following place:
https://github.com/srivasrrahul/PersistentQueueBasedOnSQlLite
I'll try to explain the overall design in next post.
Most of the JMS based queues like ActiveMQ etc store the data inside filesystem. These writes most of the time are not transactional. Hence even if the write succeeds there is still a minor chance that data writes might have failed. This is due to reason that writes are mostly done on buffer-cache on the operating system and OS flushes these writes later to the disk. This problem doesn't exists for most of the systems. However for critical systems this is a problem. Sometimes its not ok to have data losses at all.
However having DB based queues suffer from many disadvantages as compared to standard JMS based queues. I won't go into reasons as its well explained into many books. Crucial part is databases queries are poll based which reduces their capacity to scale as many instances of subscriber contend for the same resource.
To understand this phenomenon I created a persistent queue based on SQLLite and understand why it happens at code level and can we do something about it?
Full code is at following place:
https://github.com/srivasrrahul/PersistentQueueBasedOnSQlLite
I'll try to explain the overall design in next post.
