Just recording some experiences so that I can find them later, and in case they help others. At work we develop a J2EE application that runs on WebLogic (it also runs on WebSphere but that's not relevant here). Part of the application receives data via calls to session beans. The data is placed in a JMS queue for processing by message driven beans. This lets us smooth out peaks in the rate at which data is received as it doesn't have to be processed in real time.

A problem we have is that there can be collisions between different messages. In certain circumstances two messages will try to insert a database record with the same key value. The application handles this by requeuing the message that received the exception for later processing. No data is lost but we'd like to avoid the exceptions if we can.

The solution we've used is to place the data in one of several "virtual" queues by assigning a QueueNumber to each incoming message. The QueueNumber is calculated so that messages that will cause collisions all get the same QueueNumber. If each queue is processsed serially there will be no collisions.

The virtual queues are implemented using a single JMS queue by attaching a QueueNumber message attribute to each message. Message beans are defined for the virtual queues using message selectors that only pick up messages with the correct QueueNumber. On average each virtual queue will contain the same number of messages and be processed at the same rate so this shouldn't be too bad from a JMS performance point of view according to the BEA docs. For WebLogic if the max-beans-in-free-pool element is set to 1 in the weblogic-ejb-jar.xml file then we should only get one bean processing each message in the queue, so messages that could cause collisions will be processed serially.

However, while this all appears to work perfectly initially after a while threads will start to get stuck, especially as the load increases. We can reproduce this situation consistently. Taking a thread dump doesn't provide any useful info, at least not to us, as the threads that are stuck seem to be stuck in Class.forName.

We not been able to work out what the actual cause of the problem is, despite much google searching and thinking, but we have found a cure. If we increase max-beans-in-free-pool to 2 then we get no more stuck threads. We do now get some collisions, but not too many, and at least processing continues. From looking at some pages found on google other people with stuck threads seem to have max-beans-in-free-pool set to 1. Why this should cause the problem we don't know.