Sadly I spent most of this weekend dealing with a problem caused by multi threaded Java code. Part of the system I develop is a multi threaded "server" that is used to run tasks in our main J2EE system based on schedules or the arrival of events. It was developed before the latest J2EE spec revisions so we don't use Message Beans or Timers yet.

For some time we've been getting the occasional report of the server "sticking". It looks like it is running a task that never finished. You can configure the server to run a set of tasks synchronously so if one of them sticks the rest will never run. Until this weekend our thinking has been that the problem has been with the task being run in the server not finishing. Perhaps it gets stuck in a loop or it finished but never reports back.

While testing some performance improvements we were finally able to reproduce the problem consistently in a test situation. From looking at weblogic we could see that nothing was actually running at the time, even though the server reported that something was. I took a look at the code for the server, and now that I knew what the problem wasn't it became clear that what it was was a race condition between the thread managing the queue of tasks to execute and the threads actually executing tasks. Data that was shared between threads was being updated outside of synchronized blocks. Still took a few attempts to tidy everything up but it's good to finally get the problem solved.