CUBA 7.2.13 - Scheduler Tasks not executing

Support
#1

Hello,

Facing issue with scheduler tasks on Cuba 7.2.13. Have 4 scheduler task configured as singleton to run daily on specified cron settings. Smallest task execution time is around 1 sec and for largest task its around 90 sec.
These task were executing fine. No environment changes done but now task history shows no execution record and tasks are not executed. Manually executing any task at the configured time gives no error and success is recorded in task execution history

In logs found
Exception in thread “cuba_core_scheduler-16” java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.signal(AbstractQueuedSynchronizer.java:1939)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1103)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)

Appreciate any help on this.

Thanks

#2

Hi,

How attributes of the affected scheduled tasks look like, can you show the attributes?

Also invoke the SQL select on your DB to check the state of these scheduled task:

select * from SYS_SCHEDULED_TASK

Column values for LAST_START_TIME, LAST_START_SERVER are the ones that may light up the situation.

#3

image
image1311×450 31.4 KB

Pls refer attached screenshot for attributes. Application is hosted on a single server, LAST_START_SERVER is localhost and LAST_START_TIME shows it last executed on 01 Aug 25

#4

I would suggest to set some sensible Timeout for the task. It might help.

#5

ok, have set timeout. Will update the results.

#6

No change after setting timeouts. Tasks not executed.

#7

Appreciate any other inputs to resolve this. Thanks

#8

What comes into my mind:

  1. Take thread dump from the application server and look what ScheduledRunnerThread-XXX threads are doing.
    Thread dump can be taken from the server console (jstack utility) or from JMX console - “java.lang:type=Threading” mbean.

  2. Temporarily change log settings for “com.haulmont.cuba.core.app.scheduling.Scheduling” logger to DEBUG or even better - TRACE level.
    Change threshold of your main logging appender to TRACE also.
    It can be done via Server Log administration screen.
    Check what messages are written into the log. Scheduling mechanism writes many log messages.

  3. Have you tried to restart the application server?
    Does the scheduling resume its functioning after restart?

#9

Thank you, will try 1 and 2. Have tried restarting but it does not resolve.

#10

To my surprise the thread dump from the application server has no ScheduledRunnerThread .

Is it possible that somehow the thread pool size is set to zero. Where can we check and if needed increase the threadpool size.

Thanks

#11

Take a look at these properties:

cuba.schedulingThreadPoolSize
com.haulmont.cuba.core.app.ServerConfig#getSchedulingThreadPoolSize

    /**
     * @return Maximum size of thread pool which is used to process scheduled tasks
     */
    @Property("cuba.schedulingThreadPoolSize")
    @DefaultInt(10)
    int getSchedulingThreadPoolSize();

and
cuba.schedulingActive
com.haulmont.cuba.core.app.ServerConfig#getSchedulingActive

    /**
     * @return Scheduled tasks execution control.
     */
    @Property("cuba.schedulingActive")
    @Source(type = SourceType.DATABASE)
    @DefaultBoolean(false)
    boolean getSchedulingActive();
#12

Sorry for the delayed update.

Our server is scheduled for shutdown at 12:30 am and start up at 6:30 am. Rescheduled all 4 scheduled tasks to trigger immediately after server startup at 6:35 am to 6:39 am. Added in cuba.schedulingThreadPoolSize=10 in
server.log (26.8 KB)
app.properties file

Next morning, 3rd Sept, all 4 jobs are triggered, executed successfully and recorded in scheduler history. No more scheduled tasks for the day.

Later server logs show java.lang.IllegalMonitorStateException

Exception in thread “cuba_core_scheduler-4” java.lang.IllegalMonitorStateException
at java.util.concurrent.locks.ReentrantLock$Sync.tryRelease(ReentrantLock.java:151)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1261)
at java.util.concurrent.locks.ReentrantLock.unlock(ReentrantLock.java:457)

Same exceptions happen at server shutdown

Next day, 4 Sept, immediately after server startup same java.lang.IllegalMonitorStateException occurred, and no scheduled task was triggered. Same exceptions happen at server shutdown

Relevant server log attached.

Appreciate any help to fix this.

Thanks

#13

If these exceptions happen during normal server functioning (not during shutdown), it looks like a bug somewhere in java.util.concurrent code.
But there is no clue if the bug was provoked by the CUBA code.

(here in comments you can see some information: java - Why am I getting IllegalMonitorStateException? - Stack Overflow)

What is your Java version on the server?
I would advise to upgrade it to the latest bugfix version.