Bug with cancel long reports

Hello!
I need to generate report that may build long time (longer then 10 minutes).
When I cancel It before they complete. But report remain in background tasks.

Снимок экрана от 2017-12-13 15-54-35

    BackgroundTaskThread-1156444328-51799eab-9bf7-b644-ea73-9baaa603c92b [id=6547] (RUNNABLE)
java.util.HashMap.getNode:579
java.util.HashMap.get:557
com.haulmont.yarg.reporting.DataExtractorImpl.getQueriesResult:148
com.haulmont.yarg.reporting.DataExtractorImpl.getBandData:100
com.haulmont.yarg.reporting.DataExtractorImpl.createBands:71
com.haulmont.yarg.reporting.DataExtractorImpl.extractData:55
com.haulmont.yarg.reporting.Reporting.loadBandData:139
com.haulmont.yarg.reporting.Reporting.runReport:98
com.haulmont.yarg.reporting.Reporting.runReport:82
com.haulmont.reports.ReportingBean.createReportDocument:315
com.haulmont.reports.ReportingBean.createReportDocument:284
com.haulmont.reports.ReportingBean.createReport:188
com.haulmont.reports.ReportServiceBean.createReport:40
sun.reflect.GeneratedMethodAccessor1302.invoke:-1
sun.reflect.DelegatingMethodAccessorImpl.invoke:43
java.lang.reflect.Method.invoke:498
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection:333
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint:190
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed:157
org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed:85
com.haulmont.cuba.core.sys.ServiceInterceptor.aroundInvoke:117
sun.reflect.GeneratedMethodAccessor348.invoke:-1
sun.reflect.DelegatingMethodAccessorImpl.invoke:43
java.lang.reflect.Method.invoke:498
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs:629
org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod:618
org.springframework.aop.aspectj.AspectJAroundAdvice.invoke:70
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed:168
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke:92
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed:179
org.springframework.aop.framework.JdkDynamicAopProxy.invoke:213
com.sun.proxy.$Proxy399.createReport:-1
sun.reflect.NativeMethodAccessorImpl.invoke0:-2
sun.reflect.NativeMethodAccessorImpl.invoke:62
sun.reflect.DelegatingMethodAccessorImpl.invoke:43
java.lang.reflect.Method.invoke:498
com.haulmont.cuba.core.sys.remoting.LocalServiceInvokerImpl.invoke:94
com.haulmont.cuba.web.sys.remoting.LocalServiceProxy$LocalServiceInvocationHandler.invoke:148
com.sun.proxy.$Proxy531.createReport:-1
com.haulmont.reports.gui.ReportGuiManager.getReportResult:264
com.haulmont.reports.gui.ReportGuiManager$1.run:355
com.haulmont.reports.gui.ReportGuiManager$1.run:350
com.haulmont.cuba.gui.backgroundwork.LocalizedTaskWrapper.run:55
com.haulmont.cuba.web.gui.executors.impl.WebBackgroundWorker$WebTaskExecutor.call:191
java.util.concurrent.FutureTask.run:266
com.haulmont.cuba.web.gui.executors.impl.WebBackgroundWorker$WebTaskExecutor.lambda$startExecution$1:350
com.haulmont.cuba.web.gui.executors.impl.WebBackgroundWorker$WebTaskExecutor$$Lambda$483/1505972304.run:-1
java.util.concurrent.ThreadPoolExecutor.runWorker:1149
java.util.concurrent.ThreadPoolExecutor$Worker.run:624
java.lang.Thread.run:748

Hi Sergey,
Could you provide sample report or sample application for this case?

The system provides an ability to cancel SQL statements while a report is running. Java code isn’t interrupted by report canceling.

Thanks,
Andrey

In project I use two datastores, to different databases. Main datastore - PostgreSql, additional - mssql.
Version platform now is 6.7.4. Same problem was on 6.5.4.
In report I use native SQL request. That request about 200-300 thousand rows from mssql (additional datastore). SQL request processed in time less then 30 seconds (benchmarked separate). Other time Java consume 100% of CPU load.

In platform 6.7.4:
XSLX export now can process about 150 thousands row. If rows more then 150 thousands, process never ended and need to be canceled or timed out (I set timeout to 30minutes).
Then I use report template CSV (thanks! I waited for this feature!), report can process 200-300 thousands without problems and ended about in ten minutes.
When process is cancelled manually it remain as background process (see screenshots and stacktrace), Java process consume 100% (one cpu core) until timeout.

Yes, my problem is with java processing large datasets. It’s slow.

In one report, I needed to take data from two different datastores. And in order to combine the data from all of them, I tried to do this in two ways:

  1. To retrieve data from the second database. I did it in suband with subquery one value based on parameters from parent databand.
  2. Formed the key in two queries and linked through the field.

According to my benchmark, the first method works faster.
When I do a thousand subqueries to get single value from the database is faster than to merge two different databands by one key.
Is it possible to optimize the algorithm for combining two data bands by a key field?

Hi Sergey

We plan to add an ability to interrupt loop in the DataExtractor when the report is canceled. See issue https://youtrack.cuba-platform.com/issue/YARG-50. So report thread should be interrupted. We plan to fix it in the platform version 6.7.5.

I have created an issue for optimizing the algorithm for combining two bands (try to use maps while searching by keys instead of loops). See https://youtrack.cuba-platform.com/issue/YARG-51

Thanks for reporting the problem.

Andrey

1 Like