Hello,
We see an issue with a few of our CUBA applications where the application is down with a Tomcat 404 error. After much troubleshooting, it appears that AWS updated our Tomcat instance to Tomcat 8.5 with Corretto 11 running on 64bit Amazon Linux 2/4.3.13.
After changing the version back to Tomcat 8.5 with Corretto 11 running on 64bit Amazon Linux 2/4.3.12 the application was able to load again. I’ve tested this with CUBA Platform v7.2.19 and 7.2.20 with the same results. Nothing else was changed in the build process (Just double-clicked ‘Build War’ in the CUBA pane on the left in Intellij IDEA).
I was wondering if this is a CUBA Platform issue, or if there are recommendations for solving the issue, as I’m expecting to see this occur in our other apps soon when AWS pushes the managed updates in those environments.
These are the SEVERE errors from the logs:
30-Oct-2023 00:53:01.292 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
30-Oct-2023 00:53:01.300 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
More detailed logs here:
----------------------------------------
/var/log/tomcat/localhost.2023-10-30.log
----------------------------------------
30-Oct-2023 19:54:33.913 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing AtmosphereFramework
30-Oct-2023 19:54:34.371 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log No Spring WebApplicationInitializer types detected on classpath
30-Oct-2023 19:54:49.244 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring RemotingServlet 'remoting'
30-Oct-2023 19:54:49.268 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStart Exception sending context initialized event to listener instance of class [com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener]
java.lang.RuntimeException: An error occurred while starting single WAR application
at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener.contextInitialized(SingleAppCoreServletListener.java:85)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4441)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4899)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:710)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:687)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:660)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1177)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1925)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener.contextInitialized(SingleAppCoreServletListener.java:78)
... 13 more
Caused by: java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
at org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:168)
at org.springframework.context.support.AbstractApplicationContext.getAutowireCapableBeanFactory(AbstractApplicationContext.java:357)
at com.haulmont.cuba.core.sys.CubaXmlWebApplicationContext.createBeanFactory(CubaXmlWebApplicationContext.java:29)
at org.springframework.context.support.AbstractRefreshableApplicationContext.refreshBeanFactory(AbstractRefreshableApplicationContext.java:127)
at org.springframework.context.support.AbstractApplicationContext.obtainFreshBeanFactory(AbstractApplicationContext.java:671)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553)
at org.springframework.web.servlet.FrameworkServlet.configureAndRefreshWebApplicationContext(FrameworkServlet.java:702)
at org.springframework.web.servlet.FrameworkServlet.createWebApplicationContext(FrameworkServlet.java:668)
at com.haulmont.cuba.core.sys.remoting.RemotingServlet.initWebApplicationContext(RemotingServlet.java:99)
at org.springframework.web.servlet.FrameworkServlet.initServletBean(FrameworkServlet.java:530)
at org.springframework.web.servlet.HttpServletBean.init(HttpServletBean.java:170)
at javax.servlet.GenericServlet.init(GenericServlet.java:143)
at com.haulmont.cuba.core.sys.remoting.RemotingServlet.init(RemotingServlet.java:181)
at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreContextLoader.contextInitialized(SingleAppCoreContextLoader.java:52)
... 18 more
30-Oct-2023 19:54:56.041 INFO [localhost-startStop-1] org.apache.catalina.core.ApplicationContext.log Initializing Spring SingleAppDispatcherServlet 'dispatcher'
30-Oct-2023 19:54:58.065 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.listenerStop Exception sending context destroyed event to listener instance of class [com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener]
java.lang.RuntimeException: An error occurred while destroying context of single WAR application
at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener.contextDestroyed(SingleAppCoreServletListener.java:99)
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4487)
at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5127)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:242)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:175)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:710)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:687)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:660)
at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1177)
at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1925)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener.contextDestroyed(SingleAppCoreServletListener.java:93)
... 14 more
Caused by: java.lang.IllegalStateException: BeanFactory not initialized or already closed - call 'refresh' before accessing beans via the ApplicationContext
at org.springframework.context.support.AbstractRefreshableApplicationContext.getBeanFactory(AbstractRefreshableApplicationContext.java:168)
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1172)
at com.haulmont.cuba.core.sys.AbstractWebAppContextLoader.contextDestroyed(AbstractWebAppContextLoader.java:137)
... 19 more
…and here:
----------------------------------------
/var/log/tomcat/catalina.2023-10-30.log
----------------------------------------
30-Oct-2023 19:54:20.389 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name: Apache Tomcat/8.5.94
30-Oct-2023 19:54:20.397 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built: Oct 13 2023 19:35:52 UTC
30-Oct-2023 19:54:20.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 8.5.94.0
30-Oct-2023 19:54:20.398 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name: Linux
30-Oct-2023 19:54:20.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version: 4.14.326-245.539.amzn2.x86_64
30-Oct-2023 19:54:20.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture: amd64
30-Oct-2023 19:54:20.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home: /usr/lib/jvm/java-11-amazon-corretto.x86_64
30-Oct-2023 19:54:20.399 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version: 11.0.21+9-LTS
30-Oct-2023 19:54:20.400 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor: Amazon.com Inc.
30-Oct-2023 19:54:20.400 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE: /usr/share/tomcat
30-Oct-2023 19:54:20.400 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME: /usr/share/tomcat
30-Oct-2023 19:54:20.400 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
30-Oct-2023 19:54:20.486 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
30-Oct-2023 19:54:20.591 INFO [main] org.apache.catalina.startup.Catalina.load Initialization processed in 1292 ms
30-Oct-2023 19:54:20.739 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
30-Oct-2023 19:54:20.740 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/8.5.94]
30-Oct-2023 19:54:20.769 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/ROOT]
30-Oct-2023 19:54:32.550 WARNING [localhost-startStop-1] org.apache.tomcat.jdbc.pool.ConnectionPool.checkPoolConfiguration maxIdle is smaller than minIdle, setting maxIdle to: 10
30-Oct-2023 19:54:33.872 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
30-Oct-2023 19:54:57.604 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
30-Oct-2023 19:54:57.614 SEVERE [localhost-startStop-1] org.apache.catalina.core.StandardContext.startInternal Context [] startup failed due to previous errors
30-Oct-2023 19:54:58.070 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesJdbc The web application [ROOT] registered the JDBC driver [org.postgresql.Driver] but failed to unregister it when the web application was stopped. To prevent a memory leak, the JDBC Driver has been forcibly unregistered.
30-Oct-2023 19:54:58.071 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [Thread-3] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.base@11.0.21/java.lang.Object.wait(Native Method)
java.base@11.0.21/java.lang.Object.wait(Object.java:328)
com.sun.star.lib.util.AsynchronousFinalizer$1.run(AsynchronousFinalizer.java:89)
30-Oct-2023 19:54:58.073 WARNING [localhost-startStop-1] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [JURTKiller] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
com.haulmont.cuba.core.sys.cleanup.StopThreadsCleanUp$JURTKiller.run(StopThreadsCleanUp.java:313)
30-Oct-2023 19:54:58.283 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat/webapps/ROOT] has finished in [37,512] ms
30-Oct-2023 19:54:58.291 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
30-Oct-2023 19:54:58.316 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 37725 ms
Thanks,
Adam