Tomcat parallel deployment fails (conflicting JMX MBean keys)

I’m trying make use of Tomcat parallel deployment to minimize application downtime when deploying new versions of my application.
I am packaging my app with the ‘buildWar’ gradle task, and then deploying in my own tomcat webapp folder.
At the moment this isn’t working: the newer versions of the application fail at startup since they can’t register MBeans (they have the same key of the previously deployed version).
Here is part of the stacktrace:

2017-10-19 11:13:17.044 INFO  [localhost-startStop-1] com.haulmont.cuba.core.sys.jmx.MBeanExporter - Registering beans for JMX exposure: [app.cuba:type=ConfigStorage, app.cuba:type=CachingFacade, app.cuba:type=FileUploading, app.cuba:type=JmxNodeIdentifier, app.cuba:type=JmxLogControl, app.cuba:type=ClassLoaderManager, app.cuba:type=StatisticsCounter]
2017-10-19 11:13:17.081 ERROR [localhost-startStop-1] com.haulmont.cuba.core.sys.AbstractWebAppContextLoader - Error initializing application
org.springframework.jmx.export.UnableToRegisterMBeanException: Unable to register MBean [com.haulmont.cuba.web.jmx.ConfigStorage@6daa028c] with key 'app.cuba:type=ConfigStorage'; nested exception is javax.management.InstanceAlreadyExistsException: app.cuba:type=ConfigStorage
        at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:628)
        at org.springframework.jmx.export.MBeanExporter.registerBeans(MBeanExporter.java:550)
        at org.springframework.jmx.export.MBeanExporter.afterSingletonsInstantiated(MBeanExporter.java:432)
        at com.haulmont.cuba.core.sys.jmx.MBeanExporter.afterSingletonsInstantiated(MBeanExporter.java:67)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:781)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:866)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:542)
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:139)
        at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:93)
        at com.haulmont.cuba.core.sys.CubaClassPathXmlApplicationContext.<init>(CubaClassPathXmlApplicationContext.java:27)
        at com.haulmont.cuba.core.sys.AbstractAppContextLoader.createApplicationContext(AbstractAppContextLoader.java:79)
        at com.haulmont.cuba.core.sys.AbstractAppContextLoader.initAppContext(AbstractAppContextLoader.java:56)
        at com.haulmont.cuba.core.sys.AbstractWebAppContextLoader.contextInitialized(AbstractWebAppContextLoader.java:72)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:988)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1860)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: javax.management.InstanceAlreadyExistsException: app.cuba:type=ConfigStorage
        at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
        at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
        at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
        at org.springframework.jmx.support.MBeanRegistrationSupport.doRegister(MBeanRegistrationSupport.java:195)
        at org.springframework.jmx.export.MBeanExporter.registerBeanInstance(MBeanExporter.java:682)
        at org.springframework.jmx.export.MBeanExporter.registerBeanNameOrInstance(MBeanExporter.java:618)
        ... 25 common frames omitted
19-Oct-2017 11:13:17.102 GRAVE &#91;localhost-startStop-1&#93; org.apache.catalina.core.StandardContext.startInternal One or more listeners failed to start. Full details will be found in the appropriate container log file
19-Oct-2017 11:13:17.105 GRAVE &#91;localhost-startStop-1&#93; org.apache.catalina.core.StandardContext.startInternal Context &#91;/app##002&#93; startup failed due to previous errors
11:13:17,110 |-INFO in ch.qos.logback.classic.servlet.LogbackServletContextListener@6c335933 - About to stop ch.qos.logback.classic.LoggerContext &#91;default&#93;

I’ve tries messing with the spring configuration files (core and web) to change the registration policy, but had little success. Here is what I trying to add:

<!-- Exporting metrics via JMX -->
<context:mbean-server id="mbeanServer">
<context:mbean-export server="mbeanServer" default-domain="com.company" registration="replaceExisting">
</context:mbean-export>
</context:mbean-server>

I’m not sure I’m going in the right direction; it seems to me thet ‘mbean-server id’ has a dynamic value on cuba applications.
From ‘JMX Console > Inspect MBean “JMImplementation:type=MBeanServerDelegate”’ i see that ‘MBeanServerId’ has “$servername_$timestamp” naming convention.
(Also, I’m not sure the ‘default-domain’ is of any use at all, and what value should it have.)

I’m suspecting my approach is way off!
How can I change registration policy for MBeans in a cuba application?
Are there deployment strategies alternatives for zero or minimal downtime?

MBeans are registered using the cuba.webContextName application property as a prefix, see for example cuba-spring.xml file. In theory, you could add a version number to this property with each new build, then all MBeans will have new names.

We don’t have any experience in parallel deployment, so please share your experience if you have any results. When we have the “no downtime” requirement, we usually use two separate servers with a load balancer.

Nicholas,
Did you ever get the Tomcat parallel deployment working ?

I need to know the settings I need to have in order to have multiple Cuba projects deployed to the same tomcat “webapps” folder