Session sporadically hanging on Tomcat 9 on 6.10.7

Support
#1

Received the following stack trace after deploying a Cuba app to Tomcat 9 running on Windows Server 2012R2:

09-Apr-2019 09:32:11.668 SEVERE [ajp-nio-8011-exec-8] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [app_servlet] in context with path [...] threw exception
java.lang.UnsupportedOperationException: HTTP upgrade is not supported by this protocol
    at org.apache.coyote.AbstractProcessor.doHttpUpgrade(AbstractProcessor.java:867)
    at org.apache.coyote.AbstractProcessor.action(AbstractProcessor.java:592)
    at org.apache.coyote.Request.action(Request.java:432)
    at org.apache.catalina.connector.Request.upgrade(Request.java:2031)
    at org.apache.catalina.connector.RequestFacade.upgrade(RequestFacade.java:1123)
    at org.apache.tomcat.websocket.server.UpgradeUtil.doUpgrade(UpgradeUtil.java:235)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:78)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:107)
    at com.haulmont.addon.saml.web.security.saml.SamlLoginHttpRequestFilter.doFilter(SamlLoginHttpRequestFilter.java:36)
    at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:112)
    at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:73)
    at com.haulmont.cuba.web.sys.CubaHttpFilter.doFilter(CubaHttpFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
    at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:394)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)

I haven’t been able to reproduce the error consistently, but, when it occurs, it hangs the current session indefinitely and eventually the application runs out of memory.
After some digging in the Tomcat source, I see that the WsFilter class doesn’t exist in 8.5. In 9 the final stacktrace line in AbstractProcessor (ironically) looks like:

protected void doHttpUpgrade(UpgradeToken upgradeToken) {
    // Should never happen
    throw new UnsupportedOperationException(
            sm.getString("abstractProcessor.httpupgrade.notsupported"));
}

Whatever implementation of AbstractProcessor is running (my guess is StreamProcessor) must not override this.

Any chance of a workaround? Or am I stuck downgrading Tomcat to 8.5? FWIW this is the only issue we’ve noticed with Cuba 6.10 on Tomcat 9.

#3

Hi,

Could you share your web.xml file? Did you try to deploy a solution to Tomcat 9 without com.haulmont.addon.saml application component?

#4

The project that this is extending does use the SAML addon, but this one does not. This project’s web.xml did not declare that as an app component. I’m guessing it should and I’ve added it and restarted.

Worth mentioning this is an IIS redirect to Tomcat.

I see this code at the start of Tomcat’s WsFilter:

// This filter only needs to handle WebSocket upgrade requests
if (!sc.areEndpointsRegistered() ||
        !UpgradeUtil.isWebSocketUpgradeRequest(request, response)) {
    chain.doFilter(request, response);
    return;
}

I can only imagine that an AJP or HTTP 2 request is making it to this point and not meeting these conditions, after which, the WsFilter tries to upgrade the request, which AJP and HTTP 2 don’t support. I’m currently sitting with a breakpoint right after that that block, I’ll update if I catch anything.

And as I’m typing I’ve caught the issue. A web socket upgrade request to /PUSH gets to the WsFilter. It’s over AJP and the AjpProcessor does not support HTTP upgrade.

#5

I’ve checked Tomcat 9 with Platform 6.10 and everything runs OK. Unfortunately, I cannot reproduce your setup.

The problem could be related with async property of servlets and filters. WebSocket upgrade requires that all HTTP filters marked as <async-supported>true</async-supported>. But, it is just a guess.

In such a complex case I’d recommend that you switch to Long Polling transport instead of WebSocket using cuba.web.pushLongPolling application property of Web Client:

web-app.properties:

cuba.web.pushLongPolling = true