KryoException: java.lang.IndexOutOfBoundsException

mario · June 27, 2016, 7:57pm

Hi,

i’ve a problem with a deployed application. Setup: CUBA Version 6.1.5. WAR deployment. Every layer is deployed individually on its own virtual machine. app-core is clustered, cluster is connected via TUNNEL protocol and Jgroups gossip router.

What i encountered is the following scenario: When i click through the application in the web-app all of a sudden a problem occurs. Unfortunately i can’t really give more concrete information about when the problem occurs right now.

The web app displays the following error (in the logs as well as in the UI):

19:41:10.079 INFO  c.h.c.c.s.r.ClusteredHttpInvokerRequestExecutor - Invocation of http://ip-10-0-1-210.eu-west-1.compute.internal/remoting/cuba_DataService failed: java.io.IOException: Did not receive successful HTTP response: status code = 500, status message = [Internal Server Error]
19:41:10.079 INFO  c.h.c.c.s.r.ClusteredHttpInvokerRequestExecutor - No more URL available
19:41:10.096 ERROR com.haulmont.cuba.web.log.AppLog - Exception in com.haulmont.cuba.web.toolkit.ui.CubaButton: 
com.vaadin.server.ServerRpcManager$RpcInvocationException: Unable to invoke method click in com.vaadin.shared.ui.button.ButtonServerRpc
	at com.vaadin.server.ServerRpcManager.applyInvocation(ServerRpcManager.java:160)
	at com.vaadin.server.ServerRpcManager.applyInvocation(ServerRpcManager.java:118)
	at com.vaadin.server.communication.ServerRpcHandler.handleInvocations(ServerRpcHandler.java:414)
	at com.vaadin.server.communication.ServerRpcHandler.handleRpc(ServerRpcHandler.java:274)
	at com.vaadin.server.communication.UidlRequestHandler.synchronizedHandleRequest(UidlRequestHandler.java:79)
	at com.haulmont.cuba.web.sys.CubaVaadinServletService$CubaUidlRequestHandler.lambda$synchronizedHandleRequest$82(CubaVaadinServletService.java:306)
	at com.haulmont.cuba.web.sys.CubaVaadinServletService.withUserSession(CubaVaadinServletService.java:187)
	at com.haulmont.cuba.web.sys.CubaVaadinServletService$CubaUidlRequestHandler.synchronizedHandleRequest(CubaVaadinServletService.java:306)
	at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:41)
	at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1409)
	at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:369)
	at com.haulmont.cuba.web.sys.CubaApplicationServlet.serviceAppRequest(CubaApplicationServlet.java:242)
	at com.haulmont.cuba.web.sys.CubaApplicationServlet.service(CubaApplicationServlet.java:165)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at com.haulmont.cuba.web.sys.CubaHttpFilter.handleNotFiltered(CubaHttpFilter.java:110)
	at com.haulmont.cuba.web.sys.CubaHttpFilter.doFilter(CubaHttpFilter.java:97)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672)
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2500)
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2489)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedMethodAccessor77.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.vaadin.server.ServerRpcManager.applyInvocation(ServerRpcManager.java:158)
	... 38 more
Caused by: com.vaadin.event.ListenerMethod$MethodException: Invocation of method buttonClick in com.haulmont.cuba.web.gui.components.WebButton$1 failed.
	at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:528)
	at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:198)
	at com.vaadin.event.EventRouter.fireEvent(EventRouter.java:161)
	at com.vaadin.server.AbstractClientConnector.fireEvent(AbstractClientConnector.java:1008)
	at com.vaadin.ui.Button.fireClick(Button.java:377)
	at com.haulmont.cuba.web.toolkit.ui.CubaButton.fireClick(CubaButton.java:56)
	at com.vaadin.ui.Button$1.click(Button.java:54)
	... 42 more
Caused by: org.springframework.remoting.RemoteAccessException: Could not access HTTP invoker remote service at [cuba_DataService]; nested exception is java.io.IOException: Did not receive successful HTTP response: status code = 500, status message = [Internal Server Error]
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.convertHttpInvokerAccessException(HttpInvokerClientInterceptor.java:216)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:147)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy20.loadList(Unknown Source)
	at com.haulmont.cuba.client.sys.DataManagerClientImpl.loadList(DataManagerClientImpl.java:54)
	at com.haulmont.cuba.gui.data.impl.GenericDataSupplier.loadList(GenericDataSupplier.java:102)
	at com.haulmont.cuba.gui.data.impl.CollectionDatasourceImpl.loadData(CollectionDatasourceImpl.java:520)
	at com.haulmont.cuba.gui.data.impl.CollectionDatasourceImpl.refresh(CollectionDatasourceImpl.java:146)
	at com.haulmont.cuba.gui.data.impl.CollectionDatasourceImpl.refresh(CollectionDatasourceImpl.java:106)
	at com.haulmont.cuba.gui.data.impl.DsContextImpl.lambda$registerDependency$3(DsContextImpl.java:405)
	at com.haulmont.cuba.gui.data.impl.AbstractDatasource.fireItemChanged(AbstractDatasource.java:297)
	at com.haulmont.cuba.gui.data.impl.DatasourceImpl.setItem(DatasourceImpl.java:176)
	at com.haulmont.cuba.gui.components.EditorWindowDelegate.setItem(EditorWindowDelegate.java:177)
	at com.haulmont.cuba.web.gui.WebWindow$Editor.setItem(WebWindow.java:1357)
	at com.haulmont.cuba.gui.components.AbstractEditor.setItem(AbstractEditor.java:70)
	at com.haulmont.cuba.gui.WindowManager.openEditor(WindowManager.java:688)
	at com.haulmont.cuba.gui.components.WindowDelegate.openEditor(WindowDelegate.java:245)
	at com.haulmont.cuba.web.gui.WebWindow.openEditor(WebWindow.java:458)
	at com.haulmont.cuba.gui.components.actions.CreateAction.internalOpenEditor(CreateAction.java:230)
	at com.haulmont.cuba.gui.components.actions.CreateAction.actionPerform(CreateAction.java:222)
	at com.haulmont.cuba.web.gui.components.WebButton.performAction(WebButton.java:53)
	at com.haulmont.cuba.web.gui.components.WebButton$1.buttonClick(WebButton.java:44)
	at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.vaadin.event.ListenerMethod.receiveEvent(ListenerMethod.java:508)
	... 48 more
Caused by: java.io.IOException: Did not receive successful HTTP response: status code = 500, status message = [Internal Server Error]
	at org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:186)
	at com.haulmont.cuba.core.sys.remoting.ClusteredHttpInvokerRequestExecutor.doExecuteRequest(ClusteredHttpInvokerRequestExecutor.java:60)
	at org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:138)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:194)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:176)
	at org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:144)
	... 73 more

In the app-core the following error occurs:

19:40:59.576 ERROR c.h.c.c.sys.remoting.RemotingServlet - Error processing request
com.esotericsoftware.kryo.KryoException: java.lang.IndexOutOfBoundsException: Index: 30, Size: 11
Serialization trace:
view (com.haulmont.cuba.core.global.LoadContext)
arguments (com.haulmont.cuba.core.sys.remoting.CubaRemoteInvocation)
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:144) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:396) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.DefaultArraySerializers$ObjectArraySerializer.read(DefaultArraySerializers.java:307) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:708) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:790) ~[kryo-shaded-3.0.3.jar:na]
	at com.haulmont.cuba.core.sys.serialization.KryoSerialization.deserialize(KryoSerialization.java:140) ~[cuba-global-6.1.5.jar:6.1.5]
	at com.haulmont.cuba.core.sys.serialization.SerializationSupport.deserialize(SerializationSupport.java:56) ~[cuba-global-6.1.5.jar:6.1.5]
	at com.haulmont.cuba.core.sys.remoting.HttpServiceExporter.doReadRemoteInvocation(HttpServiceExporter.java:96) ~[cuba-core-6.1.5.jar:6.1.5]
	at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.readRemoteInvocation(HttpInvokerServiceExporter.java:117) ~[spring-web-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.readRemoteInvocation(HttpInvokerServiceExporter.java:96) ~[spring-web-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at com.haulmont.cuba.core.sys.remoting.HttpServiceExporter.handleRequest(HttpServiceExporter.java:76) ~[cuba-core-6.1.5.jar:6.1.5]
	at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:51) ~[spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at com.haulmont.cuba.core.sys.remoting.RemotingServlet.doService(RemotingServlet.java:150) [cuba-core-6.1.5.jar:6.1.5]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) [spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) [spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [servlet-api.jar:na]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) [spring-webmvc-4.2.0.RELEASE.jar:4.2.0.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [catalina.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.33]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.33]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.33]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.33]
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.33]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [catalina.jar:8.0.33]
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-coyote.jar:8.0.33]
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-coyote.jar:8.0.33]
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2500) [tomcat-coyote.jar:8.0.33]
	at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.run(AprEndpoint.java:2489) [tomcat-coyote.jar:8.0.33]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72-internal]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72-internal]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.33]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72-internal]
Caused by: java.lang.IndexOutOfBoundsException: Index: 30, Size: 11
	at java.util.ArrayList.rangeCheck(ArrayList.java:653) ~[na:1.8.0_72-internal]
	at java.util.ArrayList.get(ArrayList.java:429) ~[na:1.8.0_72-internal]
	at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:60) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:834) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:706) ~[kryo-shaded-3.0.3.jar:na]
	at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-shaded-3.0.3.jar:na]
	... 44 common frames omitted

Does anyone have a glue what is going on? Any help would be appreciated!

EDIT:
What i found are a few issues regarding the kryo library:
Kryo IndexOutofBoundsException in MapReferenceResolver · Issue #428 · EsotericSoftware/kryo · GitHub, Kryo 3.0.0 java.lang.IndexOutOfBoundsException · Issue #351 · EsotericSoftware/kryo · GitHub, Kryo IndexOutofBoundsException in MapReferenceResolver · Issue #363 · EsotericSoftware/kryo · GitHub and a few others: Issues · EsotericSoftware/kryo · GitHub

Unfortunately i can’t really figure out from the descriptions of the issues what solves the problem.

Bye,
Mario

krivopustov · June 28, 2016, 5:55am

Hi Mario,
Thank you for reporting the problem.
Unfortunately, Kryo has shown itself as more performant but not stable enough for our tasks, so we have switched to standard Java serialization by default in Platform 6.2. You can upgrade your project to 6.2 or stay on 6.1 and set the following app property on both middleware and clients:


cuba.serialization.impl = com.haulmont.cuba.core.sys.serialization.StandardSerialization

Meanwhile, we will try to reproduce and fix the issue.

mario · June 28, 2016, 12:13pm

Hi Konstantin,

thanks for the solution. I updated the platform version to 6.2.1. The Kyro Exception disappeared. Nevertheless another very similar exception occurs:


11:59:29.480 ERROR c.h.c.c.sys.remoting.RemotingServlet - Error processing request
java.lang.IllegalArgumentException: Failed to deserialize object
	at com.haulmont.cuba.core.sys.serialization.StandardSerialization.deserialize(StandardSerialization.java:71) ~[cuba-global-6.2.1.jar:6.2.1]
	at com.haulmont.cuba.core.sys.serialization.SerializationSupport.deserialize(SerializationSupport.java:56) ~[cuba-global-6.2.1.jar:6.2.1]
	at com.haulmont.cuba.core.sys.remoting.HttpServiceExporter.doReadRemoteInvocation(HttpServiceExporter.java:96) ~[cuba-core-6.2.1.jar:6.2.1]
	at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.readRemoteInvocation(HttpInvokerServiceExporter.java:117) ~[spring-web-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.readRemoteInvocation(HttpInvokerServiceExporter.java:96) ~[spring-web-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at com.haulmont.cuba.core.sys.remoting.HttpServiceExporter.handleRequest(HttpServiceExporter.java:76) ~[cuba-core-6.2.1.jar:6.2.1]
	at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:51) ~[spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:959) [spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) [spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at com.haulmont.cuba.core.sys.remoting.RemotingServlet.doService(RemotingServlet.java:148) [cuba-core-6.2.1.jar:6.2.1]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:968) [spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:870) [spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:648) [servlet-api.jar:na]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:844) [spring-webmvc-4.2.5.RELEASE.jar:4.2.5.RELEASE]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) [servlet-api.jar:na]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) [catalina.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.33]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) [tomcat-websocket.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) [catalina.jar:8.0.33]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106) [catalina.jar:8.0.33]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141) [catalina.jar:8.0.33]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) [catalina.jar:8.0.33]
	at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616) [catalina.jar:8.0.33]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) [catalina.jar:8.0.33]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:522) [catalina.jar:8.0.33]
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1095) [tomcat-coyote.jar:8.0.33]
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:672) [tomcat-coyote.jar:8.0.33]
	at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(AprEndpoint.java:2437) [tomcat-coyote.jar:8.0.33]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_72-internal]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72-internal]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-util.jar:8.0.33]
	at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72-internal]
Caused by: java.io.OptionalDataException: null
	at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1363) ~[na:1.8.0_72-internal]
	at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) ~[na:1.8.0_72-internal]
	at com.haulmont.cuba.core.sys.serialization.StandardSerialization.deserialize(StandardSerialization.java:69) ~[cuba-global-6.2.1.jar:6.2.1]
	... 34 common frames omitted

Directly before this exception occurs there is a warning that i saw which i already saw and wanted to fix anyway. Nevertheless - i’m not sure if it is related:

11:59:27.988 WARN  c.h.c.c.sys.remoting.RemotingServlet -  Invalid configuration parameters that may cause problems:
*****
***** cuba.webHostName=ip-10-0-2-74.eu-west-1.compute.internal, actual=ip-10-0-1-210.eu-west-1.compute.internal
***** cuba.webPort=8080, actual=80
***** cuba.webContextName=app-core, actual=
*****

The exception occurs on both middleware servers that are part of the cluster.

Bye,
Mario

krivopustov · June 28, 2016, 12:42pm

Looks like the problem is caused by the communication channel.
Could you send us all Tomcat logs zipped?

mario · June 28, 2016, 6:56pm

Well, what should i say… it quitely disappeared. I’m just not able to reproduce it.
The only thing that i changed is the that there was an warning i didn’t show that the logs above. It is about the setting:

cuba.trustedClientPermittedIpList

that didn’t matched with the real ip of the web-app (in the logs of the core-app). I changed it just to don’t bother you with this additional errors - since i could login anyway, which is kind of wired btw., i just wanted to remove it. Is it possible that this has anything to do with this situation?

Can you elaborate a little bit on the additional mentioned warning:

*****
***** cuba.webHostName=ip-10-0-2-74.eu-west-1.compute.internal, actual=ip-10-0-1-210.eu-west-1.compute.internal
***** cuba.webPort=8080, actual=80
***** cuba.webContextName=app-core, actual=
*****

What exactly is the problem from it and also how the app is able to get the actual values, since it is jailed in a Docker container and only the Docker port mapping

-p "80:8080"

“knows” about the actual port. Or does it parses the stuff from incoming HTTP requests?

I’d leave the discussion open for now because i have a bad feeling that the error might happen again. In this case i’ll drop the logs here.

Thanks!

Bye,
Mario

krivopustov · June 29, 2016, 6:40am

Hi Mario,
I don’t think cuba.trustedClientPermittedIpList could affect the issue. It is used only when a client block gets a user session without providing the user’s password. This is possible for “trusted” clients like web client and requires passing a special token (see cuba.trustedClientPassword) and for the client block being in the list of trusted IPs. If such requests were rejected, you would see warnings from LoginWorkerBean in the log.

Regarding the warning from RemotingServlet.
The mentioned properties together form the ServerId, which is very important for coordinating scheduled tasks. That’s why we notify if the parameters do not match the actual environment. But in fact, the ServerId just must be unique within the cluster (regardless of actual values), so I think you can ignore the warning as soon as you are sure that cuba.webHostName is set to different values on your servers.

Please let us know if the exception appears again.