Issue with build UberJar

abd.ibrahim.allam · August 12, 2020, 9:19am

Hi team ,

Currently we are facing strange behavior ,

we have two environment UAT and Pre-Prod both are connected to Postgres AWS-RDS both DB are created from dumb taken from our local DB

Our Application is deployed over 2 Amazon Ec2 Instances using Jenkins pipeline
Application is deployed as Uber Jar

Currently Application is build and deployed on UAT without any issue
but on Pre-Prod we are facing issue related to cuba.useLocalServiceInvocation

Even same issue is happening if we are running application-jar locally
Although if we are running application from Intellj it is working fine

I think issue is related to some saved info in DB as we did many trials for that

If we are connecting to the original DB that the current RDS is created from it is working fine without the issue appearing
If we are connecting same build to the other DB which is RDS also but it is created from local Postgres UAT it is also working fine

I tried to set cuba.useLocalServiceInvocation to true and even remove it which will be true by default as per documentation

below is log exception which we are getting which running application locally

12:24:50.900 INFO  o.e.j.server.handler.ContextHandler     - Started o.e.j.w.WebAppContext@3d121db3{/vpay,jar:file:/home/abd.allam/vpay/build/distributions/uberJar/vpay.jar!/LIB-INF/app,AVAILABLE}
12:24:50.916 INFO  o.e.jetty.server.AbstractConnector      - Started ServerConnector@4e1d422d{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
12:24:50.917 INFO  org.eclipse.jetty.server.Server         - Started @9703ms
12:24:56.637 WARN  org.eclipse.jetty.server.session        - Problem checking if id node0hfugcq1emwdbwutjoo9dxn0m0 is in use

And this is from the server if we are running build over the server

2020-08-11 17:11:15.711 WARN  [qtp1359484306-20] org.eclipse.jetty.server.session - Problem checking if id node0ye4qf9j020aopvupa4ic46030 is in use
2020-08-11 17:11:16.609 DEBUG [qtp1359484306-15] com.haulmont.cuba.web.App - Initializing application
2020-08-11 17:13:16.613 ERROR [qtp1359484306-15] com.haulmont.cuba.web.AppUI - Unable to init ui
com.haulmont.cuba.web.sys.remoting.LocalServiceAccessException: Unable to connect to middleware block 'vpay-core'. No middleware blocks are registered in the application server. Possible causes:
'cuba.useLocalServiceInvocation' property enabled by mistake for unsupported deployment methods,

Thanks

abd.ibrahim.allam · August 12, 2020, 4:52pm

Hi Team ,

Also below exceptions is appearing on the logs

java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanEntityReference(XMLDocumentFragmentScannerImpl.java:1786) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2985) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:534) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:888) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:824) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:635) ~[na:na]
               at java.xml/com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse(SAXParserImpl.java:324) ~[na:na]
               at org.eclipse.jetty.xml.XmlParser.parse(XmlParser.java:255) ~[vpay.jar:na]
               at org.eclipse.jetty.xml.XmlConfiguration.<init>(XmlConfiguration.java:197) ~[vpay.jar:na]
               at org.eclipse.jetty.xml.XmlConfiguration.<init>(XmlConfiguration.java:214) ~[vpay.jar:na]
               at org.eclipse.jetty.plus.webapp.EnvConfiguration.configure(EnvConfiguration.java:117) ~[vpay.jar:na]
               at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:517) ~[vpay.jar:na]
               at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1454) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:854) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[vpay.jar:na]
               at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:119) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:167) ~[vpay.jar:na]
               at org.eclipse.jetty.server.Server.start(Server.java:418) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:110) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[vpay.jar:na]
               at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[vpay.jar:na]
               at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[vpay.jar:na]
               at com.haulmont.uberjar.CubaJettyServer.start(CubaJettyServer.java:127) ~[vpay.jar:na]
               at com.haulmont.uberjar.ServerRunner.execute(ServerRunner.java:249) ~[vpay.jar:na]
               at com.haulmont.uberjar.ServerRunner.main(ServerRunner.java:39) ~[vpay.jar:na]
15:55:39.732 INFO  o.e.j.w.StandardDescriptorProcessor     - NO JSP Support for /vpay, did not find org.eclipse.jetty.jsp.JettyJspServlet
15:55:39.796 INFO  org.eclipse.jetty.server.session        - DefaultSessionIdManager workerName=node0
15:55:39.796 INFO  org.eclipse.jetty.server.session        - No SessionScavenger set, using defaults
15:55:39.798 INFO  org.eclipse.jetty.server.session        - node0 Scavenging every 600000ms
15:55:40.010 INFO  c.h.c.c.s.AbstractWebAppContextLoader   - Initializing 'web' block, servlet context path: /vpay
15:55:40.139 INFO  c.h.cuba.core.sys.AppComponents         - Using app components: [com.haulmont.cuba, com.haulmont.charts, it.nexbit.cuba.security.forgotpassword, com.haulmont.addon.helium, com.haulmont.addon.restapi, de.diedavids.cuba.wizard]
15:55:40.189 INFO  c.h.c.c.s.AbstractWebAppContextLoader   - Loading app properties from classpath:com/company/vp/web-app.properties
15:55:40.193 INFO  c.h.c.c.s.AbstractWebAppContextLoader   - Loading app properties from /WEB-INF/local.app.properties
15:55:40.206 INFO  c.h.c.c.s.AbstractWebAppContextLoader   - cuba.webContextName: vpay
15:55:40.286 INFO  c.h.c.c.s.e.EnvironmentChecksRunner     - Environment checks on web module completed successfully
15:55:43.230 INFO  c.h.c.c.s.r.d.StaticServerSelector      - Server URLs: [http://localhost:8080/vpay-core/remoting]
15:55:43.253 INFO  c.h.c.w.s.r.WebRemoteProxyBeanCreator   - Configuring proxy beans for local service invocations: [cuba_AuthenticationService, cuba_TrustedClientService, cuba_ServerInfoService, cuba_UserSettingService, cuba_UserManagementService, cuba_DataService, cuba_ConfigStorageService, cuba_PersistenceManagerService, cuba_LockService, cuba_UserSessionService, cuba_FoldersService, cuba_EmailService, cuba_FileStorageService, cuba_EntitySnapshotService, cuba_UniqueNumbersService, cuba_LocalizedMessageService, cuba_ResourceService, cuba_ScheduledTasksService, cuba_EntityLogService, cuba_TestingService, cuba_DomainDescriptionService, cuba_RelatedEntitiesService, cuba_EntitySqlGenerationService, cuba_NumberIdService, cuba_QueryResultsService, cuba_DynamicAttributesCacheService, cuba_EntityRestoreService, cuba_LocalFileExchangeService, cuba_EntityImportExportService, cuba_ConstraintLocalizationService, cuba_AttributeAccessService, cuba_ExceptionReportService, cuba_BulkEditorDataService, cuba_PersistenceSecurityService, cuba_AccessGroupsService, cuba_AttributeOptionsLoaderService, cuba_RolesService, cuba_SecurityScopesService]
15:55:43.254 INFO  c.h.c.w.s.r.WebRemoteProxyBeanCreator   - Configuring proxy beans for local service invocations: [extsec_UserManagementService]
15:55:43.255 INFO  c.h.c.w.s.r.WebRemoteProxyBeanCreator   - Configuring proxy beans for local service invocations: [restapi_ServerTokenStore]
15:55:43.255 INFO  c.h.c.w.s.r.WebRemoteProxyBeanCreator   - Configuring proxy beans for local service invocations: [vp_LookupService, vp_RegexpService, vp_WorkScheduleService, vp_CurrencyConversionRateService, vp_VPDateUtilsService, vp_PayElementService, vp_PayElementBankAccountService, vp_PayElementGLLinkageService, vp_EmployeeAllowanceService, vp_EmailsConfigurationsPoolService, vp_PayrollService, vp_ApplicabilityRuleService, vp_EmployeeService, vp_CalculationRuleService, vp_PayrollCalendarService, vp_ProrataRuleService, vp_NumberUtilsService, vp_ReportService, vp_EmailService, vp_DataLoaderService, vp_CountryService, vp_BankService, vp_UserService, vp_PayElementPaymentMethodService, vp_ApplicationRoleTemplateService, vp_LicenseService, vp_LeaveService, vp_PublicHolidayService, vp_SectorService, vp_CountryUserService, vp_CurrencyService, vp_EmployeesGroupService, vp_DashBoardService, vp_GLAccountService, vp_WorkflowService, vp_AttributeService, vp_OrganizationService, vp_GlobeService, vp_ReportingStructureChangeService, vp_EmployeeRequestService, vp_EmployeesGroupRequestService, vp_OrganizationRequestService, vp_CountryRequestService, vp_GlobeRequestService, vp_SequencerService, vp_LeaveRequestService, vp_ExpenseClaimService]
15:55:45.862 INFO  c.h.c.c.s.CubaThreadPoolTaskScheduler   - Initializing ExecutorService 'scheduler'
15:55:46.334 INFO  c.h.cuba.core.sys.jmx.MBeanExporter     - Registering beans for JMX exposure: [vpay.cuba:type=ConfigStorage, vpay.cuba:type=CachingFacade, vpay.cuba:type=FileUploading, vpay.cuba:type=JmxNodeIdentifier, vpay.cuba:type=JmxLogControl, vpay.cuba:type=ClassLoaderManager, vpay.cuba:type=StatisticsCounter]
15:55:46.527 INFO  c.h.cuba.core.sys.MetadataImpl          - Initializing metadata
15:55:46.866 WARN  c.h.cuba.core.sys.MetaModelLoader       - Class it.nexbit.cuba.security.forgotpassword.entity.ResetPasswordLinkInfo is not loaded into metadata
15:55:46.867 WARN  c.h.cuba.core.sys.MetaModelLoader       - Class it.nexbit.cuba.security.forgotpassword.entity.ResetPasswordLinkInfo is not loaded into metadata
15:55:46.966 WARN  c.h.cuba.core.sys.MetaModelLoader       - Class com.company.vp.entity.dto.DepartmentSalariesDashboard is not loaded into metadata
15:55:47.051 WARN  c.h.cuba.core.sys.MetaModelLoader       - Class com.company.vp.entity.dto.DepartmentSalariesDashboard is not loaded into metadata
15:55:47.566 INFO  c.h.cuba.core.sys.MetadataImpl          - Metadata initialized in 1039 ms
15:55:47.836 INFO  o.e.j.s.handler.ContextHandler.vpay     - Initializing Spring RestAPIDispatcherServlet 'rest_api'
15:55:47.836 INFO  c.h.a.r.a.r.RestAPIDispatcherServlet    - Initializing Servlet 'rest_api'
15:55:51.151 INFO  s.d.s.w.PropertySourcedRequestMappingHandlerMapping- Mapped URL path [/swagger-api] onto method [springfox.documentation.swagger2.web.Swagger2Controller#getDocumentation(String, HttpServletRequest)]
15:55:51.182 INFO  s.d.s.w.p.DocumentationPluginsBootstrapper- Context refreshed
15:55:51.200 INFO  s.d.s.w.p.DocumentationPluginsBootstrapper- Found 1 custom documentation plugin(s)
15:55:51.345 INFO  s.d.s.w.s.ApiListingReferenceScanner    - Scanning for api listing references
15:55:51.408 INFO  c.h.a.r.a.r.RestAPIDispatcherServlet    - Completed initialization in 3572 ms
15:55:51.412 INFO  c.h.c.c.s.AbstractWebAppContextLoader   - AppContext started
15:55:51.448 INFO  o.e.j.s.handler.ContextHandler.vpay     - Initializing Spring CubaDispatcherServlet 'dispatcher'
15:55:51.448 INFO  c.h.c.web.sys.CubaDispatcherServlet     - Initializing Servlet 'dispatcher'
15:55:51.772 INFO  c.h.c.web.sys.CubaDispatcherServlet     - Completed initialization in 324 ms
15:55:51.773 INFO  o.e.j.server.handler.ContextHandler     - Started o.e.j.w.WebAppContext@17f9d882{/vpay,jar:file:/home/tghonamy/vpay/build/distributions/uberJar/vpay.jar!/LIB-INF/app,AVAILABLE}
15:55:51.809 INFO  o.e.jetty.server.AbstractConnector      - Started ServerConnector@2aa3cd93{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
15:55:51.810 INFO  org.eclipse.jetty.server.Server         - Started @13794ms
15:59:50.639 ERROR com.haulmont.cuba.web.AppUI             - Unable to init ui
com.haulmont.cuba.web.sys.remoting.LocalServiceAccessException: Unable to connect to middleware block 'vpay-core'. No middleware blocks are registered in the application server. Possible causes:
'cuba.useLocalServiceInvocation' property enabled by mistake for unsupported deployment methods,
or 'core' application missing from 'webapps' folder (in case of fast deployment).
               at com.haulmont.cuba.web.sys.remoting.LocalServiceProxy$LocalServiceInvocationHandler.createServiceAccessException(LocalServiceProxy.java:209) ~[classes/:na]
               at com.haulmont.cuba.web.sys.remoting.LocalServiceProxy$LocalServiceInvocationHandler.invoke(LocalServiceProxy.java:108) ~[classes/:na]
               at com.sun.proxy.$Proxy31.healthCheck(Unknown Source) ~[na:na]
               at com.haulmont.cuba.web.AppUI.init(AppUI.java:330) ~[classes/:na]
               at com.vaadin.ui.UI.doInit(UI.java:738) ~[shared/:na]
               at com.vaadin.server.communication.UIInitHandler.getBrowserDetailsUI(UIInitHandler.java:209) ~[shared/:na]
               at com.vaadin.server.communication.UIInitHandler.synchronizedHandleRequest(UIInitHandler.java:67) ~[shared/:na]
               at com.vaadin.server.SynchronizedRequestHandler.handleRequest(SynchronizedRequestHandler.java:40) ~[shared/:na]
               at com.vaadin.server.VaadinService.handleRequest(VaadinService.java:1578) ~[shared/:na]
               at com.vaadin.server.VaadinServlet.service(VaadinServlet.java:425) ~[shared/:na]
               at com.haulmont.cuba.web.sys.CubaApplicationServlet.serviceAppRequest(CubaApplicationServlet.java:329) ~[classes/:na]
               at com.haulmont.cuba.web.sys.CubaApplicationServlet.service(CubaApplicationServlet.java:215) ~[classes/:na]
               at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:876) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623) ~[vpay.jar:na]
               at org.springframework.web.filter.CompositeFilter$VirtualFilterChain.doFilter(CompositeFilter.java:108) ~[shared/:na]
               at org.springframework.web.filter.CompositeFilter.doFilter(CompositeFilter.java:74) ~[shared/:na]
               at com.haulmont.cuba.web.sys.CubaHttpFilter.doFilter(CubaHttpFilter.java:93) ~[classes/:na]
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[vpay.jar:na]
               at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:214) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) ~[vpay.jar:na]
               at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) ~[vpay.jar:na]
               at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) ~[vpay.jar:na]
               at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480) ~[vpay.jar:na]
               at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152) ~[vpay.jar:na]
               at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) ~[vpay.jar:na]
               at org.eclipse.jetty.server.Server.handle(Server.java:505) ~[vpay.jar:na]
               at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370) ~[vpay.jar:na]
               at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267) ~[vpay.jar:na]
               at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305) ~[vpay.jar:na]
               at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[vpay.jar:na]
               at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:781) ~[vpay.jar:na]
               at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:917) ~[vpay.jar:na]
               at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]

albudarov · August 13, 2020, 7:14am

Hi,
Can you present use the whole startup log file from the very beginning?
It looks like the middleware part of your application fails to start with an error.

abd.ibrahim.allam · August 13, 2020, 7:41am

Hi Alex ,

kindly find attached app.log

app.log (67.9 KB)

subbotin · August 13, 2020, 10:14am

Hi,

Most likely the files (jetty-env.xml or core/WEB-INF/web.xml) have an invalid XML format. That file contains evWvcy string (error from the log file).

Thank you

abd.ibrahim.allam · August 13, 2020, 2:33pm

Hi @subbotin ,

Thanks , You are right jetty-env was containing & inside password field and that was the issue When changing password and remove & if fixed the issue

Thanks