Elastic Beanstalk Issues with multitenancy addon

Support
#1

Hello,

We (the team I’m working with and I) are new to the CUBA Platform. We are currently developing a portal application which deployed to a simple AWS Elastic Beanstalk environment without issue. Recently, one of the developers modified the application to include the multitenancy addon. The application works fine in CUBA Studio but fails once it is deployed to Elastic Beanstalk, even when being deployed to a completely fresh environment from the ground up.

Here is a stack trace that I pulled from the EC2 app.log folder:

2021-08-04 18:25:28.237 ERROR [localhost-startStop-1] com.haulmont.cuba.core.sys.AbstractWebAppContextLoader - Error initializing application
org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: java.lang.IllegalArgumentException: Property 'deleteTs' not found in membership_SubscriptionGroupLink
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:448) ~[spring-orm-5.2.9.RELEASE.jar:5.2.9.RELEASE]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400) ~[spring-tx-5.2.9.RELEASE.jar:5.2.9.RELEASE]
        at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373) ~[spring-tx-5.2.9.RELEASE.jar:5.2.9.RELEASE]
        at com.haulmont.cuba.core.sys.TransactionImpl.<init>(TransactionImpl.java:63) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.TransactionsImpl.create(TransactionsImpl.java:55) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.PersistenceImpl.createTransaction(PersistenceImpl.java:139) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.security.auth.AuthenticationManagerBean.login(AuthenticationManagerBean.java:114) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.security.auth.AnonymousSessionHolder.loginAnonymous(AnonymousSessionHolder.java:118) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.security.auth.AnonymousSessionHolder.initializeAnonymousSessions(AnonymousSessionHolder.java:103) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.security.auth.AnonymousSessionHolder.applicationStarted(AnonymousSessionHolder.java:54) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.AppContext.startContext(AppContext.java:239) ~[cuba-global-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.AppContext$Internals.startContext(AppContext.java:302) ~[cuba-global-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.AbstractWebAppContextLoader.contextInitialized(AbstractWebAppContextLoader.java:94) ~[cuba-global-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.AppContextLoader.contextInitialized(AppContextLoader.java:52) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreContextLoader.contextInitialized(SingleAppCoreContextLoader.java:46) ~[cuba-core-7.2.14.jar:7.2.14]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at com.haulmont.cuba.core.sys.singleapp.SingleAppCoreServletListener.contextInitialized(SingleAppCoreServletListener.java:78) ~[cuba-shared-lib-7.2.14.jar:7.2.14]
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4705) ~[catalina.jar:8.5.63]
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5171) ~[catalina.jar:8.5.63]
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) ~[catalina.jar:8.5.63]
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743) ~[catalina.jar:8.5.63]
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719) ~[catalina.jar:8.5.63]
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705) ~[catalina.jar:8.5.63]
        at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1085) ~[catalina.jar:8.5.63]
        at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1802) ~[catalina.jar:8.5.63]
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

Caused by: javax.persistence.PersistenceException: java.lang.IllegalArgumentException: Property 'deleteTs' not found in membership_SubscriptionGroupLink
        at org.eclipse.persistence.internal.jpa.EntityManagerSetupImpl.deploy(EntityManagerSetupImpl.java:852) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.getAbstractSession(EntityManagerFactoryDelegate.java:219) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryDelegate.createEntityManagerImpl(EntityManagerFactoryDelegate.java:327) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManagerImpl(EntityManagerFactoryImpl.java:350) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryImpl.createEntityManager(EntityManagerFactoryImpl.java:313) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]
        at org.springframework.orm.jpa.JpaTransactionManager.createEntityManagerForTransaction(JpaTransactionManager.java:466) ~[spring-orm-5.2.9.RELEASE.jar:5.2.9.RELEASE]
        at org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:391) ~[spring-orm-5.2.9.RELEASE.jar:5.2.9.RELEASE]
        ... 32 common frames omitted

Caused by: java.lang.IllegalArgumentException: Property 'deleteTs' not found in membership_SubscriptionGroupLink
        at com.haulmont.chile.core.model.impl.MetaClassImpl.getPropertyNN(MetaClassImpl.java:107) ~[cuba-global-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.persistence.mapping.UuidMappingProcessor.process(UuidMappingProcessor.java:52) ~[cuba-core-7.2.14.jar:7.2.14]
        at com.haulmont.cuba.core.sys.persistence.EclipseLinkSessionEventListener.preLogin(EclipseLinkSessionEventListener.java:124) ~[cuba-core-7.2.14.jar:7.2.14]
        at org.eclipse.persistence.sessions.SessionEventManager.preLogin(SessionEventManager.java:622) ~[org.eclipse.persistence.core-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.preConnectDatasource(DatabaseSessionImpl.java:853) ~[org.eclipse.persistence.core-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.sessions.DatabaseSessionImpl.login(DatabaseSessionImpl.java:829) ~[org.eclipse.persistence.core-2.7.3-16-cuba.jar:na]
        at org.eclipse.persistence.internal.jpa.EntityManagerFactoryProvider.login(EntityManagerFactoryProvider.java:258) ~[org.eclipse.persistence.jpa-2.7.3-16-cuba.jar:na]

When trying to reach the application through Elastic Beanstalk, we get a 404 error.

Here are some other relevant details:

  • Elastic Beanstalk Platform: Tomcat 8.5 with Corretto 11 running on 64bit Amazon Linux 2

  • CUBA Platform version: 7.2.14

  • Project deployment type: WAR

  • Database: Postgres 12.5 RDS instance created through Elastic Beanstalk console (for demo purposes)

Let me know if there are any other details that I can provide that would be helpful. As a note, I did see in a related post that it might be helpful to include the com.haulmont.addon.sdbmt component in the application TestContainer classes, but that hasn’t seemed to help either.

Any help would be appreciated!

#2

Hi,
Can you show the source code of the membership_SubscriptionGroupLink entity?
According to the error message, it looks something is wrong with the declaration of this entity.

#3

Sure!

import com.haulmont.addon.sdbmt.entity.StandardTenantEntity;

import javax.persistence.Column;
import javax.persistence.Entity;
import javax.persistence.Table;
import javax.validation.constraints.NotNull;

@Table(name = "MEMBERSHIP_SUBSCRIPTION_GROUP_LINK")
@Entity(name = "membership_SubscriptionGroupLink")
public class SubscriptionGroupLink extends StandardTenantEntity {
    private static final long serialVersionUID = -1618545165939470845L;

    @NotNull
    @Column(name = "SUBSCRIPTION_ID", nullable = false)
    private String subscription_id;

    public String getSubscription_id() {
        return subscription_id;
    }

    public void setSubscription_id(String subscription_id) {
        this.subscription_id = subscription_id;
    }

}
#4

Well, the entity looks OK.

I can only suggest that maybe a re-deployment glitch.
Maybe restarting Tomcat instances on the Beanstalk service will help.

#5

Still no luck, unfortunately. I tried restarting Tomcat on the instances and even just recreating the entire environment just in case the issue had something to do with the updated database scripts. I’m still seeing the Property 'deleteTs' not found message for the SubscriptionGroupLink entity (plus another entity, the declaration of which seems as valid as the mentioned entity).

#8

UPDATE: The problem has changed somewhat.

I deployed the initial application (without multitenancy) to a completely new local environment (starting from an empty Postgres schema) with cuba.automaticDatabaseUpdate set to true this time, and this deployment worked as expected. I then tried to deploy the problematic application version with multitenancy enabled to this same local environment in CUBA Studio and got some different errors.

It seems like the problem does have something to do with the updated database scripts, after all. The original Property 'deleteTs' not found message no longer shows up for any entities. Instead, what I’m seeing are the following errors in CUBA Studio:

Task :app-core:updateDb FAILED
No init scripts from 50-SDBMT have been executed, running init scripts
Executing SQL script: C:\Users\dev1\StudioProjects\app\modules\core\build\db\50-SDBMT\init\postgres\10.create-db.sql
Aug 06, 2021 3:33:28 PM groovy.sql.Sql execute
WARNING: Failed to execute: -- begin CUBASDBMT_TENANT
WARNING: Failed to execute: -- begin CUBASDBMT_TENANT

create table CUBASDBMT_TENANT (
    ID uuid,
    VERSION integer not null,
    CREATE_TS timestamp,
    CREATED_BY varchar(50),
    UPDATE_TS timestamp,
    UPDATED_BY varchar(50),
    DELETE_TS timestamp,
    DELETED_BY varchar(50),
    TENANT_ID varchar(255),
    --
    NAME varchar(255) not null,
    ACCESS_GROUP_ID uuid not null,
    ADMIN_ID uuid not null,
    --
    primary key (ID)
) because: ERROR: relation "cubasdbmt_tenant" already exists
Aug 06, 2021 3:33:28 PM groovy.sql.Sql executeUpdate
WARNING: Failed to execute: insert into SYS_DB_CHANGELOG (SCRIPT_NAME, IS_INIT) values (?, ?) because: ERROR: duplicate key value violates unique constraint "sys_db_changelog_pkey"
  Detail: Key (script_name)=(50-SDBMT/update/postgres/19/281019-migrationTenantId.sql) already exists.

WARNING: Failed to execute: insert into SYS_DB_CHANGELOG (SCRIPT_NAME, IS_INIT) values (?, ?) because: ERROR: duplicate key value violates unique constraint "sys_db_changelog_pkey"

These are the same issues I encountered in Elastic Beanstalk when I replicated the experiment there.

At this point, is this a separate question entirely meriting its own separate post?

#9

Hi!

I would like to ask a few questions in order to clarify the sequence of actions.

Did you remove the add-on from your app? If so, did you rebuild the war?

Does it mean you added the add-on again, rebuilt the war and redeployed it or you just runned the app in CUBA Studio?

It seems like problem occurs because the database has not been completely cleaned up. The app without multitenancy add-on doesn’t have the entity CUBASDBMT_TENANT.

Regards,
Nadezhda.

#10

Sure!

Yes; I rolled back to the last known working version of the application in version control (which did not include the add-on). In this case I did not rebuild the war since I was just deploying the application through CUBA Studio.

The latter; I added the add-on again by pulling the latest version of the application from the repository, and then I ran that version again in CUBA Studio.

Are there steps that I am missing in between deployments that would facilitate the proper cleaning up of the database? I have thus far only tried to update the Main Data Store through the Update Database option in CUBA Studio.

#11

The problem occurs due after add-on removing table cubasdbmt_tenant has not been removed from database automatically. There is a workaround: create database after add-on removing.

Regards,
Nadezhda.