How do I set up a loading raw data via CSV

Support
#1

Hi, am still learning and am looking to load data via CSV file into the Cuba Platform. I have structured Cuba and exported out the Excel format and I would like to load a lot of raw data back into the version so I can continue working on the development on some live data.

Looking to simply add a upload button that captures all the raw line data from csv into the templates - not just load the file.

Thanks

#2

Hi Peter,

A simplest way of importing data from a CSV file can be as follows:

  • Add FileUploadField to a screen controller:

<table id="customersTable">
...
    <buttonsPanel id="buttonsPanel">
...
        <upload id="uploadField"/>
    </buttonsPanel>
</table>
  • In the screen controller, add an upload field listener which obtains the uploaded file, processes it and deletes:

public class CustomerBrowse extends AbstractLookup {

    @Inject
    private FileUploadField uploadField;
    @Inject
    private FileUploadingAPI fileUploadingAPI;

    @Override
    public void init(Map<String, Object> params) {
        uploadField.addFileUploadSucceedListener(e -> {
            UUID fileId = uploadField.getFileId();
            File file = fileUploadingAPI.getFile(fileId);

            processFile(file);

            try {
                fileUploadingAPI.deleteFile(fileId);
            } catch (FileStorageException ex) {
                throw new RuntimeException(ex);
            }
        });
    }

    private void processFile(File file) {
        // import data
    }
}
  • In the processFile() method, load the file from the disk, parse CSV, create corresponding entity instances and use DataManager or a custom service to store them. See the data-manipulation for how to work with data programmatically.
#3

Exactly, but beware the processFile method cannot take too much time or the browser will be disconnected from the server at end.

#4

I this is of any help, I have used the supercsv library to easily import csv to entities/attributes.
http://super-csv.github.io/super-csv/index.html
There are also other alternative java libraries that can help do the job.

Also, if you are just trying to load back data that you previously created in your Cuba application, the EntityInspector screen has an export and import function that may be helpful.

Regards

#5

Thanks, I am looking to load new data from other application sources currently stored as CSV or excel files. Finding cuba a little challenging to navigate when it come to these add-ons and other needs whilst still working through the system. I looked ‘using third party add-on’ Using a Third-party Vaadin Component - CUBA Platform. Developer’s Manual but this only seems to track Vaadin add-on with maven dependency.

How can i install the supercsv through the cuba platform instead of going out opening code in files?. This is where I am finding in general a lot of sections on the cuba side not at all clear and hard to follow - when looking to do simple activity for someone whom is not 100% code driven but more function and purpose driven.

Can i move through the UI Component generation with supercsv?

Thanks so much for the feedback

#6

Hi,

? Adding supercsv dependency inCuba Studio UI

Go to Project Properties > Edit > Advanced tab.
In this tab, under the Module where you want to use the library, click the “+” sign, and add the reference address for the library - in this case “net.sf.supercsv:super-csv:2.4.0”. (See screenshot)

? Can i move through the UI Component generation with supercsv?

I am not sure what you mean exactly here. Supercsv is not a Cuba plugin but a java library, so after having added supercsv you will need to write some code to assign csv data to your entities - supercsv only makes this a bit easier to do by providing functions that read a csv file and generate entity instances for each line of csv with given attributes.

There is currently no GUI wizard to read a csv file and generate entities / entity instances as far as I know.

Regards,

addLib

#7

  1. You need to use some CSV handling library, like the SuperCSV mentioned earlier or the Apache Common-csv.

  2. Your mileage will vary depending on the quality of the CSV data (or the excel table if it from excel). There is no wizard or any other GUI helper to import CSV.

  3. I usually make a service, for data import/export. The service home is the core module, so You need to add the maven repository to the core module (in the Project settings, advanced tab). Of course you can do it in GUI module too, and add a the dependency to the GUI module in this case.

  4. Beware the import time and memory usage. If the import takes too much time, you will block the GUI thread, and after the import the client will be in disconnected state. To prevent this, long running import must be done in a Background task.

For the background task you wull need some GUI to show to the user, that system is busy. For this there is too possibility:

A. Undetermined progress window, you don’t know how far the process goes, the progress bar will be in undetermined mode (left right movement)

Put such a code in a button event handler or there where You want to start the import:


BackgroundWorkWindow.show(new BackgroundTask<Void, Void>(1200, this) {
                @Override
                public Void run(TaskLifeCycle<Void> taskLifeCycle) throws Exception {
                    persistenceService.importCSV(importfile);
                    return null;
                }
            }, windowTitle, windowMessage, cancel);

a. 1200 in the 1st line is the max time how long can run the background task, it must be longer than the normal execution time, or you will get a timeout error, and the execution will be aborted.

b. The persistenceService.ImportCSV (importfile) wil lprocess the whole CSV file, record by record.

c. You can specify the Window title in windowTitle, and the progress bar message in the windowMessage parameter. If the cancel parameter is true then you will get a cancel button in the window wher you can cancel the background task.

B. If you know how long itt will takes, then you can use the determined version of this window, but in this case importCSV method must yield the progress to the GUI, and it means that the importCSV method must be aware of such a need.

I prefer to make a general import methods so, the method can be called from several place with GUI or without so I don’t use a determined progresswindow…

#8

Hi, my question is:

  1. can I use a Groovy script like this The Structure of Groovy scripts - CUBA Platform. Developer’s Manual in order to load from CSV file standard data (like tax-codes downloaded from institutional web-sites) after and only after database creation, so it will be executed after every db (re)creation initiated by Studio or Gradle? It seems Groovy scripts are executed at every server startup The Execution of Database Scripts by the Server - CUBA Platform. Developer’s Manual. In either cases where do I have to put Groovy scripts in order to be leaded and executed by server?

  2. can I use Groovy Script for database migration. So can I load custom database schema of previous application in a specific project and create additional entity/tables from new consolidated model of new application (with generate model or simply by database creation scripts in cuba global module) and then create Groovy Scripts to migrate data from old to new schema using entity object and Persistence after having added/recreated Java source for entity in new schema?

Thank you in advance,
Fabrizio De Massis