Data Store Import

The general syntax of the Data Store import command is:

Before we discuss the command execution, let us look at the operations and configurations available for this command in detail.

Operations

There are three operations that can be performed using the Data Store import command:

• Insert: Inserts new rows of data into the table from the CSV file
• Update: Updates existing rows of data in the table
• Upsert: Enables both insert and update operations at the same time. If the record already exists, the row is updated. If the record does not exist, a new row with the data is inserted.

You can specify the operation to be executed in the input JSON configuration file.

Input CSV File

As mentioned earlier, the import operation bulk writes records from a CSV file to a table. You can provide the CSV file in two ways:

1. You can upload the CSV file containing the records in the File Store before you begin executing this operation. In this case, you must specify the unique ID generated for the file in the File Store, in the JSON configuration file that you pass during the command execution.
2. If you do not pass any JSON configuration file with the command, you must specify the path of the CSV file in your system in your command execution, as shown in the general syntax above. Catalyst will then upload the CSV file you provide to your project’s File Store and automatically obtain it’s File ID for execution. You can select the folder to save the file in the File Store.

Catalyst will consider the CSV file specified with the command as the higher priority, over the File ID specified in the JSON file. If the CSV file is not specified in either places, the CLI will display an error message during command execution.

JSON Configuration File Structure

A sample JSON configuration file specifying the requirements of the import operation, that you can optionally pass during the command execution, is shown below:

    

        
copy
        

            

        
    
    

         {
	"table_identifier" : "ZylkerEmpDatabase",
	"operation" : "upsert",
	"file_id" : 105000000206598,
	"callback" : {
		"url" : "https://hr.zylker.com/en/EmpRecords/_callback.php.com",
		"headers" : {
			"src" : "ZCatalyst",
			"operation" : "bulkwriteAPI"
		}
	},
	"find_by" : "EmpID",
	"fk_mapping" : [
		{
			"local_column" : "EmpID",
			"reference_column" : "EmployeeID"
		},
		{
			"local_column" : "DeptID",
			"reference_column" : "DepartmentID"
		}
	]
}


    

The parameters supported by the JSON configuration file are defined below:

Callback Properties

Import Job Processing States

There are three job processing states for the import operation:

• In-Progress: The job enters this state as soon as you execute the import command. The CLI will display a job_id parameter that you can use to check the status of the job execution for subsequent responses manually, if you have not configured a callback URL in the JSON.
  
  If you have configured a callback URL, a state response with the job_id will also be posted to the URL in the format you defined.
• Success: If the import operation is successful, Catalyst will provide a download URL where you can download a CSV report file that contains the details of the job execution. Catalyst also enables you to download the report file directly to your system through the CLI. If you have configured a callback URL, this response will be posted to it as well.
• Failed: If the import operation fails, the CLI will display the details of the errors that occurred. If you have configured a callback URL, this response will be posted to it as well.

Import Options

The import command execution process is described in detail in this section, for each option. The Data Store import command supports the following options:

--config <path>

The --config option enables you to define the path of the JSON configuration file in your system.

For example, you can execute the import operation and pass the JSON file path as shown below:

The CLI will then automatically schedule the job execution and mark it as “in-progress”. It will display a live stream of the execution process, along with the job_id .

If the job executes successfully, the CLI will prompt you to download the report file to your system. Type ‘y ‘, then press Enter to download.

The ZIP file containing the report file will be downloaded to your project directory.

You can use the download URL provided by Catalyst to download the report even when you are not working in your CLI. You must execute it as an API command .

--table <name|id>

The --table option enables you to specify the table name or Table ID of the table, to which the records must be written.

As mentioned earlier, if you don’t specify the table identifier in the JSON file, or if you don’t pass a JSON file during the command execution, you must specify it using this option.

Catalyst will consider the table specified in the option as the higher priority, over the one specified in the JSON file. If the table is not specified in either places, the CLI will display an error message during command execution.

For example, you can specify the table identifier along with the CSV file path in the following way:

If you specify the CSV file path, you must select the folder in your project’s File Store to which the file must be uploaded. The CLI will then execute the import operation, bulk write the records to the specified table, and provide the report file in the same way.

--production

The –production option enables you to directly execute the import operation in the production environment of your Catalyst project. If you use this option, the records from the input CSV file will be bulk written to the specified table directly in production.