import data reference

yb-voyager import data

The following page describes the usage of the following import commands:

import data

For offline migration, Import the data to the YugabyteDB database.

For live migration (with fall-forward and fall-back), the import data command is an alias of import data to target which imports the data from the data directory to the target database, and starts ingesting the new changes captured by export data to the target database.

Syntax

Syntax for import data is as follows:

Usage: yb-voyager import data [ <arguments> ... ]

Syntax for import data to target is as follows:

Usage: yb-voyager import data to target [ <arguments> ... ]

Arguments

The following table lists the valid CLI flags and parameters for import data command.

When run at the same time, flags take precedence over configuration flag settings.

CLI flag
 Config file parameter Description
--run-guardrails-checks 
import-data:
  run-guardrails-checks:
Run guardrails checks during migration.
Default: true
Accepted values: true, false, yes, no, 0, 1
--batch-size 
import-data:
  batch-size:
Size of batches in the number of rows generated for ingestion during import data.
Default: 20000 rows
Example: yb-voyager import data ... --batch-size 20000
--disable-pb 
import-data:
  disable-pb:
Use this argument to disable progress bar or statistics during data import.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
--enable-upsert 
import-data:
  enable-upsert:
Enable UPSERT mode on target tables while importing data.
Note: Ensure that tables on the target YugabyteDB database do not have secondary indexes. If a table has secondary indexes, setting this flag to true may lead to corruption of the indexes.
Default: false
Usage for disabling the mode: yb-voyager import data ... --enable-upsert false
Accepted parameters: true, false, yes, no, 0, 1
--table-list 
import-data:
  table-list:
Comma-separated list of names of source database tables whose data is to be imported. Table names can also be glob patterns containing wildcard characters, such as an asterisk () (matches zero or more characters) or question mark (?) (matches one character). To use a glob pattern for table names, enclose the list in single quotes ('').
For example, --table-list '"Products", order'.
This argument is unsupported for live migration.
--exclude-table-list 
import-data:
  exclude-table-list:
Comma-separated list of names of source database tables for which data needs to be excluded during import. Table names follow the same convention as --table-list.
This argument is unsupported for live migration.
--table-list-file-path 
import-data:
  table-list-file-path:
Path of the file containing the list of names of source database tables (comma separated or line separated) to import. Table names use the same convention as --table-list.
--exclude-table-list-file-path 
import-data:
  exclude-table-list-file-path:
Path of the file containing the list of names of source database tables (comma separated or line separated) to exclude while importing data of those exported tables. Table names follow the same convention as --table-list.
--enable-adaptive-parallelism 
import-data:
  enable-adaptive-parallelism:
Adapt parallelism based on the resource usage (CPU, memory) of the target YugabyteDB cluster.
Default: true
Accepted parameters: true, false, yes, no, 0, 1
--adaptive-parallelism-max 
import-data:
  adaptive-parallelism-max:
Number of maximum parallel jobs to use while importing data when adaptive parallelism is enabled. By default, voyager tries to determine the total number of cores N and use N/2 as the maximum parallel jobs.
--parallel-jobs 
import-data:
  parallel-jobs:
Number of parallel jobs to use while importing data. Depending on the YugabyteDB database configuration, the value of --parallel-jobs should be tweaked such that at most 50% of target cores are utilised.
Default: If yb-voyager can determine the total number of cores N in the YugabyteDB database cluster, it uses N/2 as the default for import data and N/4 for import data to target. Otherwise, it defaults to twice the number of nodes in the cluster.
--skip-replication-checks 
import-data:
  skip-replication-checks:
It is NOT recommended to have any form of replication (CDC/xCluster) running on the target YugabyteDB cluster during data import as it may lead to a rapid increase in disk use. If detected, data import is aborted. Use this flag to turn off the checks and continue importing data.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
--truncate-tables 
import-data:
  truncate-tables:
Truncate tables on target YugabyteDB database before importing data. This option is only valid if --start-clean is set to true.
Default: false
--use-public-ip 
import-data:
  use-public-ip:
Use the node public IP addresses to distribute --parallel-jobs uniformly on data import.
Default: false
Note that you may need to configure the database with public IP addresses by setting server-broadcast-addresses.
Example: yb-voyager import data ... --use-public-ip true
Accepted parameters: true, false, yes, no, 0, 1
--target-endpoints 
import-data:
  target-endpoints:
Comma-separated list of node's endpoint to use for parallel import of data
Default: Use all the nodes in the cluster. For example: "host1:port1,host2:port2" or "host1,host2". Note: use-public-ip flag will be ignored if this is used.
-e, --export-dir 
export-dir:
Path to the export directory. This directory is a workspace used to store exported schema DDL files, export data files, migration state, and a log file.
--send-diagnostics 
send-diagnostics:
Enable or disable sending diagnostics information to Yugabyte.
Default: true
Accepted parameters: true, false, yes, no, 0, 1
--target-db-host 
target:
  db-host:
Domain name or IP address of the machine on which target database server is running.
Default: "127.0.0.1"
--target-db-name 
target:
  db-name:
Target database name.
Default: yugabyte
--target-db-password 
target:
  db-password:
Password to connect to the target YugabyteDB database. Alternatively, you can also specify the password by setting the environment variable TARGET_DB_PASSWORD. If you don't provide a password via the CLI during any migration phase, yb-voyager will prompt you at runtime for a password. If the password contains special characters that are interpreted by the shell (for example, # and $), enclose the password in single quotes.
--target-db-port 
target:
  db-port:
Port number of the target database server.
Default: 5433
--target-db-schema 
target:
  db-schema:
Schema name of the target database. MySQL and Oracle migrations only.
--target-db-user 
target:
  db-user:
Username of the target database.
--target-ssl-cert 
target:
  ssl-cert:
Path to a file containing the certificate which is part of the SSL <cert,key> pair.
--target-ssl-key 
target:
  ssl-key:
Path to a file containing the key which is part of the SSL <cert,key> pair.
--target-ssl-crl 
target:
  ssl-crl:
Path to a file containing the SSL certificate revocation list (CRL).
--target-ssl-mode 
target:
  ssl-mode:
Specify the SSL mode for the target database as one of disable, allow, prefer (default), require, verify-ca, or verify-full.
--target-ssl-root-cert 
target:
  ssl-root-cert:
Path to a file containing SSL certificate authority (CA) certificate(s).
--start-clean — Starts a fresh import with data files present in the data directory.
If the target YugabyteDB database has non-empty tables, you are prompted to continue the import without truncating those tables; if you go ahead without truncating, then yb-voyager starts ingesting the data present in the data files in upsert mode.
Note that for cases where a table doesn't have a primary key, duplicate data may be inserted. You can avoid duplication by excluding the table using --exclude-table-list, or by truncating those tables manually before using the start-clean flag.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
-h, --help — Command line help.
-y, --yes — Answer yes to all prompts during the export schema operation.
Default: false

Example

Offline migration

An example for offline migration is as follows:

Configuration file:

yb-voyager import data --config-file <path-to-config-file>

CLI:

yb-voyager import data --export-dir /dir/export-dir \
        --target-db-host 127.0.0.1 \
        --target-db-user ybvoyager \
        --target-db-password 'password' \
        --target-db-name target_db \
        --target-db-schema target_schema \
        --parallel-jobs 12

Live migration

An example for all live migration scenarios is as follows:

Configuration file:

yb-voyager import data to target --config-file <path-to-config-file>

CLI:

yb-voyager import data to target --export-dir /dir/export-dir \
        --target-db-host 127.0.0.1 \
        --target-db-user ybvoyager \
        --target-db-password 'password' \
        --target-db-name target_db \
        --target-db-schema target_schema \
        --parallel-jobs 12

import data status

For offline migration, get the status report of an ongoing or completed data import operation. The report contains migration status of tables, number of rows or bytes imported, and percentage completion.

Syntax

Usage: yb-voyager import data status [ <arguments> ... ]

Arguments

The following table lists the valid CLI flags and parameters for import data status command.

When run at the same time, flags take precedence over configuration flag settings.

CLI flag
 Config file parameter Description
-e, --export-dir 
export-dir:
Path to the export directory. This directory is a workspace used to keep the exported schema, data, state, and logs.
-h, --help — Command line help.
-y, --yes — Answer yes to all prompts during the import data operation.
Default: false

Example

Configuration file:

yb-voyager import data status --config-file <path-to-config-file>

CLI:

yb-voyager import data status --export-dir /dir/export-dir

get data-migration-report

Note that this command is applicable for Live migrations only.

Provides a consolidated report of data migration per table among all the databases involved in the live migration. The report includes the number of rows exported, the number of rows imported, change events exported and imported (INSERTS, UPDATES, and DELETES), and the final row count on the database.

Syntax

Usage: yb-voyager get data-migration-report [ <arguments> ... ]

Arguments

The following table lists the valid CLI flags and parameters for get data-migration-report command.

When run at the same time, flags take precedence over configuration flag settings.

CLI flag
 Config file parameter Description
-e, --export-dir 
export-dir:
Path to the export directory. This directory is a workspace used to store exported schema DDL files, export data files, migration state, and a log file.
--source-db-password 
source:
  db-password:
Password to connect to the source database. If you don't provide a password via the CLI during any migration phase, yb-voyager will prompt you at runtime for a password. Alternatively, you can also specify the password by setting the environment variable SOURCE_DB_PASSWORD. If the password contains special characters that are interpreted by the shell (for example, # and $), enclose it in single quotes.
--source-replica-db-password 
source-replica:
  db-password:
Password to connect to the source-replica database. Alternatively, you can also specify the password by setting the environment variable SOURCE_REPLICA_DB_PASSWORD. If the password contains special characters that are interpreted by the shell (for example, # and $), enclose it in single quotes.
--target-db-password 
target:
  db-password:
Password to connect to the target YugabyteDB database. Alternatively, you can also specify the password by setting the environment variable TARGET_DB_PASSWORD. If you don't provide a password via the CLI during any migration phase, yb-voyager will prompt you at runtime for a password. If the password contains special characters that are interpreted by the shell (for example, # and $), enclose the password in single quotes.
-h, --help — Command line help.

Example

Configuration file:

yb-voyager get data-migration-report --config-file <path-to-config-file>

CLI:

yb-voyager get data-migration-report --export-dir /dir/export-dir

import data to source

Note that this command is applicable for Live migrations only.

Imports data to the source database, and streams new changes from the YugabyteDB database to the source database.

Syntax

Usage: yb-voyager import data to source [ <arguments> ... ]

Arguments

The following table lists the valid CLI flags and parameters for import data to source command.

When run at the same time, flags take precedence over configuration flag settings.

CLI flag
 Config file parameter Description
--run-guardrails-checks 
import-data-to-source:
  run-guardrails-checks:
Run guardrails checks during migration.
Default: true
Accepted values: true, false, yes, no, 0, 1
--parallel-jobs 
import-data-to-source:
  parallel-jobs:
Number of parallel jobs to use while importing data.
Default: 16(Oracle)
-e, --export-dir 
export-dir:
Path to the export directory. This directory is a workspace used to store exported schema DDL files, export data files, migration state, and a log file.
--send-diagnostics 
send-diagnostics:
Enable or disable sending diagnostics information to Yugabyte.
Default: true
Accepted parameters: true, false, yes, no, 0, 1
--source-db-password 
source:
  db-password:
Password to connect to the source database.
-h, --help — Command line help for import data to source.
--start-clean — Starts a fresh import with exported data files present in the export-dir/data directory.
If any table on YugabyteDB database is non-empty, it prompts whether you want to continue the import without truncating those tables.
Note that for the cases where a table doesn't have a primary key, this may lead to insertion of duplicate data. To avoid this, exclude the table using the --exclude-file-list or truncate those tables manually before using the start-clean flag.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
-y, --yes — Answer yes to all prompts during the migration.
Default: false

Example

Configuration file:

yb-voyager import data to source --config-file <path-to-config-file>

CLI:

yb-voyager import data to source --export-dir /dir/export-dir \
        --source-db-password 'password'

import data to source-replica

Note that this command is applicable for Live migrations only.

Imports data to the source-replica database, and streams new changes from the YugabyteDB database to the source-replica database.

Syntax

Usage: yb-voyager import data to source-replica [ <arguments> ... ]

Arguments

The following table lists the valid CLI flags and parameters for import data to source-replica command.

When run at the same time, flags take precedence over configuration flag settings.

CLI flag
 Config file parameter Description
--batch-size 
import-data-to-source-replica:
  batch-size:
Size of batches in the number of rows generated for ingestion when you import data to source-replica database.
Default: 10000000 (Oracle) or 100000 (PostgreSQL)
--disable-pb 
import-data-to-source-replica:
  disable-pb:
Use this argument to disable progress bar or statistics during data import.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
--parallel-jobs 
import-data-to-source-replica:
  parallel-jobs:
The number of parallel batches issued to the source-replica database.
Default: 16 (Oracle); or, if yb-voyager can determine the total number of cores N, then N/2, otherwise 8 (PostgreSQL)
--truncate-tables 
import-data-to-source-replica:
  truncate-tables:
Truncate tables on target YugabyteDB database before importing data. This option is only valid if --start-clean is set to true.
Default: false
-e, --export-dir 
export-dir:
Path to the export directory. This directory is a workspace used to store exported schema DDL files, export data files, migration state, and a log file.
--send-diagnostics 
send-diagnostics:
Enable or disable sending diagnostics information to Yugabyte.
Default: true
Accepted parameters: true, false, yes, no, 0, 1
--source-replica-db-host 
source-replica:
  db-host:
Domain name or IP address of the machine on which source-replica database server is running.
Default: 127.0.0.1
--source-replica-db-name 
source-replica:
  db-name:
Name of the database in the source-replica database on which import needs to be done.
--source-replica-db-password 
source-replica:
  db-password:
Password to connect to the source-replica database. Alternatively, you can also specify the password by setting the environment variable SOURCE_REPLICA_DB_PASSWORD. If you don't provide a password via the CLI or environment variable during any migration phase, yb-voyager will prompt you at runtime for a password. If the password contains special characters that are interpreted by the shell (for example, # and $), enclose the password in single quotes.
--source-replica-db-port 
source-replica:
  db-port:
Port number of the source-replica database server.
Default: 1521 (Oracle)
--source-replica-db-schema 
source-replica:
  db-schema:
Schema name of the source-replica database.
--source-replica-db-sid 
source-replica:
  db-sid:
Oracle System Identifier (SID) that you wish to use while importing data to Oracle instances. Oracle migrations only.
--source-replica-db-user 
source-replica:
  db-user:
Username to connect to the source-replica database.
--source-replica-ssl-cert 
source-replica:
  ssl-cert:
Source-replica database SSL Certificate path.
--source-replica-ssl-crl 
source-replica:
  ssl-crl:
Source-replica database SSL Root Certificate Revocation List (CRL).
--source-replica-ssl-key 
source-replica:
  ssl-key:
Source-replica database SSL key path.
--source-replica-ssl-mode 
source-replica:
  ssl-mode:
One of disable, allow, prefer(default), require, verify-ca, or verify-full.
--source-replica-ssl-root-cert 
source-replica:
  ssl-root-cert:
Source-replica database SSL Root Certificate path.
--oracle-home 
source-replica:
  oracle-home:
Path to set $ORACLE_HOME environment variable. tnsnames.ora is found in $ORACLE_HOME/network/admin. Oracle migrations only.
--oracle-tns-alias 
source-replica:
  oracle-tns-alias:
TNS (Transparent Network Substrate) alias configured to establish a secure connection with the server. Oracle migrations only.
--start-clean — Starts a fresh import with data files present in the data directory.
If the target YugabyteDB database has any non-empty tables, you are prompted to continue the import without truncating those tables; if you proceed without truncating, then yb-voyager starts ingesting the data present in the data files in non-upsert mode.
Note that for cases where a table doesn't have a primary key, duplicate data may be inserted. You can avoid duplication by excluding the table using --exclude-table-list, or by truncating those tables manually before using the start-clean flag.
Default: false
Accepted parameters: true, false, yes, no, 0, 1
-h, --help — Command line help for import data to source-replica.
-y, --yes — Answer yes to all prompts during the migration.
Default: false

Example

Configuration file:

yb-voyager import data to source-replica --config-file <path-to-config-file>

CLI:

yb-voyager import data to source-replica --export-dir /dir/export-dir \
        --source-replica-db-host 127.0.0.1 \
        --source-replica-db-user ybvoyager \
        --source-replica-db-password 'password' \
        --source-replica-db-name source_replica_db \
        --source-replica-db-schema source_replica_schema \
        --parallel-jobs 12