Release Notes 1.0
FastTransfer 1.0 is the first major release. It brings a new data path, ADBC with the Arpe.io drivers bundled in the binary, which moves data as Arrow columns from end to end. This release also adds Db2 as a source and a target, and makes the parallel methods simpler to choose.
Highlights
- ADBC connections: six new connection types for SQL Server, PostgreSQL, Oracle, Db2, Teradata and Netezza, as source and target.
- Up to 2x faster than the native connection types on cross-database transfers, and much more on Oracle targets.
- Db2 for LUW is now supported, as a source and as a target.
Physical, one parallel method for all physical splits: FastTransfer picks Ctid, Rowid, Physloc or data slices for you.- Faster SQL Server physical split: up to 3.7x faster on clustered tables.
ADBC Connections
The adbc_* connection types use the Arpe.io ADBC drivers, shipped inside FastTransfer: there is nothing else to install or configure.
| Connection Type | Database |
|---|---|
adbc_mssql | SQL Server |
adbc_pgsql | PostgreSQL |
adbc_oracle | Oracle |
adbc_db2 | Db2 for LUW (new) |
adbc_teradata | Teradata |
adbc_netezza | Netezza |
The rows are read and written as Arrow columnar data, with no conversion between the two drivers. The command line does not change: the same server, database, user, password and trusted authentication parameters, and all the parallel methods.
- Windows
- Linux
.\FastTransfer.exe `
--sourceconnectiontype "adbc_oracle" `
--sourceserver "oraserver:1521/ORCLPDB1" `
--sourceuser "TPCH" `
--sourcepassword "mypassword" `
--sourceschema "TPCH" `
--sourcetable "ORDERS" `
--targetconnectiontype "adbc_mssql" `
--targetserver "localhost" `
--targettrusted `
--targetdatabase "tpch" `
--targetschema "dbo" `
--targettable "orders" `
--parallelmethod "Physical" `
--degree 8 `
--loadmode "Truncate"
./FastTransfer \
--sourceconnectiontype "adbc_oracle" \
--sourceserver "oraserver:1521/ORCLPDB1" \
--sourceuser "TPCH" \
--sourcepassword "mypassword" \
--sourceschema "TPCH" \
--sourcetable "ORDERS" \
--targetconnectiontype "adbc_mssql" \
--targetserver "localhost" \
--targettrusted \
--targetdatabase "tpch" \
--targetschema "dbo" \
--targettable "orders" \
--parallelmethod "Physical" \
--degree 8 \
--loadmode "Truncate"
Performance
LINEITEM table (TPC-H SF1, 6 million rows), RangeId method, degree 4, median time in seconds:
| Source → Target | DB Editor Driver | Arpe.io ADBC Drivers | Gain |
|---|---|---|---|
| SQL Server → PostgreSQL | 6.7 | 3.7 | 1.8x |
| PostgreSQL → SQL Server | 5.1 | 3.3 | 1.5x |
| Oracle → SQL Server | 6.8 | 3.6 | 1.9x |
| Oracle → PostgreSQL | 6.8 | 3.2 | 2.1x |
| SQL Server → SQL Server | 5.9 | 4.4 | 1.3x |
| Oracle → Oracle (degree 1) | 47.7 | 3.4 | 14x |
ADBC needs both ends on ADBC, and runs on Windows x64 and Linux x64. The drivers need an ADBC driver block in your license file: contact Arpe.io to add it.
See ADBC Connections and the ADBC examples.
Simpler Parallel Methods
Physical
--parallelmethod Physical replaces the four engine-specific physical methods. FastTransfer chooses the technique from the source type:
| Source | Technique |
|---|---|
| PostgreSQL | Ctid |
| Oracle | Rowid |
| SQL Server | Physloc |
| Netezza | Data slices |
No key column to choose: give the source schema and table, and a degree. See Parallel Parameters - Physical.
Faster Physloc on SQL Server
The physical split on SQL Server now spreads the rows evenly over all the streams, on rowstore and on clustered columnstore tables. On a clustered rowstore table with 8 streams, the transfer went from about 17.7 s to 4.7 s.
More sources for DBPartition and Random
DBPartitionreads the data partitions of a Db2 table (adbc_db2), and works with the ADBC types of SQL Server, PostgreSQL, Oracle and Teradata.Randomuses the right modulo syntax on Db2 and Teradata.
Reliability and Security
--useworktables: the work tables are merged into the target again, and a failed merge now fails the run.NtileandDataDrivenon date keys: no rows lost between streams, and rows with a NULL key are transferred.Ntileworks on Oracle sources.- Passwords in
uri=connection strings are hidden in all the logs. - Signed build provenance (SLSA) comes with each release download, so you can check that a binary was built from the FastTransfer sources.
Upgrading from 0.18
Ctid, Rowid, Physloc and NZDataSlice still work in 1.0, with a warning. They will be removed in 2.0: use Physical.
Physloc(andPhysicalon SQL Server) needs--sourceschemaand--sourcetable. A--querycan still be added to filter the table.NZDataSlice(andPhysicalon Netezza) no longer needs--sourceschemaand--sourcetable.- Oracle sources: a
PARTITION (...)clause in theFROMof--queryis now refused. Use theDBPartitionmethod instead.
Known Limitations
- ADBC is not available on Linux ARM64.
- With ADBC, time zone offsets from SQL Server and Oracle arrive as the same instant in UTC.
- Db2 and Teradata as ADBC targets: a few data types are not supported yet (Db2
DECFLOAT, TeradataPERIOD).
Wizard
The Command Wizard offers the ADBC connection types and the Physical method.