Version: 0.16 (Latest)

Kestra

Kestra is a powerful open-source orchestration platform that can be used to automate and schedule FastTransfer data migration workflows. FastTransfer can be seamlessly integrated into Kestra flows using its Docker image.

Docker Image

FastTransfer provides an official Docker image available on Docker Hub:

Image: arpeio/fasttransfer:latest
Repository: https://hub.docker.com/repository/docker/arpeio/fasttransfer/general

Integration Methods

Kestra offers two primary methods to orchestrate FastTransfer execution:

Method 1: Using Docker Run Plugin (Recommended)

The io.kestra.plugin.docker.Run plugin provides a direct way to execute FastTransfer commands in a Docker container.

id: fasttransfer_docker_kestra_v2
namespace: arpe.io

tasks:
  - id: run
    type: io.kestra.plugin.docker.Run    
    containerImage: arpeio/fasttransfer:latest
    host: "tcp://host.docker.internal:2375"
    commands:
      - --sourceconnectiontype
      - mssql
      - --sourceserver
      - 192.168.65.254,11433
      - --sourcedatabase
      - tpch10
      - --sourceuser
      - FastLogin
      - --sourcepassword
      - FastPassword
      - --sourceschema
      - dbo
      - --sourcetable
      - orders
      - --targetconnectiontype
      - pgcopy
      - --targetserver
      - 192.168.65.254:25433
      - --targetdatabase
      - postgres
      - --targetuser
      - postgres
      - --targetpassword
      - postgres
      - --targetschema
      - public
      - --targettable
      - orders
      - --loadmode
      - Truncate
      - --mapmethod
      - Name
      - --method
      - Ntile
      - --degree
      - "12"
      - --distributekeycolumn
      - o_orderkey
      - --nobanner

Key Parameters:

containerImage: Specifies the FastTransfer Docker image
commands: Array of FastTransfer CLI arguments

Method 2: Using Shell Commands with Docker Task Runner

Alternatively, you can use the io.kestra.plugin.scripts.shell.Commands plugin with a Docker task runner:

id: fasttransfer_docker_kestra
namespace: arpe.io

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: arpeio/fasttransfer:latest
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      host: "tcp://host.docker.internal:2375"
    commands:
      - >-
        /usr/local/bin/FastTransfer
        --sourceconnectiontype mssql
        --sourceserver "192.168.65.254,11433"
        --sourcedatabase "tpch10"
        --sourceuser "FastLogin"
        --sourcepassword "FastPassword"
        --sourceschema "dbo"
        --sourcetable "orders"
        --targetconnectiontype pgcopy
        --targetserver "192.168.65.254:25433"
        --targetdatabase "postgres"
        --targetuser "postgres"
        --targetpassword "postgres"
        --targetschema "public"
        --targettable "orders"
        --loadmode Truncate
        --mapmethod Name
        --method Ntile
        --degree 12
        --distributekeycolumn "o_orderkey"
        --nobanner

This method allows you to execute shell commands within the FastTransfer Docker container, providing more flexibility for complex workflows.

Securing Sensitive Data with Kestra Secrets

To protect sensitive information like passwords and connection strings, Kestra provides a built-in secrets management feature. You can store credentials securely and reference them in your flows using the {{ secret('SECRET_NAME') }} syntax.

Creating Secrets in Kestra

To create secrets in Kestra:

Navigate to the Kestra UI
Go to Tenant → Secrets
Add your secrets with appropriate keys:
- FASTTRANSFER_MSSQL_USER
- FASTTRANSFER_MSSQL_PASSWORD
- FASTTRANSFER_PG_USER
- FASTTRANSFER_PG_PASSWORD

Example with Secrets

id: fasttransfer_docker_kestra_v3
namespace: arpe.io

tasks:
  - id: run
    type: io.kestra.plugin.docker.Run
    containerImage: arpeio/fasttransfer:latest
    host: "tcp://host.docker.internal:2375"
    commands:
      - --sourceconnectiontype
      - mssql
      - --sourceserver
      - 192.168.65.254,11433
      - --sourcedatabase
      - tpch10
      - --sourceuser
      - "{{ secret('FASTTRANSFER_MSSQL_USER') }}"
      - --sourcepassword
      - "{{ secret('FASTTRANSFER_MSSQL_PASSWORD') }}"
      - --sourceschema
      - dbo
      - --sourcetable
      - orders
      - --targetconnectiontype
      - pgcopy
      - --targetserver
      - 192.168.65.254:25433
      - --targetdatabase
      - postgres
      - --targetuser
      - "{{ secret('FASTTRANSFER_PG_USER') }}"
      - --targetpassword
      - "{{ secret('FASTTRANSFER_PG_PASSWORD') }}"
      - --targetschema
      - public
      - --targettable
      - orders
      - --loadmode
      - Truncate
      - --mapmethod
      - Name
      - --method
      - Ntile
      - --degree
      - "12"
      - --distributekeycolumn
      - o_orderkey
      - --nobanner

This approach ensures that sensitive credentials are never exposed in your flow definitions and can be managed centrally.

Logging

All FastTransfer execution logs are captured by Kestra and can be viewed directly in the Kestra UI. To access the logs:

Navigate to the Kestra UI
Go to Executions
Select your flow execution
Click on the Logs tab

The logs provide detailed information about the transfer process, including configuration parameters, connection details, data transfer progress, and performance metrics.

Example Log Output

2026-03-02 15:51:45.3752026-03-02T15:51:45.375 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- The "FastTransfer_Settings.json" file does not exist. Using default settings. Console Only with loglevel=Information
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- FastTransfer - running in trial mode – trial mode will end on 2026‑03‑27 - normal licensed mode will then start (24 day(s) left).
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Starting
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- FastTransfer Version : 0.16.0.0 Architecture : X64 - Framework : .NET 8.0.24
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- OS : Debian GNU/Linux 13 (trixie)
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Process ID : 1
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Generated Run ID : ca195802-59f7-4b13-85e8-0bb73e8f7b32
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Connection Type : mssql
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source SqlInstance : sql22,1433
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Database : tpch
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Trusted Connection : False
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source User : migadmin
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Schema : tpch10
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Table : orders
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Type : pgcopy
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Server : host.docker.internal:5432
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Database : tpch
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Schema : tpch_1_copy
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Table : orders
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Trusted Connection : False
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target User : pytabextract_pguser
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Columns Map Method : Name
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Degree : -4
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Distribute Method : Ntile
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Distribute Column : o_orderkey
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Bulkcopy Batch Size : 1048576
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Load Mode : Truncate
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Use Work Tables : False
2026-03-02 15:51:45.3792026-03-02T15:51:45.379 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Encoding used : Unicode (UTF-8) - 65001 - utf-8
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Connection String : Data Source=sql22,1433;Initial Catalog=tpch;User ID=migadmin;Password=xxxxx;Connect Timeout=120;Encrypt=True;Trust Server Certificate=True;Application Name=FastTransfer;Application Intent=ReadOnly;Command Timeout=10800
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Connection String : Host=host.docker.internal;Port=5432;Database=tpch;Trust Server Certificate=True;Application Name=FastTransfer;Timeout=15;Command Timeout=10800;Username=pytabextract_pguser;Password=xxxxx
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Database Version : Microsoft SQL Server 2022 (RTM-CU2) (KB5023127) - 16.0.4015.1 (X64)  	Feb 27 2023 15:40:01  	Copyright (C) 2022 Microsoft Corporation 	Developer Edition (64-bit) on Linux (Ubuntu 20.04.5 LTS) <X64>
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Database Version : PostgreSQL 15.15 (Ubuntu 15.15-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0, 64-bit
2026-03-02 15:51:45.6342026-03-02T15:51:45.634 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Degree of parallelism was computed to 5 (=> 20\4)
2026-03-02 15:51:45.8702026-03-02T15:51:45.870 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Ntile DataSegments Computation Completed in 235 ms
2026-03-02 15:51:45.8722026-03-02T15:51:45.872 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.8842026-03-02T15:51:45.884 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.8912026-03-02T15:51:45.891 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.9002026-03-02T15:51:45.900 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.9112026-03-02T15:51:45.911 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:54.2332026-03-02T15:51:54.233 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 3 for o_orderkey between 36000003 and 48000003 : 3000001 rows x 9 columns in 8360ms
2026-03-02 15:51:54.2382026-03-02T15:51:54.238 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 1 for o_orderkey between 12000001 and 24000001 : 3000001 rows x 9 columns in 8365ms
2026-03-02 15:51:54.2412026-03-02T15:51:54.241 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 2 for o_orderkey between 24000002 and 36000002 : 3000001 rows x 9 columns in 8368ms
2026-03-02 15:51:54.4432026-03-02T15:51:54.443 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 0 for o_orderkey between 1 and 12000000 : 3000000 rows x 9 columns in 8570ms
2026-03-02 15:51:54.4432026-03-02T15:51:54.443 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 4 for o_orderkey between 48000004 and 60000004 : 3000001 rows x 9 columns in 8570ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total rows : 15000000
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total columns : 9
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total cells : 135000000
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Rows Throughput : 1766084 rows/s
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Cells Throughput : 15894756 cells/s
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Transfert time : Elapsed=8494 ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total time : Elapsed=9142 ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load
2026-03-02 15:51:54.448Command succeed with exit code 0
2026-03-02 15:51:54.450Container deleted: ca0de79e3eb4ede961668cb5080e4c20f858960c118d10731b8e0ade3ad87685
2026-03-02 15:51:54.455Volume deleted: a6b822d7677064a91d2dd555c360dc61ec49e3954149eacd5c22549c2a300850

Docker Image​

Integration Methods​

Method 1: Using Docker Run Plugin (Recommended)​

Method 2: Using Shell Commands with Docker Task Runner​

Securing Sensitive Data with Kestra Secrets​

Creating Secrets in Kestra​

Example with Secrets​

Logging​

Example Log Output​