Skip to main content
Version: 1.0 (Latest)

Kestra

Kestra is a powerful open-source orchestration platform that can be used to automate and schedule FastTransfer data migration workflows. FastTransfer can be seamlessly integrated into Kestra flows using its Docker image.

Docker Image​

FastTransfer provides an official Docker image available on Docker Hub: arpeio/fasttransfer:latest.

Integration Methods​

Kestra offers two primary methods to orchestrate FastTransfer execution:

Docker Run pluginRecommended

  • Task typeio.kestra.plugin.docker.Run
  • Argumentscommands, one FastTransfer CLI argument per item
  • Best forA direct way to execute FastTransfer commands in a Docker container
Shell commands with Docker task runner

  • Task typeio.kestra.plugin.scripts.shell.Commands
  • ArgumentsA shell command line calling /usr/local/bin/FastTransfer
  • Best forMore flexibility for complex workflows

The io.kestra.plugin.docker.Run plugin provides a direct way to execute FastTransfer commands in a Docker container.

id: fasttransfer_docker_kestra_v2
namespace: arpe.io

tasks:
- id: run
type: io.kestra.plugin.docker.Run
containerImage: arpeio/fasttransfer:latest
host: "tcp://host.docker.internal:2375"
commands:
- --sourceconnectiontype
- mssql
- --sourceserver
- 192.168.65.254,11433
- --sourcedatabase
- tpch10
- --sourceuser
- FastLogin
- --sourcepassword
- FastPassword
- --sourceschema
- dbo
- --sourcetable
- orders
- --targetconnectiontype
- pgcopy
- --targetserver
- 192.168.65.254:25433
- --targetdatabase
- postgres
- --targetuser
- postgres
- --targetpassword
- postgres
- --targetschema
- public
- --targettable
- orders
- --loadmode
- Truncate
- --mapmethod
- Name
- --parallelmethod
- Ntile
- --degree
- "12"
- --distributekeycolumn
- o_orderkey
- --nobanner

Key Parameters​

containerImage
Specifies the FastTransfer Docker image
commands
Array of FastTransfer CLI arguments

Securing Sensitive Data with Kestra Secrets​

To protect sensitive information like passwords and connection strings, Kestra provides a built-in secrets management feature. You can store credentials securely and reference them in your flows using the {{ secret('SECRET_NAME') }} syntax.

Creating Secrets in Kestra​

  1. Add the FastTransfer secrets
    Kestra UI›Tenant›Secrets
    1. Navigate to the Kestra UI
    2. Go to Tenant → Secrets
    3. Add your secrets with appropriate keys:
      • FASTTRANSFER_MSSQL_USER
      • FASTTRANSFER_MSSQL_PASSWORD
      • FASTTRANSFER_PG_USER
      • FASTTRANSFER_PG_PASSWORD

Example with Secrets​

id: fasttransfer_docker_kestra_v3
namespace: arpe.io

tasks:
- id: run
type: io.kestra.plugin.docker.Run
containerImage: arpeio/fasttransfer:latest
host: "tcp://host.docker.internal:2375"
commands:
- --sourceconnectiontype
- mssql
- --sourceserver
- 192.168.65.254,11433
- --sourcedatabase
- tpch10
- --sourceuser
- "{{ secret('FASTTRANSFER_MSSQL_USER') }}"
- --sourcepassword
- "{{ secret('FASTTRANSFER_MSSQL_PASSWORD') }}"
- --sourceschema
- dbo
- --sourcetable
- orders
- --targetconnectiontype
- pgcopy
- --targetserver
- 192.168.65.254:25433
- --targetdatabase
- postgres
- --targetuser
- "{{ secret('FASTTRANSFER_PG_USER') }}"
- --targetpassword
- "{{ secret('FASTTRANSFER_PG_PASSWORD') }}"
- --targetschema
- public
- --targettable
- orders
- --loadmode
- Truncate
- --mapmethod
- Name
- --parallelmethod
- Ntile
- --degree
- "12"
- --distributekeycolumn
- o_orderkey
- --nobanner

This approach ensures that sensitive credentials are never exposed in your flow definitions and can be managed centrally.

Logging​

All FastTransfer execution logs are captured by Kestra and can be viewed directly in the Kestra UI:

  1. Open the execution logs
    Kestra UI›Executions
    1. Navigate to the Kestra UI
    2. Go to Executions
    3. Select your flow execution
    4. Click on the Logs tab

Example Log Output​

The logs provide detailed information about the transfer process:

  • Configuration parameters
  • Connection details
  • Data transfer progress
  • Performance metrics
Full log of the example flow execution
2026-03-02 15:51:45.3752026-03-02T15:51:45.375 +00:00 -|-  -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- The "FastTransfer_Settings.json" file does not exist. Using default settings. Console Only with loglevel=Information
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- FastTransfer - running in trial mode – trial mode will end on 2026‑03‑27 - normal licensed mode will then start (24 day(s) left).
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Starting
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- FastTransfer Version : 0.16.0.0 Architecture : X64 - Framework : .NET 8.0.24
2026-03-02 15:51:45.3772026-03-02T15:51:45.377 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- OS : Debian GNU/Linux 13 (trixie)
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Process ID : 1
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Generated Run ID : ca195802-59f7-4b13-85e8-0bb73e8f7b32
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Connection Type : mssql
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source SqlInstance : sql22,1433
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Database : tpch
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Trusted Connection : False
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source User : migadmin
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Schema : tpch10
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Table : orders
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Type : pgcopy
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Server : host.docker.internal:5432
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Database : tpch
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Schema : tpch_1_copy
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Table : orders
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Trusted Connection : False
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target User : pytabextract_pguser
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Columns Map Method : Name
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Degree : -4
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Distribute Method : Ntile
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Distribute Column : o_orderkey
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Bulkcopy Batch Size : 1048576
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Load Mode : Truncate
2026-03-02 15:51:45.3782026-03-02T15:51:45.378 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Use Work Tables : False
2026-03-02 15:51:45.3792026-03-02T15:51:45.379 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Encoding used : Unicode (UTF-8) - 65001 - utf-8
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Connection String : Data Source=sql22,1433;Initial Catalog=tpch;User ID=migadmin;Password=xxxxx;Connect Timeout=120;Encrypt=True;Trust Server Certificate=True;Application Name=FastTransfer;Application Intent=ReadOnly;Command Timeout=10800
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Connection String : Host=host.docker.internal;Port=5432;Database=tpch;Trust Server Certificate=True;Application Name=FastTransfer;Timeout=15;Command Timeout=10800;Username=pytabextract_pguser;Password=xxxxx
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Source Database Version : Microsoft SQL Server 2022 (RTM-CU2) (KB5023127) - 16.0.4015.1 (X64) Feb 27 2023 15:40:01 Copyright (C) 2022 Microsoft Corporation Developer Edition (64-bit) on Linux (Ubuntu 20.04.5 LTS) <X64>
2026-03-02 15:51:45.5172026-03-02T15:51:45.517 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Target Database Version : PostgreSQL 15.15 (Ubuntu 15.15-1.pgdg22.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04.2) 11.4.0, 64-bit
2026-03-02 15:51:45.6342026-03-02T15:51:45.634 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Degree of parallelism was computed to 5 (=> 20\4)
2026-03-02 15:51:45.8702026-03-02T15:51:45.870 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Ntile DataSegments Computation Completed in 235 ms
2026-03-02 15:51:45.8722026-03-02T15:51:45.872 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.8842026-03-02T15:51:45.884 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.8912026-03-02T15:51:45.891 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.9002026-03-02T15:51:45.900 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:45.9112026-03-02T15:51:45.911 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Start Loading Data using distribution method NTile
2026-03-02 15:51:54.2332026-03-02T15:51:54.233 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 3 for o_orderkey between 36000003 and 48000003 : 3000001 rows x 9 columns in 8360ms
2026-03-02 15:51:54.2382026-03-02T15:51:54.238 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 1 for o_orderkey between 12000001 and 24000001 : 3000001 rows x 9 columns in 8365ms
2026-03-02 15:51:54.2412026-03-02T15:51:54.241 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 2 for o_orderkey between 24000002 and 36000002 : 3000001 rows x 9 columns in 8368ms
2026-03-02 15:51:54.4432026-03-02T15:51:54.443 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 0 for o_orderkey between 1 and 12000000 : 3000000 rows x 9 columns in 8570ms
2026-03-02 15:51:54.4432026-03-02T15:51:54.443 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load Query 4 for o_orderkey between 48000004 and 60000004 : 3000001 rows x 9 columns in 8570ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total rows : 15000000
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total columns : 9
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total cells : 135000000
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Rows Throughput : 1766084 rows/s
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Cells Throughput : 15894756 cells/s
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Transfert time : Elapsed=8494 ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Total time : Elapsed=9142 ms
2026-03-02 15:51:54.4462026-03-02T15:51:54.446 +00:00 -|- -|- ca195802-59f7-4b13-85e8-0bb73e8f7b32 -|- INFORMATION -|- tpch.tpch_1_copy.orders -|- Completed Load
2026-03-02 15:51:54.448Command succeed with exit code 0
2026-03-02 15:51:54.450Container deleted: ca0de79e3eb4ede961668cb5080e4c20f858960c118d10731b8e0ade3ad87685
2026-03-02 15:51:54.455Volume deleted: a6b822d7677064a91d2dd555c360dc61ec49e3954149eacd5c22549c2a300850
Related pages
Copyright © 2026 Architecture & Performance.