Skip to main content
Version: 0.15 (Latest)

Kestra

Kestra is a powerful open-source orchestration platform that can be used to automate and schedule FastTransfer data migration workflows. FastTransfer can be seamlessly integrated into Kestra flows using its Docker image.

Docker Image​

FastTransfer provides an official Docker image available on Docker Hub:

Image: arpeio/fasttransfer:latest
Repository: https://hub.docker.com/repository/docker/arpeio/fasttransfer/general

Integration Methods​

Kestra offers two primary methods to orchestrate FastTransfer execution:

The io.kestra.plugin.docker.Run plugin provides a direct way to execute FastTransfer commands in a Docker container.

id: fasttransfer_docker_kestra_v2
namespace: arpe.io

tasks:
  - id: run
    type: io.kestra.plugin.docker.Run    
    containerImage: arpeio/fasttransfer:latest
    host: "tcp://host.docker.internal:2375"
    commands:
      - --sourceconnectiontype
      - mssql
      - --sourceserver
      - 192.168.65.254,11433
      - --sourcedatabase
      - tpch10
      - --sourceuser
      - FastLogin
      - --sourcepassword
      - FastPassword
      - --sourceschema
      - dbo
      - --sourcetable
      - orders
      - --targetconnectiontype
      - pgcopy
      - --targetserver
      - 192.168.65.254:25433
      - --targetdatabase
      - postgres
      - --targetuser
      - postgres
      - --targetpassword
      - postgres
      - --targetschema
      - public
      - --targettable
      - orders
      - --loadmode
      - Truncate
      - --mapmethod
      - Name
      - --method
      - Ntile
      - --degree
      - "12"
      - --distributekeycolumn
      - o_orderkey
      - --nobanner

Key Parameters:

  • containerImage: Specifies the FastTransfer Docker image
  • commands: Array of FastTransfer CLI arguments

Method 2: Using Shell Commands with Docker Task Runner​

Alternatively, you can use the io.kestra.plugin.scripts.shell.Commands plugin with a Docker task runner:

id: fasttransfer_docker_kestra
namespace: arpe.io

tasks:
  - id: shell
    type: io.kestra.plugin.scripts.shell.Commands
    containerImage: arpeio/fasttransfer:latest
    taskRunner:
      type: io.kestra.plugin.scripts.runner.docker.Docker
      host: "tcp://host.docker.internal:2375"
    commands:
      - >-
        /usr/local/bin/FastTransfer
        --sourceconnectiontype mssql
        --sourceserver "192.168.65.254,11433"
        --sourcedatabase "tpch10"
        --sourceuser "FastLogin"
        --sourcepassword "FastPassword"
        --sourceschema "dbo"
        --sourcetable "orders"
        --targetconnectiontype pgcopy
        --targetserver "192.168.65.254:25433"
        --targetdatabase "postgres"
        --targetuser "postgres"
        --targetpassword "postgres"
        --targetschema "public"
        --targettable "orders"
        --loadmode Truncate
        --mapmethod Name
        --method Ntile
        --degree 12
        --distributekeycolumn "o_orderkey"
        --nobanner

This method allows you to execute shell commands within the FastTransfer Docker container, providing more flexibility for complex workflows.

Securing Sensitive Data with Kestra Secrets​

To protect sensitive information like passwords and connection strings, Kestra provides a built-in secrets management feature. You can store credentials securely and reference them in your flows using the {{ secret('SECRET_NAME') }} syntax.

Creating Secrets in Kestra​

To create secrets in Kestra:

  1. Navigate to the Kestra UI
  2. Go to Tenant → Secrets
  3. Add your secrets with appropriate keys:
    • FASTTRANSFER_MSSQL_USER
    • FASTTRANSFER_MSSQL_PASSWORD
    • FASTTRANSFER_PG_USER
    • FASTTRANSFER_PG_PASSWORD

Example with Secrets​

id: fasttransfer_docker_kestra_v3
namespace: arpe.io

tasks:
  - id: run
    type: io.kestra.plugin.docker.Run
    containerImage: arpeio/fasttransfer:latest
    host: "tcp://host.docker.internal:2375"
    commands:
      - --sourceconnectiontype
      - mssql
      - --sourceserver
      - 192.168.65.254,11433
      - --sourcedatabase
      - tpch10
      - --sourceuser
      - "{{ secret('FASTTRANSFER_MSSQL_USER') }}"
      - --sourcepassword
      - "{{ secret('FASTTRANSFER_MSSQL_PASSWORD') }}"
      - --sourceschema
      - dbo
      - --sourcetable
      - orders
      - --targetconnectiontype
      - pgcopy
      - --targetserver
      - 192.168.65.254:25433
      - --targetdatabase
      - postgres
      - --targetuser
      - "{{ secret('FASTTRANSFER_PG_USER') }}"
      - --targetpassword
      - "{{ secret('FASTTRANSFER_PG_PASSWORD') }}"
      - --targetschema
      - public
      - --targettable
      - orders
      - --loadmode
      - Truncate
      - --mapmethod
      - Name
      - --method
      - Ntile
      - --degree
      - "12"
      - --distributekeycolumn
      - o_orderkey
      - --nobanner

This approach ensures that sensitive credentials are never exposed in your flow definitions and can be managed centrally.

Logging​

All FastTransfer execution logs are captured by Kestra and can be viewed directly in the Kestra UI. To access the logs:

  1. Navigate to the Kestra UI
  2. Go to Executions
  3. Select your flow execution
  4. Click on the Logs tab

The logs provide detailed information about the transfer process, including configuration parameters, connection details, data transfer progress, and performance metrics.

Example Log Output​

2026-02-20 10:11:36.8232026-02-20T09:11:37.315 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- The "FastTransfer_Settings.json" file does not exist. Using default settings. Console Only with loglevel=Information
2026-02-20 10:11:36.8352026-02-20T09:11:37.318 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- FastTransfer - running in trial mode – trial mode will end on 2026‑02‑27 - normal licensed mode will then start (6 day(s) left).
2026-02-20 10:11:37.0642026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Starting
2026-02-20 10:11:37.3212026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- FastTransfer Version : 0.15.0.0 Architecture : X64 - Framework : .NET 8.0.23
2026-02-20 10:11:37.3212026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- OS : Debian GNU/Linux 13 (trixie)
2026-02-20 10:11:37.3212026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Process ID : 1
2026-02-20 10:11:37.3222026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Generated Run ID : 75c7227b-f62f-44b9-b33d-e980632fce6b
2026-02-20 10:11:37.3222026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Database : tpch10
2026-02-20 10:11:37.3222026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source SqlInstance : host.docker.internal,11433
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Connection Type : mssql
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Trusted Connection : False
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source User : FastLogin
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Schema : dbo
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Type : pgcopy
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Database : postgres
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Server : host.docker.internal:25433
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Table : orders
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Schema : public
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Table : orders
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Trusted Connection : False
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target User : postgres
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Columns Map Method : Name
2026-02-20 10:11:37.3232026-02-20T09:11:37.319 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Degree : 12
2026-02-20 10:11:37.3232026-02-20T09:11:37.320 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Distribute Method : Ntile
2026-02-20 10:11:37.3232026-02-20T09:11:37.320 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Distribute Column : o_orderkey
2026-02-20 10:11:37.3232026-02-20T09:11:37.320 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Bulkcopy Batch Size : 1048576
2026-02-20 10:11:37.3232026-02-20T09:11:37.320 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Load Mode : Truncate
2026-02-20 10:11:37.3232026-02-20T09:11:37.320 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Use Work Tables : False
2026-02-20 10:11:37.3232026-02-20T09:11:37.322 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Encoding used : Unicode (UTF-8) - 65001 - utf-8
2026-02-20 10:11:37.3232026-02-20T09:11:37.542 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Connection String : Data Source=host.docker.internal,11433;Initial Catalog=tpch10;User ID=FastLogin;Password=xxxxx;Connect Timeout=120;Encrypt=True;Trust Server Certificate=True;Application Name=FastTransfer;Application Intent=ReadOnly;Command Timeout=10800
2026-02-20 10:11:37.3232026-02-20T09:11:37.542 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Source Database Version : Microsoft SQL Server 2022 (RTM-GDR) (KB5073031) - 16.0.1165.1 (X64)  	Nov 24 2025 19:08:45  	Copyright (C) 2022 Microsoft Corporation 	Developer Edition (64-bit) on Windows 10 Home 10.0 <X64> (Build 26200: ) (Hypervisor)
2026-02-20 10:11:37.3252026-02-20T09:11:37.542 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Connection String : Host=host.docker.internal;Port=25433;Database=postgres;Trust Server Certificate=True;Application Name=FastTransfer;Timeout=15;Command Timeout=10800;Username=postgres;Password=xxxxx
2026-02-20 10:11:37.5442026-02-20T09:11:37.542 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Target Database Version : PostgreSQL 16.10, compiled by Visual C++ build 1944, 64-bit
2026-02-20 10:11:37.5442026-02-20T09:11:38.577 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Ntile DataSegments Computation Completed in 731 ms
2026-02-20 10:11:37.5442026-02-20T09:11:38.581 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:37.5442026-02-20T09:11:38.632 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.5792026-02-20T09:11:38.640 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.5822026-02-20T09:11:38.647 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6332026-02-20T09:11:38.657 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6422026-02-20T09:11:38.677 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6492026-02-20T09:11:38.687 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6592026-02-20T09:11:38.717 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6852026-02-20T09:11:38.725 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.6902026-02-20T09:11:38.739 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.7222026-02-20T09:11:38.765 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.7282026-02-20T09:11:38.893 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Start Loading Data using distribution method NTile
2026-02-20 10:11:38.7402026-02-20T09:12:12.109 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 2 for o_orderkey between 10000002 and 15000002 : 1250001 rows x 9 columns in 32808ms
2026-02-20 10:12:13.7732026-02-20T09:12:19.879 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 9 for o_orderkey between 45000033 and 50000033 : 1250001 rows x 9 columns in 40578ms
2026-02-20 10:12:16.0162026-02-20T09:12:29.092 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 3 for o_orderkey between 15000003 and 20000003 : 1250001 rows x 9 columns in 49046ms
2026-02-20 10:12:19.3052026-02-20T09:12:30.415 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 1 for o_orderkey between 5000001 and 10000001 : 1250001 rows x 9 columns in 50369ms
2026-02-20 10:12:19.8812026-02-20T09:12:33.966 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 0 for o_orderkey between 1 and 5000000 : 1250000 rows x 9 columns in 53920ms
2026-02-20 10:12:29.0942026-02-20T09:12:34.503 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load Query 8 for o_orderkey between 40000032 and 45000032 : 1250001 rows x 9 columns in 54457ms
2026-02-20 10:12:30.4172026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Total cells : 135000000
2026-02-20 10:12:33.9672026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Total rows : 15000000
2026-02-20 10:12:34.5102026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Cells Throughput : 2445813 cells/s
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Total columns : 9
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Rows Throughput : 271756 rows/s
2026-02-20 10:12:34.5112026-02-20T09:12:34.508 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Total time : Elapsed=55795 ms
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Transfert time : Elapsed=55196 ms
2026-02-20 10:12:34.5112026-02-20T09:12:34.508 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Completed Load
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Total time : Elapsed=55795 ms
2026-02-20 10:12:34.5112026-02-20T09:12:34.507 +00:00 -|-  -|- 75c7227b-f62f-44b9-b33d-e980632fce6b -|- INFORMATION -|-  -|- Transfert time : Elapsed=55196 ms
2026-02-20 10:12:34.744Command succeed with exit code 0
2026-02-20 10:12:34.824Container deleted: ca0de79e3eb4ede961668cb5080e4c20f858960c118d10731b8e0ade3ad87685
2026-02-20 10:12:34.832Volume deleted: a6b822d7677064a91d2dd555c360dc61ec49e3954149eacd5c22549c2a300850
Copyright © 2026 Architecture & Performance.