Create mirror - PeerDB Docs: Setup your ETL in minutes with SQL.

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "name": "id",
          "ordering": 1
        },
        {
          "name": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "name": "id",
          "ordering": 1
        },
        {
          "name": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

POST api/v1/flows/cdc/create

This endpoint is used to create a CDC mirror.

Request Fields

connection_configs

object

required

Connection configuration

Show connection_configs

flow_job_name

string

required

The name of the mirror to be created.

source_name

string

required

The name of the source peer (data store).

destination_name

string

required

The name of the destination peer (data store).

table_mappings

array

required

Table mappings

Show table_mappings

source_table_identifier

string

required

The source table identifier.

destination_table_identifier

string

required

The destination table identifier.

exclude

string

Columns from the source table to exclude, separated by commas.

columns

object

For Clickhouse Only And Optional. Ordering key setting for Clickhouse target tables created by PeerDB.

Show columns

name

string

required

Name of the column

ordering

number

A number indicating the rank of this column in the ordering key of ORDER BY in ReplacingMergeTree.

do_initial_snapshot

boolean

required

Whether to do an initial snapshot.

max_batch_size

number

default:"1000000"

The maximum number of rows to sync in a batch.

idle_timeout_seconds

number

default:"60"

How often the mirror syncs, in seconds.

publication_name

string

The name of the publication to use. PeerDB will try to create a publication if not provided here.

snapshot_num_rows_per_partition

number

default:"1000000"

The number of rows per partition to sync during the initial snapshot. Default is 1 million rows.

snapshot_max_parallel_workers

number

default:"4"

The maximum number of parallel workers to use during the initial snapshot.

snapshot_num_tables_in_parallel

number

default:"1"

The number of tables to sync in parallel during the initial snapshot.

resync

boolean

default:"false"

Whether to resync the mirror. The mirror must be dropped before resyncing.

initial_snapshot_only

boolean

default:"false"

Whether to do only the initial snapshot and not perform CDC.

soft_delete_col_name

string

The name of the column that indicates a soft delete.

synced_at_col_name

string

The name of the column that indicates when the row was last synced.

Response Fields

workflowId

string

The ID of the parent workflow created for this mirror.

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "name": "id",
          "ordering": 1
        },
        {
          "name": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

Create a Snowflake peer Peer info

API Endpoints

​Request Fields

​Response Fields

Request Fields

Response Fields