Create mirror - PeerDB Docs: Setup your ETL in minutes with SQL.

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "sourceName": "id",
          "ordering": 1
        },
        {
          "sourceName": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

{
	"code": 2,
	"message": "invalid mirror: mirror with name mirror_api_kick_off already exists",
	"details": []
}

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "sourceName": "id",
          "ordering": 1
        },
        {
          "sourceName": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

{
	"code": 2,
	"message": "invalid mirror: mirror with name mirror_api_kick_off already exists",
	"details": []
}

POST api/v1/flows/cdc/create

This endpoint is used to create a CDC mirror.

Request Fields

object

required

Connection configuration

Show connection_configs

string

required

The name of the mirror to be created.

string

required

The name of the source peer (data store).

string

required

The name of the destination peer (data store).

array

required

Table mappings

Show table_mappings

string

required

The source table identifier.

string

required

The destination table identifier.

string

Columns from the source table to exclude, separated by commas.

object

For Clickhouse Only And Optional. Ordering key setting for Clickhouse target tables created by PeerDB.

Show columns

string

required

Name of the column

number

A number indicating the rank of this column in the ordering key of ORDER BY in ReplacingMergeTree.

boolean

required

Whether to do an initial snapshot.

number

default:"1000000"

The maximum number of rows to sync in a batch.

number

default:"60"

How often the mirror syncs, in seconds.

string

The name of the publication to use. PeerDB will try to create a publication if not provided here.

number

default:"1000000"

The number of rows per partition to sync during the initial snapshot. Default is 1 million rows.

number

default:"4"

The maximum number of parallel workers to use during the initial snapshot.

number

default:"1"

The number of tables to sync in parallel during the initial snapshot.

boolean

default:"false"

Whether to resync the mirror. The mirror must be dropped before resyncing.

boolean

default:"false"

Whether to do only the initial snapshot and not perform CDC.

string

The name of the column that indicates a soft delete.

string

The name of the column that indicates when the row was last synced.

Response Fields

string

The ID of the parent workflow created for this mirror.

curl --request POST \
  --url localhost:3000/api/v1/flows/cdc/create \
  --header 'Authorization: Basic OnJsYWNrd3dhbmEyMw==' \
  --header 'Content-Type: application/json' \
  --data '
{
"connection_configs": {
  "flow_job_name": "mirror_api_kick_off",
	 "source_name": "rds_peer",
  "destination_name": "ch_peer",
  "table_mappings": [
    {
      "source_table_identifier": "public.users",
      "destination_table_identifier": "users_api"
    },
    {
      "source_table_identifier": "public.payments",
      "destination_table_identifier": "payments_api"
    },
    {
      "source_table_identifier": "public.optional_ordering_key",
      "destination_table_identifier": "optional_ordering_key",
      "columns": [
        {
          "sourceName": "id",
          "ordering": 1
        },
        {
          "sourceName": "created_at",
          "ordering": 2
        }
      ]
    },
  ],
  "max_batch_size": 1000,
  "idle_timeout_seconds": 300,
  "publication_name": "",
  "do_initial_snapshot": true,
  "snapshot_num_rows_per_partition": 5000,
  "snapshot_max_parallel_workers": 4,
  "snapshot_num_tables_in_parallel": 2,
  "resync": false,
  "initial_snapshot_only": false,
  "soft_delete_col_name": "_peerdb_is_deleted",
  "synced_at_col_name": "_peerdb_synced_at"
}
}'

{
	"workflowId": "mirror_kick_off-peerflow-2d269226-9757-4d9f-8cfd-6adb1ca29c0e"
}

{
	"code": 2,
	"message": "invalid mirror: mirror with name mirror_api_kick_off already exists",
	"details": []
}