Mirror status - PeerDB Docs: Setup your ETL in minutes with SQL.

curl --request POST \
  --url http://localhost:3000/api/v1/mirrors/status \
  --header 'Authorization: Basic OnBlZXJkYg==' \
  --header 'Content-Type: application/json' \
  --data '{
	`flowJobName`:`testing_bq_2`,
	`includeFlowInfo`:false
}'

curl --request POST \
  --url http://localhost:3000/api/v1/mirrors/status \
  --header 'Authorization: Basic OnBlZXJkYg==' \
  --header 'Content-Type: application/json' \
  --data '{
	`flowJobName`:`testing_bq_2`,
	`includeFlowInfo`:true
}'

{
	"flowJobName": "testing_bq_2",
	"errorMessage": "",
	"currentFlowState": "STATUS_RUNNING",
	"ok": true
}

{
	"flowJobName": "testing_bq_2",
	"cdcStatus": {
		"config": {
			"flowJobName": "testing_bq_2",
			"tableMappings": [
				{
					"sourceTableIdentifier": "public.sales",
					"destinationTableIdentifier": "public_sales",
					"partitionKey": "",
					"exclude": [] // excluded columns, comma-separated
				},
			],
			"maxBatchSize": 1000000,
			"idleTimeoutSeconds": "60",
			"cdcStagingPath": "",
			"publicationName": "",
			"replicationSlotName": "",
			"doInitialSnapshot": true,
			"snapshotNumRowsPerPartition": 1000000,
			"snapshotStagingPath": "",
			"snapshotMaxParallelWorkers": 4,
			"snapshotNumTablesInParallel": 1,
			"resync": false,
			"initialSnapshotOnly": false,
			"softDelete": true,
			"softDeleteColName": "_PEERDB_IS_DELETED",
			"syncedAtColName": "_PEERDB_SYNCED_AT",
			"script": "",
			"system": "Q",
			"sourceName": "postgres_local",
			"destinationName": "bq_peer"
		},
		"snapshotStatus": {
			"clones": [
				{
					"tableName": "public_sales",
					"startTime": "2024-06-27T14:37:48.701204Z",
					"numPartitionsCompleted": 1,
					"numPartitionsTotal": 1,
					"numRowsSynced": "4",
					"avgTimePerPartitionMs": "11381",
					"flowJobName": "clone_testing_bq_2_public_sales_dc500df7_2606_4c19_96c0_e40e0df4e5ec",
					"sourceTable": "public.sales",
					"fetchCompleted": true,
					"consolidateCompleted": true,
					"mirrorName": "testing_bq_2"
				}
			]
		},
		"cdcSyncs": [], // coming soon
		"sourceType": "POSTGRES",
		"destinationType": "BIGQUERY"
	},
	"errorMessage": "",
	"currentFlowState": "STATUS_RUNNING",
	"ok": true
}

{
	"flowJobName": "fake_mirror",
	"errorMessage": "unable to get the workflow ID of mirror fake_mirror",
	"currentFlowState": "STATUS_UNKNOWN",
	"ok": false
}

POST api/v1/mirrors/status

This endpoint is used to get the status of a mirror. Status of the mirror could be:

STATUS_SETUP: The mirror is in setup flow, where it creates target tables and metadata tables.
STATUS_SNAPSHOT: The mirror is currently performing initial load.
STATUS_RUNNING: The mirror has completed initial load and is in the phase of CDC.
STATUS_PAUSED: The mirror is in CDC phase and is paused.
STATUS_PAUSING: The mirror is in CDC phase and is in the process of pausing.
STATUS_TERMINATED: The mirror has been deleted/terminated.
STATUS_UNKNOWN: The mirror is not found in the catalog, or its status cannot be obtained due to some issue.

Request Body

The request consists of the mirror name specified as flowJobName and includeFlowInfo as a boolean, which when set, will include in its response additional, non-status related information about the mirror such as mirror configuration and initial load information of tables.

flowJobName

string

required

The name of the mirror to get the status of.

includeFlowInfo

boolean

When set to true, will include in its response additional, non-status related information about the mirror such as mirror configuration and initial load information of tables.

Response Fields

flowJobName

string

The name of the mirror.

errorMessage

string

Error message if any.

currentFlowState

string

The current state of the mirror.

boolean

Indicates if the request was successful.

Additional flow information when includeFlowInfo is set to true:

Show child attributes

cdcStatus

object

Information about the mirror configuration and initial load of tables.

Show config

config

object

Mirror configuration.

Show child attributes

flowJobName

string

The name of the mirror.

tableMappings

object array

Array of table mapping objects. Each object contains the following fields.

Show child attributes

sourceTableIdentifier

string

The source table identifier. Will be of the form schema.table.

destinationTableIdentifier

string

The destination table identifier.

exclude

string array

Column names to exclude

maxBatchSize

number

Pull batch size: the maximum number of rows to pull in a single batch, if idleTimeoutSeconds has not been reached.

idleTimeoutSeconds

string

The time in seconds after which the mirror will flush pulled rows to the target data store, if pull batch size has not already been reached.

publicationName

string

User provided publication name for the mirror, if any.

replicationSlotName

string

User provided replication slot name for the mirror, if any.

doInitialSnapshot

boolean

Indicates if initial snapshot was enabled for this mirror.

snapshotNumRowsPerPartition

number

The number of rows to sync per partition during initial snapshot.

snapshotMaxParallelWorkers

number

The maximum number of parallel workers to use during initial snapshot.

snapshotNumTablesInParallel

number

The number of tables to sync in parallel during initial snapshot.

resync

boolean

Indicates if this mirror is a resync

initialSnapshotOnly

boolean

Indicates if this mirror was configured to only run initial load and not CDC.

softDelete

boolean

Indicates if soft delete is enabled.

softDeleteColName

string

The name of the column used for soft delete.

syncedAtColName

string

The name of the column used for tracking sync time.

script

string

Lua script configured for the mirror.

system

Q | PG

The data type system of the mirror.

sourceName

string

The source peer name.

destinationName

string

The destination peer name.

snapshotStatus

object

Information about the initial load of tables.

Show child attributes

clones

object array

Array of clone objects - each indicating initial load information of a table. Each object contains the following fields.

Show child attributes

tableName

string

The name of the destination table.

startTime

string

The time when the clone started.

numPartitionsCompleted

number

The number of partitions completed.

numPartitionsTotal

number

The total number of partitions.

numRowsSynced

string

The number of rows synced.

avgTimePerPartitionMs

string

The average time taken to sync each partition, in milliseconds.

flowJobName

string

The name of the clone job.

sourceTable

string

The source table.

fetchCompleted

boolean

Indicates if fetching is completed.

consolidateCompleted

boolean

Indicates if consolidation (loading of stage files to target tables) is completed.

mirrorName

string

The name of the mirror.

sourceType

string

The source peer type. Example: "POSTGRES"

destinationType

string

The destination peer type. Example: "BIGQUERY"

curl --request POST \
  --url http://localhost:3000/api/v1/mirrors/status \
  --header 'Authorization: Basic OnBlZXJkYg==' \
  --header 'Content-Type: application/json' \
  --data '{
	`flowJobName`:`testing_bq_2`,
	`includeFlowInfo`:false
}'

curl --request POST \
  --url http://localhost:3000/api/v1/mirrors/status \
  --header 'Authorization: Basic OnBlZXJkYg==' \
  --header 'Content-Type: application/json' \
  --data '{
	`flowJobName`:`testing_bq_2`,
	`includeFlowInfo`:true
}'

{
	"flowJobName": "testing_bq_2",
	"errorMessage": "",
	"currentFlowState": "STATUS_RUNNING",
	"ok": true
}

{
	"flowJobName": "testing_bq_2",
	"cdcStatus": {
		"config": {
			"flowJobName": "testing_bq_2",
			"tableMappings": [
				{
					"sourceTableIdentifier": "public.sales",
					"destinationTableIdentifier": "public_sales",
					"partitionKey": "",
					"exclude": [] // excluded columns, comma-separated
				},
			],
			"maxBatchSize": 1000000,
			"idleTimeoutSeconds": "60",
			"cdcStagingPath": "",
			"publicationName": "",
			"replicationSlotName": "",
			"doInitialSnapshot": true,
			"snapshotNumRowsPerPartition": 1000000,
			"snapshotStagingPath": "",
			"snapshotMaxParallelWorkers": 4,
			"snapshotNumTablesInParallel": 1,
			"resync": false,
			"initialSnapshotOnly": false,
			"softDelete": true,
			"softDeleteColName": "_PEERDB_IS_DELETED",
			"syncedAtColName": "_PEERDB_SYNCED_AT",
			"script": "",
			"system": "Q",
			"sourceName": "postgres_local",
			"destinationName": "bq_peer"
		},
		"snapshotStatus": {
			"clones": [
				{
					"tableName": "public_sales",
					"startTime": "2024-06-27T14:37:48.701204Z",
					"numPartitionsCompleted": 1,
					"numPartitionsTotal": 1,
					"numRowsSynced": "4",
					"avgTimePerPartitionMs": "11381",
					"flowJobName": "clone_testing_bq_2_public_sales_dc500df7_2606_4c19_96c0_e40e0df4e5ec",
					"sourceTable": "public.sales",
					"fetchCompleted": true,
					"consolidateCompleted": true,
					"mirrorName": "testing_bq_2"
				}
			]
		},
		"cdcSyncs": [], // coming soon
		"sourceType": "POSTGRES",
		"destinationType": "BIGQUERY"
	},
	"errorMessage": "",
	"currentFlowState": "STATUS_RUNNING",
	"ok": true
}

{
	"flowJobName": "fake_mirror",
	"errorMessage": "unable to get the workflow ID of mirror fake_mirror",
	"currentFlowState": "STATUS_UNKNOWN",
	"ok": false
}

CDC batches Change mirror state

⌘I

​Request Body

​Response Fields

Request Body

Response Fields