Neon Postgres Source Setup Guide

This is a guide on how to create a Neon PostgreSQL peer which you can use for replication in PeerDB. Make sure you’re signed in to your Neon console for this setup.

​

Creating a user with permissions

Let’s create a new user for PeerDB with the necessary permissions suitable for CDC, and also create a publication that we’ll use for replication.

For this, you can head over to the SQL Console tab. Here, we can run the following SQL commands:

  CREATE USER peerdb_user PASSWORD 'peerdb_password';
  GRANT USAGE ON SCHEMA "public" TO peerdb_user;
  GRANT SELECT ON ALL TABLES IN SCHEMA "public" TO peerdb_user;
  ALTER DEFAULT PRIVILEGES IN SCHEMA "public" GRANT SELECT ON TABLES TO peerdb_user;

-- Give replication permission to the USER
  ALTER USER peerdb_user REPLICATION;

-- Create a publication. We will use this when creating the mirror
  CREATE PUBLICATION peerdb_publication FOR ALL TABLES;

Click on Run to have a publication and a user ready.

​

Enable Logical Replication

In Neon, you can enable logical replication through the UI. This is necessary for PeerDB’s CDC to replicate data. Head over to the Settings tab and then to the Logical Replication section.

Click on Enable to be all set here. You should see the below success message once you enable it.

Let’s verify the below settings in your Neon Postgres instance:

SHOW wal_level; -- should be logical
SHOW max_wal_senders; -- should be 10
SHOW max_replication_slots; -- should be 10

​

IP Whitelisting (For Neon Enterprise plan)

If you have Neon Enterprise plan, you can whitelist the PeerDB Cloud IP addresses (or the equivalent for PeerDB OSS/Enterprise) to allow replication from PeerDB Cloud to your Neon Postgres instance. To do this you can click on the Settings tab and go to the IP Allow section.

​

Copy Connection Details

Now that we have the user, publication ready and replication enabled, we can copy the connection details to create a Neon Postgres peer in PeerDB. Head over to the Dashboard and at the text box where it shows the connection string, change the view to Parameters Only. We will need these parameters for our next step.

​

Create Neon Postgres Peer in PeerDB UI

Now that we have the connection details, we can create a Neon Postgres peer in PeerDB. Head over to the PeerDB UI and click on Create Peer. Select Neon as the source.

Fill in the Neon connection details that we copied earlier in the following form.

Click on Validate and once that’s green, you can go ahead and click on Create to create the peer!

​

Important Gotchas

In Neon databases that are idle (no activity), slots can be dropped. To prevent this, you need to ensure that the replication slot actively receives database changes. You can use pg_logical_emit_message()—a system function for emitting a logical decoding message into the WAL. PeerDB picks up the message as part of its WAL processing and flushes the slot at the frequency of the CDC sync interval.

You’d need to grant EXECUTE permissions for the peerdb replication user on this function:

GRANT EXECUTE ON FUNCTION pg_logical_emit_message( transactional boolean, prefix text, content text )
TO <postgres_peer_user>;

Edit the current value fields of the following settings in the Settings tab in UI:

• PEERDB_ENABLE_WAL_HEARTBEAT: Set this to true (no quotes or anything)
• PEERDB_WAL_HEARTBEAT_QUERY: Set this to the emit_message function call command to be run periodically:

SELECT pg_logical_emit_message(false,'heartbeat','')

Currently, PeerDB emits a message every 12 minutes. This will soon be configurable.