Make sure you have the following dependencies available:
git
(used for source version control)ssh
client (used to authenticate with GitHub)go
and rust
setup on your local machine (used by flow
and nexus
respectively)Run the following steps to set up your environment:
git clone --recursive git@github.com:PeerDB-io/peerdb.git
command.psql postgres
. To set up a dev environment, it is important to have a user named postgres because end-to-end tests use postgres
userwal_level
to logical, run ALTER SYSTEM SET wal_level = 'logical'
max_wal_senders
, run ALTER SYSTEM SET max_wal_senders = 10
max_replication_slots
, run ALTER SYSTEM SET max_replication_slots = 4
brew services restart postgresql
to reload the configuration for the changes to take effect.wal_level
has been set to logical, run SHOW wal_level;
max_wal_senders
has been set to 10, run SHOW max_wal_senders;
max_replication_slots
has been set to 4, run SHOW max_replication_slots;
You should have a GCP account and Project set up. The Project should be associated with a billing account. If you don’t have a credit card, you can set up the bigquery sandbox by following the instructions here. We will be using a Service Account and Key file to authenticate bigquery from our local machine.
After all the edits, the bq-creds.json file should look something like this -
To configure bigquery as a peer, you will have to modify the environment variable TEST_BQ_CREDS
to point to the path of the JSON file. If you are using VSCode, you can add the below entry under settings.json file.
You should have a Snowflake account and warehouse set up. If you don’t you can sign up for a free-trial and create an account with Snowflake. Your role should have enough permissions to create a database, schema and tables. PeerDB uses snowflake’s key pair authentication. Please follow the steps below -
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
pbcopy < rsa_key.pub
ALTER USER jsmith SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...';
DESC USER jsmith;
For more information, please follow the official link here
The JSON used to authenticate snowflake from inside peerDB is of the below format -
account_id
: <organization_name>-<account_name>
. You can find this information under Admin Page -> Accounts.
username
: <user_name>
. Go to Admin -> Users & Roles -> Users, find the relevant user_name
private_key
: <string_private_key>
The rsa_key.p8 file generated in the above step needs to be converted to string, which means all new lines should be converted to the “\n” symbol.
database
: “peerdb”. Create a new database. You can use an existing one, but make sure your role does have permissions to read/write to the database.
schema
: “peerdb”
warehouse
: “COMPUTE_WH” or whatever is available under Admin -> Warehouses
role
: “ACCOUNTADMIN” or whatever is available under Admin -> Users & Roles -> Role
query_timeout
: 300
Save the above json file into sf-creds.json. After all the edits, the bq-creds.json file should look something like this -
To configure snowflake as a peer, you will have to modify the environment variable TEST_SF_CREDS
to point to the path of the JSON file. If you are using VSCode, you can add the below entry under settings.json file.
You can run go tests, but before that, keep a timeout of 300s or higher. If you are using vscode, the default timeout is 60s, which will make the test fail.
In vscode, change the settings.json to have an entry
"go.testTimeout": "300s"
You are all set! Start testing the code and contributing!
Make sure you have the following dependencies available:
git
(used for source version control)ssh
client (used to authenticate with GitHub)go
and rust
setup on your local machine (used by flow
and nexus
respectively)Run the following steps to set up your environment:
git clone --recursive git@github.com:PeerDB-io/peerdb.git
command.psql postgres
. To set up a dev environment, it is important to have a user named postgres because end-to-end tests use postgres
userwal_level
to logical, run ALTER SYSTEM SET wal_level = 'logical'
max_wal_senders
, run ALTER SYSTEM SET max_wal_senders = 10
max_replication_slots
, run ALTER SYSTEM SET max_replication_slots = 4
brew services restart postgresql
to reload the configuration for the changes to take effect.wal_level
has been set to logical, run SHOW wal_level;
max_wal_senders
has been set to 10, run SHOW max_wal_senders;
max_replication_slots
has been set to 4, run SHOW max_replication_slots;
You should have a GCP account and Project set up. The Project should be associated with a billing account. If you don’t have a credit card, you can set up the bigquery sandbox by following the instructions here. We will be using a Service Account and Key file to authenticate bigquery from our local machine.
After all the edits, the bq-creds.json file should look something like this -
To configure bigquery as a peer, you will have to modify the environment variable TEST_BQ_CREDS
to point to the path of the JSON file. If you are using VSCode, you can add the below entry under settings.json file.
You should have a Snowflake account and warehouse set up. If you don’t you can sign up for a free-trial and create an account with Snowflake. Your role should have enough permissions to create a database, schema and tables. PeerDB uses snowflake’s key pair authentication. Please follow the steps below -
openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM -out rsa_key.p8 -nocrypt
openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub
pbcopy < rsa_key.pub
ALTER USER jsmith SET RSA_PUBLIC_KEY='MIIBIjANBgkqh...';
DESC USER jsmith;
For more information, please follow the official link here
The JSON used to authenticate snowflake from inside peerDB is of the below format -
account_id
: <organization_name>-<account_name>
. You can find this information under Admin Page -> Accounts.
username
: <user_name>
. Go to Admin -> Users & Roles -> Users, find the relevant user_name
private_key
: <string_private_key>
The rsa_key.p8 file generated in the above step needs to be converted to string, which means all new lines should be converted to the “\n” symbol.
database
: “peerdb”. Create a new database. You can use an existing one, but make sure your role does have permissions to read/write to the database.
schema
: “peerdb”
warehouse
: “COMPUTE_WH” or whatever is available under Admin -> Warehouses
role
: “ACCOUNTADMIN” or whatever is available under Admin -> Users & Roles -> Role
query_timeout
: 300
Save the above json file into sf-creds.json. After all the edits, the bq-creds.json file should look something like this -
To configure snowflake as a peer, you will have to modify the environment variable TEST_SF_CREDS
to point to the path of the JSON file. If you are using VSCode, you can add the below entry under settings.json file.
You can run go tests, but before that, keep a timeout of 300s or higher. If you are using vscode, the default timeout is 60s, which will make the test fail.
In vscode, change the settings.json to have an entry
"go.testTimeout": "300s"
You are all set! Start testing the code and contributing!