Docs
Search…
⌃K

PostgreSQL

Relay WAL events from your Postgres server to a Batch collection

Postgres Setup

Your postgres server must be started with the option wal_level=logical and have at least one available replication slot for plumber to read from
Ensure postgres is started with the correct Write Ahead Log level, and has at least 1 replication slot
postgres=# show wal_level;
wal_level
-----------
logical
(1 row)
postgres=# show max_replication_slots;
max_replication_slots
-----------------------
10
(1 row)

Create a publication

A publication is required for postgres to know which changes to ship to a replication slot. See postgres documentation for options: https://www.postgresql.org/docs/current/sql-createpublication.html
You may name the publication anything you wish. In this example, we'll use batchsh_plumber.
CREATE PUBLICATION batchsh_plumber FOR ALL TABLES;

Create a replication slot

For this example, we'll name ours plumber_slot.
The most important thing to note here is that pgoutput is the output plugin.
SELECT * FROM pg_create_logical_replication_slot('plumber_slot', 'pgoutput');
slot_name | lsn
--------------+-----------
plumber_slot | 0/168A3B8
(1 row)

Relay events to your batch collection

Relay via plumber docker container
docker run -d --name postgres -p 8080:8080 \
-e PLUMBER_RELAY_CDCPOSTGRES_HOSTNAME=localhost \
-e PLUMBER_RELAY_CDCPOSTGRES_USERNAME=postgres \
-e PLUMBER_RELAY_CDCPOSTGRES_PASSWORD=postgres \
-e PLUMBER_RELAY_CDCPOSTGRES_DATABASE=mydb \
-e PLUMBER_RELAY_CDCPOSTGRES_SLOT=plumber_slot \
-e PLUMBER_RELAY_CDCPOSTGRES_PUBLISHER=batchsh_plumber \
-e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
batchcorp/plumber postgres
Relay via plumber binary
PLUMBER_RELAY_CDCPOSTGRES_HOSTNAME=localhost \
PLUMBER_RELAY_CDCPOSTGRES_USERNAME=postgres \
PLUMBER_RELAY_CDCPOSTGRES_PASSWORD=postgres \
PLUMBER_RELAY_CDCPOSTGRES_DATABASE=mydb \
PLUMBER_RELAY_CDCPOSTGRES_SLOT=plumber_slot \
PLUMBER_RELAY_CDCPOSTGRES_PUBLISHER=batchsh_plumber \
PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
plumber relay postgres

View events in your Batch collection

Change events should start appearing in your batch collection within seconds of them being made in your postgresql database.