PostgreSQL

Relay WAL events from your Postgres server to a Batch collection

Postgres Setup

Your postgres server must be started with the option wal_level=logical and have at least one available replication slot for plumber to read from

Ensure postgres is started with the correct Write Ahead Log level, and has at least 1 replication slot

postgres=# show wal_level;
 wal_level
-----------
 logical
(1 row)

postgres=# show max_replication_slots;
 max_replication_slots
-----------------------
 10
(1 row)

Create a publication

A publication is required for postgres to know which changes to ship to a replication slot. See postgres documentation for options: https://www.postgresql.org/docs/current/sql-createpublication.html

You may name the publication anything you wish. In this example, we'll use batchsh_plumber.

CREATE PUBLICATION batchsh_plumber FOR ALL TABLES;

Create a replication slot

For this example, we'll name ours plumber_slot.

The most important thing to note here is that pgoutput is the output plugin.

SELECT * FROM pg_create_logical_replication_slot('plumber_slot', 'pgoutput');

    slot_name   |    lsn
--------------+-----------
 plumber_slot | 0/168A3B8
(1 row)

Relay events to your batch collection

Relay via plumber docker container

docker run -d --name postgres -p 8080:8080 \
    -e PLUMBER_RELAY_CDCPOSTGRES_HOSTNAME=localhost \
    -e PLUMBER_RELAY_CDCPOSTGRES_USERNAME=postgres \
    -e PLUMBER_RELAY_CDCPOSTGRES_PASSWORD=postgres \
    -e PLUMBER_RELAY_CDCPOSTGRES_DATABASE=mydb \
    -e PLUMBER_RELAY_CDCPOSTGRES_SLOT=plumber_slot \
    -e PLUMBER_RELAY_CDCPOSTGRES_PUBLISHER=batchsh_plumber \
    -e PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
    batchcorp/plumber postgres

Relay via plumber binary

PLUMBER_RELAY_CDCPOSTGRES_HOSTNAME=localhost \
PLUMBER_RELAY_CDCPOSTGRES_USERNAME=postgres \
PLUMBER_RELAY_CDCPOSTGRES_PASSWORD=postgres \
PLUMBER_RELAY_CDCPOSTGRES_DATABASE=mydb \
PLUMBER_RELAY_CDCPOSTGRES_SLOT=plumber_slot \
PLUMBER_RELAY_CDCPOSTGRES_PUBLISHER=batchsh_plumber \
PLUMBER_RELAY_TOKEN=$YOUR-BATCHSH-TOKEN-HERE \
plumber relay postgres

View events in your Batch collection

Change events should start appearing in your batch collection within seconds of them being made in your postgresql database.