Plumber (Golang)

Plumber is a Swiss army knife CLI tool for interacting with various messaging systems.

Source Code

GitHub - batchcorp/plumber: A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.GitHub

Installation

Mac

Install via Homebrew

$ brew tap batchcorp/public
$ brew install plumber

Download Latest Binary

https://github.com/batchcorp/plumber/releases/latest/download/plumber-darwin

Linux

Install via Homebrew

$ brew tap batchcorp/public
$ brew install plumber

Download Latest Binary

https://github.com/batchcorp/plumber/releases/latest/download/plumber-linux

Windows

Download latest binary

https://github.com/batchcorp/plumber/releases/latest/download/plumber-windows.exe

Docker

docker pull batchcorp/plumber:latest

Compile From Source

$ git clone https://github.com/batchcorp/plumber.git
$ cd plumber
$ make build

Compiled binaries will be available under the build/ directory

History of Plumber

Here at Batch, we're obviously big fans of messaging systems and event-driven architecture. While these paradigms make for wonderfully resilient platforms, they can be difficult to test, often requiring one-off scripts to emit whatever event is needed. This can quickly get messy, especially when dealing with serialization formats like protocol buffers.

We created Plumber as a handy tool for our engineers to test and develop our own platform, allowing any number of system components to be tested simply by emitting the necessary event with the necessary data. This tool became so indispensable to us that we decided to open-source and expand it for the benefit of the software engineering community.

What does plumber do?

Simply put: Plumber allows you to read and write data to various messaging systems via the command line, without having to write hacky scripts or learn the complicated APIs of most messaging systems.

Supported Messaging Systems

• Kafka

• RabbitMQ

• Google Cloud Platform PubSub

• MQTT

• Amazon SQS

• Amazon SNS (Publishing)

• ActiveMQ

• Azure Service Bus

• NATS

• Redis PubSub

• Redis Streams

Writing Messages

Plumber can write the following formats:

Protocol Buffers

Plumber can take your .proto definitions, a JSON representation of the message, and emit the protocol buffer message without any additional tools.

plumber write kafka \
  --topics new_orders \
  --protobuf-dirs path/to/myprotos \
  --protobuf-root-message Order \
  --encode-type jsonpb \
  --input-file proto_message.json

AVRO

By specifying the path to an .avsc AVRO schema file, plumber can easily emit AVRO serialized messages.

plumber write kafka \
  --topics new_orders \
  --avro-schema-file path/to/schema.avsc \
  --input-file path/to/message.json

Binary

Binary file:

plumber write kafka --topics new_orders --input-file some.bin

Plain text

Plain text file:

plumber write kafka --topics new_orders --input-file message.json

Plain text string:

plumber write kafka --topics new_orders --input-data "Hello world!"

Reading Messages

In addition to outputting the raw message contents, Plumber can also automatically decode multiple serialization/wire formats and display the decoded result.

Protocol Buffers

Plumber will output a JSON representation of the protobuf message

plumber read kafka \
  --topics new_orders \
  --protobuf-dirs path/to/myprotos \
  --protobuf-root-message Order

AVRO

plumber read kafka \
  --topics new_orders \
  --avro-schema-file=path/to/schema.avsc

GZIP

plumber read kafka --topics new_orders --convert-output gzip

Base64

plumber read kafka --topics new_orders --convert-output base64

Relaying

Plumber's relay mode is used to ship messages from your messaging system directly to a Batch collection.

For more information on how to relay, check the relay docs.