Plumber (Golang)

Search…
Plumber (Golang)
​
Plumber is a Swiss army knife CLI tool for interacting with various messaging systems.
Source Code
GitHub - batchcorp/plumber: A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.
GitHub
Installation
Mac
Linux
Windows
Docker
Compile From Source
Install via Homebrew
1
$ brew tap batchcorp/public
2
$ brew install plumber
Copied!
Download Latest Binary
https://github.com/batchcorp/plumber/releases/latest/download/plumber-darwin
Install via Homebrew
1
$ brew tap batchcorp/public
2
$ brew install plumber
Copied!
Download Latest Binary
https://github.com/batchcorp/plumber/releases/latest/download/plumber-linux
Download latest binary
https://github.com/batchcorp/plumber/releases/latest/download/plumber-windows.exe
github.com
1
docker pull batchcorp/plumber:latest
Copied!
1
$ git clone https://github.com/batchcorp/plumber.git
2
$ cd plumber
3
$ make build
Copied!
Compiled binaries will be available under the build/ directory
History of Plumber
Here at Batch, we're obviously big fans of messaging systems and event-driven architecture. While these paradigms make for wonderfully resilient platforms, they can be difficult to test, often requiring one-off scripts to emit whatever event is needed. This can quickly get messy, especially when dealing with serialization formats like protocol buffers.
We created Plumber as a handy tool for our engineers to test and develop our own platform, allowing any number of system components to be tested simply by emitting the necessary event with the necessary data. This tool became so indispensable to us that we decided to open-source and expand it for the benefit of the software engineering community.
What does plumber do?
Simply put: Plumber allows you to read and write data to various messaging systems via the command line, without having to write hacky scripts or learn the complicated APIs of most messaging systems.
Supported Messaging Systems
Kafka
RabbitMQ
Google Cloud Platform PubSub
MQTT
Amazon SQS
Amazon SNS (Publishing)
ActiveMQ
Azure Service Bus
NATS
Redis PubSub
Redis Streams
Writing Messages
Plumber can write the following formats:
Protocol Buffers
AVRO
Binary
Plain text
Plumber can take your .proto definitions, a JSON representation of the message, and emit the  protocol buffer message without any additional tools.
1
plumber write kafka \
2
  --topic new_orders
3
  --protobuf-dir path/to/myprotos \
4
  --protobuf-root-message Order \
5
  --input-type jsonpb \
6
  --input-file proto_message.json
Copied!
By specifying the path to an .avsc AVRO schema file, plumber can easily emit AVRO serialized messages.
1
plumber write kafka \
2
  --topic new_orders \
3
  --avro-schema path/to/schema.avsc \
4
  --input-file path/to/message.json
Copied!
Binary file:
1
plumber write kafka --topic new_orders --input-file some.bin
Copied!
Base64 encoded string:
1
plumber write kafka \
2
  --topic new_orders \
3
  --input-type base64 \
4
  --input-data=d2mUgbG92ZSBtZXNzYWdp1bmc=
Copied!
Plain text file:
1
plumber write kafka --topic new_orders --input-file message.json
Copied!
Plain text string: 
1
plumber write kafka --topic new_orders --input-data "Hello world!"
Copied!
Reading Messages
In addition to outputting the raw message contents, Plumber can also automatically decode multiple serialization/wire formats and display the decoded result.
Protocol Buffers
AVRO
GZIP
Base64
Plumber will output a JSON representation of the protobuf message
1
plumber read kafka \
2
  --topic new_orders \
3
  --protobuf-dir path/to/myprotos \
4
  --protobuf-root-message Order
Copied!
1
plumber read kafka \
2
  --topic new_orders \
3
  --avro-schema=path/to/schema.avsc
Copied!
1
plumber read kafka --topic new_orders --convert gzip
Copied!
1
plumber read kafka --topic new_orders --convert base64
Copied!
Relaying
Plumber's relay mode is used to ship messages from your messaging system directly to a Batch collection.
For more information on how to relay, check the relay docs.
Resources - Previous
Open Source
Rabbit library (Golang)
Last modified 6mo ago
Copy link
Contents
Source Code
Installation
History of Plumber
What does plumber do?
Supported Messaging Systems
Writing Messages
Relaying