Skip to content

Latest commit

 

History

History
151 lines (103 loc) · 4.86 KB

README.md

File metadata and controls

151 lines (103 loc) · 4.86 KB
Raw

kafcat

Kafcat is a Rust fully async rewrite of kafkacat.

I was trying to copy some data from kafka on a remote server to localhost for testing purpose. Kafkacat was the only usable tool to do so. However, it is written in C and does not support binary data at all. I was working on a pull request, but it haven't been merged yet, and the code was hard to maintain and error prone. So I wrote my own kafcat.

Features

It mimicks kafkacat's cli usage, but with extra features. It can

  • read from stdin and send to kafka topic
  • read from kafka to stdin
  • copy from one topic to another(even on different servers)
  • import/export data in json format
  • swappable backend(only librdkafka for now, kafka-rust coming soon)

Drawbacks

Kafcat is still on early development, so

  • does not support TLS yet

  • does not support Apache AVRO

  • does not support complex input/output format

Basic Usage

  1. Compile kafcat. You can find the binary at target/release/kafcat

    cargo build --release

  2. Make sure Kafka is running (We assume at localhost:9092). You can usedocker-compose -f tests/plaintext-server.yml up(note that kafka in docker on MacOS is problematic, I couldn't even setup the proper testing environment via docker)

  3. Setup the listener, assuming kafcat is in current directory

    ./kafcat -C --topic test

  4. Setup the producer in another terminal,

    ./kafcat -P --topic test

  5. Copy a topic. the format is ./kafcat copy <from> -- <to>, from and to are exactly as used consumer and producer

    ./kafcat copy --topic test -- --topic test2

  6. Type any key and value with the default delimiter :. For example, hello:world.

  7. You should see hello:world in the consumer terminal.

  8. Detailed help can be found at ./kafcat --help, ./kafcat -C --help, ./kafcat -P --help, etc. You can also check out kafkacat for reference.

    kafcat-consume 

USAGE:
    kafcat {consume, -C} [FLAGS] [OPTIONS] --topic <topic>

FLAGS:
    -e, --exit
            Exit successfully when last message received

    -h, --help
            Prints help information

    -V, --version
            Prints version information


OPTIONS:
    -b, --brokers <brokers>
            Broker list in kafka format [default: localhost:9092]

    -s, --format <format>
            Serialize/Deserialize format [default: text]

    -G, --group-id <group-id>
            Consumer group id. (Kafka >=0.9 balanced consumer groups) [default: kafcat]

    -K <key-delimiter>
            Delimiter to split input key and message [default: :]

    -D <msg-delimiter>
            Delimiter to split input into messages(currently only supports '\n') [default: 
            ]

    -o <offset>
            Offset to start consuming from:
                                 beginning | end | stored |
                                 <value>  (absolute offset) |
                                 -<value> (relative offset from end)
                                 s@<value> (timestamp in ms to start at)
                                 e@<value> (timestamp in ms to stop at (not included))[default:
            beginning]

    -p, --partition <partition>
            Partition

    -t, --topic <topic>
            Topic
kafcat-produce 

USAGE:
    kafcat {produce, -P} [OPTIONS] --topic <topic>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -b, --brokers <brokers>        Broker list in kafka format [default: localhost:9092]
    -s, --format <format>          Serialize/Deserialize format [default: text]
    -G, --group-id <group-id>      Consumer group id. (Kafka >=0.9 balanced consumer groups)
                                   [default: kafcat]
    -K <key-delimiter>             Delimiter to split input key and message [default: :]
    -D <msg-delimiter>             Delimiter to split input into messages(currently only supports
                                   '\n') [default: 
                                   ]
    -p, --partition <partition>    Partition
    -t, --topic <topic>            Topic
kafcat-copy 
Copy mode accepts two parts of arguments <from> and <to>, the two parts are separated by [--].
<from> is the exact as Consumer mode, and <to> is the exact as Producer mode.

USAGE:
    kafcat copy <from>... [--] <to>...

ARGS:
    <from>...    
    <to>...      

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

Programming Style

  • git rebase onto master branch when possible
  • Squash commits before push