This repository contains various Apache Flink connectors to connect to AWS Kinesis data sources and sinks.
This Producer allows Flink applications to push directly to Kinesis Firehose.
Configure and instantiate a FlinkKinesisFirehoseProducer:
Properties config = new Properties();
outputProperties.setProperty(ConsumerConfigConstants.AWS_REGION, region);
FlinkKinesisFirehoseProducer<String> sink = new FlinkKinesisFirehoseProducer<>(streamName, new SimpleStringSchema(), config);Follow the example instructions to create an end to end application:
- Write data into a Kinesis Data Stream
- Process the streaming data using Kinesis Data Analytics
- Write the results to a Kinesis Firehose using the
FlinkKinesisFirehoseProducer - Store the data in an S3 Bucket
- You will need to install Java 1.8+ and Maven
- Clone the repository from Github
- Build using Maven from the project root directory:
$ mvn clean install
Flink maintain backwards compatibility for the Sink interface used by the Firehose Producer. This project is compatible with Flink 1.x, there is no guarantee it will support Flink 2.x should it release in the future.
| Connector Version | Flink Version | Release Date |
|---|---|---|
| 2.1.0 | 1.x | Feb, 2023 |
| 2.0.0 | 1.x | Jul, 2020 |
| 1.1.0 | 1.x | Dec, 2019 |
| 1.0.1 | 1.x | Dec, 2018 |
| 1.0.0 | 1.x | Dec, 2018 |
This library is licensed under the Apache 2.0 License.