HTTP Sink MVP

[davidsloan]

Work to date on a HTTP sink.

Not yet covered:

• deletion (via tombstones)
• tests with tls certificates

This is a work in progress and should not be used in a production environment.

Configuration
For speed of development the configuration is provided via a Kafka Connect configuration property (connect.http.config), through the means of a Json template. Here is an example:

{
   "authentication":{
      "username":"user",
      "password":"pass",
      "type":"BasicAuthentication"
   },
   "method":"Put",
   "endpoint":"http://{{clientSubDomain}}.example.com",
   "content":"<cost><costCentre>{{clientCostCentre}}</costCentre><spender>{{employee}}</spender><description>{{costDescription}}</description></body></cost>",
   "headers":[
      [
         "Content-Type",
         "application/xml"
      ],
      [
         "X-Client-Cost-Centre",
         "{{clientCostCentre}}"
      ]
   ]
}

There was a discussion about using KCQL however it was decided that this is not suitable for this use case.

Templating

Two different types of templates can be provided:

XML is used here as an example, in fact any markup language or text content can be supported.

Single Template

For when you want to substitute the contents of a single message into the template content. It could be used for multiple messages but this wouldn't make much sense as there is no looping part.

      <xml>
          <message>
            <topic>{{topic}}</topic>
            <employee>{{value.employeeId}}</employee>
            <order>{{value.orderNo}}</order>
            <groupDomain>{{value.groupDomain}}</groupDomain>
          </message>
      </xml>

Multi Template

For when you want to merge multiple messages inside a single HTTP request. It loops through every message.

      <xml>
        <messages>
          {{#message}}
          <message>
            <topic>{{topic}}</topic>
            <employee>{{value.employeeId}}</employee>
            <order>{{value.orderNo}}</order>
            <groupDomain>{{value.groupDomain}}</groupDomain>
          </message>
          {{/message}}
        </messages>
      </xml>

[cmcmteixeira]

It's a nit pick :)
I would simply pattern match on the result of extractFromKey and throw on the left branch. It's a bit more explicit behavior wise

[GoMati-MU]

I guess the functional tests are to be added later? Do we need that line then?

[GoMati-MU]

I guess the functional tests are to be added later? Do we need that line then?

[cmcmteixeira]

There's no need to extend LazyLogging here right ?

[cmcmteixeira]

Let's have a look at this on Monday maybe.

Unfortunately, doing it like this means that it will be ever running and cannot be stopped.
Running the test from HttpSinkTaskIT you can see that even tho the test terminates this process keeps running until the jvm is terminated. Ideally we would want to have a "stop" method that could interrupt this scheduling.