Add custom operator support

[nwangtw]

This is a replacement of the following PR (it can't be reopen after force push...):
#3049

This is the implementation of CustomOperator based on this design doc:
https://docs.google.com/document/d/1XzF0IlfuaaW8Gx3cPx1xLtP-kgCFK0TRNS5aAzuMuMg/edit#

CustomOperator and CustomOperatorOutput are included in this PR.

Future works:

• add/refactor windowed custom operator
• refactor/simplify existing operators (v2 design in doc)

[jerrypeng]

@nwangtw can you please provide an example topology that uses the custom operator? Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API? Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt)? So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

[nwangtw]

can you please provide an example topology that uses the custom operator?

Sure.

Also, isn't the point of the custom operator to provide some sort of compatibility with the "Storm" API? Instead of forcing users to copy their code to the corresponding "prepare" and "execute" methods in the custom operator, couldn't we just do something like CustomOperator(Bolt). So that the user that just use their existing spout or bolt implementation? and the CustomOperator is just a wrapper for bolts which allows them to be used in streamlet topologies.

Not really in this step/PR.

The goal of this step is to provide a way for users to write their own operators when needed. The low level Bolt is supported by previous PRs.

[jerrypeng]

@nwangtw If that is the case. I don't think that is appropriate then. Streamlet aims to break away from the spout and bolt API. With this change we are kind of bring the streamlet API back to the bolt and spout API.

[nwangtw]

It does blur the line a bit to allow users to write their own operators with all functions available in low level API. However these operators still has type safety and can be used the same way as other pre-defined operators. Therefore, from API point of view, I don't think it "bring the streamlet API back to the bolt and spout API." Instead, it brings more powers from the bolt and spout API to streamlet.

Basically Streamlet is a functional API, and function API is overall nice and elegant. However I believe, a functional API is HARDLY a general purpose API and it is only good for "nice" problems. It might enforce uses to change the way of thinking and solving problems, which can be good sometimes (use general/standard operations) and bad in other times (more like hacks). Therefore I think supporting custom operator is important for Heron.

That being said, the original feature I needed was to reuse existing Bolts and Spouts which has been completed. This design doc and PR are just an extension from the original feature so it is not as urgent and we still have time to discuss.