Not able to define rules with array of complex types

[arunmahadevan]

In the below schema the 'policy' is an array of record (nested) type. Below is the response from the Streamline API, but the UI does not provide a way to enter an index and then select a member field. Eg. policy[0].f1 = 100.

Response from streamline API:

{"id":15,"topologyId":2,"topologyComponentBundleId":1,"versionId":7,"name":"KAFKA-1","description":"","config":{"properties":{"cluster":"1","consumerGroupId":"aaa","retryDelayPeriodMs":2,"maximumUncommittedOffsets":10000000,"maxRecordsPerPoll":"500","maximumRetries":2147483647,"parallelism":1,"keytab":"b","emitNullTuples":false,"principal":"a","partitionRefreshPeriodMs":2000,"securityProtocol":"SASL_PLAINTEXT","bootstrapServers":"ctr-e137-1514896590304-34122-01-000002.hwx.site:6667,ctr-e137-1514896590304-34122-01-000003.hwx.site:6667,ctr-e137-1514896590304-34122-01-000005.hwx.site:6667,ctr-e137-1514896590304-34122-01-000007.hwx.site:6667","kafkaServiceName":"kafka","retryDelayMaximumMs":10000,"fetchMaximumBytesPerPartition":"1048576","firstPollOffsetStrategy":"UNCOMMITTED_EARLIEST","sslTruststoreType":"JKS","clusters":"cl1","fetchMinimumBytes":"1","sslKeystoreType":"JKS","sslTrustManagerAlgo":"PKIX","retryInitialDelayMs":0,"sslKeyManagerAlgo":"SunX509","sslProtocol":"TLS","topic":"testarray","readerSchemaVersion":"1","pollTimeoutMs":200,"offsetCommitPeriodMs":30000}},"reconfigure":false,"outputStreams":[{"id":15,"versionId":7,"streamId":"kafka_stream_15","description":null,"topologyId":2,"fields":[{"name":"user_id","type":"STRING","optional":false},{"name":"review_id","type":"STRING","optional":false},{"name":"stars","type":"INTEGER","optional":false},{"name":"date","type":"STRING","optional":false},{"name":"policy","type":"ARRAY","optional":false,"members":[{"name":"AttributeMapping","type":"NESTED","optional":true,"fields":[{"name":"f1","type":"LONG","optional":false},{"name":"f2","type":"LONG","optional":false}]}]},{"name":"type","type":"STRING","optional":false},{"name":"votes","type":"NESTED","optional":true,"fields":[{"name":"funny","type":"INTEGER","optional":false},{"name":"useful","type":"INTEGER","optional":false},{"name":"cool","type":"INTEGER","optional":false}]}],"timestamp":1517358172805}],"timestamp":1517358094852}

SR Schema :

{
 "type": "record",
 "namespace": "com.hortonworks.registries",
 "name": "yelp_reviews",
 "fields": [
  {
   "name": "user_id",
   "type": "string"
  },
  {
   "name": "review_id",
   "type": "string"
  },
  {
   "name": "stars",
   "type": "int"
  },
  {
   "name": "date",
   "type": "string"
  },
  {
   "name": "policy",
   "type": {
    "type": "array",
    "items": {
     "name": "AttributeMapping",
     "type": "record",
     "fields": [
      {
       "name": "f1",
       "type": "long"
      },
      {
       "name": "f2",
       "type": "long"
      }
     ]
    }
   }
  },
  {
   "name": "type",
   "type": "string"
  },
  {
   "name": "votes",
   "type": {
    "name": "votes_entry",
    "type": "record",
    "fields": [
     {
      "name": "funny",
      "type": "int"
     },
     {
      "name": "useful",
      "type": "int"
     },
     {
      "name": "cool",
      "type": "int"
     }
    ]
   }
  }
 ]
}

[shahsank3t]

@arunmahadevan
FYI: currently we do not have any support for "array" type throughout the application on UI.

[arunmahadevan]

@shahsank3t , in the kafka source the array type is accepted (see field policy) and displays the output fields. However it does not display the type of the members of the array.

We need to fix all the processors and sinks before enabling end-end support for "array" types.