Releases: aliyun/aliyun-emapreduce-datasources
Releases · aliyun/aliyun-emapreduce-datasources
Release v2.2.0
Release v2.1.0
add latest datasource package (#484) * add datasource pre-build package 2.1.0 * remove last latest datasource package * add latest datasource package
Release v2.0.0
Merge pull request #431 from aliyun/release-200 Release 200
Release v1.9.0
add latest jars (#402)
Release v1.8.0
- New features
- Improvements
- Bug fix
- #334: Fix get datahub schema
- #337: Wrong type convert for ots record
- #341: Failed to read logstore tag info
- #344: Datahub source job recovered from a wrong starting offset
- #349: Fix bug for ObjectMetadataDeserializer has the wrong date format
- #369: Remove explicit calls to System.gc()
- #350: Fix gson producing different date formats in diferent os/locale
- #378: Duplicate data when persist InternalRow
Release v1.7.0
- New features
- Support datahub datasource in Spark Structured Streaming
- Add jdbc datasource support
- Add hbase datasource support
- Loghub relation support write
- Add druid sink source
- Improvements
- Better support loghub datasource schema type convert
- Check if shard has finished after pull logs
- Add retry for internal server error
- Dont use SinkLog when there is no config
- Loghub table should be defined with schema
- Optimize to reduce extra count job when generate loghub RDD
- Optimize logstore read step to avoid exceeding read quota limits
- Bug fix
- Fix NPE when no data fetched from datahub
- Could not generate RDD from a parent for unifying at time
- Sometimes failed to get loghub schema
- DatahubRDD 'count' should initial in each ShardPartition
Release v1.6.0
- New feature
- Spark Streaming SQL Test tools.
- support loghub datasource in spark streaming sql.
- support datahub direct api implementation in Spark Streaming.
- Improvement
- loghub python function support direct api.
- Bug fix
- OnsUtilsHelper.createDefaultStreams lost ONS body message.
- Failed to insert values into table store.
Release v1.5.0
-
Improvement
- Spark Structured Streaming support Loghub datasource.
- Support parallel batch processing in loghub shard.
- Cost too much time when update checkpoint in loghub direct api.
- Add createRDD() in LoghubUtilsHelper.
- Add java constructor for direct loghub dstream.
-
Bugfix
- Fix wrong loghub RDD partition index.
- Direct loghub dstream data dose not contain tag information.
- Direct loghub dstream dose not support to process data from specific position.
- OdpsUtils.runSQL connection timeout.
- ODPS STRING type convert failed.
- Failed to re-run structured loghub job.
- structured loghub job failed with org.I0Itec.zkclient.exception.ZkNoNodeException.
- Wrong endTime in LoghubBatchRDD.
Release v1.4.4
- Spark Streaming support DataHub
- DirectLoghubInputDStream aupport multiple actions