From 8ba66a1ccec30c8bb44f6f705cbd947307e949a5 Mon Sep 17 00:00:00 2001 From: fe2s Date: Mon, 13 Aug 2018 23:50:48 +0300 Subject: [PATCH 1/6] update README.md with version compatibility and branching strategy details --- README.md | 35 ++++++++++++----------------------- 1 file changed, 12 insertions(+), 23 deletions(-) diff --git a/README.md b/README.md index a637ae5f..2a898526 100644 --- a/README.md +++ b/README.md @@ -7,22 +7,20 @@ Spark-Redis provides access to all of Redis' data structures - String, Hash, Lis Spark-Redis also provides Spark-Streaming support. -## Minimal requirements -You'll need the the following to use Spark-Redis: +## Version compatibility and branching + +The library has several branches, each corresponds to a different supported Spark version. For example, 'branch-2.3' works with any Spark 2.3.x version. +The master branch contains the recent development for the next release. + +| Spark-Redis | Spark | Redis | Supported Scala Versions | +| ----------- | ------------- | ---------------- | ------------------------ | +| 2.3 | 2.3 | >=2.9.0 | 2.11 | +| 1.4 | 1.4 | | 2.10 | - - Apache Spark v1.4.0 - - Scala v2.10.4 - - Jedis v2.7 - - Redis v2.8.12 or v3.0.3 ## Known limitations * Java, Python and R API bindings are not provided at this time -* The package was only tested with the following stack: - - Apache Spark v1.4.0 - - Scala v2.10.4 - - Jedis v2.7 and v2.8 pre-release (see [below](#jedis-and-read-only-redis-cluster-slave-nodes) for details) - - Redis v2.8.12 and v3.0.3 ## Additional considerations This library is work in progress so the API may change before the official release. @@ -35,29 +33,20 @@ cd spark-redis mvn clean package -DskipTests ``` -### Jedis and read-only Redis cluster slave nodes -Jedis' current version - v2.7 - does not support reading from Redis cluster's slave nodes. This functionality will only be included in its upcoming version, v2.8. - -To use Spark-Redis with Redis cluster's slave nodes, the library's source includes a pre-release of Jedis v2.8 under the `with-slaves` branch. Switch to that branch by entering the following before running `mvn clean install`: -``` -git checkout with-slaves -``` - ## Using the library Add Spark-Redis to Spark with the `--jars` command line option. For example, use it from spark-shell, include it in the following manner: ``` -$ bin/spark-shell --jars /spark-redis-.jar,/jedis-.jar +$ bin/spark-shell --jars /spark-redis--jar-with-dependencies.jar Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ - /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 + /___/ .__/\_,_/_/ /_/\_\ version 2.3.1 /_/ -Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79) -... +Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101) ``` The following sections contain code snippets that demonstrate the use of Spark-Redis. To use the sample code, you'll need to replace `your.redis.server` and `6379` with your Redis database's IP address or hostname and port, respectively. From c3b672fc1f89cbcdddfa7d9dacec1e569c63ab17 Mon Sep 17 00:00:00 2001 From: Guy Korland Date: Tue, 14 Aug 2018 09:45:25 +0300 Subject: [PATCH 2/6] Add Maven badge --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 2a898526..437250c1 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,5 @@ [![Build Status](https://travis-ci.org/RedisLabs/spark-redis.svg)](https://travis-ci.org/RedisLabs/spark-redis) +[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis) # Spark-Redis A library for reading and writing data from and to [Redis](http://redis.io) with [Apache Spark](http://spark.apache.org/) From a230da21dbf9ae91ab133f91b0287371efbc8883 Mon Sep 17 00:00:00 2001 From: Guy Korland Date: Tue, 21 Aug 2018 12:51:16 +0300 Subject: [PATCH 3/6] Added Maven example --- README.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/README.md b/README.md index 437250c1..22bcc11b 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,20 @@ The master branch contains the recent development for the next release. This library is work in progress so the API may change before the official release. ## Getting the library + +### Maven + +```xml + + + com.redislabs + spark-redis + 2.3.0 + + +``` + +### Build form source You can download the library's source and build it: ``` git clone https://github.com/RedisLabs/spark-redis.git From 2f04abddd916daaf5d0f572a58f97dcda176590e Mon Sep 17 00:00:00 2001 From: Guy Korland Date: Wed, 22 Aug 2018 18:33:14 +0300 Subject: [PATCH 4/6] add eclipse files to .gitignore --- .gitignore | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index ac9d358f..00dae253 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,6 @@ hs_err_pid*.log nohup.out +scalastyle-output.xml .idea *.iml **/.idea @@ -34,4 +35,6 @@ project/plugins/project/ build/*.jar # eclipse -.project \ No newline at end of file +.project +.classpath +/.settings/ \ No newline at end of file From a44da0af21b560461925f6c2a28ac28fc0663700 Mon Sep 17 00:00:00 2001 From: fe2s Date: Wed, 12 Sep 2018 22:15:49 +0300 Subject: [PATCH 5/6] fix issue #82: IndexOutOfBoundsException when streaming from Redis list --- README.md | 6 ++++-- .../provider/redis/streaming/RedisInputDStream.scala | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 22bcc11b..44c9133d 100644 --- a/README.md +++ b/README.md @@ -266,7 +266,8 @@ import org.apache.spark.storage.StorageLevel import com.redislabs.provider.redis._ val ssc = new StreamingContext(sc, Seconds(1)) val redisStream = ssc.createRedisStream(Array("foo", "bar"), storageLevel = StorageLevel.MEMORY_AND_DISK_2) -redisStream.print +redisStream.print() +ssc.start() ssc.awaitTermination() ``` @@ -279,7 +280,8 @@ import org.apache.spark.storage.StorageLevel import com.redislabs.provider.redis._ val ssc = new StreamingContext(sc, Seconds(1)) val redisStream = ssc.createRedisStreamWithoutListname(Array("foo", "bar"), storageLevel = StorageLevel.MEMORY_AND_DISK_2) -redisStream.print +redisStream.print() +ssc.start() ssc.awaitTermination() ``` diff --git a/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala b/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala index 79a2741a..d57fa1a1 100644 --- a/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala +++ b/src/main/scala/com/redislabs/provider/redis/streaming/RedisInputDStream.scala @@ -50,8 +50,8 @@ private class RedisReceiver[T: ClassTag](keys: Array[String], try { while(!isStopped) { val response = conn.blpop(2, key) - if (response == null) { - + if (response == null || response.isEmpty) { + // no-op } else if (classTag[T] == classTag[String]) { store(response.get(1).asInstanceOf[T]) } else if (classTag[T] == classTag[(String, String)]) { From c7222ea87627e487fe47134a93597c06d39e6996 Mon Sep 17 00:00:00 2001 From: fe2s Date: Thu, 13 Sep 2018 16:15:56 +0300 Subject: [PATCH 6/6] increment spark-redis version to 2.3.1 --- pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pom.xml b/pom.xml index 1500ce47..ba316e69 100644 --- a/pom.xml +++ b/pom.xml @@ -3,7 +3,7 @@ 4.0.0 com.redislabs spark-redis - 2.3.0-SNAPSHOT + 2.3.1-SNAPSHOT Spark-Redis A Spark library for Redis http://github.com/RedisLabs/spark-redis