Skip to content

Commit

Permalink
Merge pull request #88 from fe2s/branch-2.3
Browse files Browse the repository at this point in the history
preparing for release 2.3.1
  • Loading branch information
gkorland authored Oct 10, 2018
2 parents 0cd0d9b + ba49e92 commit 0f81ca3
Show file tree
Hide file tree
Showing 4 changed files with 38 additions and 29 deletions.
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
hs_err_pid*.log
nohup.out
scalastyle-output.xml
.idea
*.iml
**/.idea
Expand Down Expand Up @@ -34,4 +35,6 @@ project/plugins/project/
build/*.jar

# eclipse
.project
.project
.classpath
/.settings/
56 changes: 31 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
[![Build Status](https://travis-ci.org/RedisLabs/spark-redis.svg)](https://travis-ci.org/RedisLabs/spark-redis)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis/badge.svg)](https://maven-badges.herokuapp.com/maven-central/com.redislabs/spark-redis)

# Spark-Redis
A library for reading and writing data from and to [Redis](http://redis.io) with [Apache Spark](http://spark.apache.org/)
Expand All @@ -7,57 +8,60 @@ Spark-Redis provides access to all of Redis' data structures - String, Hash, Lis

Spark-Redis also provides Spark-Streaming support.

## Minimal requirements
You'll need the the following to use Spark-Redis:
## Version compatibility and branching

The library has several branches, each corresponds to a different supported Spark version. For example, 'branch-2.3' works with any Spark 2.3.x version.
The master branch contains the recent development for the next release.

| Spark-Redis | Spark | Redis | Supported Scala Versions |
| ----------- | ------------- | ---------------- | ------------------------ |
| 2.3 | 2.3 | >=2.9.0 | 2.11 |
| 1.4 | 1.4 | | 2.10 |

- Apache Spark v1.4.0
- Scala v2.10.4
- Jedis v2.7
- Redis v2.8.12 or v3.0.3

## Known limitations

* Java, Python and R API bindings are not provided at this time
* The package was only tested with the following stack:
- Apache Spark v1.4.0
- Scala v2.10.4
- Jedis v2.7 and v2.8 pre-release (see [below](#jedis-and-read-only-redis-cluster-slave-nodes) for details)
- Redis v2.8.12 and v3.0.3

## Additional considerations
This library is work in progress so the API may change before the official release.

## Getting the library

### Maven

```xml
<dependencies>
<dependency>
<groupId>com.redislabs</groupId>
<artifactId>spark-redis</artifactId>
<version>2.3.0</version>
</dependency>
</dependencies>
```

### Build form source
You can download the library's source and build it:
```
git clone https://github.com/RedisLabs/spark-redis.git
cd spark-redis
mvn clean package -DskipTests
```

### Jedis and read-only Redis cluster slave nodes
Jedis' current version - v2.7 - does not support reading from Redis cluster's slave nodes. This functionality will only be included in its upcoming version, v2.8.

To use Spark-Redis with Redis cluster's slave nodes, the library's source includes a pre-release of Jedis v2.8 under the `with-slaves` branch. Switch to that branch by entering the following before running `mvn clean install`:
```
git checkout with-slaves
```

## Using the library
Add Spark-Redis to Spark with the `--jars` command line option. For example, use it from spark-shell, include it in the following manner:

```
$ bin/spark-shell --jars <path-to>/spark-redis-<version>.jar,<path-to>/jedis-<version>.jar
$ bin/spark-shell --jars <path-to>/spark-redis-<version>-jar-with-dependencies.jar
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.4.0
/___/ .__/\_,_/_/ /_/\_\ version 2.3.1
/_/
Using Scala version 2.10.4 (OpenJDK 64-Bit Server VM, Java 1.7.0_79)
...
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_101)
```

The following sections contain code snippets that demonstrate the use of Spark-Redis. To use the sample code, you'll need to replace `your.redis.server` and `6379` with your Redis database's IP address or hostname and port, respectively.
Expand Down Expand Up @@ -262,7 +266,8 @@ import org.apache.spark.storage.StorageLevel
import com.redislabs.provider.redis._
val ssc = new StreamingContext(sc, Seconds(1))
val redisStream = ssc.createRedisStream(Array("foo", "bar"), storageLevel = StorageLevel.MEMORY_AND_DISK_2)
redisStream.print
redisStream.print()
ssc.start()
ssc.awaitTermination()
```

Expand All @@ -275,7 +280,8 @@ import org.apache.spark.storage.StorageLevel
import com.redislabs.provider.redis._
val ssc = new StreamingContext(sc, Seconds(1))
val redisStream = ssc.createRedisStreamWithoutListname(Array("foo", "bar"), storageLevel = StorageLevel.MEMORY_AND_DISK_2)
redisStream.print
redisStream.print()
ssc.start()
ssc.awaitTermination()
```

Expand Down
2 changes: 1 addition & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.redislabs</groupId>
<artifactId>spark-redis</artifactId>
<version>2.3.0-SNAPSHOT</version>
<version>2.3.1-SNAPSHOT</version>
<name>Spark-Redis</name>
<description>A Spark library for Redis</description>
<url>http://github.com/RedisLabs/spark-redis</url>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ private class RedisReceiver[T: ClassTag](keys: Array[String],
try {
while(!isStopped) {
val response = conn.blpop(2, key)
if (response == null) {

if (response == null || response.isEmpty) {
// no-op
} else if (classTag[T] == classTag[String]) {
store(response.get(1).asInstanceOf[T])
} else if (classTag[T] == classTag[(String, String)]) {
Expand Down

0 comments on commit 0f81ca3

Please sign in to comment.