Skip to content

Commit

Permalink
better installation instructions
Browse files Browse the repository at this point in the history
  • Loading branch information
raganhan committed Dec 20, 2018
1 parent d8b9811 commit 54e5273
Showing 1 changed file with 28 additions and 26 deletions.
54 changes: 28 additions & 26 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,49 +4,52 @@ A Apache Hive SerDe (short for serializer/deserializer) for the Ion file format.

### Features
* Read data stored in Ion format both binary and text.
* Supports all Ion types including nested data structures, see [Type mapping documentation](docs/type-mapping.md)
* Supports all Ion types including nested data structures, see [Type mapping documentation](docs/type-mapping.md)
for more information.
* Supports flattening of Ion documents through [path extraction](https://github.com/amzn/ion-java-path-extraction).
* Supports importing shared symbol tables and custom symbol table catalogs.
* `IonInputFormat` and `IonOutputFormat` are able to handle both Ion binary and Ion text.
* Configurable through [SerDe properties](docs/serde-properties.md).

### Installation
Download the latest binaries from maven central and place the JARs into `hive/lib` or use `ADD JAR` in Hive.
Download the latest `ion-hive-serde-all-<version-number>.jar` from [https://github.com/amzn/ion-hive-serde/releases]
and place the JARs into `hive/lib` or use `ADD JAR` in Hive. That jar contains the SerDe and all its dependencies.

To build it locally run :`./gradlew :serde:singleJar`

### Building
Project is separated into two modules:
Project is separated into two modules:
1. `serde`: with the SerDe code and unit tests.
1. `integration-tests`: integration tests using a dockerized hive installation.
1. `integration-tests`: integration tests using a dockerized hive installation.

To build only the SerDe code:
To build only the SerDe code:
```
./gradlew :serde:build
./gradlew :serde:build
```

To build the SerDe including integration tests:
To build the SerDe including integration tests:
```
./gradlew build
./gradlew build
```

Integration tests require docker to be installed, but the build itself will take care of creating the necessary
containers, starting and stopping them. See [integration-tests/README.md](integration-test/README.md) for more
Integration tests require docker to be installed, but the build itself will take care of creating the necessary
containers, starting and stopping them. See [integration-tests/README.md](integration-test/README.md) for more
information, including how to run the integration tests on your IDE.

### Examples
Examples shown using Ion text for readability but for better performance and compression Ion binary is recommended in
production systems.
Examples shown using Ion text for readability but for better performance and compression Ion binary is recommended in
production systems.

#### Simple query
```
~$ cat test.ion
{
name: "foo",
{
name: "foo",
age: 32
}
{
name: "bar",
{
name: "bar",
age: 28
}
Expand All @@ -73,17 +76,17 @@ foo 32
bar 28
```

#### Flattening
#### Flattening
```
~$ cat test.ion
{
{
personal_info: { name: "foo", age: 32 }
professional_info: { job_title: "software engineer" }
professional_info: { job_title: "software engineer" }
}
{
{
personal_info: { name: "bar", age: 28 }
professional_info: { job_title: "designer" }
professional_info: { job_title: "designer" }
}
Expand All @@ -102,7 +105,7 @@ hive> CREATE EXTERNAL TABLE test (
WITH SERDEPROPERTIES (
"ion.name.path_extractor" = "(personal_info name)",
"ion.age.path_extractor" = "(personal_info age)",
"ion.jobtitle.path_extractor" = "(professional_info job_title)",
"ion.jobtitle.path_extractor" = "(professional_info job_title)",
)
STORED AS
INPUTFORMAT 'software.amazon.ionhiveserde.formats.IonInputFormat'
Expand All @@ -114,12 +117,11 @@ OK
foo 32 software engineer
bar 28 designer
```
```

## Contributing
See [CONTRIBUTING](CONTRIBUTING.md)
See [CONTRIBUTING](CONTRIBUTING.md)

## License

This library is licensed under the Apache 2.0 License.

This library is licensed under the Apache 2.0 License.

0 comments on commit 54e5273

Please sign in to comment.