Skip to content

Commit

Permalink
feat: add cool shell script
Browse files Browse the repository at this point in the history
  • Loading branch information
tinyAdapter committed Aug 13, 2024
1 parent da9a1d8 commit edfe393
Show file tree
Hide file tree
Showing 2 changed files with 82 additions and 28 deletions.
45 changes: 17 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ mvn clean package

### Required sources

1. **dataset file**: a CSV file with "," delimiter (normally dumped from a database table), and the table header removed.
1. **dataset file**: a CSV file with "," delimiter (normally dumped from a database table) and the table header removed.
2. **dataset schema file**: a `table.yaml` file specifying the dataset's columns and their measure fields.
3. **query file**: a YAML file specifying the parameters for the running query server.

Expand All @@ -46,8 +46,7 @@ mvn clean package
Before query processing, we need to load the dataset into COOL native format. The sample code to load csv dataset with data loader can be found in [CsvLoader.java](cool-core/src/main/java/com/nus/cool/functionality/CsvLoader.java).

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CsvLoader \
./cool load \
dataset \
path/to/your/.yaml \
path/to/your/datafile \
Expand All @@ -56,7 +55,7 @@ java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \

The five arguments in the command have the following meaning:

1. a unique dataset name given under the directory
1. the dataset name
2. the `table.yaml` (the third required source)
3. the dataset file (the first required source)
4. the output directory for the compacted dataset
Expand All @@ -70,35 +69,31 @@ We provide an example for cohort query processing in [CohortAnalysis.java](cool-
#### Cohort Selection

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CohortSelection \
./cool cohortselection \
path/to/output/datasource/directory \
path/to/your/queryfile
```

#### Cohort Query

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CohortAnalysis \
./cool cohortquery \
path/to/output/datasource/directory \
path/to/your/cohortqueryfile
```

#### Funnel Query

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.FunnelAnalysis \
./cool funnelquery \
path/to/output/datasource/directory \
path/to/your/funnelqueryfile
```

#### OLAP Query

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.IcebergLoader \
./cool olapquery \
path/to/output/datasource/directory \
path/to/your/queryfile
```
Expand All @@ -112,8 +107,7 @@ We have provided examples in `sogamo` directory and `health_raw` directory. Now
The COOL system supports CSV data format by default, and you can load `sogamo` dataset with the following command.

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CsvLoader \
./cool load \
sogamo \
datasets/sogamo/table.yaml \
datasets/sogamo/data.csv \
Expand Down Expand Up @@ -161,8 +155,7 @@ There will be a cube generated under the `./CubeRepo` directory, which is named
Similarly, load the `health_raw` dataset with:

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CsvLoader \
./cool load \
health_raw \
datasets/health_raw/table.yaml \
datasets/health_raw/data.csv \
Expand All @@ -176,24 +169,23 @@ We use the `health_raw` dataset for example to demonstrate the cohort analysis.
#### Select the specific users

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CohortSelection \
./cool cohortselection \
./CubeRepo \
datasets/health_raw/sample_query_selection/query.json
```

where the arguments are:

1. `./CubeRepo`: the output directory for the compacted dataset
2. `datasets/health_raw/sample_query_selection/query.json`: the JSON file for the cohort query
2. `datasets/health_raw/sample_query_selection/query.json`: the cohort query (in JSON)

<!--
- Display all selected records of the cohort in the terminal for exploration
```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CohortExploration \
CubeRepo \
./CubeRepo \
health_raw \
sample_query_selection
```
Expand All @@ -202,8 +194,7 @@ java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
#### Execute cohort query

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CohortAnalysis \
./cool cohortquery \
./CubeRepo \
datasets/health_raw/sample_query_average/query.json
```
Expand All @@ -213,8 +204,7 @@ java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
We use the `sogamo` dataset for example to demonstrate the funnel analysis.

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.FunnelAnalysis \
./cool funnelquery \
./CubeRepo \
datasets/sogamo/sample_funnel_analysis/query.json
```
Expand All @@ -228,8 +218,7 @@ We have provided examples in `olap-tpch` directory.
The COOL system supports CSV data format by default, and you can load `tpc-h` dataset with the following command.

```bash
java -cp ./cool-core/target/cool-core-0.1-SNAPSHOT.jar \
com.nus.cool.functionality.CsvLoader \
./cool load \
tpc-h-10g \
datasets/olap-tpch/table.yaml \
datasets/olap-tpch/scripts/data.csv \
Expand All @@ -244,10 +233,10 @@ Run Server

1. put the `application.property` file at the same level as the .jar file.
2. edit the server configuration in the `application.property` file.
3. run the below command line.
3. run the below command.

```bash
java -jar cool-queryserver/target/cool-queryserver-0.1-SNAPSHOT.jar
./cool server
```

## CONNECT TO EXTERNAL STORAGE SERVICES
Expand Down
65 changes: 65 additions & 0 deletions cool
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
#!/usr/bin/env bash

COOL_CORE_PATH="${COOL_CORE_JAR_PATH:-./cool-core/target/cool-core-0.1-SNAPSHOT.jar}"
COOL_QUERY_SERVER_PATH="${COOL_QUERY_SERVER_PATH:-./cool-queryserver/target/cool-queryserver-0.1-SNAPSHOT.jar}"


main_help() {
COOL_HELP_LEFT_ALIGN="%-17s"

echo "Usage: $0 <command> [<args>]"
echo
echo Commands
printf " $COOL_HELP_LEFT_ALIGN %s\n" "help" "Show this help menu"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "load" "Load dataset"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "cohortselection" "Perform cohort selection"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "cohortquery" "Perform cohort query"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "funnelquery" "Perform funnel query"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "olapquery" "Perform OLAP query"
printf " $COOL_HELP_LEFT_ALIGN %s\n" "server" "Start query server"

exit ${1:-0}
}

main_load() {
java -cp $COOL_CORE_PATH com.nus.cool.functionality.CsvLoader "$@"
}

main_cohortselection() {
java -cp $COOL_CORE_PATH com.nus.cool.functionality.CohortSelection "$@"
}

main_cohortquery() {
java -cp $COOL_CORE_PATH com.nus.cool.functionality.CohortAnalysis "$@"
}

main_funnelquery() {
java -cp $COOL_CORE_PATH com.nus.cool.functionality.FunnelAnalysis "$@"
}

main_olapquery() {
java -cp $COOL_CORE_PATH com.nus.cool.functionality.IcebergLoader "$@"
}

main_server() {
java -jar $COOL_QUERY_SERVER_PATH
}

function main() {
if (($# == 0)); then
main_help 0
fi

case ${1} in
help | load | cohortselection | cohortquery | funnelquery | olapquery | server)
"main_$1" "${@:2}"
;;
*)
echo "unknown command: $1"
main_help 1
exit 1
;;
esac
}

main "$@"

0 comments on commit edfe393

Please sign in to comment.