This Data Processing Library Scala example shows how to use the HERE Data SDK to build a compiler pipeline that elevates and aggregates data to a higher level.
In this example, the DirectMToNCompiler functional pattern is used,
since this pattern covers use cases when we need to do geometric transformations, such as processing neighboring tiles or merging and splitting tiles.
The compiler that we are going to develop is designed to take HERE Map Content input data, merge the incoming lower level tiles into a single tile,
and produce output tiles at a higher HEREtile level. The output of this compiler is written
to lifted-topology-geometry layer in the same Protobuf format with HEREtile partitioning
on a higher level.
For more information about the DirectMToNCompiler functional pattern, see the Data Processing Library Developer Guide.
To run this example, you need two sets of credentials:
- Platform credentials: To get access to the platform data and resources, including HERE Map Content data for your pipeline input.
- Repository credentials: To download HERE Data SDK for Java & Scala libraries and Maven archetypes to your environment.
For more details on how to set up your credentials, see the Identity & Access Management Developer Guide.
For more details on how to verify that your platform credentials are configured correctly, see the Verify Your Credentials tutorial.
In the commands that follow, replace the variable placeholders with the following values:
$CATALOG_IDis your output catalog's ID.$CATALOG_HRNis your output catalog'sHRN(returned by theolp catalog createcommand).$HRN_PARTITIONis the platform environment you are in. The value should behere.$REALMis the ID of your organization, also called a realm. Consult your platform invitation letter to learn your organization ID.$CATALOG_RIBis the HRN of the public HERE Map Content catalog in your pipeline configuration (HERE environment.
Note: We recommend that you set values to variables, so that you can easily copy and execute the following commands.
As mentioned above, we will use the public HERE Map Content input catalog, however, we need to create our own output catalog to store the lifted geometry that the compiler generates.
To run this compiler locally, use a local output catalog as described below. If you want to know more about local catalogs, see the SDK tutorial about local development and testing and the OLP CLI documentation.
- Use the
olp local catalog createcommand to create a local catalog.
olp local catalog create geometry-lifter-scala geometry-lifter-scala --summary "Geometry lifter example catalog" \
--description "Geometry lifter example catalog"The local catalog will have the HRN hrn:local:data:::geometry-lifter-scala.
- Use the
olp local catalog layer addcommand to add twoversionedlayers to your catalog:
| Layer ID | Layer Type | Partitioning | Zoom Level | Content Type | Content Encoding | Schema |
|---|---|---|---|---|---|---|
| lifted-topology-geometry | Versioned | HEREtile | 10 | application/x-protobuf | uncompressed | RIB 2 Topology Geometry |
| state | Versioned | Generic | N.A. | application/octet-stream | uncompressed | - |
olp local catalog layer add hrn:local:data:::geometry-lifter-scala lifted-topology-geometry lifted-topology-geometry --versioned --summary "lifted topology-geometry" \
--description "lifted topology-geometry" --partitioning heretile:10 \
--content-type application/x-protobuf \
--schema hrn:$HRN_PARTITION:schema::$REALM:com.here.schema.rib:topology-geometry_v2:2.12.1
olp local catalog layer add hrn:local:data:::geometry-lifter-scala state state --versioned --summary "state" \
--description "state" --partitioning Generic \
--content-type application/octet-streamTo run the compiler locally, you will need to run the entry point to the compiler:
com.here.platform.data.processing.example.scala.geometry.lifter.Main
As arguments, you must provide the -Dspark.master parameter with the address of the Spark server
master to connect to, and any configuration parameters you want to override. Alternatively, you
can add those parameters to the application.conf file.
Additionally, you also need to specify the -Dpipeline-config.file and -Dpipeline-job.file parameters
to define the location of a configuration file that contains the catalogs as well as job-specific versions
of the catalogs, to read and write to.
For local runs, a bounding box filter is provided in the
config/here/local-application.conf to
limit the number of partitions to be processed. This speeds up the compilation process. In this
example, we use a bounding box around the cities of Berlin. You can edit the bounding box coordinates to compile a different
partition of HERE Map Content. Make sure you update the layer coverage to reflect the different
geographical region. In order to use this configuration file, you need to use the -Dconfig.file
parameter.
Set the environment variable $PATH_TO_CONFIG_FOLDER to ./config/here,
for the HERE platform environment.
Use all the above settings to run the following command in the geometry-lifter
directory to run the Geometry Lifter Compiler.
For the HERE platform environment:
sbt run \
-Dpipeline-config.file=./config/here/local-pipeline-config.conf \
-Dpipeline-job.file=./config/here/pipeline-job.conf \
-Dconfig.file=./config/here/local-application.conf \
-Dspark.master=local[*]After one run, in the HERE platform environment, you can inspect the local catalog with the OLP CLI:
olp local catalog inspect hrn:local:data:::geometry-lifter-scala
You should see the following result:
To follow this example, you will need a project. A project is a collection of platform resources (catalogs, pipelines, and schemas) with controlled access. You can create a project through the HERE platform portal.
Alternatively, use the OLP CLI olp project create command to create the project:
olp project create $PROJECT_ID $PROJECT_NAMEThe command returns the HERE Resource Name (HRN) of your new project. Note down this HRN as you will need it later in this tutorial.
You do not have to provide a
--scopeparameter if your app has a default scope. For details on how to set a default project scope for an app, see the Specify a default Project for Apps chapter of the Identity & Access Management Developer Guide.
For more information on how to work with projects, see the Organize your work in projects tutorial.
The catalog you need to create is used to store the lifted geometry that the compiler generates.
- Use the
olp catalog createcommand to create the catalog. Make sure to note down the HRN returned by the following command for later use:
olp catalog create $CATALOG_ID $CATALOG_ID --summary "Geometry lifter example catalog" \
--description "Geometry lifter example catalog" \
--scope $PROJECT_HRN- Use the
olp catalog layer addcommand to add twoversionedlayers to your catalog:
olp catalog layer add $CATALOG_HRN lifted-topology-geometry lifted-topology-geometry --versioned --summary "lifted topology-geometry" \
--description "lifted topology-geometry" --partitioning heretile:10 \
--content-type application/x-protobuf \
--schema hrn:$HRN_PARTITION:schema::$REALM:com.here.schema.rib:topology-geometry_v2:2.12.1 \
--scope $PROJECT_HRN
olp catalog layer add $CATALOG_HRN state state --versioned --summary "state" \
--description "state" --partitioning Generic \
--content-type application/octet-stream --scope $PROJECT_HRNIf a billing tag is required in your realm, use the
--billing-tags: "YOUR_BILLING_TAG"parameter.
- Use the
olp project resource linkcommand to link the HERE Map Content catalog to your project:
olp project resource link $PROJECT_HRN $CATALOG_RIB- For more details on catalog commands, see Catalog Commands.
- For more details on layer commands, see Layer Commands.
- For more details on project commands, see Project Commands.
- For instructions on how to link a resource to a project, see Project Resource Link command.
From the SDK examples directory, open the data-processing/scala/geometry-lifter project in your
Integrated Development Environment (IDE).
The compiler/config/here/pipeline-config.conf file contains
the permanent configuration of the data sources for the compiler.
Pick the file that corresponds to your platform environment. For example, the pipeline configuration for the HERE platform environment looks like:
pipeline.config {
output-catalog {hrn = "YOUR_OUTPUT_CATALOG_HRN"}
input-catalogs {
rib {hrn = "hrn:here:data::olp-here:rib-2"}
}
}Replace YOUR_OUTPUT_CATALOG_HRN with the HRN of your lifted geometry catalog.
To find the HRN, in the HERE platform portal,
navigate to your catalog. The HRN is displayed in the upper
left corner of the page.
The config/here/pipeline-job.conf file contains the compiler's run configuration.
In this file, modify version = 1 to reflect the version of the HERE Map Content catalog you want
to process. To find the version of the HERE Map Content catalog, in the
HERE platform,
navigate to the HERE Map Content catalog, and view the current catalog's version in the Catalog info section.
The remainder of the configuration is specified in the application.conf file that can be found in the
src/main/resources directory of the compiler project. However, you do not have to modify it unless
you want to change the behavior of the compiler.
Run the sbt assembly command in the geometry-lifter
directory to generate a fat JAR file.
sbt assemblyOnce the previous command is finished, your JAR is then available at the target directory, and you
can upload it using the HERE pipelines UI
or the OLP CLI.
You can use the OLP CLI to create pipeline components and activate the pipeline version with the following commands:
- Create pipeline components:
For this example, a bounding box filter is provided by --runtime-config parameter to
limit the number of partitions to be processed. This speeds up the compilation process. In this
example, we use a bounding box around the cities of Berlin.
You can edit the bounding box coordinates to compile a different
partition of HERE Map Content. Make sure you update the layer coverage to reflect the different
geographical region.
olp pipeline create $COMPONENT_NAME_Pipeline --scope $PROJECT_HRN
olp pipeline template create $COMPONENT_NAME_Template batch-4.0 $PATH_TO_JAR \
com.here.platform.data.processing.example.scala.geometry.lifter.Main \
--workers=4 --worker-units=3 --supervisor-units=2 --input-catalog-ids=rib \
--scope $PROJECT_HRN
olp pipeline version create $COMPONENT_NAME_version $PIPELINE_ID $PIPELINE_TEMPLATE_ID \
"$PATH_TO_CONFIG_FOLDER/pipeline-config.conf" \
--runtime-config here.platform.data-processing.executors.partitionKeyFilters.0.className=BoundingBoxFilter \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.north=52.67551 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.south=52.338261 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.east=13.76116 \
here.platform.data-processing.executors.partitionKeyFilters.0.param.boundingBox.west=13.08835 \
--scope $PROJECT_HRN- Activate the pipeline version:
olp pipeline version activate $PIPELINE_ID $PIPELINE_VERSION_ID \
--input-catalogs "$PATH_TO_CONFIG_FOLDER/pipeline-job.conf" \
--scope $PROJECT_HRNYou do not have to specify the input catalog's version, unless you want to. The latest version will be automatically used.
In the HERE platform portal, navigate to your pipeline to see its status.
In the HERE platform portal, select the Data tab and find your catalog.
- Open the
lifted-topology-geometrylayer and select the Inspect tab. - On the map, navigate to the location of your bounding box and set the zoom to level 10.
- Finally, select any highlighted partition to view the results displayed on the map.
The results should be visible on the map.