From 0f06d3cc0c7b570b0084619f02335c92cbf40a97 Mon Sep 17 00:00:00 2001 From: "Deployment Bot (from Travis CI)" Date: Tue, 16 Jan 2024 19:13:44 +0000 Subject: [PATCH] Deploy kalisio/krawler to github.com/kalisio/krawler.git:gh-pages --- 404.html | 2 +- about/contact.html | 2 +- about/contributing.html | 2 +- about/introduction.html | 2 +- about/license.html | 2 +- about/roadmap.html | 2 +- ..._hooks.md.2f6bf6ef.js => reference_hooks.md.3cc0f923.js} | 4 ++-- ...2f6bf6ef.lean.js => reference_hooks.md.3cc0f923.lean.js} | 2 +- examples/index.html | 2 +- guides/extending-krawler.html | 2 +- guides/index.html | 2 +- guides/installing-krawler.html | 2 +- guides/understanding-krawler.html | 2 +- guides/using-krawler.html | 2 +- hashmap.json | 2 +- index.html | 2 +- reference/hooks.html | 6 +++--- reference/index.html | 2 +- reference/known-issues.html | 2 +- reference/services.html | 2 +- 20 files changed, 23 insertions(+), 23 deletions(-) rename assets/{reference_hooks.md.2f6bf6ef.js => reference_hooks.md.3cc0f923.js} (78%) rename assets/{reference_hooks.md.2f6bf6ef.lean.js => reference_hooks.md.3cc0f923.lean.js} (83%) diff --git a/404.html b/404.html index 4f1b19a3..fa125f2e 100644 --- a/404.html +++ b/404.html @@ -16,7 +16,7 @@
Krawler

Appearance

404

PAGE NOT FOUND

But if you don't change your direction, and if you keep looking, you may end up where you are heading.
Take me home

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/about/contact.html b/about/contact.html index dc1282fd..fad6ff3d 100644 --- a/about/contact.html +++ b/about/contact.html @@ -19,7 +19,7 @@
Krawler

Appearance

Contact ​

Please feel free to join our slack channel using the invitation link.

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/about/contributing.html b/about/contributing.html index 45310273..d9806f90 100644 --- a/about/contributing.html +++ b/about/contributing.html @@ -19,7 +19,7 @@
Krawler

Appearance

Contributing ​

Submission guidelines ​

Report a bug ​

Before creating an issue please make sure you have checked out the docs, you might want to also try searching Github. It's pretty likely someone has already asked a similar question.

Issues can be reported in the issue tracker.

Pull Requests ​

We love pull requests and we're continually working to make it as easy as possible for people to contribute.

We prefer small pull requests with minimal code changes. The smaller they are the easier they are to review and merge. A core team member will pick up your PR and review it as soon as they can. They may ask for changes or reject your pull request. This is not a reflection of you as an engineer or a person. Please accept feedback graciously as we will also try to be sensitive when providing it.

Although we generally accept many PRs they can be rejected for many reasons. We will be as transparent as possible but it may simply be that you do not have the same context or information regarding the roadmap that the core team members have. We value the time you take to put together any contributions so we pledge to always be respectful of that time and will try to be as open as possible so that you don't waste it.

Commit message guidelines ​

We follow the Conventional commits specifications which provides a set of rules to make commit messages more readable when looking through the project history. But also, we use the git commit messages to generate the change log.

Commit message format ​

The commit message should be structured as follows:

<type>: <subject> [optional `breaking`]
<type>: <subject> [optional `breaking`]

Where type must be one of the following:

  • build: changes that affect the build system (external dependencies)
  • ci: changes to our CI configuration files and scripts
  • chore: changes that affect the project structure
  • docs: changes that affect the documentation only
  • feat: a new feature
  • fix: a bug fix
  • perf: a code change that improves performance
  • refactor: a code change that neither fixes a bug nor adds a feature
  • revert: revert changes
  • style: changes that do not affect the meaning of the code (lint issues)
  • test: adding missing tests or correcting existing tests

Use the optional [ breaking ] keyword to declare a BREAKING CHANGE.

Examples ​

  • Commit message with description and breaking change in body
feat: allow provided config object to extend other configs [ breaking ]
feat: allow provided config object to extend other configs [ breaking ]
  • Commit message with no body
docs: correct spelling in the contributing.md file
docs: correct spelling in the contributing.md file
  • Commit message for a fix using an issue number.
fix: fix minor issue in code (#12)
fix: fix minor issue in code (#12)

Versioning guidelines ​

We rely on Semantic Versioning for versioning a release. Indeed, given a version number MAJOR.MINOR.PATCH, increment the:

  • MAJOR version when you make a major evolution leading to breaking changes,
  • MINOR version when you add functionality in a backwards-compatible manner
  • PATCH version when you make backwards-compatible bug fixes.

The command npm run release:<type>, where <type> is either patch, minor or major, helps you to do the release.

It performs the following task for you:

  • increase the package version number in the package.json file
  • generate the change log
  • create a tag accordingly in the git repository and push it

Contributor Code of Conduct ​

As contributors and maintainers of this project, we pledge to respect all people who contribute through reporting issues, posting feature requests, updating documentation, submitting pull requests or patches, and other activities.

We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, or religion.

Examples of unacceptable behavior by participants include the use of sexual language or imagery, derogatory comments or personal attacks, trolling, public or private harassment, insults, or other unprofessional conduct.

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct. Project maintainers who do not follow the Code of Conduct may be removed from the project team.

Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by opening an issue or contacting one or more of the project maintainers.

This Code of Conduct is adapted from the Contributor Covenant, version 1.0.0, available at http://contributor-covenant.org/version/1/0/0/

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/about/introduction.html b/about/introduction.html index 602ef4bd..09c29811 100644 --- a/about/introduction.html +++ b/about/introduction.html @@ -19,7 +19,7 @@
Krawler

Appearance

Introduction ​

Krawler aims at making the automated process of extracting and processing (geographic) data from heterogeneous sources easy. It can be viewed as a minimalist Extract, Transform, Load (ETL). ETL refers to a process where data is

  1. extracted from heterogeneous data sources (e.g. databases or web services);
  2. transformed in a target format or structure for the purposes of querying and analysis (e.g. JSON or CSV);
  3. loaded into a final target data store (e.g. a file system or a database).

ETL

ETL naturally leads to the concept of a pipeline: a set of processing functions (called hooks in krawler) connected in series, often executed in parallel, where the output of one function is the input of the next one. The execution of a given pipeline on an input dataset to produce the associated output is a job performed by krawler.

A set of introduction articles to krawler have been written and detail:

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/about/license.html b/about/license.html index c675fab0..4fc4fad3 100644 --- a/about/license.html +++ b/about/license.html @@ -19,7 +19,7 @@
Krawler

Appearance

License ​

The MIT License (MIT)

Copyright (c) 2017-2020 Kalisio

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/about/roadmap.html b/about/roadmap.html index d62da933..dd8e08ea 100644 --- a/about/roadmap.html +++ b/about/roadmap.html @@ -19,7 +19,7 @@
Krawler

Appearance

Roadmap ​

The roadmap is available on Github.

Milestones ​

The milestones are available on Github

Release Notes ​

The changelog is available in Github

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/assets/reference_hooks.md.2f6bf6ef.js b/assets/reference_hooks.md.3cc0f923.js similarity index 78% rename from assets/reference_hooks.md.2f6bf6ef.js rename to assets/reference_hooks.md.3cc0f923.js index a7046883..d17fa709 100644 --- a/assets/reference_hooks.md.2f6bf6ef.js +++ b/assets/reference_hooks.md.3cc0f923.js @@ -106,7 +106,7 @@ import{_ as o,o as e,c as t,Q as s}from"./chunks/framework.e713b733.js";const m= }, pick: ['onlyThisPropertyWillBeKept'], omit: ['onlyThisPropertyWillBeRemoved'], -merge: { newProperty: 'will be added to the final objects' }

TIP

The transformations are applied in the order of the documentation, e.g. filtering occurs before mapping.

mergeJson(options) ​

Generate a in-memory JSON object from a set of input in-memory JSON objects, hook options are the following:

The input hook result is expected to be an array of tasks which output will be read in-memory.

writeTemplate(options) ​

Generate a file from an input template and injected in-memory JSON values, hook options are the following:

TIP

Learn more about templating

GeoJSON ​

source

convertToGeoJson(options) ​

Convert in-memory JSON values to a GeoJSON collection. For each in-memory object, the hook generates a corresponding GeoJSON feature using specific properties to build the geometry property. For now, it only allows to generate features of type of Point. Moreover, the coordinate reference system is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84), with longitude and latitude expressed in decimal degrees. The entire object is stored under the properties property of the feature. Hook options are the following:

convertOSMToGeoJson(options) ​

Convert in-memory OSM JSON values to a GeoJSON collection. It relies on osmtogeojson. Hook options are the following:

reprojectGeoJson(options) ​

Reproject a GeoJSON from a given projection system to another one, hook options are the following:

KML ​

source

readKML(options) ​

Read a KML from an input stream/store and convert it to in-memory JSON values, hook options are the following:

MongoDB ​

source

connectMongo(options) ​

Connect to a MongoDB database. The connection options of the client are defined in the hook options plus:

TIP

Since Krawler relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option useNewUrlParser: true when connecting to the database.

disconnectMongo(options) ​

Disconnect from a MongoDB database. Hook options are the following:

dropMongoCollection(options) ​

Drop if exists a collection in a MongoDB database. Hook options are the following:

createMongoCollection(options) ​

Create a collection in a MongoDB database. Hook options are the following:

readMongoCollection(options) ​

Read JSON documents from an existing collection. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

updateMongoCollection(options) ​

Updates JSON into an existing collection (uses updateOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be updated into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

deleteMongoCollection(options) ​

Removes documents from an existing collection (uses deleteMany operations under-the-hood). Hook options are the following:

createMongoAggregation(options) ​

Creates an aggregation pipeline on an existing collection. Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

dropMongoBucket(options) ​

Drop if exists a bucket in a MongoDB database. Hook options are the following:

createMongoBucket(options) ​

Create a bucket in a MongoDB database. Hook options are the following:

readMongoBucket(options) ​

Read file from an existing bucket. Hook options are the following:

writeMongoBucket(options) ​

Insert file into an existing bucket. Hook options are the following:

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections

Feathers ​

source

connectFeathers(options) ​

Connect to a Feathers API. The connection options of the client are defined in the hook options plus:

TIP

Krawler uses on the version 5 of the Feathers client.

disconnectFeathers(options) ​

Disconnect from a Feathers API. Hook options are the following:

callFeathersServiceMethod(options) ​

Performs a service operation using the API. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

Numerical Weather Prediction ​

source

Numerical Weather Prediction (NWP) data are now available from the major meteorological agencies and institutions on a day-to-day basis. These hooks aim at gathering weather forecast data generated by forecast models easily.

Each forecast model output hundreds of forecast elements (a.k.a. meteorological elements) such as temperature, wind direction, etc. The production of a set of forecast data is called a run of the model and occurs on a regular daily basis, e.g. every 6 hours. The spatial properties of a model are completely defined by a longitude/latitude grid and a set of altitude levels (meter or pressure scale). The temporal properties are defined by interval values describing at which frequency/time the forecast data are produced (a.k.a. run interval) and which time steps are available (a.k.a. forecast interval).

generateNwpTasks(options) ​

Generate tasks to download data for each variable. It is intended to be used a job hook and the required hook options (can be overriden by input data) are the following:

A task will be generated for each element, level and gathered forecast time with the following properties: level, runTime, forecastTime, timeOffset.

TIP

This hook is intended to work with task templating to generate the actual download tasks (e.g. HTTP or WCS request)

OGC ​

source

getCapabilities(options) ​

Execute a GetCapabilties request to get the general information about an OGC service such as WMS, WCS, WPS... Hook options are the following:

The following example illustrates how to use this hook:

getCapabilities: {
+merge: { newProperty: 'will be added to the final objects' }

TIP

The transformations are applied in the order of the documentation, e.g. filtering occurs before mapping.

mergeJson(options) ​

Generate a in-memory JSON object from a set of input in-memory JSON objects, hook options are the following:

The input hook result is expected to be an array of tasks which output will be read in-memory.

writeTemplate(options) ​

Generate a file from an input template and injected in-memory JSON values, hook options are the following:

TIP

Learn more about templating

GeoJSON ​

source

convertToGeoJson(options) ​

Convert in-memory JSON values to a GeoJSON collection. For each in-memory object, the hook generates a corresponding GeoJSON feature using specific properties to build the geometry property. For now, it only allows to generate features of type of Point. Moreover, the coordinate reference system is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84), with longitude and latitude expressed in decimal degrees. The entire object is stored under the properties property of the feature. Hook options are the following:

convertOSMToGeoJson(options) ​

Convert in-memory OSM JSON values to a GeoJSON collection. It relies on osmtogeojson. Hook options are the following:

reprojectGeoJson(options) ​

Reproject a GeoJSON from a given projection system to another one, hook options are the following:

KML ​

source

readKML(options) ​

Read a KML from an input stream/store and convert it to in-memory JSON values, hook options are the following:

MongoDB ​

source

connectMongo(options) ​

Connect to a MongoDB database. The connection options of the client are defined in the hook options plus:

TIP

Since Krawler relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option useNewUrlParser: true when connecting to the database.

disconnectMongo(options) ​

Disconnect from a MongoDB database. Hook options are the following:

dropMongoCollection(options) ​

Drop if exists a collection in a MongoDB database. Hook options are the following:

createMongoCollection(options) ​

Create a collection in a MongoDB database. Hook options are the following:

dropMongoIndex(options) ​

Drop if exists a collection index in a MongoDB database. Hook options are the following:

createMongoIndex(options) ​

Create a collection index in a MongoDB database. Hook options are the following:

readMongoCollection(options) ​

Read JSON documents from an existing collection. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

updateMongoCollection(options) ​

Updates JSON into an existing collection (uses updateOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be updated into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

deleteMongoCollection(options) ​

Removes documents from an existing collection (uses deleteMany operations under-the-hood). Hook options are the following:

createMongoAggregation(options) ​

Creates an aggregation pipeline on an existing collection. Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

dropMongoBucket(options) ​

Drop if exists a bucket in a MongoDB database. Hook options are the following:

createMongoBucket(options) ​

Create a bucket in a MongoDB database. Hook options are the following:

readMongoBucket(options) ​

Read file from an existing bucket. Hook options are the following:

writeMongoBucket(options) ​

Insert file into an existing bucket. Hook options are the following:

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections

Feathers ​

source

connectFeathers(options) ​

Connect to a Feathers API. The connection options of the client are defined in the hook options plus:

TIP

Krawler uses on the version 5 of the Feathers client.

disconnectFeathers(options) ​

Disconnect from a Feathers API. Hook options are the following:

callFeathersServiceMethod(options) ​

Performs a service operation using the API. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

Numerical Weather Prediction ​

source

Numerical Weather Prediction (NWP) data are now available from the major meteorological agencies and institutions on a day-to-day basis. These hooks aim at gathering weather forecast data generated by forecast models easily.

Each forecast model output hundreds of forecast elements (a.k.a. meteorological elements) such as temperature, wind direction, etc. The production of a set of forecast data is called a run of the model and occurs on a regular daily basis, e.g. every 6 hours. The spatial properties of a model are completely defined by a longitude/latitude grid and a set of altitude levels (meter or pressure scale). The temporal properties are defined by interval values describing at which frequency/time the forecast data are produced (a.k.a. run interval) and which time steps are available (a.k.a. forecast interval).

generateNwpTasks(options) ​

Generate tasks to download data for each variable. It is intended to be used a job hook and the required hook options (can be overriden by input data) are the following:

A task will be generated for each element, level and gathered forecast time with the following properties: level, runTime, forecastTime, timeOffset.

TIP

This hook is intended to work with task templating to generate the actual download tasks (e.g. HTTP or WCS request)

OGC ​

source

getCapabilities(options) ​

Execute a GetCapabilties request to get the general information about an OGC service such as WMS, WCS, WPS... Hook options are the following:

The following example illustrates how to use this hook:

getCapabilities: {
   url: 'http://geoserver.kalisio.xyz/geoserver/Kalisio/wms',
   service: 'WMS'
 }
getCapabilities: {
@@ -126,4 +126,4 @@ import{_ as o,o as e,c as t,Q as s}from"./chunks/framework.e713b733.js";const m=
   database: 'test',
   port: 5432,
   clientPath: 'taskTemplate.client'
-}

disconnectPG(options) ​

Disconnect from a PostgresSQL database. Hook options are the following:

dropPGTable(options) ​

Drop if exists a table in a PostgreSQL database. Hook options are the following:

createPGTable(options) ​

Create a table in a PostgreSQL database with the following structure:

For now the structure has been defined to store GeoJSON collection. Hook options are the following:

writePGTable(options) ​

Inserts a GeoJSON collection or an array of features into an existing table. THe table must have the same structured as a table created using the createPGTable hook. Hook options are the following:

Raster ​

source

readGeoTiff(options) ​

Read a GeoTiff from an input stream/store and convert it to in-memory JSON values, hook options are the following:

computeStatistics(options) ​

Computes minimum and maximum values on a GeoTiff file, hook options are the following:

Store ​

source

createStores(options) ​

Create (a set of) store(s), hook options are the (array of) following the following:

removeStores(options) ​

Remove (a set of) store(s), hook options are (array of) the following:

TIP

As a shortcut the options provided can only be store IDs when storePath is not used

discardIfExistsInStore(options) ​

Discard the task if a target file already exists in an output store, hook options are the following:

copyToStore(options) ​

Copy the item(s) from an input store to an output store, hook options are the following:

gzipToStore(options) ​

Gzip the item(s) from an input store to an output store, hook options are the following:

gunzipFromStore(options) ​

Gunzip the item(s) from an input store to an output store, hook options are the following:

unzipFromStore(options) ​

Unzip the item(s) from an input store to an output store, hook options are the following:

System ​

source

tar(options) ​

Tar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

untar(options) ​

Untar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

runCommand(options) ​

Run a system command. Hook options are the following:

TIP

Learn more about templating

envsubst(options) ​

Provides file-level environment variable substitution. Hook options are the following:

TXT ​

source

readTXT(options) ​

Read a TXT from an input stream/store and convert it to in-memory JSON values, hook options are the following:

Utils ​

source

generateId(options) ​

Generate a UUID (V1) for the item using node-uuid.

template(options) ​

Perform templating of the options using the item as context and merge it with item.

discardIf(options) ​

Discard all subsequent hooks and task if the input data passes the given match filter options, filter options are similar to the match filter described in common options.

apply(options) ​

Apply a given function to the hook item(s), hook options are the following:

healthcheck(options) ​

Apply a given function to the hook item(s) and healthcheck structure, hook options are the following:

addOutputs(outputs) ​

Declare a new output for the job/task, hook options is an array of objects with the following properties:

Tasks and write hooks automatically track generated outputs but sometimes outputs are generated by an external process (eg. command hook) so that you need to declare it in order to properly clean it with the clearOutputs hook.

runTask(options) ​

Run a given task, hook options are those of a task.

emitEvent(options) ​

Emit a 'krawler' event on the underlying service, hook options are the following:

XML ​

source

readXML(options) ​

Read an XML file from a store and convert it to in-memory JSON values, hook options are the following:

YAML ​

source

readYAML(options) ​

Read a YAML file from a store and convert it to in-memory JSON values, hook options are the following:

writeYAML(options) ​

Generate a YAML file from in-memory JSON values, hook options are the following:

`,334),r=[n];function l(i,p,c,h,d,u){return e(),t("div",null,r)}const f=o(a,[["render",l]]);export{m as __pageData,f as default}; +}

disconnectPG(options) ​

Disconnect from a PostgresSQL database. Hook options are the following:

dropPGTable(options) ​

Drop if exists a table in a PostgreSQL database. Hook options are the following:

createPGTable(options) ​

Create a table in a PostgreSQL database with the following structure:

For now the structure has been defined to store GeoJSON collection. Hook options are the following:

writePGTable(options) ​

Inserts a GeoJSON collection or an array of features into an existing table. THe table must have the same structured as a table created using the createPGTable hook. Hook options are the following:

Raster ​

source

readGeoTiff(options) ​

Read a GeoTiff from an input stream/store and convert it to in-memory JSON values, hook options are the following:

computeStatistics(options) ​

Computes minimum and maximum values on a GeoTiff file, hook options are the following:

Store ​

source

createStores(options) ​

Create (a set of) store(s), hook options are the (array of) following the following:

removeStores(options) ​

Remove (a set of) store(s), hook options are (array of) the following:

TIP

As a shortcut the options provided can only be store IDs when storePath is not used

discardIfExistsInStore(options) ​

Discard the task if a target file already exists in an output store, hook options are the following:

copyToStore(options) ​

Copy the item(s) from an input store to an output store, hook options are the following:

gzipToStore(options) ​

Gzip the item(s) from an input store to an output store, hook options are the following:

gunzipFromStore(options) ​

Gunzip the item(s) from an input store to an output store, hook options are the following:

unzipFromStore(options) ​

Unzip the item(s) from an input store to an output store, hook options are the following:

System ​

source

tar(options) ​

Tar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

untar(options) ​

Untar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

runCommand(options) ​

Run a system command. Hook options are the following:

TIP

Learn more about templating

envsubst(options) ​

Provides file-level environment variable substitution. Hook options are the following:

TXT ​

source

readTXT(options) ​

Read a TXT from an input stream/store and convert it to in-memory JSON values, hook options are the following:

Utils ​

source

generateId(options) ​

Generate a UUID (V1) for the item using node-uuid.

template(options) ​

Perform templating of the options using the item as context and merge it with item.

discardIf(options) ​

Discard all subsequent hooks and task if the input data passes the given match filter options, filter options are similar to the match filter described in common options.

apply(options) ​

Apply a given function to the hook item(s), hook options are the following:

healthcheck(options) ​

Apply a given function to the hook item(s) and healthcheck structure, hook options are the following:

addOutputs(outputs) ​

Declare a new output for the job/task, hook options is an array of objects with the following properties:

Tasks and write hooks automatically track generated outputs but sometimes outputs are generated by an external process (eg. command hook) so that you need to declare it in order to properly clean it with the clearOutputs hook.

runTask(options) ​

Run a given task, hook options are those of a task.

emitEvent(options) ​

Emit a 'krawler' event on the underlying service, hook options are the following:

XML ​

source

readXML(options) ​

Read an XML file from a store and convert it to in-memory JSON values, hook options are the following:

YAML ​

source

readYAML(options) ​

Read a YAML file from a store and convert it to in-memory JSON values, hook options are the following:

writeYAML(options) ​

Generate a YAML file from in-memory JSON values, hook options are the following:

`,340),r=[n];function l(i,p,c,h,d,u){return e(),t("div",null,r)}const f=o(a,[["render",l]]);export{m as __pageData,f as default}; diff --git a/assets/reference_hooks.md.2f6bf6ef.lean.js b/assets/reference_hooks.md.3cc0f923.lean.js similarity index 83% rename from assets/reference_hooks.md.2f6bf6ef.lean.js rename to assets/reference_hooks.md.3cc0f923.lean.js index 82a84331..3026918e 100644 --- a/assets/reference_hooks.md.2f6bf6ef.lean.js +++ b/assets/reference_hooks.md.3cc0f923.lean.js @@ -1 +1 @@ -import{_ as o,o as e,c as t,Q as s}from"./chunks/framework.e713b733.js";const m=JSON.parse('{"title":"Hooks","description":"","frontmatter":{},"headers":[],"relativePath":"reference/hooks.md","filePath":"reference/hooks.md"}'),a={name:"reference/hooks.md"},n=s("",334),r=[n];function l(i,p,c,h,d,u){return e(),t("div",null,r)}const f=o(a,[["render",l]]);export{m as __pageData,f as default}; +import{_ as o,o as e,c as t,Q as s}from"./chunks/framework.e713b733.js";const m=JSON.parse('{"title":"Hooks","description":"","frontmatter":{},"headers":[],"relativePath":"reference/hooks.md","filePath":"reference/hooks.md"}'),a={name:"reference/hooks.md"},n=s("",340),r=[n];function l(i,p,c,h,d,u){return e(),t("div",null,r)}const f=o(a,[["render",l]]);export{m as __pageData,f as default}; diff --git a/examples/index.html b/examples/index.html index f76d54e2..1fd36c33 100644 --- a/examples/index.html +++ b/examples/index.html @@ -67,7 +67,7 @@ "link": "https://s3.eu-central-1.amazonaws.com/krawler/bc185cb0-7983-11e8-883f-a333a7402f4a.png" } ]

Seeder ​

The seeder take advantage of Kargo to seed a dataset. It relies on the seeding capabilities of MapProxy. The global approach is to subdivide the job into multiple tasks and run mapprroxy-seed utility for each task. To subdivide the job, we use a spatial grid and each cell is used as a coverage entry to limit the extend of the corresponding task. All the tasks, i.e. mapproxy-seed share the same MapProxy configuration file and use a generated seed file.

We use the same image of MapProxy as the one used in Kargo, but for now we do not use the benefits of a Swarm infrastructure to deploy the task. Meanwhile, if you plan to seed a layer with a source exposed by TileserverGL, you can easily scale the number of instances of TileserverGL to fit the required charge.

- + \ No newline at end of file diff --git a/guides/extending-krawler.html b/guides/extending-krawler.html index 5081f31e..f4f81256 100644 --- a/guides/extending-krawler.html +++ b/guides/extending-krawler.html @@ -57,7 +57,7 @@ } } hooks.registerHook('custom', hook)

After that you can use your custom hook like the built-in ones with the CLI.

TIP

For this to work you need to add krawler as a dependency in the package.json of your job module or link to it in development mode. Please refer to our installation section

Complete example ​

A complete example of extension is available in the samples.

- + \ No newline at end of file diff --git a/guides/index.html b/guides/index.html index 5d9aaf9c..ff4a7d34 100644 --- a/guides/index.html +++ b/guides/index.html @@ -19,7 +19,7 @@
Krawler

Appearance

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/guides/installing-krawler.html b/guides/installing-krawler.html index edebc3aa..33149d18 100644 --- a/guides/installing-krawler.html +++ b/guides/installing-krawler.html @@ -43,7 +43,7 @@ yarn link @kalisio/krawler

Please refer to the KDK documentation to setup your development environment.

A native command-line executable can be generated using pkg eg for windows:

bash
pkg . --target node8-win-x86
pkg . --target node8-win-x86

Because it relies on the GDAL native bindings you will need to deploy the gdal.node file (usually found in node_modules\gdal\lib\binding) to the same directory as the executable. Take care to generate the executable with the same architecture than your Node.js version.

As a Docker container ​

When using krawler as a Docker container the arguments to the CLI have to be provided through the ARGS environment variable, along with any other required variables and the data volume to make inputs accessible within the container and get output files back:

bash
docker pull kalisio/krawler
 docker run --name krawler --rm -v /mnt/data:/opt/krawler/data -e "ARGS=/opt/krawler/data/jobfile.js" -e S3_BUCKET=krawler kalisio/krawler
docker pull kalisio/krawler
 docker run --name krawler --rm -v /mnt/data:/opt/krawler/data -e "ARGS=/opt/krawler/data/jobfile.js" -e S3_BUCKET=krawler kalisio/krawler
- + \ No newline at end of file diff --git a/guides/understanding-krawler.html b/guides/understanding-krawler.html index d144ecfa..a2b8da99 100644 --- a/guides/understanding-krawler.html +++ b/guides/understanding-krawler.html @@ -19,7 +19,7 @@
Krawler

Appearance

Understanding Krawler ​

krawler is powered by Feathers and rely on two of its main abstractions: services and hooks. We assume you are familiar with this technology.

Main concepts ​

krawler manipulates three kind of entities:

  • a store define where the extracted/processed data will reside,
  • a task define what data to be extracted and how to query it,
  • a job define what tasks to be run to fulfill a request (i.e. sequencing).

On top of this hooks provide a set of functions that can be typically run before/after a task/job such as a conversion after a download or task generation before a job run. More or less, this allows to create a processing pipeline.

Regarding the store management we rely on abstract-blob-store, which abstracts a lot of different storage backends (local file system, AWS S3, Google Drive, etc.), and is already used by feathers-blob.

Global overview ​

The following figure depicts the global architecture and all concepts at play:

Architecture

What is inside ? ​

krawler is possible and mainly powered by the following stack:

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/guides/using-krawler.html b/guides/using-krawler.html index 29ffa3a2..4664654e 100644 --- a/guides/using-krawler.html +++ b/guides/using-krawler.html @@ -119,7 +119,7 @@ ... ] }

TIP

When running the krawler as a web API note that only the hooks pipeline is mandatory in the job file. Indeed, job and task objects will be then sent by requesting the exposed web services.

Healthcheck ​

Healthcheck endpoint ​

When running the krawler as a cron job note that it provides a healthcheck endpoint e.g. on localhost:3030/api/healthcheck. The following JSON structure is returned:

The returned HTTP code is 500 whenever an error has occured in the last run, 200 otherwise.

TIP

You can add your custom data in the healthcheck structure using the healthcheck hook.

Healthcheck command ​

For convenience the krawler also includes a built-in healthcheck script that could be used e.g. by Docker. This script uses similar options than the CLI plus some specific options:

TIP

Templates are generated with healthcheck structure and environment variables as context, learn more about templating.

- + \ No newline at end of file diff --git a/hashmap.json b/hashmap.json index d92e36c1..7745dd9f 100644 --- a/hashmap.json +++ b/hashmap.json @@ -1 +1 @@ -{"guides_index.md":"8bac4ba9","about_roadmap.md":"2f902165","reference_index.md":"76bdc228","reference_hooks.md":"2f6bf6ef","about_license.md":"51fe04ca","reference_known-issues.md":"29bbda91","reference_services.md":"4fa3200f","about_contributing.md":"30a97c28","about_introduction.md":"b4a8b11a","examples_index.md":"2db99c6c","guides_extending-krawler.md":"ccca42a7","guides_installing-krawler.md":"60d1a08e","guides_understanding-krawler.md":"56d783c6","about_contact.md":"8fe8c179","guides_using-krawler.md":"e66b4f7b","index.md":"ee3add83"} +{"about_contact.md":"8fe8c179","about_license.md":"51fe04ca","about_roadmap.md":"2f902165","examples_index.md":"2db99c6c","guides_extending-krawler.md":"ccca42a7","guides_index.md":"8bac4ba9","guides_understanding-krawler.md":"56d783c6","guides_using-krawler.md":"e66b4f7b","index.md":"ee3add83","reference_hooks.md":"3cc0f923","guides_installing-krawler.md":"60d1a08e","about_contributing.md":"30a97c28","about_introduction.md":"b4a8b11a","reference_services.md":"4fa3200f","reference_index.md":"76bdc228","reference_known-issues.md":"29bbda91"} diff --git a/index.html b/index.html index b356e15a..7e2b5a96 100644 --- a/index.html +++ b/index.html @@ -19,7 +19,7 @@
Krawler

Appearance

Krawler

A minimalist Geospatial ETL

Want more details ?
krawler

Minimalist ETL

Make automated process of extracting and processing (geographic) data from heterogeneous sources with ease

Open and extensible

Take benefits of the existing functionalities and add your owns operations to address your specific needs

Web oriented

Deploy your jobs as web services in a matter of minutes

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/reference/hooks.html b/reference/hooks.html index 6f21fa6c..17553cc9 100644 --- a/reference/hooks.html +++ b/reference/hooks.html @@ -11,7 +11,7 @@ - + @@ -126,7 +126,7 @@ }, pick: ['onlyThisPropertyWillBeKept'], omit: ['onlyThisPropertyWillBeRemoved'], -merge: { newProperty: 'will be added to the final objects' }

TIP

The transformations are applied in the order of the documentation, e.g. filtering occurs before mapping.

mergeJson(options) ​

Generate a in-memory JSON object from a set of input in-memory JSON objects, hook options are the following:

The input hook result is expected to be an array of tasks which output will be read in-memory.

writeTemplate(options) ​

Generate a file from an input template and injected in-memory JSON values, hook options are the following:

TIP

Learn more about templating

GeoJSON ​

source

convertToGeoJson(options) ​

Convert in-memory JSON values to a GeoJSON collection. For each in-memory object, the hook generates a corresponding GeoJSON feature using specific properties to build the geometry property. For now, it only allows to generate features of type of Point. Moreover, the coordinate reference system is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84), with longitude and latitude expressed in decimal degrees. The entire object is stored under the properties property of the feature. Hook options are the following:

convertOSMToGeoJson(options) ​

Convert in-memory OSM JSON values to a GeoJSON collection. It relies on osmtogeojson. Hook options are the following:

reprojectGeoJson(options) ​

Reproject a GeoJSON from a given projection system to another one, hook options are the following:

KML ​

source

readKML(options) ​

Read a KML from an input stream/store and convert it to in-memory JSON values, hook options are the following:

MongoDB ​

source

connectMongo(options) ​

Connect to a MongoDB database. The connection options of the client are defined in the hook options plus:

TIP

Since Krawler relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option useNewUrlParser: true when connecting to the database.

disconnectMongo(options) ​

Disconnect from a MongoDB database. Hook options are the following:

dropMongoCollection(options) ​

Drop if exists a collection in a MongoDB database. Hook options are the following:

createMongoCollection(options) ​

Create a collection in a MongoDB database. Hook options are the following:

readMongoCollection(options) ​

Read JSON documents from an existing collection. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

updateMongoCollection(options) ​

Updates JSON into an existing collection (uses updateOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be updated into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

deleteMongoCollection(options) ​

Removes documents from an existing collection (uses deleteMany operations under-the-hood). Hook options are the following:

createMongoAggregation(options) ​

Creates an aggregation pipeline on an existing collection. Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

dropMongoBucket(options) ​

Drop if exists a bucket in a MongoDB database. Hook options are the following:

createMongoBucket(options) ​

Create a bucket in a MongoDB database. Hook options are the following:

readMongoBucket(options) ​

Read file from an existing bucket. Hook options are the following:

writeMongoBucket(options) ​

Insert file into an existing bucket. Hook options are the following:

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections

Feathers ​

source

connectFeathers(options) ​

Connect to a Feathers API. The connection options of the client are defined in the hook options plus:

TIP

Krawler uses on the version 5 of the Feathers client.

disconnectFeathers(options) ​

Disconnect from a Feathers API. Hook options are the following:

callFeathersServiceMethod(options) ​

Performs a service operation using the API. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

Numerical Weather Prediction ​

source

Numerical Weather Prediction (NWP) data are now available from the major meteorological agencies and institutions on a day-to-day basis. These hooks aim at gathering weather forecast data generated by forecast models easily.

Each forecast model output hundreds of forecast elements (a.k.a. meteorological elements) such as temperature, wind direction, etc. The production of a set of forecast data is called a run of the model and occurs on a regular daily basis, e.g. every 6 hours. The spatial properties of a model are completely defined by a longitude/latitude grid and a set of altitude levels (meter or pressure scale). The temporal properties are defined by interval values describing at which frequency/time the forecast data are produced (a.k.a. run interval) and which time steps are available (a.k.a. forecast interval).

generateNwpTasks(options) ​

Generate tasks to download data for each variable. It is intended to be used a job hook and the required hook options (can be overriden by input data) are the following:

A task will be generated for each element, level and gathered forecast time with the following properties: level, runTime, forecastTime, timeOffset.

TIP

This hook is intended to work with task templating to generate the actual download tasks (e.g. HTTP or WCS request)

OGC ​

source

getCapabilities(options) ​

Execute a GetCapabilties request to get the general information about an OGC service such as WMS, WCS, WPS... Hook options are the following:

The following example illustrates how to use this hook:

getCapabilities: {
+merge: { newProperty: 'will be added to the final objects' }

TIP

The transformations are applied in the order of the documentation, e.g. filtering occurs before mapping.

mergeJson(options) ​

Generate a in-memory JSON object from a set of input in-memory JSON objects, hook options are the following:

The input hook result is expected to be an array of tasks which output will be read in-memory.

writeTemplate(options) ​

Generate a file from an input template and injected in-memory JSON values, hook options are the following:

TIP

Learn more about templating

GeoJSON ​

source

convertToGeoJson(options) ​

Convert in-memory JSON values to a GeoJSON collection. For each in-memory object, the hook generates a corresponding GeoJSON feature using specific properties to build the geometry property. For now, it only allows to generate features of type of Point. Moreover, the coordinate reference system is a geographic coordinate reference system, using the World Geodetic System 1984 (WGS 84), with longitude and latitude expressed in decimal degrees. The entire object is stored under the properties property of the feature. Hook options are the following:

convertOSMToGeoJson(options) ​

Convert in-memory OSM JSON values to a GeoJSON collection. It relies on osmtogeojson. Hook options are the following:

reprojectGeoJson(options) ​

Reproject a GeoJSON from a given projection system to another one, hook options are the following:

KML ​

source

readKML(options) ​

Read a KML from an input stream/store and convert it to in-memory JSON values, hook options are the following:

MongoDB ​

source

connectMongo(options) ​

Connect to a MongoDB database. The connection options of the client are defined in the hook options plus:

TIP

Since Krawler relies on the version 3.1.13 of the MongoDB driver, it automatically adds the option useNewUrlParser: true when connecting to the database.

disconnectMongo(options) ​

Disconnect from a MongoDB database. Hook options are the following:

dropMongoCollection(options) ​

Drop if exists a collection in a MongoDB database. Hook options are the following:

createMongoCollection(options) ​

Create a collection in a MongoDB database. Hook options are the following:

dropMongoIndex(options) ​

Drop if exists a collection index in a MongoDB database. Hook options are the following:

createMongoIndex(options) ​

Create a collection index in a MongoDB database. Hook options are the following:

readMongoCollection(options) ​

Read JSON documents from an existing collection. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

updateMongoCollection(options) ​

Updates JSON into an existing collection (uses updateOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be updated into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

deleteMongoCollection(options) ​

Removes documents from an existing collection (uses deleteMany operations under-the-hood). Hook options are the following:

createMongoAggregation(options) ​

Creates an aggregation pipeline on an existing collection. Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

dropMongoBucket(options) ​

Drop if exists a bucket in a MongoDB database. Hook options are the following:

createMongoBucket(options) ​

Create a bucket in a MongoDB database. Hook options are the following:

readMongoBucket(options) ​

Read file from an existing bucket. Hook options are the following:

writeMongoBucket(options) ​

Insert file into an existing bucket. Hook options are the following:

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections

Feathers ​

source

connectFeathers(options) ​

Connect to a Feathers API. The connection options of the client are defined in the hook options plus:

TIP

Krawler uses on the version 5 of the Feathers client.

disconnectFeathers(options) ​

Disconnect from a Feathers API. Hook options are the following:

callFeathersServiceMethod(options) ​

Performs a service operation using the API. Hook options are the following:

WARNING

Due to templating restricted to string output any ISO date string or comparison operator value in the query object will be automatically converted back to native types so that matching will work as expected in JS

writeMongoCollection(options) ​

Inserts JSON into an existing collection (uses insertOne operations under-the-hood). Hook options are the following:

TIP

If the input data is a GeoJSON collection the array of features will be pushed into the collection not the root object, this is to conform with MongoDB geospatial capabilities that can not handle recursive collections.

Numerical Weather Prediction ​

source

Numerical Weather Prediction (NWP) data are now available from the major meteorological agencies and institutions on a day-to-day basis. These hooks aim at gathering weather forecast data generated by forecast models easily.

Each forecast model output hundreds of forecast elements (a.k.a. meteorological elements) such as temperature, wind direction, etc. The production of a set of forecast data is called a run of the model and occurs on a regular daily basis, e.g. every 6 hours. The spatial properties of a model are completely defined by a longitude/latitude grid and a set of altitude levels (meter or pressure scale). The temporal properties are defined by interval values describing at which frequency/time the forecast data are produced (a.k.a. run interval) and which time steps are available (a.k.a. forecast interval).

generateNwpTasks(options) ​

Generate tasks to download data for each variable. It is intended to be used a job hook and the required hook options (can be overriden by input data) are the following:

A task will be generated for each element, level and gathered forecast time with the following properties: level, runTime, forecastTime, timeOffset.

TIP

This hook is intended to work with task templating to generate the actual download tasks (e.g. HTTP or WCS request)

OGC ​

source

getCapabilities(options) ​

Execute a GetCapabilties request to get the general information about an OGC service such as WMS, WCS, WPS... Hook options are the following:

The following example illustrates how to use this hook:

getCapabilities: {
   url: 'http://geoserver.kalisio.xyz/geoserver/Kalisio/wms',
   service: 'WMS'
 }
getCapabilities: {
@@ -147,7 +147,7 @@
   port: 5432,
   clientPath: 'taskTemplate.client'
 }

disconnectPG(options) ​

Disconnect from a PostgresSQL database. Hook options are the following:

dropPGTable(options) ​

Drop if exists a table in a PostgreSQL database. Hook options are the following:

createPGTable(options) ​

Create a table in a PostgreSQL database with the following structure:

For now the structure has been defined to store GeoJSON collection. Hook options are the following:

writePGTable(options) ​

Inserts a GeoJSON collection or an array of features into an existing table. THe table must have the same structured as a table created using the createPGTable hook. Hook options are the following:

Raster ​

source

readGeoTiff(options) ​

Read a GeoTiff from an input stream/store and convert it to in-memory JSON values, hook options are the following:

computeStatistics(options) ​

Computes minimum and maximum values on a GeoTiff file, hook options are the following:

Store ​

source

createStores(options) ​

Create (a set of) store(s), hook options are the (array of) following the following:

removeStores(options) ​

Remove (a set of) store(s), hook options are (array of) the following:

TIP

As a shortcut the options provided can only be store IDs when storePath is not used

discardIfExistsInStore(options) ​

Discard the task if a target file already exists in an output store, hook options are the following:

copyToStore(options) ​

Copy the item(s) from an input store to an output store, hook options are the following:

gzipToStore(options) ​

Gzip the item(s) from an input store to an output store, hook options are the following:

gunzipFromStore(options) ​

Gunzip the item(s) from an input store to an output store, hook options are the following:

unzipFromStore(options) ​

Unzip the item(s) from an input store to an output store, hook options are the following:

System ​

source

tar(options) ​

Tar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

untar(options) ​

Untar files or directories using node-tar, hook options are the following:

TIP

file, files and cwd options can be templates, learn more about templating

runCommand(options) ​

Run a system command. Hook options are the following:

TIP

Learn more about templating

envsubst(options) ​

Provides file-level environment variable substitution. Hook options are the following:

TXT ​

source

readTXT(options) ​

Read a TXT from an input stream/store and convert it to in-memory JSON values, hook options are the following:

Utils ​

source

generateId(options) ​

Generate a UUID (V1) for the item using node-uuid.

template(options) ​

Perform templating of the options using the item as context and merge it with item.

discardIf(options) ​

Discard all subsequent hooks and task if the input data passes the given match filter options, filter options are similar to the match filter described in common options.

apply(options) ​

Apply a given function to the hook item(s), hook options are the following:

healthcheck(options) ​

Apply a given function to the hook item(s) and healthcheck structure, hook options are the following:

addOutputs(outputs) ​

Declare a new output for the job/task, hook options is an array of objects with the following properties:

Tasks and write hooks automatically track generated outputs but sometimes outputs are generated by an external process (eg. command hook) so that you need to declare it in order to properly clean it with the clearOutputs hook.

runTask(options) ​

Run a given task, hook options are those of a task.

emitEvent(options) ​

Emit a 'krawler' event on the underlying service, hook options are the following:

XML ​

source

readXML(options) ​

Read an XML file from a store and convert it to in-memory JSON values, hook options are the following:

YAML ​

source

readYAML(options) ​

Read a YAML file from a store and convert it to in-memory JSON values, hook options are the following:

writeYAML(options) ​

Generate a YAML file from in-memory JSON values, hook options are the following:

- + \ No newline at end of file diff --git a/reference/index.html b/reference/index.html index c4a8cd0a..65306fc1 100644 --- a/reference/index.html +++ b/reference/index.html @@ -19,7 +19,7 @@
Krawler

Appearance

MIT Licensed | Copyright © 2017-20xx Kalisio

- + \ No newline at end of file diff --git a/reference/known-issues.html b/reference/known-issues.html index f140639b..bcd979eb 100644 --- a/reference/known-issues.html +++ b/reference/known-issues.html @@ -89,7 +89,7 @@ clearOutputs: {} } } - + \ No newline at end of file diff --git a/reference/services.html b/reference/services.html index 3d364484..6e275cbf 100644 --- a/reference/services.html +++ b/reference/services.html @@ -131,7 +131,7 @@ .catch(error => { console.log(error.message) }) - + \ No newline at end of file