From 14f7de082d18c93ce6ff71df2398708b8fd21a2f Mon Sep 17 00:00:00 2001 From: Sebastian Tramp Date: Thu, 5 Oct 2023 09:51:00 +0200 Subject: [PATCH] update cmemc docu, extend variable docu, add variable tag --- .../command-reference/admin/client/index.md | 6 + .../command-reference/admin/index.md | 4 - .../command-reference/admin/metrics/index.md | 4 - .../command-reference/admin/store/index.md | 13 +- .../command-reference/admin/user/index.md | 14 +- .../admin/workspace/index.md | 2 - .../admin/workspace/python/index.md | 37 +++- .../command-reference/config/index.md | 93 +++++----- .../command-reference/dataset/index.md | 47 ++++-- .../dataset/resource/index.md | 2 - .../command-reference/graph/index.md | 5 - .../command-reference/index.md | 11 +- .../command-reference/project/index.md | 11 +- .../project/variable/index.md | 159 ++++++++++++++++++ .../command-reference/query/index.md | 10 -- .../vocabulary/cache/index.md | 1 - .../command-reference/vocabulary/index.md | 3 - .../command-reference/workflow/index.md | 17 +- docs/build/.pages | 2 +- .../di-var-add-variable.png | Bin .../di-var-email-defined.png | Bin .../di-var-email-definition.png | Bin .../di-var-email-usage-2.png | Bin .../di-var-email-usage.png | Bin .../di-var-password-definition.png | Bin .../di-var-password-usage.png | Bin .../di-var-project-select.png | Bin .../di-var-value-evaluation.png | Bin .../{di-variables => variables}/index.md | 20 ++- mkdocs.yml | 2 + 30 files changed, 336 insertions(+), 127 deletions(-) create mode 100644 docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md rename docs/build/{di-variables => variables}/di-var-add-variable.png (100%) rename docs/build/{di-variables => variables}/di-var-email-defined.png (100%) rename docs/build/{di-variables => variables}/di-var-email-definition.png (100%) rename docs/build/{di-variables => variables}/di-var-email-usage-2.png (100%) rename docs/build/{di-variables => variables}/di-var-email-usage.png (100%) rename docs/build/{di-variables => variables}/di-var-password-definition.png (100%) rename docs/build/{di-variables => variables}/di-var-password-usage.png (100%) rename docs/build/{di-variables => variables}/di-var-project-select.png (100%) rename docs/build/{di-variables => variables}/di-var-value-evaluation.png (100%) rename docs/build/{di-variables => variables}/index.md (91%) diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/client/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/client/index.md index eb66189e..47dd76d9 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/client/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/client/index.md @@ -10,6 +10,8 @@ tags: # admin client Command Group + + List client accounts, get or generate client account secrets. This command group is an opinionated interface to the Keycloak realm of your Corporate Memory instance. In order to be able to use the commands in this group, the configured cmemc connection account needs to be equipped with the `manage-clients` role in the used realm. @@ -30,6 +32,8 @@ $ cmemc admin client list [OPTIONS] + + Outputs a list of client accounts, which can be used to get an overview as well as a reference for the other commands of the `admin client` command group. !!! note @@ -57,6 +61,8 @@ $ cmemc admin client secret [OPTIONS] CLIENT_ID + + This command retrieves or generates a new secret for a client account from a realm. diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/index.md index ea75c3e1..42c4c1fb 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/index.md @@ -46,20 +46,17 @@ $ cmemc config list | parallel --ctag cmemc -c {} admin status in the table, 'overall.healthy' with result in UP in case all health flags are UP as well (otherwise DOWN). - --exit-1 [never|error|always] Specify, when this command returns with exit code 1. Available options are 'never' (exit 0 on errors and warnings), 'error' (exit 1 on errors, exit 0 on warnings), 'always' (exit 1 on errors and warnings). [default: never] - --enforce-table A single value with --key will be returned as plain text instead of a table with one row and the header. This default behaviour allows for more easy integration with scripts. This flag enforces the use of tabular output, even for single row tables. - --raw Outputs combined raw JSON output of the health/info endpoints. ``` @@ -92,7 +89,6 @@ Please be aware that this command can reveal secrets which you might not want to --raw Outputs raw JSON. Note that this option will always try to fetch a new JSON token response. In case you are working with OAUTH_GRANT_TYPE=prefetched_token, this may lead to an error. - --decode Decode the access token and outputs the raw JSON. Note that the access token is only decoded and esp. not validated. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/metrics/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/metrics/index.md index 32b76ca1..a2a23372 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/metrics/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/metrics/index.md @@ -37,7 +37,6 @@ A metric of a specific job is identified by a metric ID. Possible metric IDs of --job [DP] The job from which the metrics data is fetched. [default: DP] - --filter ... A set of label name/value pairs in order to filter the samples of the requested metric family. Each metric has a different set of labels with different @@ -46,13 +45,11 @@ A metric of a specific job is identified by a metric ID. Possible metric IDs of option. The label names are then shown as column headers and label values as cell values of this column. - --enforce-table A single sample value will be returned as plain text instead of the normal table. This allows for more easy integration with scripts. This flag enforces the use of tabular output, even for single row tables. - --raw Outputs raw prometheus sample classes. ``` @@ -99,7 +96,6 @@ For each metric, the output table shows the metric ID, the type of the metric, a --job [DP] The job from which the metrics data is fetched. [default: DP] --id-only Lists metric identifier only. This is useful for piping the IDs into other commands. - --raw Outputs (sorted) JSON dict, parsed from the metrics API output. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/store/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/store/index.md index 5290f304..1cd44f04 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/store/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/store/index.md @@ -39,16 +39,14 @@ This command creates a showcase scenario of multiple graphs including integratio --scale INTEGER The scale factor provides a way to set the target size of the scenario. A value of 10 results in around 40k triples, a value of 50 in around 350k triples. [default: 10] - --create Delete old showcase data if present and create new showcase databased on the given scale factor. - --delete Delete existing showcase data if present. ``` ## admin store bootstrap -Update/Import bootstrap data. +Update/Import or remove bootstrap data. ```shell-session title="Usage" $ cmemc admin store bootstrap [OPTIONS] @@ -57,10 +55,12 @@ $ cmemc admin store bootstrap [OPTIONS] -This command imports the bootstrap data needed for managing shapes and configuration objects. +Use ``--import`` to import the bootstrap data needed for managing shapes and configuration objects. This will remove the old data first. + +Use ``--remove`` to delete bootstrap data. !!! note - The command will first remove all existing bootstrap data (identified with the isSystemResource flag) and will then import the new data to the corresponding graphs (shape catalog, vocabulary catalog, configuration graph). + The removal of existing bootstrap data will search for resources which are flagged with the isSystemResource property. @@ -69,7 +69,8 @@ This command imports the bootstrap data needed for managing shapes and configura ```text --import Delete existing bootstrap data if present and import bootstrap - data which was delivered + data which was delivered with Corporate Memory. + --remove Delete existing bootstrap data if present. ``` ## admin store export diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/user/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/user/index.md index d89c92a0..52bf39de 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/user/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/user/index.md @@ -10,6 +10,8 @@ tags: # admin user Command Group + + List, create, delete and modify user accounts. This command group is an opinionated interface to the Keycloak realm of your Corporate Memory instance. In order to be able to manage user data, the configured cmemc connection account needs to be equipped with the `manage-users` role in the used realm. @@ -30,6 +32,8 @@ $ cmemc admin user list [OPTIONS] + + Outputs a list of user accounts, which can be used to get an overview as well as a reference for the other commands of the `admin user` command group. @@ -53,6 +57,8 @@ $ cmemc admin user create USERNAME + + This command creates a new user account. !!! note @@ -72,6 +78,8 @@ $ cmemc admin user update [OPTIONS] USERNAME + + This command updates metadata and group assignments of a user account. For each data value, a separate option needs to be used. All options can be combined in a single execution. @@ -103,6 +111,8 @@ $ cmemc admin user delete USERNAME + + This command deletes a user account from a realm. !!! note @@ -122,6 +132,8 @@ $ cmemc admin user password [OPTIONS] USERNAME + + With this command, the password of a user account can be changed. The default execution mode of this command is an interactive prompt which asks for the password (twice). In order automate password changes, you can use the ``--value`` option. !!! warning @@ -135,10 +147,8 @@ With this command, the password of a user account can be changed. The default ex --value TEXT With this option, the new password can be set in a non- interactive way. - --temporary If enabled, the user must change the password on next login. - --request-change If enabled, will send a email to user to reset the password. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md index 5f3e8047..60b0d6ff 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/index.md @@ -33,10 +33,8 @@ The file name is optional and will be generated with by the template if absent. -o, --overwrite Overwrite existing files. This is a dangerous option, so use it with care. - --type TEXT Type of the exported workspace file. [default: xmlZip] - -t, --filename-template TEXT Template for the export file name. Possible placeholders are (Jinja2): {{connection}} (from the --connection option) and {{date}} diff --git a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/python/index.md b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/python/index.md index 942f8653..3c8b5532 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/python/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/admin/workspace/python/index.md @@ -37,10 +37,10 @@ You can install a package by uploading a source distribution .tar.gz file, by up ## admin workspace python uninstall -Uninstall a python package from the workspace. +Uninstall a python packages from the workspace. ```shell-session title="Usage" -$ cmemc admin workspace python uninstall PACKAGE_NAME +$ cmemc admin workspace python uninstall [OPTIONS] [PACKAGE_NAME]... ``` @@ -50,6 +50,14 @@ This command is essentially a `pip uninstall` in the remote python environment. +??? info "Options" + ```text + + -a, --all This option removes all installed packages from the system, + leaving only the pre-installed mandatory packages in the + environment. + ``` + ## admin workspace python list List installed python packages. @@ -70,9 +78,13 @@ It outputs a table of python package identifiers with version information. ??? info "Options" ```text - --raw Outputs raw JSON. - --id-only Lists only package identifier. This is useful for piping the IDs - into other commands. + --raw Outputs raw JSON. + --id-only Lists only package identifier. This is useful for piping the + IDs into other commands. + --available Instead of listing installed packages, this option lists + installable packages from pypi.org, which are prefixed with + 'cmem-plugin-' and so are most likely Corporate Memory plugin + packages. ``` ## admin workspace python list-plugins @@ -102,3 +114,18 @@ This commands lists all discovered plugins. --package-id-only Lists only plugin package identifier. ``` +## admin workspace python open + +Open a package pypi.org page in the browser. + +```shell-session title="Usage" +$ cmemc admin workspace python open PACKAGE +``` + + + + +With this command, you can open the pypi.org page of a published package in your browser. From there, you can follow links, review the version history as well as the origin of the package, and read the provided documentation. + + + diff --git a/docs/automate/cmemc-command-line-interface/command-reference/config/index.md b/docs/automate/cmemc-command-line-interface/command-reference/config/index.md index f95305cb..29eaf334 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/config/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/config/index.md @@ -12,50 +12,51 @@ tags: ```text List and edit configs as well as get config values. -Configurations are identified by the section identifier in the -config file. Each configuration represent a Corporate Memory deployment -with its specific access method as well as credentials. - -A minimal configuration which uses client credentials has the following -entries: - - -[example.org] -CMEM_BASE_URI=https://cmem.example.org/ -OAUTH_GRANT_TYPE=client_credentials -OAUTH_CLIENT_ID=cmem-service-account -OAUTH_CLIENT_SECRET=my-secret-account-pass - -Note that OAUTH_GRANT_TYPE can be either client_credentials, password or -prefetched_token. - -In addition to that, the following config parameters can be used as well: - - -SSL_VERIFY=False - for ignoring certificate issues (not recommended) -DP_API_ENDPOINT=URL - to point to a non-standard DataPlatform location -DI_API_ENDPOINT=URL - to point to a non-standard DataIntegration location -OAUTH_TOKEN_URI=URL - to point to an external IdentityProvider location -OAUTH_USER=username - only if OAUTH_GRANT_TYPE=password -OAUTH_PASSWORD=password - only if OAUTH_GRANT_TYPE=password -OAUTH_ACCESS_TOKEN=token - only if OAUTH_GRANT_TYPE=prefetched_token - -In order to get credential information from an external process, you can -use the parameter OAUTH_PASSWORD_PROCESS, OAUTH_CLIENT_SECRET_PROCESS and -OAUTH_ACCESS_TOKEN_PROCESS to setup an external executable. - - -OAUTH_CLIENT_SECRET_PROCESS=/path/to/getpass.sh -OAUTH_PASSWORD_PROCESS=["getpass.sh", "parameter1", "parameter2"] - -The credential executable can use the cmemc environment for fetching the -credential (e.g. CMEM_BASE_URI and OAUTH_USER). -If the credential executable is not given with a full path, cmemc -will look into your environment PATH for something which can be executed. -The configured process needs to return the credential on the first line -of stdout. In addition to that, the process needs to exit with exit -code 0 (without failure). There are examples available in the online -manual. + Configurations are identified by the section identifier in the + config file. Each configuration represent a Corporate Memory deployment + with its specific access method as well as credentials. + + A minimal configuration which uses client credentials has the following + entries: + +  + [example.org] + CMEM_BASE_URI=https://cmem.example.org/ + OAUTH_GRANT_TYPE=client_credentials + OAUTH_CLIENT_ID=cmem-service-account + OAUTH_CLIENT_SECRET=my-secret-account-pass + + Note that OAUTH_GRANT_TYPE can be either client_credentials, password or + prefetched_token. + + In addition to that, the following config parameters can be used as well: + +  + SSL_VERIFY=False - for ignoring certificate issues (not recommended) + DP_API_ENDPOINT=URL - to point to a non-standard DataPlatform location + DI_API_ENDPOINT=URL - to point to a non-standard DataIntegration location + OAUTH_TOKEN_URI=URL - to point to an external IdentityProvider location + OAUTH_USER=username - only if OAUTH_GRANT_TYPE=password + OAUTH_PASSWORD=password - only if OAUTH_GRANT_TYPE=password + OAUTH_ACCESS_TOKEN=token - only if OAUTH_GRANT_TYPE=prefetched_token + + In order to get credential information from an external process, you can + use the parameter OAUTH_PASSWORD_PROCESS, OAUTH_CLIENT_SECRET_PROCESS and + OAUTH_ACCESS_TOKEN_PROCESS to setup an external executable. + +  + OAUTH_CLIENT_SECRET_PROCESS=/path/to/getpass.sh + OAUTH_PASSWORD_PROCESS=["getpass.sh", "parameter1", "parameter2"] + + The credential executable can use the cmemc environment for fetching the + credential (e.g. CMEM_BASE_URI and OAUTH_USER). + If the credential executable is not given with a full path, cmemc + will look into your environment PATH for something which can be executed. + The configured process needs to return the credential on the first line + of stdout. In addition to that, the process needs to exit with exit + code 0 (without failure). There are examples available in the online + manual. + ``` ## config list @@ -104,10 +105,10 @@ $ cmemc config edit Get the value of a known cmemc configuration key. ```shell-session title="Usage" -$ cmemc config get [CMEM_BASE_URI|SSL_VERIFY|REQUESTS_CA_BUNDLE|DP_API_END +$ cmemc config get {CMEM_BASE_URI|SSL_VERIFY|REQUESTS_CA_BUNDLE|DP_API_END POINT|DI_API_ENDPOINT|KEYCLOAK_BASE_URI|KEYCLOAK_REALM_ID|OAUTH_T OKEN_URI|OAUTH_GRANT_TYPE|OAUTH_USER|OAUTH_PASSWORD|OAUTH_CLIENT_ - ID|OAUTH_CLIENT_SECRET|OAUTH_ACCESS_TOKEN] + ID|OAUTH_CLIENT_SECRET|OAUTH_ACCESS_TOKEN} ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/dataset/index.md b/docs/automate/cmemc-command-line-interface/command-reference/dataset/index.md index f4845b48..3ee605aa 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/dataset/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/dataset/index.md @@ -41,10 +41,8 @@ Output and filter a list of available datasets. Each dataset is listed with its can be one of the following values: project, regex, tag, type. The options for the second parameter depend on the first parameter. - --raw Outputs raw JSON objects of the dataset search API response. - --id-only Lists only dataset IDs and no labels or other metadata. This is useful for piping the IDs into other cmemc commands. @@ -78,12 +76,10 @@ This command deletes existing datasets in integration projects from Corporate Me -a, --all Delete all datasets. This is a dangerous option, so use it with care. - --project TEXT In combination with the '--all' flag, this option allows for deletion of all datasets of a certain project. The behaviour is similar to the 'dataset list --project' command. - --filter ... Delete datasets based on metadata. First parameter --filter CHOICE can be one of ['project', 'regex', 'tag', 'type']. The second parameter is based on @@ -201,28 +197,22 @@ $ cmemc dataset create --project my-project --type csv my-file.csv -t, --type TEXT The dataset type of the dataset to create. Example types are 'csv','json' and 'eccencaDataPlatform' (-> Knowledge Graph). - --project TEXT The project, where you want to create the dataset in. If there is only one project in the workspace, this option can be omitted. - -p, --parameter ... A set of key/value pairs. Each dataset type has different parameters (such as charset, arraySeparator, ignoreBadLines, ...). In order to get a list of possible parameter, use the'--help-parameter' option. - --replace Replace remote file resources in case there already exists a file with the same name. - --id TEXT The dataset ID of the dataset to create. The dataset ID will be automatically created in case it is not present. - --help-types Lists all possible dataset types on given Corporate Memory instance. Note that this option already needs access to the instance. - --help-parameter Lists all possible (optional and mandatory) parameter for a dataset type. Note that this option already needs access to the instance. @@ -245,3 +235,40 @@ The command accepts multiple dataset IDs which results in opening multiple brows +## dataset update + +Update a dataset. + +```shell-session title="Usage" +$ cmemc dataset update [OPTIONS] DATASET_ID +``` + + + + +With this command, you can update the configuration of an existing dataset. Similar to the `dataset create` command, you need to use configuration key/value pairs on the ``--parameter`` option. + +To get more information about the available configuration parameters on a dataset, use the ``--help-parameter`` option. + +```shell-session title="Example" +$ cmemc dataset update my-project:my-csv -p separator ";" +``` + + + + +??? info "Options" + ```text + + -p, --parameter ... A configuration parameter key/value pair. + Each dataset type has different parameters + (such as charset, arraySeparator, + ignoreBadLines, ...). In order to get a list + of possible parameter, use the'--help- + parameter' option. + --help-parameter Lists all possible (optional and mandatory) + configuration parameter for a given dataset. + Note that this option already needs access + to the instance. + ``` + diff --git a/docs/automate/cmemc-command-line-interface/command-reference/dataset/resource/index.md b/docs/automate/cmemc-command-line-interface/command-reference/dataset/resource/index.md index 281a3769..253bca6d 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/dataset/resource/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/dataset/resource/index.md @@ -35,7 +35,6 @@ Outputs a table or a list of dataset resources (files). --id-only Lists only resource names and no other metadata. This is useful for piping the IDs into other commands. - --filter ... Filter file resources based on metadata. First parameter CHOICE can be one of ['project', 'regex']. The second parameter is based on CHOICE, @@ -63,7 +62,6 @@ There are three selection mechanisms: with specific IDs, only those specified re --force Delete resource even if in use by a task. -a, --all Delete all resources. This is a dangerous option, so use it with care. - --filter ... Filter file resources based on metadata. First parameter CHOICE can be one of ['project', 'regex']. The second parameter is based on CHOICE, diff --git a/docs/automate/cmemc-command-line-interface/command-reference/graph/index.md b/docs/automate/cmemc-command-line-interface/command-reference/graph/index.md index a3471411..d15c0ca7 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/graph/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/graph/index.md @@ -87,7 +87,6 @@ $ cmemc graph list [OPTIONS] --id-only Lists only graph identifier (IRIs) and no labels or other metadata. This is useful for piping the IRIs into other commands. - --filter ... Filter graphs based on effective access conditions or import closure. First parameter CHOICE can be 'access' or 'imported-by'. The @@ -118,12 +117,10 @@ In case of file export, data from all selected graphs will be concatenated in on --include-imports Export selected graph(s) and all graphs which are imported from these selected graph(s). - --create-catalog In addition to the .ttl and .graph files, cmemc will create an XML catalog file (catalog-v001.xml) which can be used by applications such as Protégé. - --output-dir DIRECTORY Export to this directory. --output-file FILE Export to this file. [default: -] -t, --filename-template TEXT Template for the export file name(s). Used @@ -135,7 +132,6 @@ In case of file export, data from all selected graphs will be concatenated in on the current date as YYYY-MM-DD. The file suffix will be appended. Needed directories will be created. [default: {{hash}}] - --mime-type [application/n-triples|text/turtle] Define the requested mime type [default: application/n-triples] @@ -183,7 +179,6 @@ If input is a file, content will be uploaded to IRI. If `--replace` is set, the --replace Replace / overwrite the graph - instead of just adding new triples the graph. - --skip-existing Skip importing a file if the target graph already exists in the store. Note that the graph list is fetched once at the beginning of the process, so that you can still add diff --git a/docs/automate/cmemc-command-line-interface/command-reference/index.md b/docs/automate/cmemc-command-line-interface/command-reference/index.md index 64db829f..ae25d4b3 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/index.md @@ -27,7 +27,7 @@ tags: | [admin metrics](admin/metrics/index.md) | [inspect](admin/metrics/index.md#admin-metrics-inspect) | Inspect a metric. | | [admin metrics](admin/metrics/index.md) | [list](admin/metrics/index.md#admin-metrics-list) | List metrics for a specific job. | | [admin store](admin/store/index.md) | [showcase](admin/store/index.md#admin-store-showcase) | Create showcase data. | -| [admin store](admin/store/index.md) | [bootstrap](admin/store/index.md#admin-store-bootstrap) | Update/Import bootstrap data. | +| [admin store](admin/store/index.md) | [bootstrap](admin/store/index.md#admin-store-bootstrap) | Update/Import or remove bootstrap data. | | [admin store](admin/store/index.md) | [export](admin/store/index.md#admin-store-export) | Backup all knowledge graphs to a ZIP archive. | | [admin store](admin/store/index.md) | [import](admin/store/index.md#admin-store-import) | Restore graphs from a ZIP archive. | | [admin user](admin/user/index.md) | [list](admin/user/index.md#admin-user-list) | List user accounts. | @@ -40,9 +40,10 @@ tags: | [admin workspace](admin/workspace/index.md) | [import](admin/workspace/index.md#admin-workspace-import) | Import the workspace from a file. | | [admin workspace](admin/workspace/index.md) | [reload](admin/workspace/index.md#admin-workspace-reload) | Reload the workspace from the backend. | | [admin workspace python](admin/workspace/python/index.md) | [install](admin/workspace/python/index.md#admin-workspace-python-install) | Install a python package to the workspace. | -| [admin workspace python](admin/workspace/python/index.md) | [uninstall](admin/workspace/python/index.md#admin-workspace-python-uninstall) | Uninstall a python package from the workspace. | +| [admin workspace python](admin/workspace/python/index.md) | [uninstall](admin/workspace/python/index.md#admin-workspace-python-uninstall) | Uninstall a python packages from the workspace. | | [admin workspace python](admin/workspace/python/index.md) | [list](admin/workspace/python/index.md#admin-workspace-python-list) | List installed python packages. | | [admin workspace python](admin/workspace/python/index.md) | [list-plugins](admin/workspace/python/index.md#admin-workspace-python-list-plugins) | List installed workspace plugins. | +| [admin workspace python](admin/workspace/python/index.md) | [open](admin/workspace/python/index.md#admin-workspace-python-open) | Open a package pypi.org page in the browser. | | [config](config/index.md) | [list](config/index.md#config-list) | List configured connections. | | [config](config/index.md) | [edit](config/index.md#config-edit) | Edit the user-scope configuration file. | | [config](config/index.md) | [get](config/index.md#config-get) | Get the value of a known cmemc configuration key. | @@ -54,6 +55,7 @@ tags: | [dataset](dataset/index.md) | [inspect](dataset/index.md#dataset-inspect) | Display metadata of a dataset. | | [dataset](dataset/index.md) | [create](dataset/index.md#dataset-create) | Create a dataset. | | [dataset](dataset/index.md) | [open](dataset/index.md#dataset-open) | Open datasets in the browser. | +| [dataset](dataset/index.md) | [update](dataset/index.md#dataset-update) | Update a dataset. | | [dataset resource](dataset/resource/index.md) | [list](dataset/resource/index.md#dataset-resource-list) | List available file resources. | | [dataset resource](dataset/resource/index.md) | [delete](dataset/resource/index.md#dataset-resource-delete) | Delete file resources. | | [dataset resource](dataset/resource/index.md) | [inspect](dataset/resource/index.md#dataset-resource-inspect) | Display all metadata of a file resource. | @@ -72,6 +74,11 @@ tags: | [project](project/index.md) | [delete](project/index.md#project-delete) | Delete projects. | | [project](project/index.md) | [create](project/index.md#project-create) | Create projects. | | [project](project/index.md) | [reload](project/index.md#project-reload) | Reload projects from the workspace provider. | +| [project variable](project/variable/index.md) | [list](project/variable/index.md#project-variable-list) | List available project variables. | +| [project variable](project/variable/index.md) | [get](project/variable/index.md#project-variable-get) | Get the value or other data of a project variable. | +| [project variable](project/variable/index.md) | [delete](project/variable/index.md#project-variable-delete) | Delete a project variable. | +| [project variable](project/variable/index.md) | [create](project/variable/index.md#project-variable-create) | Create a new project variable. | +| [project variable](project/variable/index.md) | [update](project/variable/index.md#project-variable-update) | Update data of an existing project variable. | | [query](query/index.md) | [execute](query/index.md#query-execute) | Execute queries which are loaded from files or the query catalog. | | [query](query/index.md) | [list](query/index.md#query-list) | List available queries from the catalog. | | [query](query/index.md) | [open](query/index.md#query-open) | Open queries in the editor of the query catalog in your browser. | diff --git a/docs/automate/cmemc-command-line-interface/command-reference/project/index.md b/docs/automate/cmemc-command-line-interface/command-reference/project/index.md index 6acafda6..deae7a54 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/project/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/project/index.md @@ -90,16 +90,13 @@ $ cmemc config list | parallel -I% cmemc -c % project export --all -t "dump/{{co -a, --all Export all projects. -o, --overwrite Overwrite existing files. This is a dangerous option, so use it with care. - --output-dir DIRECTORY The base directory, where the project files will be created. If this directory does not exist, it will be silently created. [default: .] - --type TEXT Type of the exported project file(s). Use the --help-types option or tab completion to see a list of possible types. [default: xmlZip] - -t, --filename-template TEXT Template for the export file name(s). Possible placeholders are (Jinja2): {{id}} (the project ID), {{connection}} (from the --connection @@ -107,7 +104,6 @@ $ cmemc config list | parallel -I% cmemc -c % project export --all -t "dump/{{co YYYY-MM-DD). The file suffix will be appended. Needed directories will be created. [default: {{date}}-{{connection}}-{{id}}.project] - --extract Export projects to a directory structure instead of a ZIP archive. Note that the --filename-template option is ignored here. @@ -115,7 +111,6 @@ $ cmemc config list | parallel -I% cmemc -c % project export --all -t "dump/{{co is created under the output directory. Also note that not all export types are extractable. - --help-types Lists all possible export types. ``` @@ -201,6 +196,12 @@ This command creates one or more new projects. Existing projects will not be ove when using the mapping suggestion of a transformation task. You need the task ID of the transformation task. + --label TEXT Give the label of the project. You can give more + than one label if you create more than one + project. + --description TEXT Give the description of the project. You can + give more than one description if you create + more than one project. ``` ## project reload diff --git a/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md b/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md new file mode 100644 index 00000000..08369a03 --- /dev/null +++ b/docs/automate/cmemc-command-line-interface/command-reference/project/variable/index.md @@ -0,0 +1,159 @@ +--- +title: "cmemc: Command Group - project variable" +description: "List, create, delete or get data from project variables." +icon: material/variable-box +tags: + - Variables + - cmemc +--- +# project variable Command Group + + +List, create, delete or get data from project variables. + +Project variables can be used in dataset and task parameters, and in the template transform operator. Variables are either based on a static value or based on a template. They may use templates that access globally configured variables or other preceding variables from the same project. + +Variables are identified by a `VARIABLE_ID`. To get a list of existing variables, execute the list command or use tab-completion. The `VARIABLE_ID` is a concatenation of a `PROJECT_ID` and a `VARIABLE_NAME`, such as `my-project:my-variable`. + + +## project variable list + +List available project variables. + +```shell-session title="Usage" +$ cmemc project variable list [OPTIONS] +``` + + + + +Outputs a table or a list of project variables. + + + +??? info "Options" + ```text + + --raw Outputs raw JSON. + --id-only Lists only variables names and no other metadata. + This is useful for piping the IDs into other + commands. + --filter ... Filter variables based on metadata. First parameter + CHOICE can be one of ['project', 'regex']. The + second parameter is based on CHOICE, e.g. a project + ID or a regular expression string. + ``` + +## project variable get + +Get the value or other data of a project variable. + +```shell-session title="Usage" +$ cmemc project variable get [OPTIONS] VARIABLE_ID +``` + + + + +Use the ``--key`` option to specify which information you want to get. + +!!! note + Only the `value` key is always available on a project variable. Static value variables have no `template` key, and the `description` key is optional for both types of variables. + + + + +??? info "Options" + ```text + + --key [value|template|description] + Specify the name of the value you want to + get. [default: value] + --raw Outputs raw json. + ``` + +## project variable delete + +Delete a project variable. + +```shell-session title="Usage" +$ cmemc project variable delete VARIABLE_ID +``` + + + + +!!! note + You can not delete a variable which is used by another (template based) variable. In order to do so, delete the template based variable first. + + + + +## project variable create + +Create a new project variable. + +```shell-session title="Usage" +$ cmemc project variable create [OPTIONS] VARIABLE_NAME +``` + + + + +Variables need to be created with a value or a template (not both). In addition to that, a project ID and a name are mandatory. + +```shell-session title="Example" +$ cmemc project variable create my_var --project my_project --value abc +``` + + +!!! note + cmemc is currently not able to manage the order of the variables in a project. This means you have to create plain value variables in advance, before you can create template based variables, which access these values. + + + + +??? info "Options" + ```text + + --value TEXT The value of the new project variable. + --template TEXT The template of the new project variable. You can use + Jinja template syntax, e.g. use '{{global.myVar}}' for + accessing global variables, or '{{project.myVar}}' for + accessing variables from the same project. + --description TEXT The optional description of the new project variable. + --project TEXT The project, where you want to create the variable in. + If there is only one project in the workspace, this + option can be omitted. + ``` + +## project variable update + +Update data of an existing project variable. + +```shell-session title="Usage" +$ cmemc project variable update [OPTIONS] VARIABLE_ID +``` + + + + +With this command you can update the value or the template, as well as the description of a project variable. + +!!! note + If you update the template of a static variable, it will be transformed to a template based variable. If you want to change the value of a template based variable, an error will be shown. + + + + +??? info "Options" + ```text + + --value TEXT The new value of the project variable. + --template TEXT The new template of the project variable. You can use + Jinja template syntax, e.g. use '{{global.myVar}}' for + accessing global variables, or '{{project.myVar}}' for + accessing variables from the same project. + --description TEXT The new description of the project variable. + ``` + diff --git a/docs/automate/cmemc-command-line-interface/command-reference/query/index.md b/docs/automate/cmemc-command-line-interface/command-reference/query/index.md index 78cbecf0..ca5032b8 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/query/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/query/index.md @@ -52,19 +52,16 @@ Limitations: All optional parameters (e.g. accept, base64, ...) are provided for refer to the Corporate Memory system manual for a list of accepted mime types. [default: default] - --no-imports Graphs which include other graphs (using owl:imports) will be queried as merged overall-graph. This flag disables this default behaviour. The flag has no effect on update queries. - --base64 Enables base64 encoding of the query parameter for the SPARQL requests (the response is not touched). This can be useful in case there is an aggressive firewall between cmemc and Corporate Memory. - -p, --parameter ... In case of a parameterized query (placeholders with the '{{key}}' syntax), this option fills all placeholder with a @@ -72,18 +69,14 @@ Limitations: All optional parameters (e.g. accept, base64, ...) are provided for executed.Pairs of placeholder/value need to be given as a tuple 'KEY VALUE'. A key can be used only once. - --limit INTEGER Override or set the LIMIT in the executed SELECT query. Note that this option will never give you more results than the LIMIT given in the query itself. - --offset INTEGER Override or set the OFFSET in the executed SELECT query. - --distinct Override the SELECT query by make the result set DISTINCT. - --timeout INTEGER Set max execution time for query evaluation (in milliseconds). ``` @@ -150,7 +143,6 @@ You can filter queries based on status and runtime in order to investigate slow --id-only Lists only query identifier and no labels or other metadata. This is useful for piping the ids into other cmemc commands. - --raw Outputs raw JSON response of the query status API. --filter ... Filter queries based on execution status and time. First parameter --filter CHOICE can be one of @@ -193,13 +185,11 @@ The optional output file is the same JSON document which is used as input, but e --loops INTEGER Number of loops to run the replay file. [default: 1] --wait INTEGER Number of seconds to wait between query executions. [default: 0] - --output-file FILE Save the optional output to this file. Input and output of the command can be the same file. The output is written at the end of a successful command execution. The output can be stdout ('-') - in this case, the execution statistic output is oppressed. - --run-label TEXT Optional label of this replay run. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/cache/index.md b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/cache/index.md index 93539e3c..fa503b08 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/cache/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/cache/index.md @@ -48,7 +48,6 @@ $ cmemc vocabulary cache list [OPTIONS] --id-only Lists only vocabulary term identifier (IRIs) and no labels or other metadata. This is useful for piping the ids into other cmemc commands. - --raw Outputs raw JSON. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md index 805daa30..c59a1bd6 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/vocabulary/index.md @@ -49,11 +49,9 @@ Vocabularies are graphs (see `graph` command group) which consists of class and no labels or other metadata. This is useful for piping the ids into other cmemc commands. - --filter [all|installed|installable] Filter list based on status. [default: installed] - --raw Outputs raw JSON. ``` @@ -124,7 +122,6 @@ The uploaded ontology file is analysed locally in order to discover the named gr manually add a namespace prefix with this option. Example: --namespace ex https://example.org/ - --replace Replace (overwrite) existing vocabulary, if present. ``` diff --git a/docs/automate/cmemc-command-line-interface/command-reference/workflow/index.md b/docs/automate/cmemc-command-line-interface/command-reference/workflow/index.md index fafa5d28..bfa928dd 100644 --- a/docs/automate/cmemc-command-line-interface/command-reference/workflow/index.md +++ b/docs/automate/cmemc-command-line-interface/command-reference/workflow/index.md @@ -37,14 +37,14 @@ The optional `--wait` option starts the workflows in the same way, but also poll ```text -a, --all Execute all available workflows. - --wait Wait until all executed workflows are - completed. - + --wait Wait until workflows are completed. + --progress Wait until workflows are completed and show + a progress bar. --polling-interval INTEGER RANGE How many seconds to wait between status polls. Status polls are cheap, so a higher polling interval is most likely not needed. - [default: 1] + [default: 1; 0<=x<=60] ``` ## workflow io @@ -68,7 +68,6 @@ With this command, you can execute a workflow that uses variable datasets as inp -i, --input FILE From which file the input is taken. If the workflow has no defined variable input dataset, this option is not allowed. - -o, --output FILE To which file the result is written to. Use '-' in order to output the result to stdout. If the workflow has no defined variable @@ -76,24 +75,21 @@ With this command, you can execute a workflow that uses variable datasets as inp Please note that the io command will not warn you on overwriting existing output files. - --input-mimetype [application/x-plugin-csv|application/x-plugin-json|application/xml|application/x-plugin-excel|application/octet-stream|application/x-plugin-multiCsv|text/plain|guess] Which input format should be processed: If not given, cmemc will try to guess the mime type based on the file extension or will fail. - --output-mimetype [application/x-plugin-csv|application/x-plugin-excel|application/n-triples|application/n-triples|application/x-plugin-json|application/xml|guess] Which output format should be requested: If not given, cmemc will try to guess the mime type based on the file extension or will fail. In case of an output to stdout, a default mime type will be used (JSON). - --autoconfig / --no-autoconfig Setup auto configuration of input datasets, e.g. in order to process CSV files with semicolon- instead of comma-separation. - [default: True] + [default: autoconfig] ``` ## workflow list @@ -115,11 +111,9 @@ $ cmemc workflow list [OPTIONS] --filter CHOICE can be one of ['io', 'project', 'regex', 'tag']. The second parameter is based on CHOICE. - --id-only Lists only workflow identifier and no labels or other metadata. This is useful for piping the IDs into other commands. - --raw Outputs raw JSON objects of workflow task search API response. ``` @@ -142,7 +136,6 @@ $ cmemc workflow status [OPTIONS] [WORKFLOW_IDS]... --project TEXT The project, from which you want to list the workflows. Project IDs can be listed with the 'project list' command. - --raw Output raw JSON info. --filter [Idle|Not executed|Finished|Cancelled|Failed|Successful|Canceling|Running|Waiting] Show only workflows of a specific status. diff --git a/docs/build/.pages b/docs/build/.pages index b342c484..4777ff63 100644 --- a/docs/build/.pages +++ b/docs/build/.pages @@ -12,5 +12,5 @@ nav: - Connect to Snowflake: snowflake-tutorial - Active learning: active-learning - Link Intrusion Detection Systems to Open-Source INTelligence: tutorial-how-to-link-ids-to-osint - - Project and Global Varibles: di-variables + - Project and Global Variables: variables - Evaluate Template Operator: evaluate-template diff --git a/docs/build/di-variables/di-var-add-variable.png b/docs/build/variables/di-var-add-variable.png similarity index 100% rename from docs/build/di-variables/di-var-add-variable.png rename to docs/build/variables/di-var-add-variable.png diff --git a/docs/build/di-variables/di-var-email-defined.png b/docs/build/variables/di-var-email-defined.png similarity index 100% rename from docs/build/di-variables/di-var-email-defined.png rename to docs/build/variables/di-var-email-defined.png diff --git a/docs/build/di-variables/di-var-email-definition.png b/docs/build/variables/di-var-email-definition.png similarity index 100% rename from docs/build/di-variables/di-var-email-definition.png rename to docs/build/variables/di-var-email-definition.png diff --git a/docs/build/di-variables/di-var-email-usage-2.png b/docs/build/variables/di-var-email-usage-2.png similarity index 100% rename from docs/build/di-variables/di-var-email-usage-2.png rename to docs/build/variables/di-var-email-usage-2.png diff --git a/docs/build/di-variables/di-var-email-usage.png b/docs/build/variables/di-var-email-usage.png similarity index 100% rename from docs/build/di-variables/di-var-email-usage.png rename to docs/build/variables/di-var-email-usage.png diff --git a/docs/build/di-variables/di-var-password-definition.png b/docs/build/variables/di-var-password-definition.png similarity index 100% rename from docs/build/di-variables/di-var-password-definition.png rename to docs/build/variables/di-var-password-definition.png diff --git a/docs/build/di-variables/di-var-password-usage.png b/docs/build/variables/di-var-password-usage.png similarity index 100% rename from docs/build/di-variables/di-var-password-usage.png rename to docs/build/variables/di-var-password-usage.png diff --git a/docs/build/di-variables/di-var-project-select.png b/docs/build/variables/di-var-project-select.png similarity index 100% rename from docs/build/di-variables/di-var-project-select.png rename to docs/build/variables/di-var-project-select.png diff --git a/docs/build/di-variables/di-var-value-evaluation.png b/docs/build/variables/di-var-value-evaluation.png similarity index 100% rename from docs/build/di-variables/di-var-value-evaluation.png rename to docs/build/variables/di-var-value-evaluation.png diff --git a/docs/build/di-variables/index.md b/docs/build/variables/index.md similarity index 91% rename from docs/build/di-variables/index.md rename to docs/build/variables/index.md index 9553ae81..5651e894 100644 --- a/docs/build/di-variables/index.md +++ b/docs/build/variables/index.md @@ -2,12 +2,13 @@ icon: material/variable-box tags: - KnowledgeGraph + - Variables --- # Build Variables ## Introduction -Build variables are used to configure and customize build workflow and processes. +Build variables are used to configure and customize build workflows and processes. These variables define various aspects of the integration tasks, such as the source target data formats, transformation rules, mapping definitions etc. The variables are not technically typed. They can be used in most Build configuration and input fields that take inputs of the following data types: @@ -18,11 +19,11 @@ They can be used in most Build configuration and input fields that take inputs o Two kinds of variables can be defined: -**Global variables** +**Global variables** : It is defined by the administrator in the configuration file at deployment time and cannot be set by a normal user. -**Project variables (User-defined)** +**Project variables (User-defined)** : It is defined by the user in the UI. Project variables can only be used in the same project. @@ -82,9 +83,7 @@ Which is described in the following sections. ## Project Variables -### Adding Variables - -Login to eccenca Corporate Memory, select the build module and click on the project to open. +In order to add project variables, login to eccenca Corporate Memory, select the build module and click on the project to open. ![Build projects overview](di-var-project-select.png){ class="bordered" } @@ -116,7 +115,7 @@ Type name as `email_ids`, in values we have updated all the email id’s of the ![](di-var-email-defined.png){ class="bordered" } -### Using Variables +## Using Variables Let's see how these variables are useful. @@ -166,3 +165,10 @@ Click on the symbol **{#}** it turns blue in color. It means the variable's feat !!! note Parameters that are typed as password will not show the evaluated template for security reasons and should only show after you saved the operator. + +## Accessing Variables with cmemc + +In order to allow the automation of activities with build variables from external processes, the Corporate Memory command line interfaces cmemc has a dedicated [`project variable` command group](../../automate/cmemc-command-line-interface/command-reference/project/variable/index.md) for this. + +Please have a look at command group documentation to learn how to use these commands. + diff --git a/mkdocs.yml b/mkdocs.yml index 7c20f894..d90962d5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -56,6 +56,7 @@ extra: ExpertTutorial: expert Project: project Keycloak: keycloak + Variables: variables # https://squidfunk.github.io/mkdocs-material/setup/setting-up-versioning/ version: provider: mike @@ -176,6 +177,7 @@ theme: expert: material/list-status project: eccenca/artefact-project keycloak: material/openid + variables: material/variable-box # https://squidfunk.github.io/mkdocs-material/reference/annotations/ admonition: note: fontawesome/solid/note-sticky