-
Notifications
You must be signed in to change notification settings - Fork 112
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add hbase engine docs (https://github.com/apache/linkis/pull/4891) #758
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
--- | ||
title: HBase Engine | ||
sidebar_position: 15 | ||
--- | ||
|
||
# HBase engine usage documentation | ||
|
||
This article mainly introduces the installation, use and configuration of the `hbase` engine plugin in `Linkis`. | ||
|
||
## 1. Preliminary work | ||
|
||
### 1.1 Engine Verification | ||
|
||
It is strongly recommended that you check these environment variables for the executing user before executing `hbase` tasks. The specific way is | ||
|
||
``` | ||
sudo su - ${username} | ||
echo ${JAVA_HOME} | ||
``` | ||
|
||
## 2. Engine plugin installation | ||
|
||
### 2.1 Engine plugin preparation [non-default engine](./overview.md) | ||
|
||
Method 2: Compile the engine plug-in separately (requires a `maven` environment) | ||
|
||
``` | ||
# compile | ||
cd ${linkis_code_dir}/linkis-engineconn-plugins/hbase/ | ||
mvn clean install -DskipTests -Dhbase.profile=1.2|1.4|2.2|2.5 | ||
|
||
# hbase.profile 1.2|1.4|2.2|2.5, The corresponding HBase version is: 1.2.0, 1.4.3, 2.2.6, 2.5.3 | ||
# default hbase.profile=2.5, Compile plug-ins for different HBase versions on demand | ||
# The compiled engine plug-in package is located in the following directory | ||
${linkis_code_dir}/linkis-engineconn-plugins/hbase/target/out/ | ||
``` | ||
|
||
[EngineConnPlugin engine plugin installation](../deployment/install-engineconn.md) | ||
|
||
### 2.2 Upload and load engine plugins | ||
|
||
Upload the engine plug-in package in 2.1 to the engine directory of the server | ||
|
||
```bash | ||
${LINKIS_HOME}/lib/linkis-engineplugins | ||
``` | ||
The directory structure after uploading is as follows | ||
``` | ||
linkis-engineconn-plugins/ | ||
├── hbase | ||
│ ├── dist | ||
│ │ └── 2.5.3 | ||
│ │ ├── conf | ||
│ │ └── lib | ||
│ └── plugin | ||
│ └── 2.5.3 | ||
``` | ||
### 2.3 Engine refresh | ||
|
||
#### 2.3.1 Restart and refresh | ||
Refresh the engine by restarting the `linkis-cg-linkismanager` service | ||
```bash | ||
cd ${LINKIS_HOME}/sbin | ||
sh linkis-daemon.sh restart cg-linkismanager | ||
``` | ||
|
||
### 2.3.2 Check if the engine is refreshed successfully | ||
You can check whether the `last_update_time` of this table in the `linkis_engine_conn_plugin_bml_resources` in the database is the time when the refresh is triggered. | ||
|
||
```sql | ||
#Login to the linkis database | ||
select * from linkis_cg_engine_conn_plugin_bml_resources; | ||
``` | ||
|
||
|
||
## 3. Use of HBase engine | ||
|
||
The `HBase` engine of `Linkis` runs a Jruby environment in the background to execute instructions in hbase-shell, | ||
and is fully compatible with hbase-shell instructions. | ||
|
||
|
||
### 3.1 Submit tasks through `Linkis-cli` | ||
|
||
```shell | ||
sh bin/linkis-cli -engineType hbase-2.5.3 -code "list" \ | ||
-codeType shell -runtimeMap linkis.hbase.zookeeper.quorum=zk1,zk2,zk3 | ||
``` | ||
|
||
More `Linkis-Cli` command parameter reference: [`Linkis-Cli` usage](../user-guide/linkiscli-manual.md) | ||
|
||
HBase engine supports connection parameters: | ||
|
||
| Configuration name | Remarks and default value | Is it necessary | | ||
| --- | --- | --- | | ||
| linkis.hbase.zookeeper.quorum | ZooKeeper address for connecting to HBase, e.g. zk1,zk2,zk3, default value: localhost | Required | | ||
| linkis.hbase.zookeeper.property.clientPort | ZooKeeper port for connecting to HBase, default value: 2181 | Optional, usually the default value | | ||
| linkis.zookeeper.znode.parent | ZooKeeper znode path for connecting to HBase, default value: /hbase | Optional, usually the default value | | ||
| linkis.hbase.rootdir | HDFS root directory for the HBase cluster, default value: /hbase | Optional, usually the default value | | ||
| linkis.hbase.security.authentication | Authentication method for connecting to the HBase cluster: simple or kerberos, default value: simple | Optional, depending on the cluster configuration | | ||
| linkis.hbase.kerberos.principal | HBase Kerberos authentication principal, e.g. [email protected] | Required for Kerberos authentication | | ||
| linkis.hbase.keytab.file | Path to the keytab file required for HBase Kerberos authentication, e.g. /tmp/hbase.keytab | Required for Kerberos authentication | | ||
| linkis.hbase.kerberos.proxy.user | Kerberos proxy user, ensure that the Kerberos authenticated user has the privilege to proxy as a regular user | Optional, can be omitted | | ||
| linkis.hbase.regionserver.kerberos.principal | HBase RegionServer's Kerberos authentication principal, e.g. hbase/[email protected] (obtain from the cluster configuration file) | Required for Kerberos authentication | | ||
| linkis.hbase.master.kerberos.principal | HBase Master's Kerberos authentication principal, e.g. hbase/[email protected] (obtain from the cluster configuration file) | Required for Kerberos authentication | |
101 changes: 101 additions & 0 deletions
101
i18n/zh-CN/docusaurus-plugin-content-docs/current/engine-usage/hbase.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,101 @@ | ||
--- | ||
title: HBase | ||
sidebar_position: 15 | ||
--- | ||
|
||
# HBase 引擎使用文档 | ||
|
||
本文主要介绍在 `Linkis` 中,`HBase` 引擎插件的安装、使用和配置。 | ||
|
||
## 1. 前置工作 | ||
|
||
### 1.1 引擎验证 | ||
|
||
强烈建议您在执行 `HBase` 任务之前,检查下执行用户的这些环境变量。具体方式是 | ||
``` | ||
sudo su - ${username} | ||
echo ${JAVA_HOME} | ||
``` | ||
|
||
## 2. 引擎插件安装 | ||
|
||
### 2.1 引擎插件准备(二选一)[非默认引擎](./overview.md) | ||
|
||
方式一:单独编译引擎插件(需要有 `maven` 环境) | ||
|
||
``` | ||
# 编译 | ||
cd ${linkis_code_dir}/linkis-engineconn-plugins/hbase/ | ||
mvn clean install -DskipTests -Dhbase.profile=1.2|1.4|2.2|2.5 | ||
|
||
# hbase.profile 支持1.2|1.4|2.2|2.5,对应的HBase版本是1.2.0,1.4.3,2.2.6,2.5.3 | ||
# 默认hbase.profile=2.5,按需编译不同HBase版本的插件 | ||
# 编译出来的引擎插件包,位于如下目录中 | ||
${linkis_code_dir}/linkis-engineconn-plugins/hbase/target/out/ | ||
``` | ||
|
||
[EngineConnPlugin 引擎插件安装](../deployment/install-engineconn.md) | ||
|
||
### 2.2 引擎插件的上传和加载 | ||
|
||
将 2.1 中的引擎插件包上传到服务器的引擎目录下 | ||
```bash | ||
${LINKIS_HOME}/lib/linkis-engineplugins | ||
``` | ||
上传后目录结构如下所示 | ||
``` | ||
linkis-engineconn-plugins/ | ||
├── hbase | ||
│ ├── dist | ||
│ │ └── 2.5.3 | ||
│ │ ├── conf | ||
│ │ └── lib | ||
│ └── plugin | ||
│ └── 2.5.3 | ||
``` | ||
### 2.3 引擎刷新 | ||
|
||
#### 2.3.1 重启刷新 | ||
通过重启 `linkis-cg-linkismanager` 服务刷新引擎 | ||
```bash | ||
cd ${LINKIS_HOME}/sbin | ||
sh linkis-daemon.sh restart cg-linkismanager | ||
``` | ||
|
||
### 2.3.2 检查引擎是否刷新成功 | ||
可以查看数据库中的 `linkis_engine_conn_plugin_bml_resources` 这张表的 `last_update_time` 是否为触发刷新的时间。 | ||
|
||
```sql | ||
#登陆到linkis的数据库 | ||
select * from linkis_cg_engine_conn_plugin_bml_resources; | ||
``` | ||
|
||
|
||
## 3. HBase引擎的使用 | ||
|
||
`Linkis` 的 `HBase` 引擎是通过后台运行一个Jruby环境来执行hbase-shell中的指令,完全兼容hbase-shell的指令。 | ||
|
||
|
||
### 3.1 通过 `Linkis-cli` 提交任务 | ||
|
||
```shell | ||
sh bin/linkis-cli -engineType hbase-2.5.3 -code "list" \ | ||
-codeType shell -runtimeMap linkis.hbase.zookeeper.quorum=zk1,zk2,zk3 | ||
``` | ||
|
||
更多 `Linkis-Cli` 命令参数参考: [`Linkis-Cli` 使用](../user-guide/linkiscli-manual.md) | ||
|
||
HBase引擎,支持的连接参数有: | ||
|
||
| 配置名称 | 备注及默认值信息 | 是否必须 | | ||
|-----------------|----------------|----------------------------------------| | ||
| linkis.hbase.zookeeper.quorum | 连接hbase的zk地址,如:zk1,zk2,zk3, 默认值:localhost | 必须 | | ||
| linkis.hbase.zookeeper.property.clientPort | 连接hbase的zk端口,默认值:2181 | 非必须,一般就是默认值 | | ||
| linkis.zookeeper.znode.parent | 连接HBase的zk znode path,默认值:/hbase | 非必须,一般就是默认值 | | ||
| linkis.hbase.rootdir | HBase集群的hdfs根目录,默认值:/hbase | 非必须,一般就是默认值 | | ||
| linkis.hbase.security.authentication | HBase集群连接的认证方式:simple|kerberos,默认值:simple | 非必须,视自己集群情况定 | | ||
| linkis.hbase.kerberos.principal | HBase kerberos认证principal,如:[email protected] | kerberos认证方式下必须 | | ||
| linkis.hbase.keytab.file | HBase kerberos认证所需keytab文件地址,如:/tmp/hbase.keytab | kerberos认证方式下必须 | | ||
| linkis.hbase.kerberos.proxy.user | kerberos代理用户,需保证kerberos认证用户有代理普通用户的权限 | 非必须,可以不用代理用户 | | ||
| linkis.hbase.regionserver.kerberos.principal | 如:hbase/[email protected],自行从集群配置文件中获取 | kerberos认证方式下必须 | | ||
| linkis.hbase.master.kerberos.principal | 如:hbase/[email protected],自行从集群配置文件中获取 | kerberos认证方式下必须 | |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
now can be check at linkis web console