StreamPark FAQ #507

xinzhuxiansheng · 2021-12-09T16:13:56Z

StreamPark ── A magical framework make flink&spark easier!

FAQ

Here is a compilation of frequently mentioned popular issues based on user feedback. If you have a new question, please submit issue . Please do not ask your question here. This is not a question area.

这里记录总结了用户反馈较多的热门问题, 如果你有新的问题,请提issue ,不要在这里提问. 不要在这里提问. 不要在这里提问. 这里不是提问区.

xinzhuxiansheng · 2022-01-12T15:37:20Z

1. maven install error,Failed to run task: 'npm install' failed？

because the front end uses nodejs, make sure that nodejs is installed on the compiling machine when compiling, and make sure that the nodejs version is not too old. You can enter streamx-console-webapp and manually execute the cmd to try to compile: npm install, if still If it fails, you can check the information related to nodejs compilation by yourself and try to solve this problem by yourself

xinzhuxiansheng · 2022-01-12T15:40:57Z

Caused: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

配置文件缺失metastore.uri，添加上即可, 请参考#219

xinzhuxiansheng · 2022-01-12T15:41:38Z

java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory

java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:187)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9)
	at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228)
	at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory
	at com.github.dockerjava.transport.DomainSocket.<init>(DomainSocket.java:63)
	at com.github.dockerjava.transport.BsdDomainSocket.<init>(BsdDomainSocket.java:43)
	at com.github.dockerjava.transport.DomainSocket.get(DomainSocket.java:138)
	at com.github.dockerjava.transport.UnixSocket.get(UnixSocket.java:27)
	at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl$2.createSocket(ApacheDockerHttpClientImpl.java:145)
	at org.apache.hc.client5.http.impl.io.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:125)
	at org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:407)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:168)
	at org.apache.hc.client5.http.impl.classic.InternalExecRuntime.connectEndpoint(InternalExecRuntime.java:178)
	at org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:136)

Check if Docker is started

wolfboys · 2022-01-12T23:55:51Z

log4j conflict #571

methods of resolution

Dependency hierarchy, see where the log conflicts are

fix udf log4j conflict

Requirement or improvement（诉求 & 改进建议）

add java parameter
Start the job of Flink job add： -Dlog4j.ignoreTC=true

wolfboys · 2022-01-12T23:56:41Z

Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath

2021-12-24T04:43:28.901627819Z Caused by: org.apache.flink.table.api.NoMatchingTableFactoryException: Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in
2021-12-24T04:43:28.901630469Z the classpath.
2021-12-24T04:43:28.901632969Z
2021-12-24T04:43:28.901635342Z Reason: Required context properties mismatch.
2021-12-24T04:43:28.901637804Z
2021-12-24T04:43:28.901640154Z The matching candidates:
2021-12-24T04:43:28.901642578Z org.apache.flink.table.sources.CsvAppendTableSourceFactory
2021-12-24T04:43:28.901645059Z Mismatched properties:
2021-12-24T04:43:28.901647812Z 'connector.type' expects 'filesystem', but is 'kafka'
2021-12-24T04:43:28.901650202Z 'format.type' expects 'csv', but is 'json'

Flink version: 1.14.0

解决方案：flink-kafka-connector 的使用参数不对,请参考flink官网：

CREATE TABLE user_log (
user_id VARCHAR,
item_id VARCHAR,
category_id VARCHAR,
behavior VARCHAR,
ts TIMESTAMP(3)
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = 'kafka-1:9092,kafka-2:9092,kafka-3:9092',
'properties.group.id' = 'testGroup',
'scan.startup.mode' = 'earliest-offset',
'format' = 'json'
);

CREATE TABLE pvuv_sink (
dt VARCHAR primary key,
pv BIGINT,
uv BIGINT
) WITH (
'connector' = 'jdbc', -- 使用 jdbc connector
'url' = 'jdbc:mysql://test-mysql:3306/test', -- jdbc url
'table-name' = 'pvuv_sink', -- 表名
'username' = 'root', -- 用户名
'password' = '123456' -- 密码
);

INSERT INTO pvuv_sink
SELECT
DATE_FORMAT(ts, 'yyyy-MM-dd HH:00') dt,
COUNT(*) AS pv,
COUNT(DISTINCT user_id) AS uv
FROM user_log
GROUP BY DATE_FORMAT(ts, 'yyyy-MM-dd HH:00');

另外kafka的消息格式

{"user_id": "543462", "item_id":"1715", "category_id": "1464116", "behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01T01:00:00Z"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "learning flink", "ts":"2021-02-01T01:00:00Z"}

要修改为

{"user_id": "543462", "item_id":"1715", "category_id": "1464116", "behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "pv", "ts":"2021-02-01 01:00:00"}
{"user_id": "662867", "item_id":"2244074","category_id":"1575622","behavior": "learning flink", "ts":"2021-02-01 01:00:00"}

否则日志解析失败。

wolfboys · 2022-01-12T23:57:24Z

window idea environment Could not submit Flink job to remote yarn cluster

动态修改提交job的入口参数，入口点 YarnClientImpl类 , 方法 submitApplication，提交点 this.rmClient.submitApplication(request); 对入参 request的 CLASSPATH and _FLINK_CLASSPATH 参数值的分隔符 windows is ";" 替换为 linux is ":"

这个问题应该和官网上报的是一个问题
https://issues.apache.org/jira/browse/FLINK-17858

bulolo · 2022-12-14T07:35:46Z

请问 2.0.0版本 docker 部署 .env是引用已有mysql,那么数据库 streampark数据库是要手工导入sql吗？还是在docker首次启动的时候会自己创建？

0akarma · 2022-12-28T10:44:46Z

请问 2.0.0版本 docker 部署 .env是引用已有mysql,那么数据库 streampark数据库是要手工导入sql吗？还是在docker首次启动的时候会自己创建？

得自己进入mysql容器，执行sql，默认不会初始化

wolfboys · 2023-01-12T02:04:48Z

请问streampark 集成 flinkcdc不，根据日志实时同步功能支持不?

支持, 不论是datastream写的flinkcdc同步的作业还是flinksql 作业都支持, 只要是一个标准的flink作业都支持, 如果是flink sql作业的话, connector 必须是按照flink的规范实现的标准的 flink sql connector, 引入对应的依赖jar或者pom即可.

2000liux · 2023-01-17T10:25:18Z

compile using this command : mvn clean install -DskipTests -Dcheckstyle.skip -Dmaven.javadoc.skip=true

changeme2012 · 2023-06-05T05:32:00Z

提交flink sql的任务运行失败，找不到失败原因

Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.streampark.flink.client.FlinkClient$.$anonfun$proxy$1(FlinkClient.scala:80)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.$anonfun$proxy$1(FlinkShimsProxy.scala:60)
at org.apache.streampark.common.util.ClassLoaderUtils$.runAsClassLoader(ClassLoaderUtils.scala:38)
at org.apache.streampark.flink.proxy.FlinkShimsProxy$.proxy(FlinkShimsProxy.scala:60)
at org.apache.streampark.flink.client.FlinkClient$.proxy(FlinkClient.scala:75)
at org.apache.streampark.flink.client.FlinkClient$.submit(FlinkClient.scala:49)
at org.apache.streampark.flink.client.FlinkClient.submit(FlinkClient.scala)
at org.apache.streampark.console.core.service.impl.ApplicationServiceImpl.lambda$start$10(ApplicationServiceImpl.java:1544)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
... 3 more
Caused by: java.lang.NoSuchFieldError: CANCEL_ENABLE
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit(FlinkClientTrait.scala:102)
at org.apache.streampark.flink.client.trait.FlinkClientTrait.submit$(FlinkClientTrait.scala:63)
at org.apache.streampark.flink.client.impl.YarnApplicationClient$.submit(YarnApplicationClient.scala:44)
at org.apache.streampark.flink.client.FlinkClientHandler$.submit(FlinkClientHandler.scala:40)
at org.apache.streampark.flink.client.FlinkClientHandler.submit(FlinkClientHandler.scala)

Upgrading the version to 2.1.1

3yekn1 · 2023-09-04T03:18:28Z

可以通过修改依赖为 provided解决。
是否还有其他手段解决，例如fat jar?
为什么修改child-first 与 parent-first选项未生效？

caicancai · 2023-09-04T08:37:21Z

An error is reported when the source code is compiled

At present, you can comment these two files to compile, then uncomment and compile again.You can try it.

liyichencc · 2023-09-06T02:19:55Z

after streamx-console started, app.home is not set, and throw NullPointerException

streamx-console initialization check failed. If started local for development and debugging, please ensure the -Dapp.home parameter is clearly specified in vm options, more detail: http://www.streamxhub.com/docs/user-guide/development/#vm-options

http://www.streamxhub.com/docs/user-guide/development/#vm-options Link has expired,Now you can refer to this address:https://streampark.apache.org/zh-CN/docs/user-guide/deployment/

wangyg007 · 2024-09-14T07:18:23Z

dev分支编译本地部署，前端spark applications点击add new报404

xinzhuxiansheng added the question Further information is requested label Dec 9, 2021

wolfboys pinned this issue Dec 10, 2021

wolfboys mentioned this issue Dec 12, 2021

StreamXConsole 启动的时候空指针异常 #515

Closed

xinzhuxiansheng mentioned this issue Dec 30, 2021

[Bug] java.sql.SQLSyntaxErrorException: Table 'streamx.t_setting' doesn't exist #569

Closed

wolfboys unpinned this issue Jan 12, 2022

This comment was marked as resolved.

Sign in to view

apache deleted a comment from yurx2008 Jan 12, 2022

wolfboys mentioned this issue Apr 17, 2022

【开源自荐】StreamX ─ Flink极速开发框架,一站式流数据处理平台 521xueweihan/HelloGitHub#2166

Closed

wolfboys pinned this issue May 31, 2022

wolfboys changed the title ~~StreamX FAQ~~ StreamPark FAQ Sep 2, 2022

apache deleted a comment from caosl Sep 19, 2022

apache deleted a comment from fantasticKe Sep 19, 2022

wolfboys mentioned this issue Sep 26, 2022

[Bug]Error occurs when the session mode is started #1693

Open

3 tasks

apache deleted a comment from midoll3 Dec 29, 2022

apache deleted a comment from douxiaofeng99 Jan 9, 2023

apache deleted a comment from wanghuan2054 Jan 9, 2023

apache deleted a comment from Yougetadad Jan 9, 2023

apache deleted a comment from Su1024 Jan 9, 2023

apache deleted a comment from moranyuwen Jan 10, 2023

caicancai mentioned this issue Sep 4, 2023

[Bug] An error is reported when the source code is compiled #2960

Closed

3 tasks

apache deleted a comment from bluesky2535 Sep 13, 2023

apache deleted a comment from cyrusmaster Sep 13, 2023

apache deleted a comment from westPointz Sep 13, 2023

apache deleted a comment from moranyuwen Mar 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

StreamPark FAQ #507

StreamPark FAQ #507

xinzhuxiansheng commented Dec 9, 2021 •

edited by wolfboys

Loading

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

`Caused: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient`

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

`java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory`

wolfboys commented Jan 12, 2022

log4j conflict #571

wolfboys commented Jan 12, 2022

`Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath`

wolfboys commented Jan 12, 2022 •

edited

Loading

window idea environment Could not submit Flink job to remote yarn cluster

bulolo commented Dec 14, 2022 •

edited

Loading

0akarma commented Dec 28, 2022

wolfboys commented Jan 12, 2023

请问streampark 集成 flinkcdc不，根据日志实时同步功能支持不?

2000liux commented Jan 17, 2023

changeme2012 commented Jun 5, 2023 •

edited by wolfboys

Loading

3yekn1 commented Sep 4, 2023

caicancai commented Sep 4, 2023 •

edited

Loading

liyichencc commented Sep 6, 2023 •

edited

Loading

after streamx-console started, app.home is not set, and throw NullPointerException

wangyg007 commented Sep 14, 2024

StreamPark FAQ #507

StreamPark FAQ #507

Comments

xinzhuxiansheng commented Dec 9, 2021 • edited by wolfboys Loading

FAQ

xinzhuxiansheng commented Jan 12, 2022 • edited by wolfboys Loading

1. maven install error,Failed to run task: 'npm install' failed？

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

xinzhuxiansheng commented Jan 12, 2022 • edited by wolfboys Loading

Caused: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

xinzhuxiansheng commented Jan 12, 2022 • edited by wolfboys Loading

java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory

wolfboys commented Jan 12, 2022

log4j conflict #571

methods of resolution

Requirement or improvement（诉求 & 改进建议）

wolfboys commented Jan 12, 2022

Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath

wolfboys commented Jan 12, 2022 • edited Loading

window idea environment Could not submit Flink job to remote yarn cluster

bulolo commented Dec 14, 2022 • edited Loading

0akarma commented Dec 28, 2022

wolfboys commented Jan 12, 2023

请问streampark 集成 flinkcdc不，根据日志实时同步功能支持不?

2000liux commented Jan 17, 2023

changeme2012 commented Jun 5, 2023 • edited by wolfboys Loading

3yekn1 commented Sep 4, 2023

caicancai commented Sep 4, 2023 • edited Loading

liyichencc commented Sep 6, 2023 • edited Loading

after streamx-console started, app.home is not set, and throw NullPointerException

wangyg007 commented Sep 14, 2024

xinzhuxiansheng commented Dec 9, 2021 •

edited by wolfboys

Loading

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

`Caused: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient`

xinzhuxiansheng commented Jan 12, 2022 •

edited by wolfboys

Loading

`java.lang.RuntimeException: java.io.IOException: com.sun.jna.LastErrorException: [2] No such file or directory`

`Could not find a suitable table factory for 'org.apache.flink.table.factories.TableSourceFactory' in the classpath`

wolfboys commented Jan 12, 2022 •

edited

Loading

bulolo commented Dec 14, 2022 •

edited

Loading

changeme2012 commented Jun 5, 2023 •

edited by wolfboys

Loading

caicancai commented Sep 4, 2023 •

edited

Loading

liyichencc commented Sep 6, 2023 •

edited

Loading