Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][1.5.0] Linkis orchestrator module supports plugin configuration (#5082) #5083

Closed

Conversation

CharlieYan24
Copy link
Contributor

@CharlieYan24 CharlieYan24 commented Feb 7, 2024

What is the purpose of the change

In some scenarios, the Linkis orchestrator module is not required, but the orchestrator does not support plugin configuration.
Linkis orchestrator supports the use of plugins, and it is also possible to bypass Linkis orchestrator.

Related issues/PRs

Related issues: #5082

Brief change log

  • Define linkis.entrance.skip.orchestrator parameter to skip linkis orchestrator when needed;
  • Define CodeLogicalUnitSimpleExecTask class to bypass linkis orchestrator module to execute simple task.

Checklist

  • I have read the Contributing Guidelines on pull requests.
  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the Linkis mailing list first)
  • If this is a code change: I have written unit tests to fully verify the new behavior.

Test Result

Execute hive task:
image

Execute shell task:
image

@peacewong
Copy link
Contributor

ping~ @guoshupei

jobReqBuilder.build().asInstanceOf[ComputationJobReq]
}

def callSimpleExecute(request: ExecuteRequest): ExecuteResponse = {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method lacks log, process, and result response handlers, see EngineExecuteAsyncReturn

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
The key time point is not displayed

@peacewong
Copy link
Contributor

It looks good to me, there are a few points that can be further optimized:

  1. Need to consider the update of log and progress information dealLog/dealProgressWithResource

  2. Whether classes related to org.apache.linkis.orchestrator.ecm can be reused

  3. callSimpleExecute needs to consider returning an asynchronous response and asynchronous notification of task completion.

  4. Need to support retry scenario

  5. 需要考虑日志和进度信息的更新dealLog/dealProgressWithResource

  6. org.apache.linkis.orchestrator.ecm相关的类是否可以复用

  7. callSimpleExecute需要考虑返回异步响应,任务异步通知完成的方式

  8. 需要支持retry场景

@peacewong
Copy link
Contributor

Repeated PR #5120

@peacewong peacewong closed this Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants