Replies: 2 comments 1 reply
-
|
Hi @yanghua, this is friendly ping. I'd like to inquire about the current progress of KPIP-5 and see if there is anything I can do to help. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Just want to check if there is any progress for this topic ? |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
There is a basic UI in branch-1.8 |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Motivation
As a SQL Gateway On Lakehouse framework, Kyuubi is being widely used by many companies. However, so far it still doesn't have a web UI of its own. For a gateway that offers multi-tenancy, having its own Web UI will make it easier to manage the engines it supports. So we're hearing a growing call from the community. That's why I drafted this proposal.
My current company, T3Go, is one of the early practitioners of Kyuubi, and we have customized a web UI for Kyuubi very early on.1 It has played a very important role in our management of Spark SQL Engine for various scenarios. So here we look forward to sharing our experience building Kyuubi Web UI. However, due to some limitations, the technical selection of the Web UI inside T3Go is not very close to the current mainstream front-end technology in the industry. Therefore, we will combine the more mainstream technical practices in the industry and the functions we provide internally to create a better Web UI to empower the Kyuubi community.
Before pushing the Kyuubi Web UI, we have done some preparatory work. For example, at KPIP-12, we promoted the implementation of Restful API. In addition to providing the ability to be equivalent to Hive Thrift RPC, the Kyuubi Restful API has another purpose to meet the requirement of building Web UIs.
The overall architecture of Kyuubi Web UI is not complicated, you can refer to the following figure:
Goal and Non-goal
We pursue "pragmatism" in the Web UI. Our goal is to
We are not committed to showing:
Front-end technology selection
Front-end technology has been evolving rapidly, and each framework has its own design philosophy and core focus. The goal of technology selection is to make assessments and decisions based on our core concerns. For the technical selection of Kyuubi Web UI, we mainly focus on the following:
In terms of development language, we tend to use TypeScript. TypeScript is a new language developed based on JavaScript and developed by Microsoft. It is a superset of JavaScript ES2015 and includes features from newer versions of the language. You can use it to write state-of-the-art JavaScript. It can also statically analyze your code by using a combination of annotations and type inference.
There is a more subtle benefit. TypeScript is heavily influenced by Java and .NET, so if a developer has a background in one of these languages, they will probably get started faster than JavaScript.
And in terms of ecological and functional maturity, our focus is not on reinventing the wheel (yes, we have no such energy). What we expect more is that it is more convenient to build business logic based on complete functions, components, and modules to achieve goals.
Therefore, we hope that the front-end framework has a richer template, form, module system, and mature and elegant architecture to reduce our unnecessary attention.
Based on the above concerns, we finally chose Angular among Angular, React, and Vue.
Functional module
The next step after we have completed the technical selection is to confirm which features of Kyuubi we want to expose and put on the Web UI.
Kyuubi Service Management
As we all know, Kyuubi provides a CLI tool to manage Kyuubi Server. CLI is a very geeky tool. It has a certain cost and learning threshold. It is more suitable for developers with relevant backgrounds. So we consider moving the capabilities of CLI to the web, which can make it easier to manage (multiple) Kyuubi Server.
Engine Management
From the beginning, Kyuubi only supported the Spark SQL engine, and now it has been able to support Flink SQL, Trino, and more engines. Therefore, it makes sense to introduce engine management for Kyuubi Web UI, such as engine configuration parameter management, runtime management, and so on.
Session management
The session is a very important concept in Kyuubi. The thrift JDBC/ODBC connections initiated by user clients, these connection requests are maintained as Kyuubi sessions, and execution requests are supported as Kyuubi operations bound to the corresponding session. The sharing level of the Kyuubi engine describes the correspondence between sessions and engines. It determines whether and how a new session can share an existing backend engine with other sessions.
Query module
As a Gateway, Kyuubi itself can easily query the engine. Therefore, having a SQL Query Editor attached to its Web UI will make it easier for users to use it, thus forming a complete closed loop. The SQL Query Editor looks like this:
When we have this feature (plus the function of viewing Catalog in the later stage), this Web UI can assume the function of SQL Notebook.
Authentication
Authentication is a general requirement for management consoles, and its importance cannot be overstated. In view of the popularity and maturity of LDAP in the open source world (including in the Hadoop system), we hope to integrate the authentication and authentication system of LDAP.
Observability：Log+Metrics
Observability is especially important for building reliable distributed systems. Logs and metrics are an important source of truth for improving observability. Kyuubi has laid the foundation for providing these capabilities, we will see if these capabilities are moved to the Web UI to better improve the observability of the system. However, there is no doubt that the priority of this piece is relatively low. When we complete the construction of some other key modules, we will try to devote energy to this matter.
Design
The following figure shows the project folder structure of the front-end module:
The core business logic of the project exists in the "app" folder. Next, we'll interpret what these folders contain:
The "interfaces" subfolder contains a bunch of "interface" definitions, which mainly contain definitions of some data structures, such as Session, Operation, etc. TypeScript can perform static type checking on their properties.
The "pages" subfolder contains some components for building pages. For example, if we have a Dashboard page, there will be at least three files for this page:
The "services" subfolder mainly contains a bunch of service classes, which are used to coordinate requests to interact with the backend, and at the same time encapsulate the response data into the above interface.
The "share" subfolder is used to define a bunch of reusable, business-agnostic components.
The "utils" subfolder is used to define some common helper methods.
Summary
We can foresee that Kyuubi Web UI can greatly simplify Kyuubi's operating threshold and start-up cost, thereby promoting the improvement of efficiency. On the other hand, it will also promote the acceptance of Kyuubi by colleagues in the industry, which has positive significance for its entire ecology.
Beta Was this translation helpful? Give feedback.
All reactions