-
Notifications
You must be signed in to change notification settings - Fork 3.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
MSQ profile for Brokers and Historicals. #17140
Merged
Merged
Commits on Sep 24, 2024
-
MSQ profile for Brokers and Historicals.
This patch adds a profile of MSQ named "Dart" that runs on Brokers and Historicals, and which is compatible with the standard SQL query API. For more high-level description, and notes on future work, refer to apache#17139. This patch contains the following changes, grouped into packages. Controller (org.apache.druid.msq.dart.controller): The controller runs on Brokers. Main classes are, - DartSqlResource, which serves /druid/v2/sql/dart/. - DartSqlEngine and DartQueryMaker, the entry points from SQL that actually run the MSQ controller code. - DartControllerContext, which configures the MSQ controller. - DartMessageRelays, which sets up relays (see "message relays" below) to read messages from workers' DartControllerClients. - DartTableInputSpecSlicer, which assigns work based on a TimelineServerView. Worker (org.apache.druid.msq.dart.worker) The worker runs on Historicals. Main classes are, - DartWorkerResource, which supplies the regular MSQ WorkerResource, plus Dart-specific APIs. - DartWorkerRunner, which runs MSQ worker code. - DartWorkerContext, which configures the MSQ worker. - DartProcessingBuffersProvider, which provides processing buffers from sliced-up merge buffers. - DartDataSegmentProvider, which provides segments from the Historical's local cache. Message relays (org.apache.druid.messages): To avoid the need for Historicals to contact Brokers during a query, which would create opportunities for queries to get stuck, all connections are opened from Broker to Historical. This is made possible by a message relay system, where the relay server (worker) has an outbox of messages. The relay client (controller) connects to the outbox and retrieves messages. Code for this system lives in the "server" package to keep it separate from the MSQ extension and make it easier to maintain. The worker-to-controller ControllerClient is implemented using message relays. Other changes: - Controller: Added the method "hasWorker". Used by the ControllerMessageListener to notify the appropriate controllers when a worker fails. - WorkerResource: No longer tries to respond more than once in the "httpGetChannelData" API. This comes up when a response due to resolved future is ready at about the same time as a timeout occurs. - MSQTaskQueryMaker: Refactor to separate out some useful functions for reuse in DartQueryMaker. - SqlEngine: Add "queryContext" to "resultTypeForSelect" and "resultTypeForInsert". This allows the DartSqlEngine to modify result format based on whether a "fullReport" context parameter is set. - LimitedOutputStream: New utility class. Used when in "fullReport" mode. - TimelineServerView: Add getDruidServerMetadata as a performance optimization. - CliHistorical: Add SegmentWrangler, so it can query inline data, lookups, etc. - ServiceLocation: Add "fromUri" method, relocating some code from ServiceClientImpl. - FixedServiceLocator: New locator for a fixed set of service locations. Useful for URI locations.
Configuration menu - View commit details
-
Copy full SHA for a27e239 - Browse repository at this point
Copy the full SHA a27e239View commit details
Commits on Sep 25, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 68c61c2 - Browse repository at this point
Copy the full SHA 68c61c2View commit details -
Configuration menu - View commit details
-
Copy full SHA for f110ad0 - Browse repository at this point
Copy the full SHA f110ad0View commit details -
Configuration menu - View commit details
-
Copy full SHA for df9e8bc - Browse repository at this point
Copy the full SHA df9e8bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for f65075c - Browse repository at this point
Copy the full SHA f65075cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 5230880 - Browse repository at this point
Copy the full SHA 5230880View commit details -
Configuration menu - View commit details
-
Copy full SHA for b62a149 - Browse repository at this point
Copy the full SHA b62a149View commit details
Commits on Sep 26, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 680f741 - Browse repository at this point
Copy the full SHA 680f741View commit details -
Configuration menu - View commit details
-
Copy full SHA for b2958c1 - Browse repository at this point
Copy the full SHA b2958c1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 79f5f94 - Browse repository at this point
Copy the full SHA 79f5f94View commit details -
Configuration menu - View commit details
-
Copy full SHA for 642f140 - Browse repository at this point
Copy the full SHA 642f140View commit details -
Configuration menu - View commit details
-
Copy full SHA for f5fa4c0 - Browse repository at this point
Copy the full SHA f5fa4c0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 35b142f - Browse repository at this point
Copy the full SHA 35b142fView commit details
Commits on Sep 27, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 6d0ff3e - Browse repository at this point
Copy the full SHA 6d0ff3eView commit details
Commits on Sep 28, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 42e3ff6 - Browse repository at this point
Copy the full SHA 42e3ff6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 796351f - Browse repository at this point
Copy the full SHA 796351fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1d2e13b - Browse repository at this point
Copy the full SHA 1d2e13bView commit details -
Configuration menu - View commit details
-
Copy full SHA for d7c55af - Browse repository at this point
Copy the full SHA d7c55afView commit details -
Configuration menu - View commit details
-
Copy full SHA for 518c21c - Browse repository at this point
Copy the full SHA 518c21cView commit details
Commits on Sep 30, 2024
-
Configuration menu - View commit details
-
Copy full SHA for f10be74 - Browse repository at this point
Copy the full SHA f10be74View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9177f68 - Browse repository at this point
Copy the full SHA 9177f68View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2c082c9 - Browse repository at this point
Copy the full SHA 2c082c9View commit details
Commits on Oct 1, 2024
-
- Name controller threads properly. - Close worker clients when workers go offline or when the controller exits. Worker -> controller: - Cancel futures when the controller goes offline. Worker -> worker: - Close workerClient when a worker is canceled.
Configuration menu - View commit details
-
Copy full SHA for cc28689 - Browse repository at this point
Copy the full SHA cc28689View commit details -
Fix issue with results array not getting closed if an error happened …
…while writing them.
Configuration menu - View commit details
-
Copy full SHA for 85f4f26 - Browse repository at this point
Copy the full SHA 85f4f26View commit details -
Configuration menu - View commit details
-
Copy full SHA for 9307407 - Browse repository at this point
Copy the full SHA 9307407View commit details -
Configuration menu - View commit details
-
Copy full SHA for e77610e - Browse repository at this point
Copy the full SHA e77610eView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4157913 - Browse repository at this point
Copy the full SHA 4157913View commit details -
Configuration menu - View commit details
-
Copy full SHA for ccfd523 - Browse repository at this point
Copy the full SHA ccfd523View commit details -
Configuration menu - View commit details
-
Copy full SHA for 92b2d21 - Browse repository at this point
Copy the full SHA 92b2d21View commit details -
Configuration menu - View commit details
-
Copy full SHA for d123902 - Browse repository at this point
Copy the full SHA d123902View commit details
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.