Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSQ profile for Brokers and Historicals. #17140

Merged
merged 30 commits into from
Oct 1, 2024
Merged

MSQ profile for Brokers and Historicals. #17140

merged 30 commits into from
Oct 1, 2024

Commits on Sep 24, 2024

  1. MSQ profile for Brokers and Historicals.

    This patch adds a profile of MSQ named "Dart" that runs on Brokers and
    Historicals, and which is compatible with the standard SQL query API.
    For more high-level description, and notes on future work, refer to apache#17139.
    
    This patch contains the following changes, grouped into packages.
    
    Controller (org.apache.druid.msq.dart.controller):
    
    The controller runs on Brokers. Main classes are,
    
    - DartSqlResource, which serves /druid/v2/sql/dart/.
    - DartSqlEngine and DartQueryMaker, the entry points from SQL that actually
      run the MSQ controller code.
    - DartControllerContext, which configures the MSQ controller.
    - DartMessageRelays, which sets up relays (see "message relays" below) to read
      messages from workers' DartControllerClients.
    - DartTableInputSpecSlicer, which assigns work based on a TimelineServerView.
    
    Worker (org.apache.druid.msq.dart.worker)
    
    The worker runs on Historicals. Main classes are,
    
    - DartWorkerResource, which supplies the regular MSQ WorkerResource, plus
      Dart-specific APIs.
    - DartWorkerRunner, which runs MSQ worker code.
    - DartWorkerContext, which configures the MSQ worker.
    - DartProcessingBuffersProvider, which provides processing buffers from
      sliced-up merge buffers.
    - DartDataSegmentProvider, which provides segments from the Historical's
      local cache.
    
    Message relays (org.apache.druid.messages):
    
    To avoid the need for Historicals to contact Brokers during a query, which
    would create opportunities for queries to get stuck, all connections are
    opened from Broker to Historical. This is made possible by a message relay
    system, where the relay server (worker) has an outbox of messages.
    
    The relay client (controller) connects to the outbox and retrieves messages.
    Code for this system lives in the "server" package to keep it separate from
    the MSQ extension and make it easier to maintain. The worker-to-controller
    ControllerClient is implemented using message relays.
    
    Other changes:
    
    - Controller: Added the method "hasWorker". Used by the ControllerMessageListener
      to notify the appropriate controllers when a worker fails.
    - WorkerResource: No longer tries to respond more than once in the
      "httpGetChannelData" API. This comes up when a response due to resolved future
      is ready at about the same time as a timeout occurs.
    - MSQTaskQueryMaker: Refactor to separate out some useful functions for reuse
      in DartQueryMaker.
    - SqlEngine: Add "queryContext" to "resultTypeForSelect" and "resultTypeForInsert".
      This allows the DartSqlEngine to modify result format based on whether a "fullReport"
      context parameter is set.
    - LimitedOutputStream: New utility class. Used when in "fullReport" mode.
    - TimelineServerView: Add getDruidServerMetadata as a performance optimization.
    - CliHistorical: Add SegmentWrangler, so it can query inline data, lookups, etc.
    - ServiceLocation: Add "fromUri" method, relocating some code from ServiceClientImpl.
    - FixedServiceLocator: New locator for a fixed set of service locations. Useful for
      URI locations.
    gianm committed Sep 24, 2024
    Configuration menu
    Copy the full SHA
    a27e239 View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    68c61c2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f110ad0 View commit details
    Browse the repository at this point in the history
  3. Tests for OutboxImpl.

    gianm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    df9e8bc View commit details
    Browse the repository at this point in the history
  4. Additional tests.

    gianm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    f65075c View commit details
    Browse the repository at this point in the history
  5. Fix style, constants.

    gianm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    5230880 View commit details
    Browse the repository at this point in the history
  6. Style fixes.

    gianm committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    b62a149 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Configuration menu
    Copy the full SHA
    680f741 View commit details
    Browse the repository at this point in the history
  2. Additional javadoc.

    gianm committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    b2958c1 View commit details
    Browse the repository at this point in the history
  3. Additional javadoc.

    gianm committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    79f5f94 View commit details
    Browse the repository at this point in the history
  4. Fix missing format arg.

    gianm committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    642f140 View commit details
    Browse the repository at this point in the history
  5. Add coverage.

    gianm committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    f5fa4c0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    35b142f View commit details
    Browse the repository at this point in the history

Commits on Sep 27, 2024

  1. Configuration menu
    Copy the full SHA
    6d0ff3e View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2024

  1. DartWorkerRunnerTest.

    gianm committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    42e3ff6 View commit details
    Browse the repository at this point in the history
  2. Improved behavior.

    gianm committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    796351f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1d2e13b View commit details
    Browse the repository at this point in the history
  4. Tests for runtime errors.

    gianm committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    d7c55af View commit details
    Browse the repository at this point in the history
  5. Static import assertThat.

    gianm committed Sep 28, 2024
    Configuration menu
    Copy the full SHA
    518c21c View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Additional tests.

    gianm committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    f10be74 View commit details
    Browse the repository at this point in the history
  2. Review comments.

    gianm committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    9177f68 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2c082c9 View commit details
    Browse the repository at this point in the history

Commits on Oct 1, 2024

  1. Controller -> worker:

    - Name controller threads properly.
    - Close worker clients when workers go offline or when the controller exits.
    
    Worker -> controller:
    
    - Cancel futures when the controller goes offline.
    
    Worker -> worker:
    
    - Close workerClient when a worker is canceled.
    gianm committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    cc28689 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    85f4f26 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    9307407 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e77610e View commit details
    Browse the repository at this point in the history
  5. NL

    gianm committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    4157913 View commit details
    Browse the repository at this point in the history
  6. Style.

    gianm committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    ccfd523 View commit details
    Browse the repository at this point in the history
  7. Style.

    gianm committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    92b2d21 View commit details
    Browse the repository at this point in the history
  8. Cut loop in tests.

    gianm committed Oct 1, 2024
    Configuration menu
    Copy the full SHA
    d123902 View commit details
    Browse the repository at this point in the history