Merge #398

398: implement experimental ai-powered search r=ahmednfwela a=memishood

# Pull Request

## Related issue
Fixes #369 

## What does this PR do?
I've implemented all the required steps mentioned in the issue above. I made every effort to respect the existing project conventions. I hope this pull request will be helpful and add value to Meilisearch 💜

**PR:** I didn't uncomment vector search tests in search_tests.dart as I'm focused on embedders capability.

### Some technical points that I might have considered wrong in the pull-req
- Ensure `semanticRatio` is between 0.0 and 1.0
- Throw exception when embedder source is not one of them: `openAi`, `huggingFace`, `userProvided`, `rest`, `ollama`

**Any comment & feedback is appreciated!**

## PR checklist
Please check if your PR fulfills the following requirements:
- [x] Does this PR fix an existing issue, or have you listed the changes applied in the PR description (and why they are needed)?
- [x] Have you read the contributing guidelines?
- [x] Have you made sure that the title is accurate and descriptive of the changes?

Thank you so much for contributing to Meilisearch!


Co-authored-by: Emre Memis <[email protected]>
Co-authored-by: ahmednfwela <[email protected]>
Co-authored-by: Emre <[email protected]>

CONTRIBUTING.md

@@ -10,6 +10,8 @@ First of all, thank you for contributing to Meilisearch! The goal of this docume
     - [Setup ](#setup-)
     - [Tests and Linter ](#tests-and-linter-)
     - [Updating code samples](#updating-code-samples)
+    - [Run integration tests for embedders](#run-integration-tests-for-embedders)
+      - [OpenAI Model Integration](#openai-model-integration)
   - [Git Guidelines](#git-guidelines)
     - [Git Branches ](#git-branches-)
     - [Git Commits ](#git-commits-)
@@ -103,6 +105,19 @@ The process to define a new code sample is as follows:
     ```bash
     dart run ./tool/bin/meili.dart update-samples --fail-on-change
     ```
+
+### Run integration tests for embedders
+
+Integration tests for embedders are located in `test/search_test.dart`
+
+#### OpenAI Model Integration
+The tests utilize OpenAI models for embedding functionalities. Ensure you have a valid OpenAI API key to run these tests.
+
+- Generate an OpenAI API Key
+- Provide the API Key in one of two ways:
+  - Pass the key via environment variable: `export OPEN_AI_API_KEY=your_openai_api_key` (will not work on dart web)
+  - Pass the key via dart define: `dart --define=OPEN_AI_API_KEY=your_openai_api_key test --use-data-isolate-strategy` (Works on both web and native)
+
 ## Git Guidelines
 
 ### Git Branches <!-- omit in TOC -->

README.md

@@ -210,8 +210,6 @@ final res = await index
 
 This package guarantees compatibility with [version v1.x of Meilisearch](https://github.com/meilisearch/meilisearch/releases/tag/latest), but some features may not be present. Please check the [issues](https://github.com/meilisearch/meilisearch-dart/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22+label%3Aenhancement) for more info.
 
-⚠️ This package is not compatible with the [`vectoreStore` experimental feature](https://www.meilisearch.com/docs/learn/experimental/vector_search) of Meilisearch v1.6.0 and later. More information on this [issue](https://github.com/meilisearch/meilisearch-dart/issues/369).
-
 ## 💡 Learn more
 
 The following sections in our main documentation website may interest you:

lib/meilisearch.dart

@@ -8,3 +8,5 @@ export 'src/client.dart';
 export 'src/index.dart';
 export 'src/exception.dart';
 export 'src/facade.dart';
+export 'src/settings/distribution.dart';
+export 'src/settings/embedder.dart';

lib/src/index.dart

@@ -685,6 +685,35 @@ class MeiliSearchIndex {
     );
   }
 
+  /// Get the embedders settings of a Meilisearch index.
+  @RequiredMeiliServerVersion('1.6.0')
+  Future<Map<String, Embedder>?> getEmbedders() async {
+    final response = await http
+        .getMethod<Map<String, Object?>>('/indexes/$uid/settings/embedders');
+
+    return response.data?.map(
+        (k, v) => MapEntry(k, Embedder.fromMap(v as Map<String, Object?>)));
+  }
+
+  /// Update the embedders settings. Overwrite the old settings.
+  @RequiredMeiliServerVersion('1.6.0')
+  Future<Task> updateEmbedders(Map<String, Embedder>? embedders) async {
+    return await _getTask(
+      http.putMethod(
+        '/indexes/$uid/settings/embedders',
+        data: embedders?.map((k, v) => MapEntry(k, v.toMap())),
+      ),
+    );
+  }
+
+  /// Reset the embedders settings to its default value
+  @RequiredMeiliServerVersion('1.6.0')
+  Future<Task> resetEmbedders() async {
+    return await _getTask(
+      http.deleteMethod('/indexes/$uid/settings/embedders'),
+    );
+  }
+
   //
   // StopWords endpoints
   //

lib/src/query_parameters/_exports.dart

@@ -11,3 +11,4 @@ export 'multi_search_query.dart';
 export 'delete_documents_query.dart';
 export 'facet_search_query.dart';
 export 'swap_index.dart';
+export 'hybrid_search.dart';

lib/src/query_parameters/hybrid_search.dart

@@ -0,0 +1,17 @@
+class HybridSearch {
+  final String embedder;
+  final double semanticRatio;
+
+  const HybridSearch({
+    required this.embedder,
+    required this.semanticRatio,
+  }) : assert(
+          semanticRatio >= 0.0 && semanticRatio <= 1.0,
+          "'semanticRatio' must be between 0.0 and 1.0",
+        );
+
+  Map<String, Object?> toMap() => {
+        'embedder': embedder,
+        'semanticRatio': semanticRatio,
+      };
+}

lib/src/query_parameters/index_search_query.dart

@@ -1,6 +1,7 @@
 import '../annotations.dart';
 import '../filter_builder/_exports.dart';
 import '../results/matching_strategy_enum.dart';
+import 'hybrid_search.dart';
 import 'search_query.dart';
 
 @RequiredMeiliServerVersion('1.1.0')
@@ -29,6 +30,7 @@ class IndexSearchQuery extends SearchQuery {
     super.highlightPostTag,
     super.matchingStrategy,
     super.attributesToSearchOn,
+    super.hybrid,
     super.showRankingScore,
     super.vector,
     super.showRankingScoreDetails,
@@ -65,6 +67,7 @@ class IndexSearchQuery extends SearchQuery {
     String? highlightPostTag,
     MatchingStrategy? matchingStrategy,
     List<String>? attributesToSearchOn,
+    HybridSearch? hybrid,
     bool? showRankingScore,
     List<dynamic /* double | List<double> */ >? vector,
     bool? showRankingScoreDetails,
@@ -91,6 +94,7 @@ class IndexSearchQuery extends SearchQuery {
         highlightPostTag: highlightPostTag ?? this.highlightPostTag,
         matchingStrategy: matchingStrategy ?? this.matchingStrategy,
         attributesToSearchOn: attributesToSearchOn ?? this.attributesToSearchOn,
+        hybrid: hybrid ?? this.hybrid,
         showRankingScore: showRankingScore ?? this.showRankingScore,
         vector: vector ?? this.vector,
         showRankingScoreDetails:

lib/src/query_parameters/search_query.dart

@@ -21,6 +21,8 @@ class SearchQuery extends Queryable {
   final String? highlightPostTag;
   final MatchingStrategy? matchingStrategy;
   final List<String>? attributesToSearchOn;
+  @RequiredMeiliServerVersion('1.6.0')
+  final HybridSearch? hybrid;
   @RequiredMeiliServerVersion('1.3.0')
   final bool? showRankingScore;
   @RequiredMeiliServerVersion('1.3.0')
@@ -47,6 +49,7 @@ class SearchQuery extends Queryable {
     this.highlightPostTag,
     this.matchingStrategy,
     this.attributesToSearchOn,
+    this.hybrid,
     this.showRankingScore,
     this.showRankingScoreDetails,
     this.vector,
@@ -72,6 +75,7 @@ class SearchQuery extends Queryable {
       'highlightPostTag': highlightPostTag,
       'matchingStrategy': matchingStrategy?.name,
       'attributesToSearchOn': attributesToSearchOn,
+      'hybrid': hybrid?.toMap(),
       'showRankingScore': showRankingScore,
       'showRankingScoreDetails': showRankingScoreDetails,
       'vector': vector,
@@ -97,6 +101,7 @@ class SearchQuery extends Queryable {
     String? highlightPostTag,
     MatchingStrategy? matchingStrategy,
     List<String>? attributesToSearchOn,
+    HybridSearch? hybrid,
     bool? showRankingScore,
     List<dynamic>? vector,
     bool? showRankingScoreDetails,
@@ -121,6 +126,7 @@ class SearchQuery extends Queryable {
         highlightPostTag: highlightPostTag ?? this.highlightPostTag,
         matchingStrategy: matchingStrategy ?? this.matchingStrategy,
         attributesToSearchOn: attributesToSearchOn ?? this.attributesToSearchOn,
+        hybrid: hybrid ?? this.hybrid,
         showRankingScore: showRankingScore ?? this.showRankingScore,
         vector: vector ?? this.vector,
         showRankingScoreDetails:

lib/src/settings/distribution.dart

@@ -0,0 +1,31 @@
+///  Describes the mean and sigma of distribution of embedding similarity in the embedding space.
+///
+/// The intended use is to make the similarity score more comparable to the regular ranking score.
+/// This allows to correct effects where results are too "packed" around a certain value.
+class DistributionShift {
+  /// Value where the results are "packed".
+  /// Similarity scores are translated so that they are packed around 0.5 instead
+  final double currentMean;
+
+  /// standard deviation of a similarity score.
+  ///
+  /// Set below 0.4 to make the results less packed around the mean, and above 0.4 to make them more packed.
+  final double currentSigma;
+
+  DistributionShift({
+    required this.currentMean,
+    required this.currentSigma,
+  });
+
+  factory DistributionShift.fromMap(Map<String, Object?> map) {
+    return DistributionShift(
+      currentMean: map['current_mean'] as double,
+      currentSigma: map['current_sigma'] as double,
+    );
+  }
+
+  Map<String, Object?> toMap() => {
+        'current_mean': currentMean,
+        'current_sigma': currentSigma,
+      };
+}