add tutuorial for cross-encoder model on sagemaker #2607

ylwu-amzn · 2024-07-03T00:34:27Z

Description

Build tutorial for Reranking with cross-encoder model on Sagemaker.

Issues Resolved

[List any issues this PR will resolve]

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Yaliang Wu <[email protected]>

xinyual · 2024-07-03T01:03:46Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+    env=hub,
+    role=role, 
+)
+predictor = huggingface_model.deploy(


Can we also add instruction for GPU usage? Since we have batch ingestion now, ingestion throughput can benefit a lot from using GPU.
Customer can choose GPU container from https://github.com/aws/deep-learning-containers/blob/master/available_images.md#huggingface-training-containers by using proper transformer + pytorch + py version and set a GPU instance like g4dn/g5.xlarge. Then the endpoint will use GPU for inference automatically.

Let's keep this tutorial focusing on current topic. We can create a separate tutorial for GPU usage.
@xinyual , seems you have done some testing on GPU , can you help build a tutorial about how to use GPU on Sagemaker ?

Sorry for late reply. I miss this message. We also want to create some docs for tutorial of neural sparse model. Maybe we will raise them together later.

Signed-off-by: Yaliang Wu <[email protected]>

kolchfa-aws

Some comments.

kolchfa-aws · 2024-07-03T18:01:37Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+
+# Steps
+
+## 0. Deploy Model on Sagemaker


Suggested change

## 0. Deploy Model on Sagemaker

## 0. Deploy the model on Amazon SageMaker

kolchfa-aws · 2024-07-03T18:02:03Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+# Steps
+
+## 0. Deploy Model on Sagemaker
+Use this code to deploy model on Sagemaker.


Suggested change

Use this code to deploy model on Sagemaker.

Use the following code to deploy the model on Amazon SageMaker:

kolchfa-aws · 2024-07-03T18:03:00Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+    instance_type='ml.m5.xlarge' # ec2 instance type
+)
+```
+Find the model inference endpoint and note it. We will use it to create connector in next step


Suggested change

Find the model inference endpoint and note it. We will use it to create connector in next step

Note the model inference endpoint; you'll use it to create a connector in the next step.

kolchfa-aws · 2024-07-03T18:03:34Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+```
+Find the model inference endpoint and note it. We will use it to create connector in next step
+
+## 1. Create Connector and Model


Suggested change

## 1. Create Connector and Model

## 1. Create a connector and register the model

kolchfa-aws · 2024-07-03T18:04:24Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+
+## 1. Create Connector and Model
+
+If you are using self-managed Opensearch, you should supply AWS credentials:


Suggested change

If you are using self-managed Opensearch, you should supply AWS credentials:

To create a connector for the model, send the following request. If you are using self-managed OpenSearch, supply your AWS credentials:

kolchfa-aws · 2024-07-03T19:21:16Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+{ "passage_text" : "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states." }
+
+```
+### 2.2 Create reranking pipeline


Suggested change

### 2.2 Create reranking pipeline

### 2.2 Create a reranking pipeline

kolchfa-aws · 2024-07-03T19:22:07Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+    ]
+}
+```
+Note: if you provide multiple filed names in `document_fields`, it will concat the value of all fields then do rerank.


Suggested change

Note: if you provide multiple filed names in `document_fields`, it will concat the value of all fields then do rerank.

Note: if you provide multiple filed names in `document_fields`, the values of all fields are first concatenated and then reranking is performed.

kolchfa-aws · 2024-07-03T19:23:25Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+Note: if you provide multiple filed names in `document_fields`, it will concat the value of all fields then do rerank.
+### 2.2 Test reranking
+
+You can tune `size` if you want to return less result. For example, set `"size": 2` if you want to return top 2 documents.


Suggested change

You can tune `size` if you want to return less result. For example, set `"size": 2` if you want to return top 2 documents.

To return a different number of results, provide the `size` parameter. For example, set `size` to `4` to return the top four documents:

kolchfa-aws · 2024-07-03T19:24:13Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+  }
+}
+```
+Test without reranking pipeline:


Suggested change

Test without reranking pipeline:

Test the query without a reranking pipeline:

kolchfa-aws · 2024-07-03T19:24:22Z

docs/tutorials/rerank/rerank_pipeline_with_CrossEncoder_model_deployed_on_Sagemaker.md

+  }
+}
+```
+The first document in the response is `Carson City is the capital city of the American state of Nevada`, which is incorrect.


Suggested change

The first document in the response is `Carson City is the capital city of the American state of Nevada`, which is incorrect.

The first document in the response is `Carson City is the capital city of the American state of Nevada`, which is incorrect:

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn · 2024-07-04T02:06:54Z

Some comments.

Thanks , addressed all comments

* add tutuorial for cross-encoder model on sagemaker Signed-off-by: Yaliang Wu <[email protected]> * add connector helper doc link Signed-off-by: Yaliang Wu <[email protected]> * remvoe title field Signed-off-by: Yaliang Wu <[email protected]> * address commnets Signed-off-by: Yaliang Wu <[email protected]> * use a better input format to invoke model Signed-off-by: Yaliang Wu <[email protected]> --------- Signed-off-by: Yaliang Wu <[email protected]> (cherry picked from commit bffa32a)

* add tutuorial for cross-encoder model on sagemaker Signed-off-by: Yaliang Wu <[email protected]> * add connector helper doc link Signed-off-by: Yaliang Wu <[email protected]> * remvoe title field Signed-off-by: Yaliang Wu <[email protected]> * address commnets Signed-off-by: Yaliang Wu <[email protected]> * use a better input format to invoke model Signed-off-by: Yaliang Wu <[email protected]> --------- Signed-off-by: Yaliang Wu <[email protected]> (cherry picked from commit bffa32a) Co-authored-by: Yaliang Wu <[email protected]>

…t#2607) * add tutuorial for cross-encoder model on sagemaker Signed-off-by: Yaliang Wu <[email protected]> * add connector helper doc link Signed-off-by: Yaliang Wu <[email protected]> * remvoe title field Signed-off-by: Yaliang Wu <[email protected]> * address commnets Signed-off-by: Yaliang Wu <[email protected]> * use a better input format to invoke model Signed-off-by: Yaliang Wu <[email protected]> --------- Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn requested review from b4sjoo, dhrubo-os, jngz-es, model-collapse, rbhavna, zane-neo, Zhangxunmt, austintlee, HenryL27, samuel-oci and xinyual as code owners July 3, 2024 00:34

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 3, 2024 00:34 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 3, 2024 00:34 — with GitHub Actions Failure

add tutuorial for cross-encoder model on sagemaker

e6b5b3e

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn force-pushed the doc_bp branch from 4866634 to e6b5b3e Compare July 3, 2024 00:37

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 3, 2024 00:37 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 3, 2024 00:37 — with GitHub Actions Failure

add connector helper doc link

cfbf33e

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 3, 2024 00:39 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 3, 2024 00:39 — with GitHub Actions Failure

xinyual reviewed Jul 3, 2024

View reviewed changes

remvoe title field

5b66940

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 3, 2024 01:36 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 3, 2024 01:36 — with GitHub Actions Failure

kolchfa-aws reviewed Jul 3, 2024

View reviewed changes

address commnets

32217a6

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 4, 2024 00:19 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 4, 2024 00:19 — with GitHub Actions Failure

ylwu-amzn added the backport 2.x label Jul 4, 2024

use a better input format to invoke model

be76b35

Signed-off-by: Yaliang Wu <[email protected]>

ylwu-amzn temporarily deployed to ml-commons-cicd-env July 4, 2024 02:00 — with GitHub Actions Inactive

ylwu-amzn had a problem deploying to ml-commons-cicd-env July 4, 2024 02:00 — with GitHub Actions Failure

b4sjoo approved these changes Jul 4, 2024

View reviewed changes

rbhavna approved these changes Jul 4, 2024

View reviewed changes

ylwu-amzn merged commit bffa32a into opensearch-project:main Jul 4, 2024
4 of 5 checks passed

opensearch-trigger-bot bot mentioned this pull request Jul 4, 2024

[Backport 2.x] add tutuorial for cross-encoder model on sagemaker #2611

Merged

b4sjoo added the v2.16.0 Issues targeting release v2.16.0 label Jul 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add tutuorial for cross-encoder model on sagemaker #2607

add tutuorial for cross-encoder model on sagemaker #2607

ylwu-amzn commented Jul 3, 2024

xinyual Jul 3, 2024

ylwu-amzn Jul 4, 2024 •

edited

Loading

xinyual Jul 23, 2024

kolchfa-aws left a comment

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

kolchfa-aws Jul 3, 2024

ylwu-amzn commented Jul 4, 2024

	## 0. Deploy Model on Sagemaker
	## 0. Deploy the model on Amazon SageMaker

	Use this code to deploy model on Sagemaker.
	Use the following code to deploy the model on Amazon SageMaker:

	Find the model inference endpoint and note it. We will use it to create connector in next step
	Note the model inference endpoint; you'll use it to create a connector in the next step.

	## 1. Create Connector and Model
	## 1. Create a connector and register the model


		## 1. Create Connector and Model

		If you are using self-managed Opensearch, you should supply AWS credentials:

	If you are using self-managed Opensearch, you should supply AWS credentials:
	To create a connector for the model, send the following request. If you are using self-managed OpenSearch, supply your AWS credentials:

	### 2.2 Create reranking pipeline
	### 2.2 Create a reranking pipeline

	Note: if you provide multiple filed names in `document_fields`, it will concat the value of all fields then do rerank.
	Note: if you provide multiple filed names in `document_fields`, the values of all fields are first concatenated and then reranking is performed.

	You can tune `size` if you want to return less result. For example, set `"size": 2` if you want to return top 2 documents.
	To return a different number of results, provide the `size` parameter. For example, set `size` to `4` to return the top four documents:

	Test without reranking pipeline:
	Test the query without a reranking pipeline:

	The first document in the response is `Carson City is the capital city of the American state of Nevada`, which is incorrect.
	The first document in the response is `Carson City is the capital city of the American state of Nevada`, which is incorrect:

add tutuorial for cross-encoder model on sagemaker #2607

add tutuorial for cross-encoder model on sagemaker #2607

Conversation

ylwu-amzn commented Jul 3, 2024

Description

Issues Resolved

Check List

Choose a reason for hiding this comment

ylwu-amzn Jul 4, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kolchfa-aws left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ylwu-amzn commented Jul 4, 2024

ylwu-amzn Jul 4, 2024 •

edited

Loading