feat: add qwen coder model (#783)

**Reason for Change**:

Fixes: #449 

- add qwen coder model
- update inference api docs

Signed-off-by: jerryzhuang <[email protected]>

cmd/workspace/models.go

@@ -9,4 +9,5 @@ import (
 	_ "github.com/kaito-project/kaito/presets/workspace/models/mistral"
 	_ "github.com/kaito-project/kaito/presets/workspace/models/phi2"
 	_ "github.com/kaito-project/kaito/presets/workspace/models/phi3"
+	_ "github.com/kaito-project/kaito/presets/workspace/models/qwen"
 )

docs/inference/README.md

@@ -96,6 +96,89 @@ For detailed `InferenceSpec` API definitions, refer to the [documentation](https
 
 The OpenAPI specification for the inference API is available at [vLLM API](../../presets/workspace/inference/vllm/api_spec.json), [transformers API](../../presets/workspace/inference/text-generation/api_spec.json).
 
+#### vLLM inference API
+
+vLLM supports OpenAI-compatible inference APIs. Check [here](https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html) for more details.
+
+#### Transformers inference API
+
+The inference service endpoint is `/chat`.
+
+**basic example**
+```
+curl -X POST "http://<SERVICE>:80/chat" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt":"YOUR_PROMPT_HERE"}'
+```
+
+**example with full configurable parameters**
+```
+curl -X POST \
+    -H "accept: application/json" \
+    -H "Content-Type: application/json" \
+    -d '{
+        "prompt":"YOUR_PROMPT_HERE",
+        "return_full_text": false,
+        "clean_up_tokenization_spaces": false,
+        "prefix": null,
+        "handle_long_generation": null,
+        "generate_kwargs": {
+                "max_length":200,
+                "min_length":0,
+                "do_sample":true,
+                "early_stopping":false,
+                "num_beams":1,
+                "num_beam_groups":1,
+                "diversity_penalty":0.0,
+                "temperature":1.0,
+                "top_k":10,
+                "top_p":1,
+                "typical_p":1,
+                "repetition_penalty":1,
+                "length_penalty":1,
+                "no_repeat_ngram_size":0,
+                "encoder_no_repeat_ngram_size":0,
+                "bad_words_ids":null,
+                "num_return_sequences":1,
+                "output_scores":false,
+                "return_dict_in_generate":false,
+                "forced_bos_token_id":null,
+                "forced_eos_token_id":null,
+                "remove_invalid_values":null
+            }
+        }' \
+        "http://<SERVICE>:80/chat"
+```
+
+**parameters**
+- `prompt`: The initial text provided by the user, from which the model will continue generating text.
+- `return_full_text`: If False only generated text is returned, else full text is returned.
+- `clean_up_tokenization_spaces`: True/False, determines whether to remove potential extra spaces in the text output.
+- `prefix`: Prefix added to the prompt.
+- `handle_long_generation`: Provides strategies to address generations beyond the model's maximum length capacity.
+- `max_length`: The maximum total number of tokens in the generated text.
+- `min_length`: The minimum total number of tokens that should be generated.
+- `do_sample`: If True, sampling methods will be used for text generation, which can introduce randomness and variation.
+- `early_stopping`: If True, the generation will stop early if certain conditions are met, for example, when a satisfactory number of candidates have been found in beam search.
+- `num_beams`: The number of beams to be used in beam search. More beams can lead to better results but are more computationally expensive.
+- `num_beam_groups`: Divides the number of beams into groups to promote diversity in the generated results.
+- `diversity_penalty`: Penalizes the score of tokens that make the current generation too similar to other groups, encouraging diverse outputs.
+- `temperature`: Controls the randomness of the output by scaling the logits before sampling.
+- `top_k`: Restricts sampling to the k most likely next tokens.
+- `top_p`: Uses nucleus sampling to restrict the sampling pool to tokens comprising the top p probability mass.
+- `typical_p`: Adjusts the probability distribution to favor tokens that are "typically" likely, given the context.
+- `repetition_penalty`: Penalizes tokens that have been generated previously, aiming to reduce repetition.
+- `length_penalty`: Modifies scores based on sequence length to encourage shorter or longer outputs.
+- `no_repeat_ngram_size`: Prevents the generation of any n-gram more than once.
+- `encoder_no_repeat_ngram_size`: Similar to `no_repeat_ngram_size` but applies to the encoder part of encoder-decoder models.
+- `bad_words_ids`: A list of token ids that should not be generated.
+- `num_return_sequences`: The number of different sequences to generate.
+- `output_scores`: Whether to output the prediction scores.
+- `return_dict_in_generate`: If True, the method will return a dictionary containing additional information.
+- `pad_token_id`: The token ID used for padding sequences to the same length.
+- `eos_token_id`: The token ID that signifies the end of a sequence.
+- `forced_bos_token_id`: The token ID that is forcibly used as the beginning of a sequence token.
+- `forced_eos_token_id`: The token ID that is forcibly used as the end of a sequence when max_length is reached.
+- `remove_invalid_values`: If True, filters out invalid values like NaNs or infs from model outputs to prevent crashes.
+
 # Inference workload
 
 Depending on whether the specified model supports distributed inference or not, the Kaito controller will choose to use either Kubernetes **apps.deployment** workload (by default) or Kubernetes **apps.statefulset** workload (if the model supports distributed inference) to manage the inference service, which is exposed using a Cluster-IP type of Kubernetes `service`.

examples/inference/kaito_workspace_qwen_2.5_coder_7b-instruct.yaml

@@ -0,0 +1,12 @@
+apiVersion: kaito.sh/v1alpha1
+kind: Workspace
+metadata:
+  name: workspace-qwen-2.5-coder-7b-instruct
+resource:
+  instanceType: "Standard_NC24ads_A100_v4"
+  labelSelector:
+    matchLabels:
+      apps: qwen-2.5-coder
+inference:
+  preset:
+    name: qwen2.5-coder-7b-instruct

presets/workspace/models/falcon/README.md

@@ -11,79 +11,4 @@
 
 ## Usage
 
-The inference service endpoint is `/chat`.
-
-### Basic example
-```
-curl -X POST "http://<SERVICE>:80/chat" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt":"YOUR_PROMPT_HERE"}'
-```
-
-### Example with full configurable parameters
-```
-curl -X POST \
-    -H "accept: application/json" \
-    -H "Content-Type: application/json" \
-    -d '{
-        "prompt":"YOUR_PROMPT_HERE",
-        "return_full_text": false,
-        "clean_up_tokenization_spaces": false, 
-        "prefix": null,
-        "handle_long_generation": null,
-        "generate_kwargs": {
-                "max_length":200,
-                "min_length":0,
-                "do_sample":true,
-                "early_stopping":false,
-                "num_beams":1,
-                "num_beam_groups":1,
-                "diversity_penalty":0.0,
-                "temperature":1.0,
-                "top_k":10,
-                "top_p":1,
-                "typical_p":1,
-                "repetition_penalty":1,
-                "length_penalty":1,
-                "no_repeat_ngram_size":0,
-                "encoder_no_repeat_ngram_size":0,
-                "bad_words_ids":null,
-                "num_return_sequences":1,
-                "output_scores":false,
-                "return_dict_in_generate":false,
-                "forced_bos_token_id":null,
-                "forced_eos_token_id":null,
-                "remove_invalid_values":null
-            }
-        }' \
-        "http://<SERVICE>:80/chat"
-```
-
-### Parameters
-- `prompt`: The initial text provided by the user, from which the model will continue generating text.
-- `return_full_text`: If False only generated text is returned, else full text is returned.
-- `clean_up_tokenization_spaces`: True/False, determines whether to remove potential extra spaces in the text output.
-- `prefix`: Prefix added to the prompt.
-- `handle_long_generation`: Provides strategies to address generations beyond the model's maximum length capacity.
-- `max_length`: The maximum total number of tokens in the generated text.
-- `min_length`: The minimum total number of tokens that should be generated.
-- `do_sample`: If True, sampling methods will be used for text generation, which can introduce randomness and variation.
-- `early_stopping`: If True, the generation will stop early if certain conditions are met, for example, when a satisfactory number of candidates have been found in beam search.
-- `num_beams`: The number of beams to be used in beam search. More beams can lead to better results but are more computationally expensive.
-- `num_beam_groups`: Divides the number of beams into groups to promote diversity in the generated results.
-- `diversity_penalty`: Penalizes the score of tokens that make the current generation too similar to other groups, encouraging diverse outputs.
-- `temperature`: Controls the randomness of the output by scaling the logits before sampling.
-- `top_k`: Restricts sampling to the k most likely next tokens.
-- `top_p`: Uses nucleus sampling to restrict the sampling pool to tokens comprising the top p probability mass.
-- `typical_p`: Adjusts the probability distribution to favor tokens that are "typically" likely, given the context.
-- `repetition_penalty`: Penalizes tokens that have been generated previously, aiming to reduce repetition.
-- `length_penalty`: Modifies scores based on sequence length to encourage shorter or longer outputs.
-- `no_repeat_ngram_size`: Prevents the generation of any n-gram more than once.
-- `encoder_no_repeat_ngram_size`: Similar to `no_repeat_ngram_size` but applies to the encoder part of encoder-decoder models.
-- `bad_words_ids`: A list of token ids that should not be generated.
-- `num_return_sequences`: The number of different sequences to generate.
-- `output_scores`: Whether to output the prediction scores.
-- `return_dict_in_generate`: If True, the method will return a dictionary containing additional information.
-- `pad_token_id`: The token ID used for padding sequences to the same length.
-- `eos_token_id`: The token ID that signifies the end of a sequence.
-- `forced_bos_token_id`: The token ID that is forcibly used as the beginning of a sequence token.
-- `forced_eos_token_id`: The token ID that is forcibly used as the end of a sequence when max_length is reached.
-- `remove_invalid_values`: If True, filters out invalid values like NaNs or infs from model outputs to prevent crashes.
+See [document](../../../../docs/inference/README.md).

presets/workspace/models/mistral/README.md

@@ -10,79 +10,4 @@
 
 ## Usage
 
-The inference service endpoint is `/chat`.
-
-### Basic example
-```
-curl -X POST "http://<SERVICE>:80/chat" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt":"YOUR_PROMPT_HERE"}'
-```
-
-### Example with full configurable parameters
-```
-curl -X POST \
-    -H "accept: application/json" \
-    -H "Content-Type: application/json" \
-    -d '{
-        "prompt":"YOUR_PROMPT_HERE",
-        "return_full_text": false,
-        "clean_up_tokenization_spaces": false, 
-        "prefix": null,
-        "handle_long_generation": null,
-        "generate_kwargs": {
-                "max_length":200,
-                "min_length":0,
-                "do_sample":true,
-                "early_stopping":false,
-                "num_beams":1,
-                "num_beam_groups":1,
-                "diversity_penalty":0.0,
-                "temperature":1.0,
-                "top_k":10,
-                "top_p":1,
-                "typical_p":1,
-                "repetition_penalty":1,
-                "length_penalty":1,
-                "no_repeat_ngram_size":0,
-                "encoder_no_repeat_ngram_size":0,
-                "bad_words_ids":null,
-                "num_return_sequences":1,
-                "output_scores":false,
-                "return_dict_in_generate":false,
-                "forced_bos_token_id":null,
-                "forced_eos_token_id":null,
-                "remove_invalid_values":null
-            }
-        }' \
-        "http://<SERVICE>:80/chat"
-```
-
-### Parameters
-- `prompt`: The initial text provided by the user, from which the model will continue generating text.
-- `return_full_text`: If False only generated text is returned, else full text is returned.
-- `clean_up_tokenization_spaces`: True/False, determines whether to remove potential extra spaces in the text output.
-- `prefix`: Prefix added to the prompt.
-- `handle_long_generation`: Provides strategies to address generations beyond the model's maximum length capacity.
-- `max_length`: The maximum total number of tokens in the generated text.
-- `min_length`: The minimum total number of tokens that should be generated.
-- `do_sample`: If True, sampling methods will be used for text generation, which can introduce randomness and variation.
-- `early_stopping`: If True, the generation will stop early if certain conditions are met, for example, when a satisfactory number of candidates have been found in beam search.
-- `num_beams`: The number of beams to be used in beam search. More beams can lead to better results but are more computationally expensive.
-- `num_beam_groups`: Divides the number of beams into groups to promote diversity in the generated results.
-- `diversity_penalty`: Penalizes the score of tokens that make the current generation too similar to other groups, encouraging diverse outputs.
-- `temperature`: Controls the randomness of the output by scaling the logits before sampling.
-- `top_k`: Restricts sampling to the k most likely next tokens.
-- `top_p`: Uses nucleus sampling to restrict the sampling pool to tokens comprising the top p probability mass.
-- `typical_p`: Adjusts the probability distribution to favor tokens that are "typically" likely, given the context.
-- `repetition_penalty`: Penalizes tokens that have been generated previously, aiming to reduce repetition.
-- `length_penalty`: Modifies scores based on sequence length to encourage shorter or longer outputs.
-- `no_repeat_ngram_size`: Prevents the generation of any n-gram more than once.
-- `encoder_no_repeat_ngram_size`: Similar to `no_repeat_ngram_size` but applies to the encoder part of encoder-decoder models.
-- `bad_words_ids`: A list of token ids that should not be generated.
-- `num_return_sequences`: The number of different sequences to generate.
-- `output_scores`: Whether to output the prediction scores.
-- `return_dict_in_generate`: If True, the method will return a dictionary containing additional information.
-- `pad_token_id`: The token ID used for padding sequences to the same length.
-- `eos_token_id`: The token ID that signifies the end of a sequence.
-- `forced_bos_token_id`: The token ID that is forcibly used as the beginning of a sequence token.
-- `forced_eos_token_id`: The token ID that is forcibly used as the end of a sequence when max_length is reached.
-- `remove_invalid_values`: If True, filters out invalid values like NaNs or infs from model outputs to prevent crashes.
+See [document](../../../../docs/inference/README.md).

presets/workspace/models/phi2/README.md

@@ -9,79 +9,4 @@
 
 ## Usage
 
-The inference service endpoint is `/chat`.
-
-### Basic example
-```
-curl -X POST "http://<SERVICE>:80/chat" -H "accept: application/json" -H "Content-Type: application/json" -d '{"prompt":"YOUR_PROMPT_HERE"}'
-```
-
-### Example with full configurable parameters
-```
-curl -X POST \
-    -H "accept: application/json" \
-    -H "Content-Type: application/json" \
-    -d '{
-        "prompt":"YOUR_PROMPT_HERE",
-        "return_full_text": false,
-        "clean_up_tokenization_spaces": false, 
-        "prefix": null,
-        "handle_long_generation": null,
-        "generate_kwargs": {
-                "max_length":200,
-                "min_length":0,
-                "do_sample":true,
-                "early_stopping":false,
-                "num_beams":1,
-                "num_beam_groups":1,
-                "diversity_penalty":0.0,
-                "temperature":1.0,
-                "top_k":10,
-                "top_p":1,
-                "typical_p":1,
-                "repetition_penalty":1,
-                "length_penalty":1,
-                "no_repeat_ngram_size":0,
-                "encoder_no_repeat_ngram_size":0,
-                "bad_words_ids":null,
-                "num_return_sequences":1,
-                "output_scores":false,
-                "return_dict_in_generate":false,
-                "forced_bos_token_id":null,
-                "forced_eos_token_id":null,
-                "remove_invalid_values":null
-            }
-        }' \
-        "http://<SERVICE>:80/chat"
-```
-
-### Parameters
-- `prompt`: The initial text provided by the user, from which the model will continue generating text.
-- `return_full_text`: If False only generated text is returned, else full text is returned.
-- `clean_up_tokenization_spaces`: True/False, determines whether to remove potential extra spaces in the text output.
-- `prefix`: Prefix added to the prompt.
-- `handle_long_generation`: Provides strategies to address generations beyond the model's maximum length capacity.
-- `max_length`: The maximum total number of tokens in the generated text.
-- `min_length`: The minimum total number of tokens that should be generated.
-- `do_sample`: If True, sampling methods will be used for text generation, which can introduce randomness and variation.
-- `early_stopping`: If True, the generation will stop early if certain conditions are met, for example, when a satisfactory number of candidates have been found in beam search.
-- `num_beams`: The number of beams to be used in beam search. More beams can lead to better results but are more computationally expensive.
-- `num_beam_groups`: Divides the number of beams into groups to promote diversity in the generated results.
-- `diversity_penalty`: Penalizes the score of tokens that make the current generation too similar to other groups, encouraging diverse outputs.
-- `temperature`: Controls the randomness of the output by scaling the logits before sampling.
-- `top_k`: Restricts sampling to the k most likely next tokens.
-- `top_p`: Uses nucleus sampling to restrict the sampling pool to tokens comprising the top p probability mass.
-- `typical_p`: Adjusts the probability distribution to favor tokens that are "typically" likely, given the context.
-- `repetition_penalty`: Penalizes tokens that have been generated previously, aiming to reduce repetition.
-- `length_penalty`: Modifies scores based on sequence length to encourage shorter or longer outputs.
-- `no_repeat_ngram_size`: Prevents the generation of any n-gram more than once.
-- `encoder_no_repeat_ngram_size`: Similar to `no_repeat_ngram_size` but applies to the encoder part of encoder-decoder models.
-- `bad_words_ids`: A list of token ids that should not be generated.
-- `num_return_sequences`: The number of different sequences to generate.
-- `output_scores`: Whether to output the prediction scores.
-- `return_dict_in_generate`: If True, the method will return a dictionary containing additional information.
-- `pad_token_id`: The token ID used for padding sequences to the same length.
-- `eos_token_id`: The token ID that signifies the end of a sequence.
-- `forced_bos_token_id`: The token ID that is forcibly used as the beginning of a sequence token.
-- `forced_eos_token_id`: The token ID that is forcibly used as the end of a sequence when max_length is reached.
-- `remove_invalid_values`: If True, filters out invalid values like NaNs or infs from model outputs to prevent crashes.
+See [document](../../../../docs/inference/README.md).