Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Example] Use new ggml backend with llama options support #52

Merged
merged 16 commits into from
Nov 3, 2023
Merged

[Example] Use new ggml backend with llama options support #52

Changes from 1 commit
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
e60315b
[Example] Use new ggml backend with llama options support
dm4 Oct 19, 2023
450a92c
[Example] Change the branch of the second-state/wasmedge-wasi-nn crat…
dm4 Oct 20, 2023
ad4af72
[Example] Use serde_json for llama options
dm4 Oct 20, 2023
46e25f3
[Example] Update wasm binary
dm4 Oct 20, 2023
274d494
[Example] Update README.md of llama interactive
dm4 Oct 22, 2023
75c8b1a
[Example] Use metadata in the llama2-chat example instead of the env …
hydai Nov 2, 2023
e794afe
[Example] Remove basic ggml-llama example
dm4 Nov 2, 2023
b839dde
[Example] Update README.md of llama interactive
dm4 Nov 2, 2023
3c14961
[Example] Handle streaming output
dm4 Nov 2, 2023
c55bb63
[Example] Merge wasmedge-ggml-llama into interactive example
dm4 Nov 2, 2023
77bfb42
[Example] Use arguments for non-interactive mode
dm4 Nov 2, 2023
e6b9477
[CI] Fix llama interactive workflow
dm4 Nov 2, 2023
559f195
[Example] Use original wasm instead of AOT wasm
dm4 Nov 2, 2023
fac6415
[Example] Add the default parameter values to the README and synchron…
dm4 Nov 2, 2023
bf83171
[Example] Update README for llama.cpp version
dm4 Nov 3, 2023
27073e8
[Example] Update README for n-gpu-layers in macOS
dm4 Nov 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 18 additions & 2 deletions wasmedge-ggml-llama-interactive/src/main.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,10 +28,26 @@ fn main() {
let system_prompt = String::from("<<SYS>>You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>");
let mut saved_prompt = String::new();

// Set options to input with index 1
let options = r#"{"enable-log": true, "ctx-size": 1024}"#;
dm4 marked this conversation as resolved.
Show resolved Hide resolved
context
.set_input(
1,
wasi_nn::TensorType::U8,
&[1],
&options.as_bytes().to_vec(),
)
.unwrap();

// Ask a quick question to load the model
let initial_prompt = "Are you ready to answer questions? Answer yes or no.";
context
.set_input(0, wasi_nn::TensorType::U8, &[1], &initial_prompt.as_bytes().to_vec())
.set_input(
0,
wasi_nn::TensorType::U8,
&[1],
&initial_prompt.as_bytes().to_vec(),
)
.unwrap();
context.compute().unwrap();

Expand All @@ -54,7 +70,7 @@ fn main() {
context.compute().unwrap();

// Retrieve the output.
let max_output_size = 4096*6;
let max_output_size = 4096 * 6;
let mut output_buffer = vec![0u8; max_output_size];
let mut output_size = context.get_output(0, &mut output_buffer).unwrap();
output_size = std::cmp::min(max_output_size, output_size);
Expand Down