Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add in response headers to streaming endpoint #282

Merged
merged 2 commits into from
Feb 26, 2024
Merged

add in response headers to streaming endpoint #282

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
aa187f0
add in response headers to streaming endpoint
noyoshi Feb 26, 2024
c1640e2
add model id
noyoshi Feb 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions router/src/server.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -554,6 +554,28 @@ async fn generate_stream_with_callback(
});
}

let (adapter_source, adapter_parameters) = extract_adapter_params(
req.0.parameters.adapter_id.clone(),
req.0.parameters.adapter_source.clone(),
req.0.parameters.adapter_parameters.clone(),
);

let adapter_id_string = adapter_parameters
.adapter_ids
.iter()
.map(|id| id.as_str())
// filter out base model adapter id
.filter(|id| *id != BASE_MODEL_ADAPTER_ID)
.collect::<Vec<_>>()
.join(",");

if adapter_id_string.len() > 0 {
headers.insert("x-adapter-ids", adapter_id_string.parse().unwrap());
headers.insert("x-adapter-source", adapter_source.unwrap().parse().unwrap());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't there be some guards against panics here?

}

headers.insert("x-model-id", MODEL_ID.get().unwrap().parse().unwrap());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we could also include x-prompt-tokens in lieu of x-total-tokens until we solve log capture?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that can wait until another PR


let stream = async_stream::stream! {
// Inference
let mut end_reached = false;
Expand Down
Loading