Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Function calling #996

Merged
merged 93 commits into from
May 23, 2024
Merged

Function calling #996

merged 93 commits into from
May 23, 2024

Conversation

nsarrazin
Copy link
Collaborator

@nsarrazin nsarrazin commented Apr 9, 2024 •
edited
Loading

**Screenshots**
  • Probably only works well with command R+, requires a tokenizer with chat prompt templates that has multiple prompt templates for tools
  • Hardcoded a few tools
    • websearch
    • text 2 img
    • calculator
    • url fetcher (injects url content as plaintext)

@nsarrazin nsarrazin added enhancement New feature or request front This issue is related to the front-end of the app. back This issue is related to the Svelte backend or the DB assistants Related to the assistants feature labels Apr 9, 2024
@flozi00
Copy link
Contributor

flozi00 commented Apr 10, 2024

The pr looks very promising, I only have some open questions about integration.
At the moment there are built in tools with static specified functions behind, what's the workflow to be expected for adding more tools ?
Can will there be a list of links, local mounted folder with codes or a moderated pool of tools within the main repo ?

I already have some tools to be added and would love to contribute them

@wdhorton
Copy link
Contributor

It would be cool if users could specify a tool by pasting in an OpenAPI spec

@wdhorton
Copy link
Contributor

Really excited for this feature

@nsarrazin
Copy link
Collaborator Author

Hi! So we're thinking about how to best integrate this feature. We'll probably add a few built-in tools that call some nice models from the hub as a first step, and then add support for OpenAPI tools afterwards since that's quite a bit more challenging to do, but it's definitely on my roadmap.

If you have any suggestions, tips or use cases for this feature, please let me know, it's always useful to know how people plan to use a feature 😄

@nsarrazin
Copy link
Collaborator Author

I added a text2img tool, seems to work well. Now i'm adding:

  • A component to open an image when you click on it (right now it only shows a thumbnail)
  • A component for displaying & playing audio files

That way we'll have good multimodal support for tools 😄

@gururise
Copy link
Contributor

Would love to see a tool that can perform RAG given a url to a chromaDB.

Saghen and others added 13 commits May 22, 2024 09:57
@Saghen
feat: move pdf to markdown to huggingchat
4ebee73
@Saghen
fix: uploaded file width
2270575
@Saghen
feat: prompt the model when no files available
d704585
@gary149
Feat functions: UI update (#1157)
f1fab0f 
* details: close on click outside

* drag for tools

* tools ordering

* add link to community discussion

* misc

* calculator listed as last

* file style

* images with different ratios

* page scraper: longer timeout

@Saghen @mishig25 seems to work better with this

* vertical gap

* Revert "drag for tools"

This reverts commit 8eeed3b.

* chat padding on desktop

* file colors

* larger gap for all messages on xl screens
@Saghen
feat: update pdf to markdown schema
86df89d
@Saghen
fix: uploaded file container width
251e7cc
@Saghen
fix: tool name check
a4f1d18
@Saghen
fix: cohere endpoint
456e696
@Saghen
fix: use most recent message for files
6139400
@Saghen
feat: allow tools to access previous files
1cd0683
@Saghen
feat: support all file types on document parser
4a0a3a7
@Saghen
fix: use document parser in default tools
4579850
@Saghen
feat: rename url fetcher internal name
ed4b666
mishig25
mishig25 reviewed May 23, 2024
View reviewed changes
src/lib/components/chat/ChatMessage.svelte Outdated
{#if searchUpdates && searchUpdates.length > 0}
<OpenWebSearchResults
classNames={tokens.length ? "mb-3.5" : ""}
webSearchMessages={searchUpdates}
/>
{/if}

{#if toolUpdates}
{#each Object.values(toolUpdates) as tool}
{#if tool.length > 0}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
{#if tool.length > 0}
{#if tool.length}

mishig25
mishig25 approved these changes May 23, 2024
View reviewed changes
Copy link
Collaborator

@mishig25 mishig25 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after the CI is greaan, LGTM ! 🚀 🔥
Great job Liam & Nathan!

@mishig25 mishig25 changed the title Function calling for assistants Function calling May 23, 2024
@mishig25 mishig25 merged commit 989379c into main May 23, 2024
3 checks passed
@mishig25 mishig25 deleted the feat/functions branch May 23, 2024 14:52
@Saghen Saghen mentioned this pull request May 24, 2024
Merged
@YWaller YWaller mentioned this pull request May 24, 2024
Closed
ice91 pushed a commit to ice91/chat-ui that referenced this pull request Oct 30, 2024
@nsarrazin @gary149 @Saghen
Function calling (huggingface#996)
5d666ab 
* Initial work on function calling

* wip

* Add websearch as a tool

* lint

* smol

* Move tools to their own files

* directly answer check

* Add text2img tool

* group tool calls together

* show retry even if no message

* fix trailing urls

* Add image popup

* format

* wip

* clean-up

* better loading indicator

* text colors

* extra example to summarize

* switch default model

* Add fetchUrl tool

* wip

* Add latest gradio

* fix types

* version bump sharp

* Basic tools menu

* working menu

* fix menu positioning

* deps fix

* add deps

* cleanup

* more cleanup

* cleanup

* package cleanups

* moar cleanup

* ui update

* Update ChatMessage.svelte

* upgrade gradio dep

* lint

* refactor and pass results to cohere

* feat: code interpreter tool

* fix: add e2b dependency

* feat: working TGI endpoint with tool results

* feat: allow image model to expand to 90dvw

* fix: dont block on title gen

* bump sharp

* misc cleanup

* fix url fetcher, adjust tool typing

* migrate to new MessageUpdate schema

* fix lint errors

* image editing, pdf upload + parsing

* feat: file preview for non-images

* feat: image prompting, file ui, file migration, many fixes

* feat: multiple files and tool file indices

* feat: add back remote keylogging prevention

* minor nit

* minor

* resolve nits

* feat: use node vm for calculator and improve prompt

* more nits

* feat: add index to tools settings

* chore: update invalid package lock

* feat: bump gradio client to 0.19.4

* feat: move pdf to markdown to huggingchat

* fix: uploaded file width

* feat: prompt the model when no files available

* Feat functions: UI update (huggingface#1157)

* details: close on click outside

* drag for tools

* tools ordering

* add link to community discussion

* misc

* calculator listed as last

* file style

* images with different ratios

* page scraper: longer timeout

@Saghen @mishig25 seems to work better with this

* vertical gap

* Revert "drag for tools"

This reverts commit 8eeed3b.

* chat padding on desktop

* file colors

* larger gap for all messages on xl screens

* feat: update pdf to markdown schema

* fix: uploaded file container width

* fix: tool name check

* fix: cohere endpoint

* fix: use most recent message for files

* feat: allow tools to access previous files

* feat: support all file types on document parser

* fix: use document parser in default tools

* feat: rename url fetcher internal name

* feat: resolve type error

* feat: enable tools on prod command r+

* feat: truncate document markdown

* feat: bump @huggingface/inference

* feat: resolve type errors

* nit

* Feat functions misc update 2 (huggingface#1158)

* update examples

a

* add tool indicator

* llama 3 description update

* image quick fix

* mobile

* icon purple

---------

Co-authored-by: Victor Mustar <[email protected]>
Co-authored-by: Liam Dyer <[email protected]>
Co-authored-by: Mishig <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Saghen Saghen Saghen left review comments

@mishig25 mishig25 mishig25 approved these changes

Assignees
No one assigned
Labels
assistants Related to the assistants feature back This issue is related to the Svelte backend or the DB enhancement New feature or request front This issue is related to the front-end of the app.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@nsarrazin @flozi00 @wdhorton @gururise @pngwn @mishig25 @Saghen @gary149