Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Debug studio #1831

Open
wants to merge 26 commits into
base: main
Choose a base branch
from
Open

Debug studio #1831

wants to merge 26 commits into from

Conversation

zolinthecow
Copy link
Contributor

Motivation

When developing apps (especially more complex ones) with SGLang's frontend it's a bit hard to tell what is actually getting passed into the generate/ endpoint. This PR introduces a "prompt debugging studio" that allows you to turn on a debug mode and will start intercepting generate requests and saving them to a debug server, where you can go to a web interface and view all the prompts and responses for SGL functions.

For example, if you start the debug server:

python -m sglang.launch_debug_server

It will start a debug server on http://0.0.0.0:56765

Then, you can add a debug region to an sgl function:

@sgl.function
def text_qa(s, question):
    s.begin_debug_region("TEXT_QA")
    s += "Q: " + question + "\n"
    s += "A:" + sgl.gen("answer", stop="\n")

state = text_qa.run(
    question="What is the capital of France?",
    temperature=0.1,
    stream=True
)

The s.begin_debug_region will start debugging under the prompt name "TEXT_QA". You can then go to http://localhost:56765 (will have to ssh forward if you're developing on a remote server) and you should see prompts start appearing

image

Modifications

  • Adds BeginDebugRegion and EndDebugRegion to SGLExpr for the frontend language
  • Adds launch_debug_server which under the hood relies on enochian-studio (one of my packages) for the debug server
    • enochian-studio can be found here, I can say it's up to date since I maintain it, along with Etched
  • Adds posting to the debug server if enabled for all backends

Checklist

  • Format your code according to the Contributor Guide.
  • Add unit tests as outlined in the Contributor Guide.
  • Update documentation as needed, including docstrings or example tutorials.

@zolinthecow
Copy link
Contributor Author

^ Couldn't reproduce CI error locally :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant