Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: add debug option + logs, poll LLM for run status. #15

Merged
merged 15 commits into from
Dec 12, 2024

Conversation

lublagg
Copy link
Collaborator

@lublagg lublagg commented Dec 9, 2024

This PR introduces several changes:

  1. Adds debug log messages to show behind the scenes requests and responses to and from the LLM and to and from the CODAP API. The debug logs are only shown if the app is in "dev" mode (either via a URL parameter or via appConfig.json) and if showDebugLog is set to true. By default, if the app is in dev mode, showDebugLog will be set to true.
  2. Updates the startRun logic and implements a new function in assistant-model.ts to continuously poll the LLM responses so that we can execute multi-step requests, and adds logic for handling error run statuses.
  3. Switches to using a different Assistant model that has one tool function, create_request. This model has been given the CODAP API documentation in its instructions and generates the arguments for requests itself. We then construct the message from the arguments returned by the LLM, and send the request to CODAP.

@lublagg lublagg changed the base branch from refactor-with-mst-2 to main December 11, 2024 15:55
@lublagg lublagg marked this pull request as ready for review December 11, 2024 19:33
@lublagg lublagg changed the title 188589592 debug output 2 Feat: add debug option + logs to chat transcript, poll LLM for run status. Dec 11, 2024
@lublagg lublagg changed the title Feat: add debug option + logs to chat transcript, poll LLM for run status. Feat: add debug option + logs, poll LLM for run status. Dec 11, 2024
@lublagg lublagg requested a review from bgoldowsky December 11, 2024 19:44
Copy link

@bgoldowsky bgoldowsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! 👍

const { action, resource, values } = JSON.parse(toolCall.function.arguments);
const request = { action, resource, values };
self.transcriptStore.addMessage(DEBUG_SPEAKER, { description: "Request sent to CODAP", content: formatJsonMessage(request) });
const res = yield codapInterface.sendRequest(request);

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may be a later addition, but it seems like some sort of sanity checking before sending it direct to CODAP might be a good idea? Are there API calls that would cause real problems, or crash the page?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think there are any that will crash the page... I think at worst you would just get a { success: false } message.

In theory it could do something weird like create a request to delete something you don't want to delete. But, if we implement the undo/redo button functionality, I guess that would be a reversible scenario.

I think this will be something to keep in mind moving forward as we test it out with different inputs.

description: "Get a list of all attributes in a dataset",
strict: true,
name: "create_request",
description: "Create a request to get data from CODAP",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not just get data, right? It could create/alter/delete components as well.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes -- it can use all of the available CODAP API create/update/delete endpoints.

@@ -3,6 +3,22 @@ export const timeStamp = (): string => {
return now.toLocaleString();
};

const idChars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789";
export const createMessageId = (): string => {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like this should have a comment saying why you're not just using a nanoid or some other standard random identifier.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I hadn't thought of that - I just added in the nanoid package and am using that now.

src/utils/utils.ts Outdated Show resolved Hide resolved
@lublagg lublagg merged commit cb98eb8 into main Dec 12, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants