Skip to content

Commit

Permalink
Update content/blog/codegen-learnings/index.md
Browse files Browse the repository at this point in the history
Co-authored-by: Gavin <[email protected]>
  • Loading branch information
arturl and thtmnisamnstr authored Jan 4, 2025
1 parent 2c655dc commit 4b86a80
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion content/blog/codegen-learnings/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@ Solutions to many of these problems go well beyond the domain of code generation

However, the first of these problems can be addressed by the approach known as "self-debugging".

We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real time. The user is staring at the screen waiting for an answer, so latency is a major concern.
We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern.

Monitoring these typechecking errors in production can also provide valuable insight into the quality of the RAG, and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem), or a "wrong" schema brought in by an irrelevant document (a precision problem).

Expand Down

0 comments on commit 4b86a80

Please sign in to comment.