diff --git a/content/blog/codegen-learnings/index.md b/content/blog/codegen-learnings/index.md index 4d79806d348b..ee17f335c4d2 100644 --- a/content/blog/codegen-learnings/index.md +++ b/content/blog/codegen-learnings/index.md @@ -187,7 +187,7 @@ Solutions to many of these problems go well beyond the domain of code generation However, the first of these problems can be addressed by the approach known as "self-debugging". -We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real time. The user is staring at the screen waiting for an answer, so latency is a major concern. +We observed that many generated TypeScript programs that fail to typecheck contain only a few errors, and asking the LLM to fix these errors often produces a valid program. While this approach is still experimental, we see promising results where getting to a correct program requires only 1-2 iterations of self-debugging. The biggest challenge with this approach is doing it in real-time. The user is staring at the screen waiting for an answer, so latency is a major concern. Monitoring these typechecking errors in production can also provide valuable insight into the quality of the RAG, and even suggest specific solutions. For example, failure to typecheck a member-access expression is a likely indicator of a missing type schema (a recall problem), or a "wrong" schema brought in by an irrelevant document (a precision problem).