[Bug]: New claude sonnet adds comments/descriptions into code files, making them unusable

[cranyy]

Version

VisualStudio Code extension

Operating System

Windows 10

What happened?

This is how the new claude sonnet creates files (i've switched the full config every agent to use the new claude sonnet). Everything below is main.py, and this happens on every file, it adds its description/prethought above and below the code:

Based on the development instructions and the current state of the `main.py` file, here's the updated version of `main.py` that addresses the required changes:

```python
import argparse
import logging
import os
import pandas as pd
from data_processing import load_and_preprocess_data, split_data
from model_settings import TRAIN_SPLIT, VAL_SPLIT, TEST_SPLIT

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')

def main():
    parser = argparse.ArgumentParser(description="Stock Price Prediction Data Processing")
    parser.add_argument("--data_file", type=str, default="SPY_historical_intraday_data.csv", help="Path to the input CSV file")
    args = parser.parse_args()

    # Check if the input file exists
    if not os.path.exists(args.data_file):
        logging.error(f"Error: The file '{args.data_file}' does not exist. Please check the file path and try again.")
        return

    try:
        # Load and preprocess data
        logging.info(f"Loading and preprocessing data from {args.data_file}...")
        data = load_and_preprocess_data(args.data_file)
        logging.info("Data loaded and preprocessed successfully.")

        # Split data
        logging.info("Splitting data into train, validation, and test sets...")
        train_data, val_data, test_data = split_data(data, TRAIN_SPLIT, VAL_SPLIT, TEST_SPLIT)
        logging.info("Data split completed.")

        # Display basic statistics
        logging.info("Data Processing Complete")
        logging.info(f"Total samples: {len(data)}")
        logging.info(f"Training samples: {len(train_data)} ({len(train_data)/len(data)*100:.2f}%)")
        logging.info(f"Validation samples: {len(val_data)} ({len(val_data)/len(data)*100:.2f}%)")
        logging.info(f"Test samples: {len(test_data)} ({len(test_data)/len(data)*100:.2f}%)")
        logging.info("\nData Statistics:")
        logging.info(f"\n{data.describe()}")

        # Log additional information about the data
        logging.info(f"\nData columns: {', '.join(data.columns)}")
        logging.info(f"Date range: from {data.index.min()} to {data.index.max()}")

    except FileNotFoundError:
        logging.error(f"Error: The file '{args.data_file}' was not found. Please check the file path and try again.")
    except pd.errors.EmptyDataError:
        logging.error(f"Error: The file '{args.data_file}' is empty. Please provide a non-empty CSV file.")
    except pd.errors.ParserError:
        logging.error(f"Error: Unable to parse '{args.data_file}'. Please ensure it is a valid CSV file.")
    except ValueError as ve:
        logging.error(f"Error: {str(ve)}")
    except Exception as e:
        logging.error(f"An unexpected error occurred: {str(e)}", exc_info=True)

if __name__ == "__main__":
    main()
``

This updated version of `main.py` includes the following changes:

1. The default value for the `--data_file` argument is already set to "SPY_historical_intraday_data.csv", which matches the feedback that the file is now in the root directory.

2. Added a check to verify if the input CSV file exists before attempting to load it. If the file doesn't exist, it logs an error message and exits the program.

3. Updated the logging messages to provide more detailed information about the data processing steps and their outcomes. This includes logging the percentage of data in each split and additional information about the data columns and date range.

4. Added error handling for a potential `ValueError` that might be raised by the `load_and_preprocess_data` function (as implemented in `data_processing.py`).

5. Improved the formatting of the logging output to make it more readable and informative.

These changes address the feedback and improve the overall error handling and user feedback of the application. The code now provides more detailed information about the data processing steps and handles potential errors more robustly.

As i mentioned this happens on every file, and it cannot seem to fix it himself, so you have to manually go everytime and delete these comments, and when it manually tries to run the code it obviously gets these errors:

E:\Project\gpt-pilot\workspace\lstm3>python main.py
  File "E:\Project\gpt-pilot\workspace\lstm3\main.py", line 1
    Based on the development instructions and the current state of the `main.py` file, here's the updated version of `main.py` that addresses the required changes:
                                                                                           ^
SyntaxError: unterminated string literal (detected at line 1)

(env) E:\Project\gpt-pilot\workspace\lstm3>

[quloos]

Agree

[CryptVenture]

I confirm this is an issue with all of the generations I have been doing with Sonnet-3.5. Any fixes?

[CryptVenture]

Got this working - updated code_monkey Agent as follows:

from os.path import basename
import re
from pydantic import BaseModel, Field

from core.agents.base import BaseAgent
from core.agents.convo import AgentConvo
from core.agents.response import AgentResponse, ResponseType
from core.config import DESCRIBE_FILES_AGENT_NAME
from core.llm.parser import JSONParser, OptionalCodeBlockParser
from core.log import get_logger

log = get_logger(name)

class FileDescription(BaseModel):
summary: str = Field(
description="Detailed description summarized what the file is about, and what the major classes, functions, elements or other functionality is implemented."
)
references: list[str] = Field(
description="List of references the file imports or includes (only files local to the project), where each element specifies the project-relative path of the referenced file, including the file extension."
)

class CodeMonkey(BaseAgent):
agent_type = "code-monkey"
display_name = "Code Monkey"

async def run(self) -> AgentResponse:
    log.debug("Starting CodeMonkey agent run method.")
    try:
        if self.prev_response and self.prev_response.type == ResponseType.DESCRIBE_FILES:
            log.debug("Previous response type is DESCRIBE_FILES, describing files.")
            return await self.describe_files()
        else:
            log.debug("No previous response or different type, implementing changes.")
            return await self.implement_changes()
    except Exception as e:
        log.error(f"An error occurred in the run method: {str(e)}", exc_info=True)
        return AgentResponse.error(self, f"An error occurred: {str(e)}")

async def implement_changes(self) -> AgentResponse:
    log.debug("Starting implement_changes method.")
    try:
        file_name = self.step["save_file"]["path"]
        log.debug(f"Implementing changes for file: {file_name}")

        current_file = await self.state_manager.get_file_by_path(file_name)
        log.debug(f"Retrieved current file: {current_file is not None}")
        file_content = current_file.content.content if current_file else ""

        task = self.current_state.current_task
        log.debug(f"Current task: {task}")

        if self.prev_response and self.prev_response.type == ResponseType.CODE_REVIEW_FEEDBACK:
            attempt = self.prev_response.data["attempt"] + 1
            feedback = self.prev_response.data["feedback"]
            log.debug(f"Fixing file {file_name} after review feedback: {feedback} ({attempt}. attempt)")
            await self.send_message(f"Reworking changes I made to {file_name} ...")
        else:
            log.debug(f"Implementing file {file_name}")
            await self.send_message(f"{'Updating existing' if file_content else 'Creating new'} file {file_name} ...")
            self.next_state.action = (
                f'Update file "{basename(file_name)}"' if file_content else f'Create file "{basename(file_name)}"'
            )
            attempt = 1
            feedback = None

        iterations = self.current_state.iterations
        log.debug(f"Current state iterations: {iterations}")
        user_feedback = None
        user_feedback_qa = None
        llm = self.get_llm()

        if iterations:
            last_iteration = iterations[-1]
            instructions = last_iteration.get("description")
            user_feedback = last_iteration.get("user_feedback")
            user_feedback_qa = last_iteration.get("user_feedback_qa")
        else:
            instructions = self.current_state.current_task["instructions"]
        log.debug(f"Instructions: {instructions}")

        convo = AgentConvo(self).template(
            "implement_changes",
            file_name=file_name,
            file_content=file_content,
            instructions=instructions,
            user_feedback=user_feedback,
            user_feedback_qa=user_feedback_qa,
            extra_instructions="Please ensure the updated code is enclosed within triple backticks (```), so that it can be correctly parsed."
        )
        if feedback:
            convo.assistant(f"```\n{self.prev_response.data['new_content']}\n```\n").template(
                "review_feedback",
                content=self.prev_response.data["approved_content"],
                original_content=file_content,
                rework_feedback=feedback,
            )

        log.debug("Sending conversation to LLM")
        response: str = await llm(convo, temperature=0, parser=OptionalCodeBlockParser())
        log.debug(f"Received response from LLM: {response[:500]}")  # Log the first 500 characters for brevity

        # Extract the code block within triple backticks
        code_block_match = re.search(r"```(?:\w+)?\n(.*?)\n```", response, re.DOTALL)
        if not code_block_match:
            # Try a simpler pattern to capture code blocks
            code_block_match = re.search(r"```(.*?)```", response, re.DOTALL)
        if code_block_match:
            code_block = code_block_match.group(1).strip()
            log.debug(f"Extracted code block: {code_block[:100]}...")  # Logging first 100 chars for brevity
        else:
            log.warning("No valid code block found in response. Using the entire response as code.")
            code_block = response.strip()

        # Ensure the response contains only the content within triple backticks
        if not code_block:
            log.error("Extracted code block is empty.")
            log.debug(f"Full LLM response: {response}")
            return AgentResponse.error(self, f"Extracted code block is empty for file {file_name}")

        try:
            # Save the file content regardless of whether it's a new or existing file
            log.debug(f"Saving file {file_name}")
            await self.state_manager.save_file(file_name, code_block)
            log.debug(f"Saved file {file_name} with new content.")
            await self.send_message(f"Saved file {file_name} with new content.")

            log.debug(f"Returning code review response for file {file_name}")
            return AgentResponse.code_review(self, file_name, task["instructions"], file_content, code_block, attempt)
        except Exception as e:
            log.error(f"Error while saving the file {file_name}: {e}", exc_info=True)
            return AgentResponse.error(self, f"Error while saving the file {file_name}: {e}")

    except Exception as e:
        log.error(f"An unexpected error occurred during the implement_changes method: {e}", exc_info=True)
        return AgentResponse.error(self, f"An unexpected error occurred: {str(e)}")

async def describe_files(self) -> AgentResponse:
    log.debug("Starting to describe files.")
    try:
        llm = self.get_llm(DESCRIBE_FILES_AGENT_NAME)
        to_describe = {
            file.path: file.content.content for file in self.current_state.files if not file.meta.get("description")
        }
        log.debug(f"Files to describe: {list(to_describe.keys())}")

        for file in self.next_state.files:
            content = to_describe.get(file.path)
            if content is None:
                continue

            if content == "":
                file.meta = {
                    **file.meta,
                    "description": "Empty file",
                    "references": [],
                }
                continue

            log.debug(f"Describing file {file.path}")
            await self.send_message(f"Describing file {file.path} ...")

            convo = (
                AgentConvo(self)
                .template(
                    "describe_file",
                    path=file.path,
                    content=content,
                )
                .require_schema(FileDescription)
            )
            llm_response: FileDescription = await llm(convo, parser=JSONParser(spec=FileDescription))
            log.debug(f"Received description for file {file.path}")

            file.meta = {
                **file.meta,
                "description": llm_response.summary,
                "references": llm_response.references,
            }
        log.debug("Finished describing files.")
        return AgentResponse.done(self)
    except Exception as e:
        log.error(f"An error occurred during the describe_files method: {e}", exc_info=True)
        return AgentResponse.error(self, f"An error occurred while describing files: {str(e)}")