Automates the analysis of GitHub repositories specifically tailored for usage with large context LLMs. This Python script efficiently fetches README files, repository structure, and non-binary file contents. Additionally, it provides structured outputs complete with pre-formatted prompts to guide further analysis of the repository's content.
- README Retrieval: Automatically extracts the content of README.md to provide an initial insight into the repository.
- Structured Repository Traversal: Maps out the repository's structure through an iterative traversal method, ensuring thorough coverage without the limitations of recursion.
- Selective Content Extraction: Retrieves text contents from files, intelligently skipping over binary files to streamline the analysis process.
To use RepoToTextForLLMs, you'll need:
- Python installed on your system.
- The
githubPython package. - A GitHub Personal Access Token configured as an environment variable (
GITHUB_TOKEN).
- Ensure Python and the required package (
PyGithub) are installed:
pip install PyGithub tqdm- Set your GitHub Personal Access Token as an environment variable:
GITHUB_TOKEN = os.getenv('GITHUB_TOKEN', 'YOUR TOKEN HERE')- Place the script in your desired directory.
- Execute the script in your terminal:
python repototxt.py- Enter the GitHub repository URL when prompted. The script will process the repository and output its findings, including the README, structure, and file contents (excluding binary files), accompanied by analysis prompts.
Contributions to RepoToTextForLLMs are welcomed. Whether it's through submitting pull requests, reporting issues, or suggesting improvements, your input helps make this tool better for everyone.
This project is licensed under the MIT License.