Error saving index: RangeError: Invalid string length

[Gitter-MI]

Hi Steven,
vb here from the openai community forum.
When creating an index I ran into this error where the filename is too long which causes the index creation to fail.

Error: Error adding document "C:\Users\VB\source\repos\evals.git\lfs\objects\03\5f\035fbfc20caee391612ae6e8f394b0788406f493b208e67419bb1eea17f9575c": Error: Error saving index: RangeError: Invalid string length
at LocalDocumentIndex. (C:\Users\VeitB\AppData\Roaming\npm\node_modules@stevenic\codepilot\node_modules\vectra\lib\LocalDocumentIndex.js:258:23)
at Generator.throw ()
at rejected (C:\Users\VB\AppData\Roaming\npm\node_modules@stevenic\codepilot\node_modules\vectra\lib\LocalDocumentIndex.js:29:65)

This is the filename "035fbfc20caee391612ae6e8f394b0788406f493b208e67419bb1eea17f9575c". The file has no extension.

Ps. I know I should have excluded the .git folder anyways.

[Gitter-MI]

I tried to add the same project (OpenAI evals) again to vectra using Codepilot and this time made sure to exclude the directory and files that previously caused the error. But the same error was thrown later in the process. I have come to believe that this may actually be an issue with rate limiting on the side of the OpenAI account used. In this particular case the account was Tier 1 with a token limit of 1.000.000 TPM and that the error message does not correctly decribe the reason for the error.
If this assumption is correct then in this scenario it is not possible to create a project because the only way to rebuild is to create from scratch and the same error will occur again.

[Stevenic]

Yeah at some point the JSON just gets to be too big... The limit for an index is around 2,000 files currently. I'm hoping to address that in Vectra 2 with a more compact file format.