Policy for evals introducing new dependencies

[dannyhetharia]

Hi there,

I'm working on (what I think is) a pretty cool eval. There's a python package (it's extremely light, only depends on numpy), which would boost the quality of the evaluation a lot and would save me hours of work. However, if I would be responsible for this repo, I would be extremely hesitant to accept new dependencies for evals, so though I'd check in here first. I wasn't able to find anything regarding the policy about this.

Appreciate your input.

[qrdlgit]

I don't believe they've merged any evals associated with code, though perhaps I missed one. Also - you could be the first!

You can also read this regarding custom evals: #715

Full disclosure, I'm just a user like everyone else here.

[dannyhetharia]

Thanks, I wasn't aware of that discussion yet. The response to #520 was pretty unequivocal though. Feel like I would be wasting my time writing a PR with code. The docs kind of put you on the wrong foot with this.

Anybody out there who can confirm or deny that this is still a hard no?