feat(evaluate): add signwriting evaluation metrics #1104
Conversation
AmitMY
requested review from
fhieber,
tdomhan,
mjdenkowski,
xingniu and
annacurrey
as code owners
|
Hi Amit, It's great to see that you have an implementation working! We can think about two questions for adding new features to Sockeye:
For now, it looks like you have an implementation that works for your use case and that you could share with other interested users via a forked repository. In that case, you would control everything and wouldn't need to add extra layers of engineering. Running your metric with the main version of Sockeye would be more challenging. Adding support for custom metric scripts is a great idea from a science perspective. Since Sockeye is also built to run in production environments, we need to avoid security vulnerabilities like arbitrary code execution. Designing, implementing, and documenting a secure way to run user-defined metrics would significantly increase the scope of this project. I recommend using your fork of Sockeye for now and revisiting this if using the fork starts to block your work or the work of others working on SignWriting. Best, |
|
Thanks. In any case, for the meanwhile I am successfully using my fork. |
This pull request fixes #1103
Changes here are complete (this works), but we might want to figure out a different solution, in case this is too much hardcoding for you all (perhaps something like an external evaluation script, and a single custom metric that can call that script)
Note! that unlike BLEU or other metrics, this evaluation takes all factors into account, as they matter for SignWriting. Even if using a non-factored setup though, this evaluation works.
What do you think? how can we integrate this into sockeye?
This is very important to me, to push forward the adoption of SignWriting as a written form of signed languages, allowing both transcription and translation models to be trained with sockeye.
Tests fail with the following error, despite an Nvidia Tesla T4 GPU available:
Pull Request Checklist
until you can check this box.
pytest)pytest test/system)./style-check.sh)sockeye/__init__.py. Major version bump if this is a backwards incompatible change.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.