Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE REQUEST] Functionality for analyzing the differences between two Annotation objects. #67

Open
AndriyMulyar opened this issue Dec 28, 2018 · 2 comments
Labels
enhancement New feature or request

Comments

@AndriyMulyar
Copy link
Collaborator

What problem does your feature solve?
A method to do analysis of annotations (namely for the application of looking at differences between gold and predicted annotations).

Describe the solution you'd like
The Annotation class should be given some static methods like Annotation.diff(ann_object_1, ann_object_2) will output the difference between to annotation objects. Maybe some parameter for leniency to deal with fuzzy annotation matching.

Interface sklearn to compute various evaluation metrics between two annotation files (assuming one is gold and one is predicted).

Additional context
This would be very useful for result analysis and guiding the building of pipelines.

@AndriyMulyar AndriyMulyar added the enhancement New feature or request label Dec 28, 2018
@AndriyMulyar
Copy link
Collaborator Author

Currently pull request #68 begins preliminary work mentioned above.

Ideas for further improvements:

  1. Method in Dataset that will allow to compare gold and predicted over a whole corpus by utilizing the diff functionality implemented in Added Annotations.diff(); unit tests #68 .
  2. Give the diff method optional fuzzy parameters that will highlight model predictions that are almost correct (maybe off by a few characters)

@swfarnsworth
Copy link
Member

I believe the functionality you described is covered by Annotations.compare_by_index(), which has the strict parameter for fuzzy predictions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants