Create a linguistic parser

[atimmer]

Explanation

A TextContainer object should have a getTree method that returns the tree based on the text. This tree should be generated by a linguistic parser that knows how to split a text into sentences and words. The getTree method should return a tree with Sentence and Word objects.

• The Sentence object should contain the content of the sentence and the relative indexes within the text container.
• The Word object should contain the content of the word and the relative indexes within the text container.

Better suggestions for the name of the linguistic parser are welcome. The linguistic parser should use the code we already have available in the current code. So the sentence parser can be reused. I've created an issue to track the removal of the HTML specific code from the sentence parser

Tasks

• Create a method on the textContainer class
  • Create tokenizer for sentences based on current tokenizer
  • Create tokenizer for words based on current tokenizer
  • Call tokenizers in getTree method.
  • Return sentence objects, a sentence object contains word objects
• Create unit tests (copy from existing tokenizer)
  • Create unit tests covering this issue SentenceTokenizer incorrectly processes punctuation marks within words #402

Technical decisions

• The linguistic parser can use the current sentence tokenizer.

Feedback?

[maartenleenders]

I've pushed my work to 406-create-linguistic-parser (not so much yet).

Some clarifications I've gotten during my assignment:

• The issue includes building the linguistic parser helper functions.
• The issue includes building the Word and Sentence objects.

Remaining tasks added to the issue ☝️

[manuelaugustin]

Closed in favor of the following Jira issues: https://yoast.atlassian.net/browse/LIN-80?atlOrigin=eyJpIjoiNjhlOWIzYzgxNjZlNDJkNmIwMzgwMmQ3OTI4MDRhNTQiLCJwIjoiaiJ9 & https://yoast.atlassian.net/browse/LIN-164?atlOrigin=eyJpIjoiZTc5NzdhOGJlMTc0NGNjZWIyODA3NDdhMmUzMjAzMzYiLCJwIjoiaiJ9