Clean PTB trees: remove empty nodes, strip functional tags

[moreymat]

The Penn Treebank contains information that parsers do not use for training nor output:

• empty nodes (for traces),
• grammatical functions.

This PR provides the infrastructure to discard such information so that trees read from the Penn Treebank contain the same information as trees output by any parser, e.g. the Stanford Parser.

Trees without empty nodes are also easier to align with discourse annotations, cf. #13 .

[moreymat]

We need to talk about the color of the bike shed.
Then I will finish to pep8 and pylint it, and provide proper tests.

[kowey]

Also, as I understand it, you're asking me to wait for this to settle a bit? (or are you happy merging this in right away?)

[moreymat]

Please feel free to accept the PR as soon as you feel we reach a satisfying
state (for you).

2014-10-31 17:36 GMT+01:00 Eric Kow [email protected]:

Also, as I understand it, you're asking me to wait for this to settle a
bit? (or are you happy merging this in right away?)

—
Reply to this email directly or view it on GitHub
https://github.com/kowey/educe/pull/14#issuecomment-61288816.

[moreymat]

@kowey minor refactoring done, pylint+pep8 too. I think you can safely merge now.

[moreymat]

In fact this is not ready for merge yet as irit-rst-dt gather just failed. I will let you know when it is debugged.

[kowey]

Now that #15 has been merged (with apologies for my sloppiness), is it safe to merge this?

[moreymat]

I think so, thanks !
At least, irit-rst-dt gather now works on my system, on PTB trees free of empty nodes and functional tags.