Fix unusual multiword-entries

[flammie]

There is a high number of entries in current apertium-eng.eng.dix containing white-spaces that are not your typical lexical entries. These should be moved to various -separable dictionaries or just removed altogether. Examples:

<e lm="a holiday">       <i>a<b/>holiday</i><par n="house__n"/></e>
<e lm="foreign language"><i>foreign<b/>language</i><par n="house__n"/></e>
<e lm="very strong wind"><i>very<b/>strong<b/>wind</i><par n="house__n"/></e>
<e lm="shortly after">   <i>shortly<b/>after</i><par n="at__pr"/></e>
<e r="RL" lm="shortly after"><i>shortly<b/>after</i><par n="after__cnjadv"/></e>
<e r="LR" lm="that's why"><p><l>that's<b/>why</l><r>that<b/>is<b/>why</r></p><par n="after__cnjadv"/></e>

[mr-martian]

I think a lot of these phrases are originally from eng-deu. I can do some grepping in an hour or two and verify this.

[flammie]

Yeah, I can envision many of them happening because of certain specific languages, maybe some were even added from Finnish too, but we need some future proof solution for language specific hacks to monodix, e.g. -separables.

I listed a handful of examples but there're dozens if not hundreds in -eng, not all of them equally questionable...

[mr-martian]

See #35 for the ~900 that are only in eng-deu.

The biggest users of -eng multiwords seem to be fin-eng, isl-eng, and eng-deu.

[flammie]

See #35 for the ~900 that are only in eng-deu.

These seem good, but I guess they should be reviewed by a native speaker ;-)

The biggest users of -eng multiwords seem to be fin-eng, isl-eng, and eng-deu.

a large stash eng-fin ones were probably added semi-automatically from untrimmed debug output or other questionable sources and can be simply deleted... but if you have some scripts in place to easily generate a list I could have a lookl.

[mr-martian]

eng_mwe_loc.txt

Here's the multiwords not affected by #35 and what bidixes they appear in.

[flammie]

hmm yeah its a real mishmash of things ranging from acceptable lexical units to random combination of adjacent words...

I don't know if there's any good heuristic to decide if they go to language specific or monolingual part than going through the list by hand, maybe someone can come up with tactics?