Feature request - a better way to add support for illegal word combinations

[oharboe]

In our technical documentation, we want to catch combinations like "dual port", which is ambigious and replace it with clearer terms.

I've added the below in conf.py to check for such illegal word combinations.

Is there a better way to do this?

with open(os.path.join(root_folder, 'nomenclature', 'nomenclature.yaml')) as f:
    forbidden_word_combinations = list(
        map(
            lambda word: word.lower().split(),
            filter(lambda word: len(word.split()) > 1,
                   yaml.safe_load(f)['forbidden'].keys())))

forbidden_word_combinations_max = max(map(len, forbidden_word_combinations))


class MonkeyPatchedSpellingChecker(
        sphinxcontrib.spelling.checker.SpellingChecker):
    def __init__(self,
                 lang,
                 suggest,
                 word_list_filename,
                 tokenizer_lang='en_US',
                 filters=None,
                 context_line=False):
        super().__init__(lang, suggest, word_list_filename, tokenizer_lang,
                         filters, context_line)

    def check(self, text):
        """Yields bad words and suggested alternate spellings.
        """
        window = []
        for word, pos in self.tokenizer(text):
            window.append(word.lower())
            if len(window) > forbidden_word_combinations_max:
                window = window[1:forbidden_word_combinations_max + 1]
            illegal_combo = any(
                map(lambda combo: combo == window[0:len(combo)],
                    forbidden_word_combinations))
            correct = self.dictionary.check(word) and not illegal_combo
            if correct:
                continue

            suggestions = self.dictionary.suggest(word) if self.suggest else []
            line = sphinxcontrib.spelling.checker.line_of_index(
                text, pos) if self.context_line else ""

            yield (' '.join(window)
                   if illegal_combo else word), suggestions, line
        return


sphinxcontrib.spelling.checker.SpellingChecker = MonkeyPatchedSpellingChecker

[dhellmann]

I like the idea of a "jargon" detector. The jargon isn't really misspelling, so it may work better in its own extension instead of trying to add it to this one. Maybe sphinxcontrib.jargon_detector? :-)

You would need to create a builder, but you wouldn't need the directive if the jargon file is always going to be outside of the documentation. The jargon file could probably just be a plain text file with 1 phrase per line, like the spelling word list file is. That would make it simpler to read than the YAML file. Unless maybe the input file offers suggested alternatives? In that case you would want to something more structured.

To look for multi-word strings, it might be simpler to look for the entire phrase in the input text as a substring, instead of running the text through the tokenizer and building up sequences of words. A substring search would be more efficient, and would avoid problems with the tokenizer changing the input (for example, contractions come out as 2 tokens today).

[oharboe]

@dhellmann We're also investigating https://github.com/errata-ai/vale and wondering if that's really what we need to set up rules for consistent terminology.

[dhellmann]

Vale does look useful. Thanks for the link!

[oharboe]

Some more monkey patching. We haven't looked into vale yet. It may be a better way to go about this than to try to force sphinxcontrib spelling into this service.

I share the conf.py between all documents and I have a I have a .yaml file per doc where I can enable/disable forbidden words checking and exclude file names. Excluding a page from checking is useful for the .rst files that lists all the forbidden words :-)

From conf.py:

class MonkeyPatchedSpellingBuilder(sphinxcontrib.spelling.SpellingBuilder):
    def _find_misspellings(self, docname, doctree):
        excluded = Matcher(settings['forbidden-excluded-pattern'])
        self.checker.illegal_words = settings['illegal-words'] and not excluded(
            self.env.doc2path(docname, None))
        return super()._find_misspellings(docname, doctree)


sphinxcontrib.spelling.SpellingBuilder = MonkeyPatchedSpellingBuilder