Introducing the content library

[jhoetter]

Hey, community!

Since we open-sourced refinery, we faced lots of new users and fans of the application. We've heard of many great use cases you're implementing with the application, so we asked ourselves how we can

• help you even more in building projects
• and how to make it easy for you to contribute something to the project, even if you only have little time

Enter the content library! It is designed to give you access to heuristics and enrichers for refinery off-the-shelf. It is a collection of NLP code snippets we've found to be incredibly useful, with or even without refinery (though we of course think you should use refinery for your projects ;-) )

To give you an example, this is the language detection module, which is available both online and as pure open-source:

from typing import Dict, Any
from langdetect import detect

def fn_language_detection(record: Dict[str, Any]) -> str:
    """Detect language of text
        
    Args:
        record (Dict): one single record you want to process

    Returns:
        str: Language of your text
    """

    text = record["your-text"]
    language = detect(text)
    return language

Obviously, we're standing on the shoulder of giants, so many of the modules are using some kind of open-source library to work. We're hoping that this list is going to introduce you to some cool libraries you might haven't heard of yet, that you'll find useful.

If you want to contribute to the repository, please look into our contributions guideline

We're going to add further modules continuously, so it surely makes sense to star this repo. Alternatively, you can also select to "watch" the repo.

Happy to hear any feedback and ideas you people have :)

Johannes for the Kern AI team