Move corpus configuration to backend

[majsan]

Since it is complicated and time consuming to add and especially maintain corpora in the current setup, we want to move the configuration to the backend. Another benefit is that we will not have to rebuild the frontend when adding corpora.

Exactly how this will look is to be determined. Step 1 is to move as much code as possible out from the corpus configuration files.

[janiemi]

Sounds like a major undertaking. :) I recall that something like this may have been mentioned as a goal years ago.

Even though it’s probably rather early now, if the idea is still in its infancy, below are some comments, questions, wishes and other random thoughts of mine on the matter. (I haven’t asked the opinions of other FIN-CLARIN people, but I don’t think there is anything very controversial here.) I hope this text isn’t too long for a GitHub issue comment. Although I’d be glad to get some comments, I don’t expect you to answer to the questions at this stage; they just reflect what I’d find relevant and important in such a major change.

1. Possible advantages

   I think that in principle, it would be good to have corpus (configuration) information in a single place, and the backend might be a better and more natural place than the frontend, as the corpus data is there anyway and as the configuration data might also serve as metadata for the backend. In some cases, the backend might perhaps adjust some results based on the corpus configuration, or at least people using the backend as a Web service would be able to get information on the corpora from the configuration, such as corpus name and description, without having to resort to the frontend.

   I could imagine that moving corpus configuration to the backend might also make it easier to implement some features that we have been wishing for a long time:

   1. some kind of a faceted corpus selection in the frontend, based on corpus metadata properties; and
   2. cross-site searches: for example, searching in (a selection of) Swedish corpora both in Språkbanken’s Korp and in the Korp of the Language Bank of Finland in a single frontend search (or perhaps also in a single backend search).

   I acknowledge that both of these require much more work than moving corpus configurations to the backend.

2. Format and storage of configurations

   Would the corpus configurations in the backend reside in a MySQL/MariaDB database, from which the backend would query them and send as JSON to the frontend? My concern with the database approach is that it wouldn’t lend itself to version control as easily as text files of some kind. And at least I’d like to have the corpus configurations in version control.

   Or would corpus configurations be Python code, or maybe YAML, converted to JSON for the frontend? Or simply JSON that would be served to the frontend as such? I suppose the backend could perhaps store and serve the configurations directly as JavaScript, but I don’t know if it would be worth the change to move the current configuration files to the backend more or less as such.

3. Flexibility of content

   Whatever the format is, I think it should be flexible or extensible enough to be able to represent also pieces of information that might not be used at Språkbanken. For example, in the Korp of the Language Bank of Finland, we have added to corpus and corpus folder configurations additional metadata properties for e.g. the persistent identifier, licence, link to citation instructions and IPR holder of a corpus, which are shown in the corpus info box and in the sidebar. For example, if the configuration of a corpus folder is the following:

    settings.corporafolders.ylenews_sv.s = {
        title: "Yle svenska webbartiklar 2012–2018 (för alla)",
        description: "Yle svenska webbartiklar 2012–2018, blandad, Korp<br/>Variant öppen för alla: meningarna i en blandad ordning inom varje text och ingen utökad kontextvisning<br/><br/>Korpusen är indelad i delkorpusar enligt år, och artiklarna är ordnade enligt redigeringsdatumet.",
        // Contents are added later with settings.fn.add_corpus_settings
        contents: [],
        info: {
            urn: "urn:nbn:fi:lb-2019120406",
            metadata_urn: "urn:nbn:fi:lb-2019120404",
            licence: settings.licenceinfo.CC_BY,
            cite_id: "ylenews-sv-2012-2018-s-korp",
            homepage: {
                name: "Svenska Yle",
                url: "https://svenska.yle.fi",
                no_label: true,
            },
        },
    };

   its info box shows up as follows, here localized in English:
   

   As far as I understand, our corpora are also more varied in their annotation schemes than Språkbanken’s corpora, so the configuration format should allow multiple different value sets for parts-of-speech, for example.

4. Avoiding duplicating information

   Regardless of the format of the configurations, I’d like to see ways to avoid duplicating information. This covers both the facility of specifying common, reusable attribute definitions and also configuring corpora split into multiple subcorpora, which share much in common.

   We now generate the configurations for many of our corpora partly programmatically with JavaScript code in the mode files. For example, we have a historical newspaper corpus split into (sub)corpora by the year and organized into folders by the decade. The corpus configurations are practically the same except for the year in the corpus name, description and id, so I preferred to generate the definitions with JavaScript instead of copying the same configuration almost 180 times (a part of the configuration code). Another example is a Finnish dialect corpus with each dialect as its own subcorpus, in a two-level hierarchy of folders of dialect areas and regions (configuration code). How would cases like these be handled in the new approach?

   Of course, if the configuration format does not lend itself to deduplicating information, the corpus configurations could be generated from another format better suited for that. However, that would add another (maybe site-specific) format and conversion scripts, and thus another step in configuring a corpus, so I’d find it much better if the actual configuration format supported representing common information only once.

5. Effects on the frontend: corpus selector and modes

   How would moving corpus configurations to the backend affect the frontend? Would the features remain the same, or would there be something that would be more difficult or impossible to support?

   In particular, do you plan to keep the hierarchical corpus selector more or less as it is now? If so, the corpus folder hierarchy needs to be represented somehow in the configurations. As I mentioned above, we’d also like to see a faceted corpus selector, but that could be in addition to the current hierarchical one.

   Moreover, will the different modes with different corpora still be present? I’d think that having corpus configurations in the backend might allow making the modes more flexible, or at least make it easier to move a corpus from one mode to another, if a corpus configuration would specify as labels of some kind the mode (or modes) in which the corpus should appear. Or maybe a mode could be constructed dynamically based on for example the language or annotation scheme of a corpus.

6. Custom attributes, controllers and such

   How would custom attributes be represented in the backend corpus configurations? I’m thinking in particular of the attributes using a custom controller, template and/or sidebar component. Or would they be defined in the frontend and the corpus configurations would refer to them somehow?

7. Localization

   Would corpus attribute name and value translations also be moved to the backend, or would they still reside in the translations/corpora-lg.json files in the frontend? I’d find it more logical if they were also moved to the backend, as the corpus configurations use or refer to them and as they would then also be available to the backend.

   We’d also very much like to see the possibility to localize corpus names and descriptions and other pieces of corpus metadata.

[majsan]

Thanks for all the input! Some comments:

1. Possible advantages

some kind of a faceted corpus selection in the frontend, based on corpus metadata properties; and cross-site searches

This could be nice to have. Our goal is to allow users to upload their own corpora and search those in Korp. Discussions are upcoming, but we must make changes to the corpus chooser anyway and with a rewrite, I think we should at least make sure that the new/rewritten one is more extendable.

2. Format and storage of configurations

Would the corpus configurations in the backend reside in a MySQL/MariaDB database [...] Or would corpus configurations be Python code, or maybe YAML, converted to JSON for the frontend?

Not stored in a database, I agree that versioning is important. But otherwise, we don't now yet. I think either:

• The configuration is still code and generates JSON. A DSL would be nice then, to avoid the general chaos that we have now or:
• A flexible YAML/JSON that avoids duplication and use Python to generate the final JSON files

The final JSON-format could be very similar to what we have now after running the configuration files.

And of course, as you pointed out in 4., anyone with a Korp-instance may do as they choose when it comes to generating the JSON, but it would be nice to have a solution that fits everyone.

3. Flexibility of content

Whatever the format is, I think it should be flexible or extensible enough to be able to represent also pieces of information that might not be used at Språkbanken.

Noted. It will of course not be a problem with the JSON in the end, but the code that creates the JSON have to be flexible enough to accept fields that are not in the standard Korp.

As far as I understand, our corpora are also more varied in their annotation schemes than Språkbanken’s corpora, so the configuration format should allow multiple different value sets for parts-of-speech, for example.

Yes, this is something that must be included. We also have some attributes where the name is the same, but configuration differs.

4. Avoiding duplicating information

How would cases like these be handled in the new approach?

Templating/inheritance is what comes to mind. Make an abstract corpora that the similar ones will inherit/copy and agument with the needed fields. We also need this.

5. Effects on the frontend: corpus selector and modes

How would moving corpus configurations to the backend affect the frontend? Would the features remain the same, or would there be something that would be more difficult or impossible to support?
In particular, do you plan to keep the hierarchical corpus selector more or less as it is now?
Moreover, will the different modes with different corpora still be present?

I think we want to keep as much as possible to not make this task too big.

Or maybe a mode could be constructed dynamically based on for example the language or annotation scheme of a corpus.

Interesting, both this and a faceted selection creates the need for the backend to be able to answer more specific questions than "all corpora in mode X". The backend could have the configurations in a simple memory database/document store, so we can make queries. Dynamic modes seems like an easy thing to implement, but maybe part of a larger discussion first on how we actually want corpus selection to work in the future.

6 Custom attributes, controllers and such

How would custom attributes be represented in the backend corpus configurations?

We will define all custom function/components in the frontend and refer to those with strings in the configuration. (This is the part that I will work on first, to clean up our current configuration files). It is of course possible to define the code as strings and store in the backend configuration files, but will probably just make stuff more complicated. If we need it, we can add flags to the components so that they are configurable. Something like:

sidebarComponent: {
    name: 'autocomplete',
    options: {
        maxItems: '25'
    }
}

To avoid situations where the components are very alike ("autocomplete25items", "autocomplete50items" etc.).

7. Localization

Would corpus attribute name and value translations also be moved to the backend...

Yes! This is a goal, and should happen before we can call this done. I think that it will just simplify if the attribute definitions and translations are closer each other.

We’d also very much like to see the possibility to localize corpus names and descriptions and other pieces of corpus metadata.

I agree that this would be nice and at least some metadata should be easy to allow translations for.

[janiemi]

Thanks for your answers and comments! Your plans and ideas sound very promising to me. Here are some further thoughts of mine.

Our goal is to allow users to upload their own corpora and search those in Korp.

Sounds interesting: such a feature has also been our wish. We have thought it to be rather complicated to implement, but I suppose that moving corpus configuration to the backend would clear one hurdle. I’d like to hear more about that at some point, even though we aren’t using all the components perhaps required by your planned solution, such as Sparv.

2. Format and storage of configurations

I think either:

• The configuration is still code and generates JSON. A DSL would be nice then, to avoid the general chaos that we have now or:
• A flexible YAML/JSON that avoids duplication and use Python to generate the final JSON files

Of these options, I personally might prefer YAML, if it’s easy enough to support all the desired configuration features in it. And if YAML input is supported, it might not be difficult to support JSON input with the same semantics, too. Configurations as (Python) code would probably be the most flexible approach, but I agree that some kind of a DSL would then be nice to cover at least most of the cases.

4. Avoiding duplicating information

Templating/inheritance is what comes to mind. Make an abstract corpora that the similar ones will inherit/copy and agument with the needed fields. We also need this.

Right; that would work at least for corpus attributes. I’d also like to see some kind of templating within individual string values, so that in a corpus template, you could define something like name: "Corpus {year}", and when the template is instantiated, year would be passed as a parameter and filled in the string values of the template.

5. Effects on the frontend: corpus selector and modes

[…]
Interesting, both [dynamic modes] and a faceted selection creates the need for the backend to be able to answer more specific questions than "all corpora in mode X".

I don’t know if that would be required by faceted corpus selection in the frontend if it were applied to the corpora of a specific mode whose configurations had already been retrieved from the backend. However, I think faceted queries might also be useful for those using the backend as a Web service, so I’d like to see that kind of a feature.

6. Custom attributes, controllers and such

We will define all custom function/components in the frontend and refer to those with strings in the configuration.

Sounds reasonable.

It is of course possible to define the code as strings and store in the backend configuration files, but will probably just make stuff more complicated.

I think that on the one hand, it would be nice to have all the corpus configuration code in the same place (backend), but on the other hand, it would be better to have the JavaScript code of the components and functions where other JavaScript code is (frontend). Having them in the frontend might also make it easier to reuse code. And if a change in the frontend requires modifying the code of a component, you wouldn’t need to touch the configuration in the backend.

If we need it, we can add flags to the components so that they are configurable.

I think some kind of component parametrization would indeed be great.

7. Localization

[…]
I agree that this would be nice and at least some metadata should be easy to allow translations for.

I’ve long since had the idea of supporting in corpus (and in particular corpus folder) configurations something like

name: {
    fi: "Sanomalehti",
    sv: "Tidning",
    en: "Newspaper",
},

but I’ve never got around implementing it myself and I probably won’t do that now that the goal is to move configurations to the backend.

[janiemi]

(These questions might be more appropriate in the Korp backend repo, but as this issue is open here and as the backend issues seem rather inactive, I ask them here.)

What is the current state of moving Korp corpus configurations to the backend? I have noticed that the code in the dev branch of the frontend has been retrieving the configurations from the backend for some time now and that the dev branch of the korp-frontend-sb repo now only contains general Korp configuration and no longer corpus configurations. I thus assume the work has progressed quite nicely but is not in production quite yet. However, I haven’t yet seen the configuration backend nor the source format of the corpus configurations the backend uses. Do you still have much work to do on either the backend or the frontend side?

By reading the frontend code (and config.yml in the korp-frontend-sb repository), I see that the configurations are served by a development version of the Korp backend. However, the code in the korp-backend GitHub repo has last been updated over two years ago. I can also get the configurations from Språkbanken’s backend as JSON, but I assume the source format that the backend reads is different.

We’d thus like to get at least a preview of what the corpus configurations now look like, in order to estimate the work we’ll have ahead in converting our corpus configurations to the new format. You perhaps have a script for converting the JavaScript configurations to the new format (have you?), but our extensions and they ways we have used JavaScript code to generate corpus configurations might mean more work in the conversion.

If possible, we’d also like to see the backend code processing the configurations, to assess if we need to amend it somehow to cater for our needs. Moreover, it’s now a bit difficult for us to try to keep up with your frontend development code without being able to use the backend supporting corpus configurations. However, I also understand that you might not wish to disclose code (or a configuration format) that is under heavy development and that might still change significantly. In contrast, I don’t think the possible lack or sparsity of documentation would be a major problem for us at this stage.

[EDITED: I just got information from Krister Lindén who had talked to Markus Forsberg that the Min Språkbank service being developed is not yet in a state that you would make it available to others. When it is ready enough, we’re interested in getting it, too, to see if or how we could adapt it to our needs.]

Last but not least: thank you for all the great work!

[janiemi]

@MartinHammarstedt and @majsan, thank you for all the information via email and repositories! I now answer myself based on the answers I got from you, for future record and also for developers at other Korp sites. Please correct or amend me if I misunderstood something.

1. The Korp backend version 8.1.0 supporting the corpus configuration endpoint /corpus_config is now available in the Korp backend repository. According to @MartinHammarstedt, the code is functionally complete for use in Korp, but requires further testing and documentation. In addition, some extra functionality will be added for use with Min Språkbank. (I’m curious as to what kind of extra functionality you need for Min Språkbank.)

2. The corpus configuration for Språkbanken’s Korp in YAML is now available in the korp-config repo.

3. A one-off script was used to convert the JavaScript configurations to YAML. The script is available in a private GitHub repository, and I suppose @majsan can also at request invite other interested parties besides me to have read access to the repo. However, the YAML format was changed afterwards (in particular the representation of attributes, it seems), so the script would need to be modified (or the result converted) before the backend code can use the result. (We’d need to modify it anyway to cover some additions to the corpus configurations of the Korp of the Language Bank of Finland, but I don’t know if it would be easier to do that or to write a conversion script from scratch. I’ll probably return to this issue later.)

4. Discussion about moving corpus configuration to the backend can continue under this issue but if concrete problems or suggestions for improvements come up that mostly concern the backend, they should have their own issues in the backend repo.

[janiemi]

Here are some thoughts of mine on the current corpus configuration representation on the backend and some ideas or proposals for enhancements or additional (optional) features. This is once again rather a long comment, but I hope you bear with me.

A thought on specifying the containing modes and folders in a corpus configuration file

I was first a bit surprised at the fact that the modes and folders for each corpus are now specified in the configuration of the corpus and not that of the folder. I thought it would be faster to find the corpora in all the folders of a mode than to scan all corpora to find those which are to be included in a mode. However, I then realized that the current system allows adding a corpus as a single file, without having to modify any mode file.

Differences between the output of the conversion script and the current configurations

I noticed the following differences between the output of the corpus configuration conversion script and the current YAML configurations:

1. The script can in at least some cases bundle commonly used groups of attributes that can be referred to in the configuration as a group (via inherits), whereas in the current YAML configurations, each attribute is defined and referred to individually. However, at least the latest version of the script I tried didn’t seem to use inherits as extensively as I’d have expected. An attribute group may contain both positional and structural (and custom) attributes, whereas the attribute definitions in the current scheme separate positional and structural attributes, and apparently custom attributes need to be defined inline.

2. The attribute definitions generated by the script include the name of the attribute, whereas those in the current configurations only contain the actual definition that can be used for differently named attributes.

3. The current configurations have separate label and value for context and within.

4. The script generates camelCase for some dictionary keys, whereas the current configurations use snake_case.

I think the current attribute definitions are in general simpler than those generated by the script. It also seemed to me that many commonly used groups of attributes were not grouped, such as those corresponding to modernAttrs.

Did you convert the output of the conversion script to the current configurations step by step, with one-off scripts or one-liners, or how did you do that?

General thoughts regarding enhancements of the configuration format

I think any improvements to the corpus configuration format should be backward-compatible: the existing corpus configurations should continue to work as they are, so that you need not change your corpus configurations unless you wish to do so.

I admit that I’m a bit hesitant to suggest enhancements to your carefully crafted configuration representation, but it was a bit different than I had expected. In particular, I’d be keen to reduce redundancy in both the YAML source and the resulting JSON and JavaScript objects, even though that means some extra processing.

I think I should be able to implement any changes myself but I’d like to get feedback from you (@majsan and @MartinHammarstedt and whoever else might be or may wish to be involved), in particular on such features that you have already considered and rejected for some reason. And please also tell me if some of my ideas conflict with your future plans.

Many of the features would affect or require support in both the backend and frontend. I’ll list them here briefly but I can open new issues in the backend repository for more details of at least those that would primarily affect the backend.

Enhancement ideas

Then to my enhancement ideas. I have marked with a ⭐ those that I’d definitely like to have. Please let me know what you think.

1. ⭐ Allow defining common values of lists (groups) of attributes.

   pos_attributes, struct_attributes and custom_attributes could get a string value, which would refer to a file in subdirectory attribute_groups of the configuration (or something like that, perhaps with separate subdirectories for positional, structural and custom attributes), similarly to how attributes can now be defined.

   I think the values could be handled in the frontend: whenever the value of these properties in the configuration JSON returned by the backend is a string, the frontend would replace it with a reference to the named object in the configuration, such as attribute_groups["modern"].

2. Allow referring to a list of attributes and extending it with attributes defined inline.

   This is a possible extension of the above. An attribute list could be referred to as a string value (instead of a single-valued mapping) in the YAML list of attributes, converted to something like @attrlist in the JSON. The frontend could then use $.extend or the like create the actual list of attributes in JavaScript. However, I don’t know if this would add too much complication for relatively little gain, so this is not a priority of mine.

3. ⭐ Allow common values for context and within at corpus level and for opts and possibly translation at attribute level.

   I think this could be implemented similarly to attribute lists above. The properties for which this feature is supported could perhaps be configurable, as we’d like to have at least one extra one that might also be defined at corpus folder level. The values could be defined in files in subdirectories contexts, withins, options and translations of the configuration, similarly to how attributes can now be defined.

   I think the values could be handled in the frontend: whenever context, within, opts or translation has a string value, replace it with a reference to the named object in the configuration, such as opts["set_options"].

4. ⭐ Support at least one level of subdirectories in the corpora directory.

   Even though having all corpus configuration files in a single directory may have its advantages, Språkbanken’s corpus configuration currently contains more than 1,000 files in corpora, which in my opinion is makes it a bit difficult to handle, and we have even more corpora. The simplest option would be to just scan (possibly recursively) all subdirectories of corpora and process the files as now, but I propose some extensions below.

5. ⭐ Support corpus configuration templates.

   Definitions common to many corpora could be extracted to a template file, and the corpus configurations containing only the variable parts would refer to the template. The corpus configurations could perhaps also override values in the template.

   I see at least two possible (and probably complementary) ways to implement template files:

   1. Templates are in a separate directory, say, corpus_templates, and corpus configurations would refer to them explicitly, e.g., template: kubhist2-norrkopingstidningar.

   2. ⭐ A template is specified implicitly for all corpus configurations in a corpus subdirectory in a file named _template.yaml (or perhaps Template.yaml, TEMPLATE.yaml or something else if you wish to allow a corpus named _template). In this case, the corpus files need not (or perhaps should not) refer to a template explicitly.

   At least in the first form, templates could perhaps be recursive: a template could refer to another template with common settings shared by several templates.

   I don’t know if it would be better to expand templates in the backend or in the frontend. Expanding in the backend would make the JSON configurations returned by the backend more self-contained, whereas expanding in the frontend would make them more compact. One option could be to have a backend parameter that would specify if the backend should expand them, but then expansion would need to be implemented in both the backend and the frontend.

6. ⭐ Support filling in parts of string values in a template.

   A template could contain id, title and description with placeholders that would be replaced with values in a corpus configuration using the template. I see at least two, possibly complementary options:

   1. Use {} (or whatever is decided) as the placeholder and fill in the value from the corresponding property in the corpus configuration. For example, the template could contain

      id: kubhist2-norrkopingstidningar-{}
      title:
        eng: Norrköpings Tidningar {}'s
        swe: Norrköpings Tidningar {}-talet

      and a corpus configuration:

      id: 1880
      title: 1880

      which would be expanded to

      id: kubhist2-norrkopingstidningar-1880
      title:
        eng: Norrköpings Tidningar 1880's
        swe: Norrköpings Tidningar 1880-talet

   2. Use explicitly named placeholders; that is, in the template:

      id: kubhist2-norrkopingstidningar-{decade}
      title:
        eng: Norrköpings Tidningar {decade}'s
        swe: Norrköpings Tidningar {decade}-talet

      and in a corpus configuration:

      template_fill:
        decade: 1880

      This might make it simpler to have different fill values for the titles and descriptions in different languages.

7. ⭐ Allow mapping a corpus subdirectory to a folder.

   A subdirectory of the corpus configuration directory could correspond to a corpus folder. The subdirectory could contain a file named e.g. _folder.yaml that would contain information on the folder (title, description and id), the mode(s) and parent folder specified as in corpus configurations, and possible subfolder structure. The corpus configurations in the subdirectory wouldn’t then need to (or perhaps shouldn’t) specify the mode and folder, but they would be added to the folder specified in the folder file. If the folder file specifies subfolders, a corpus configuration should specify in which subfolder the corpus is to be added.

   When compiling folder information for a mode, the backend would need to add information in the _folder.yaml files in addition to the mode file.

   If multiple levels of subdirectories are allowed in the corpora directory, each level could map to a level of folders. However, I don’t know if it would be worth the added complexity.

8. Allow configurations of multiple corpora in a single file.

   In particular in conjunction with templates and value placeholders, the configurations for individual corpora might be very short, such as those in the examples above. The configurations for multiple corpora could thus be collected into the same file. Such corpus collection files could be named specially so that they can be distinguished from normal corpus files, such as _collection.yaml, or _collection_something.yaml if multiple such files are allowed in a single directory.

   As an extension to the above I have thought of supporting specifying a range of years or decades that would be used as template fill values, so that you wouldn’t need to specify each year or decade separately. I don’t know if that would be too particular, though.

   However, I understand that it may be valuable to have a separate file for each corpus, so that it may be easier to see which corpora are defined just by looking at the file listing.