ElmTextSearch full text indexer

Copyright (c) 2016 Robin Luiten

This is a full text indexing engine inspired by lunr.js and written in Elm language. See http://lunrjs.com/ for lunr.js

I am happy to hear about users of this package.

I am happy to receive contributions be they bug reports, pull requests, documentation updates or examples.

v4.0.0 will not load indexes saved with old version.

If you do not use storeToValue storeToString fromString fromValue in ElmTextSearch this update is not likely to introduce issues.

The way that filters and transforms are applied to the content of documents has changed. This is to properly fix a bug reported see #10 where stop word filters were not correctly applied. This means saved indexes from previous version of ElmTextSearch will not load in this version.

• Defaults.indexVersion has changed value.

The reason this is a Major version bump is some generalisation was done to enable future support for loading and saving of older version and types of default index configurations.

v5.0.0 updates for Elm 0.19

Result types from loading indexes are now Decode.Error not String.

v5.0.2, v5.1.0

New functions addT for add, searchT for search and removeT for remove. These replace the error type of result with a type. v5.0.2 was a goof on my part i forgot to expose new functions correctly.

Packages

Several packages were created for this project and published separately for this package to depend on.

• trie
• http://package.elm-lang.org/packages/rluiten/trie/latest
• stemmer
• http://package.elm-lang.org/packages/rluiten/stemmer/latest
• sparsevector
• http://package.elm-lang.org/packages/rluiten/sparsevector/latest

Parts of lunr.js were left out

• This does not have an event system.
• Its internal data structure is not compatible.

Notes captured along way writing this.

• lunr.js
• tokenStore.remove does not decrement length, but it doesn't use length really only save/load
• stemmer "lay" -> "lay" "try" -> "tri" is opposite to porter stemmer
• porter stemmer erlang implementation
• step5b does not use endsWithDoubleCons which is required afaik to pass the voc.txt output.txt cases

Example

See examples folder for four examples. You can run any of the examples if you navigate to the examples folder and run elm reactor and select an example in the src folder.

First example is included inline here.

IndexNewAddSearch.elm

module Main exposing (ExampleDocType, createNewIndexExample, main, resultSearchIndex, resultUpdatedMyIndexAfterAdd)

{-| Create an index and add a document, search a document

Copyright (c) 2016 Robin Luiten

-}

import Browser
import ElmTextSearch
import Html exposing (Html, button, div, text)


{-| Example document type.
-}
type alias ExampleDocType =
    { cid : String
    , title : String
    , author : String
    , body : String
    }


{-| Create an index with default configuration.
See ElmTextSearch.SimpleConfig documentation for parameter information.
-}
createNewIndexExample : ElmTextSearch.Index ExampleDocType
createNewIndexExample =
    ElmTextSearch.new
        { ref = .cid
        , fields =
            [ ( .title, 5.0 )
            , ( .body, 1.0 )
            ]
        , listFields = []
        }


{-| Add a document to an index.
-}
resultUpdatedMyIndexAfterAdd : Result String (ElmTextSearch.Index ExampleDocType)
resultUpdatedMyIndexAfterAdd =
    ElmTextSearch.add
        { cid = "id1"
        , title = "First Title"
        , author = "Some Author"
        , body = "Words in this example document with explanations."
        }
        createNewIndexExample


{-| Search the index.

The result includes an updated Index because a search causes internal
caches to be updated to improve overall performance.

-}
resultSearchIndex : Result String ( ElmTextSearch.Index ExampleDocType, List ( String, Float ) )
resultSearchIndex =
    resultUpdatedMyIndexAfterAdd
        |> Result.andThen
            (ElmTextSearch.search "explanations")


{-| Display search result.
-}
main =
    Browser.sandbox { init = 0, update = update, view = view }


type Msg
    = DoNothing


update msg model =
    case msg of
        DoNothing ->
            model


view model =
    let
        -- want only the search results not the returned index
        searchResults =
            Result.map Tuple.second resultSearchIndex
    in
    div []
        [ text
            ("Result of searching for \"explanations\" is "
                ++ Debug.toString searchResults
            )
        ]