Loading data via GET request

.github/dependabot.yml

@@ -0,0 +1,10 @@
+# Set update schedule for GitHub Actions
+
+version: 2
+updates:
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      # Check for updates to GitHub Actions every month
+      interval: "monthly"

.github/workflows/validateTeachingMaterialTable.yml

@@ -0,0 +1,25 @@
+name: validate data/teachingmaterial.tsv
+
+# Trigger the workflow on push or pull request, but only for the main branch
+on:
+  pull_request:
+  push:
+    branches: [main]
+
+jobs:
+  validate:
+    name: validate file
+    runs-on: ubuntu-latest
+    steps:
+
+    - name: Clone repo
+      uses: actions/checkout@v3
+
+    - name: Install elm and elm-test-rs as defined in elm-tooling.json
+      uses: mpizenberg/[email protected]
+
+    - name: Create module TeachingMaterialData with teachingMaterialString
+      run: ./script_teachingmaterial_tsv2elm.sh
+
+    - name: Try parsing teachingMaterialString by running the pseudo-test
+      run: elm-test-rs

.gitignore

@@ -4,7 +4,6 @@ elm-stuff
 repl-temp-*
 
 # rendered, intermediate data products
-src/MapOfComputionalArchaeology.elm
 src/TeachingMaterialData.elm
 src/TeachingMaterialEdges.elm
 elm.js

README.md

@@ -1,17 +1,29 @@
 # MapofComputationalArchaeology
 
-## Data
+This repo stores code and data for a small web app to list teaching material in the field of Computational Archaeology: https://sslarch.github.io/MapofComputationalArchaeology
 
-Raw data lives in `data`.
+The teaching material table is stored in `data/teachingmaterial.tsv` and the web app queries it directly from the master branch on GitHub. **To add new material or modify the existing entries, just edit this file and submit a pull request with your changes.** The file will then be structurally validated by a GitHub action and can be merged if it is sound.
 
-The original version of the base map (`data/Archaeologia.map`) was created with this [Fantasy-Map-Generator](https://azgaar.github.io/Fantasy-Map-Generator) and can theoretically still be edited with it. The derived file `data/comparchmap.svg` replaced it, though, for all means and purposes, as several manual steps have to be performed to wrangle the output of the map generator into a .svg file usable with the elm-charts library (simplifying layers, merging objects, replacing labels). Ideally future changes should be performed on the .svg file only. For more complex changes (coastlines) it may be necessary to go back to the .map file.
+## Developer notes
 
-Elm can not read the .svg file on the fly. Instead the drawing has to be converted to Elm code with the [svg2elm](https://github.com/pinata-llc/svg2elm) tool which directly creates a dedicated Elm module for it. Running `script_map_svg2elm.sh` executes the relevant bash command. This takes a minute, because the .svg file is fairly large. The resulting module `src/MapOfComputionalArchaeology.elm` is a derived data product and therefore not logged with Git to safe space.
+This web app was written in [Elm](https://elm-lang.org). See the `src` directory for the code and look [here](https://guide.elm-lang.org/install/elm.html) for instructions on how to run and test it locally with `elm reactor`.
 
-The teaching material table is stored in `data/teachingmaterial.tsv`. Just as the map is has to be translated to Elm code to make it accessible for the webapp. The script `script_teachingmaterial_tsv2elm.sh` creates a module `src/TeachingMaterialData.elm` with a string ready to be parsed. Just as the map module this is a derived data product and not logged by Git.
+### Deploying the app
 
-## Elm
+To prepare the app for deployment on GitHub run `script_build_elmjs.sh`, which will transpile the Elm code in `src` to a single, large `.js` file, and then reduce its size with [uglifyjs](https://github.com/mishoo/UglifyJS) to produce the minified script `elm.min.js`. This file stores the entire app. The `index.html` file finally calls this script. This is what GitHub detects and shows as the website.
 
-The webapp was written in [Elm](https://elm-lang.org/) and exists in a single file `src/Main.elm`. See the instructions [here](https://guide.elm-lang.org/install/elm.html) on how to run and explore the app with `elm reactor`.
+### Validating `data/teachingmaterial.tsv`
 
-To prepare the app for deployment on GitHub run `script_build_elmjs.sh`, which will transpile the Elm code in `src` to a single, large `.js` file and then reduce its size with [uglifyjs](https://github.com/mishoo/UglifyJS) to produce the script `elm.min.js`. This file stores the entire app with all data and code. The `index.html` file finally runs this script. This is what GitHub detects and shows as the website.
+To allow for convenient, yet safe editing of the input data in `data/teachingmaterial.tsv` an automatic GitHub action workflow runs upon every push to the main branch. This workflow is specified in `.github/workflows/validateTeachingMaterialTable.yml` and involves the following main steps:
+
+1. The Elm compiler and the testing tool elm-test-rs are installed as specified in `elm-tooling.json`.
+2. The script `script_teachingmaterial_tsv2elm.sh` is run to create a module `src/TeachingMaterialData.elm` with a string-representation of the table.
+3. elm-test-rs triggers the pseudo-test in `tests/Tests.elm`, which runs the parsing code in `src/TeachingMaterial.elm` on the string-representation in `src/TeachingMaterialData.elm`.
+
+If the parsing fails in any way, then the GitHub action will fail, reporting the specific parsing issue. If it does not fail, then the new version of `data/teachingmaterial.tsv` can be read by the app and is safe to merge.
+
+### Changing the fantasy map
+
+The original version of the base map (`data/Archaeologia.map`) was created with this [Fantasy-Map-Generator](https://azgaar.github.io/Fantasy-Map-Generator) and can theoretically still be edited with it. The derived file `data/comparchmap.svg` replaced it, though, for all means and purposes, as several manual steps have to be performed to wrangle the output of the map generator into a .svg file usable with the elm-charts library (simplifying layers, merging objects, replacing labels). Ideally future changes should be performed on the .svg file only. Only for more complex changes (coastlines) it may be necessary to go back to the .map file.
+
+If the `data/comparchmap.svg` is changed then it has to be transformed to Elm code by running `script_map_svg2elm.sh`. Elm can not read the .svg file on the fly. Instead the drawing has to be converted to Elm code with the [svg2elm](https://github.com/pinata-llc/svg2elm) tool which directly creates a dedicated Elm module (`src/MapOfComputionalArchaeology.elm`) for it. This takes a minute, because the .svg file is fairly large.

elm-tooling.json

@@ -0,0 +1,6 @@
+{
+  "tools": {
+    "elm": "0.19.1",
+    "elm-test-rs": "3.0.0"
+  }
+}

elm.json

@@ -13,6 +13,7 @@
             "elm/core": "1.0.5",
             "elm/file": "1.0.5",
             "elm/html": "1.0.0",
+            "elm/http": "2.0.0",
             "elm/svg": "1.0.1",
             "elm/virtual-dom": "1.0.3",
             "elm-community/maybe-extra": "5.3.0",
@@ -39,7 +40,11 @@
         }
     },
     "test-dependencies": {
-        "direct": {},
-        "indirect": {}
+        "direct": {
+            "elm-explorations/test": "2.2.0"
+        },
+        "indirect": {
+            "elm/random": "1.0.0"
+        }
     }
 }

src/Main.elm

@@ -3,8 +3,7 @@ module Main exposing (..)
 import MapOfComputionalArchaeology exposing (comparchmap)
 import TeachingMaterial exposing (
     TeachingResource, Difficulty (..),
-    makeDummyResource, teachingResources,
-    difficultyToString)
+    makeDummyResource, difficultyToString, parseTeachingResources)
 
 import Bootstrap.Alert as Alert
 import Bootstrap.Button as Button
@@ -16,6 +15,7 @@ import Bootstrap.Modal as Modal
 import Bootstrap.Utilities.Spacing as Spacing
 import Browser
 import Browser.Events as E
+import Browser.Navigation as Navigation
 import Chart as C
 import Chart.Attributes as CA
 import Chart.Events as CE
@@ -27,6 +27,7 @@ import FontAwesome.Styles as Icon
 import Html as H exposing (Html, a, br, button, div, h1, input, p, span, text)
 import Html.Attributes as HA exposing (href, placeholder, style)
 import Html.Events as HE exposing (onInput)
+import Http as Http
 import List exposing (map, concat, sort, any, member)
 import Maybe.Extra exposing (values)
 import Select as Select
@@ -77,6 +78,9 @@ type alias Model = {
     , selectedElement           : Maybe TeachingResource
     -- welcome
     , welcomeVisibility         : Modal.Visibility
+    -- error
+    , errorVisibility           : Modal.Visibility
+    , errorMessage              : String
     }
 
 type alias QueryModel =
@@ -102,15 +106,15 @@ type Dragging =
 -- INIT
 
 init : Int -> List TeachingResource -> ( Model, Cmd Msg )
-init wW elements =
+init wW resources =
     let model = { windowWidth = wW
                 -- table and data
-                , elements = teachingResources
+                , elements = resources
                 , tableState = Table.sortBy "ID" True
                 , multiQueryContent = [ ]
                 , multiQuery = {
                     id = "exampleMulti"
-                  , available = (map .tags teachingResources |> concat) ++ (map .programmingLanguage teachingResources |> concat) |> sort
+                  , available = (map .tags resources |> concat) ++ (map .programmingLanguage resources |> concat) |> sort
                   , itemToLabel = identity
                   , selected = [ ]
                   , selectState = Select.init ""
@@ -138,6 +142,9 @@ init wW elements =
                 , selectedElement = Nothing
                 -- welcome
                 , welcomeVisibility = Modal.shown
+                -- error
+                , errorVisibility = Modal.hidden
+                , errorMessage    = ""
                 }
     in ( model, Cmd.none )
 
@@ -154,6 +161,9 @@ filter minChars toLabel query items =
 
 type Msg =
       SetWindowWidth Int
+    -- download data
+    | SendHttpRequest
+    | DataReceived (Result Http.Error String)
     -- table and data
     | SetMultiQuery (MultiQueryMsg String)
     | ButtonAddToQuery String
@@ -173,12 +183,23 @@ type Msg =
     | ShowModal (Maybe TeachingResource)
     -- welcome
     | CloseWelcome
+    -- error
+    | CloseError
+    | ShowError String
 
 update : Msg -> Model -> ( Model, Cmd Msg )
 update msg model =
     case msg of
         SetWindowWidth w ->
             ({ model | windowWidth = w }, Cmd.none)
+        -- download data
+        SendHttpRequest -> ( model, getTeachingResources )
+        DataReceived res -> case res of
+            Err e -> update (ShowError "Failed to download data from GitHub.") model
+            Ok  x ->
+                case parseTeachingResources x of
+                    Err e -> update (ShowError ("Can't parse data. " ++ e)) model
+                    Ok  y -> ({ model | elements = y }, Cmd.none)
         -- map
         OnMouseClick closestPoint ->
             case (List.head (filterClosestPointToRealEntries closestPoint)) of
@@ -252,7 +273,20 @@ update msg model =
             ({ model | modalVisibility = Modal.shown, selectedElement = element } , Cmd.none)
         -- welcome
         CloseWelcome ->
-            ({ model | welcomeVisibility = Modal.hidden } , Cmd.none)
+            ({ model | welcomeVisibility = Modal.hidden }, getTeachingResources)
+        -- error
+        CloseError ->
+            ({ model | errorVisibility = Modal.hidden }, Navigation.reload)
+        ShowError e ->
+            ({ model | errorVisibility = Modal.shown, errorMessage = e }, Cmd.none)
+
+getTeachingResources : Cmd Msg
+getTeachingResources =
+    Http.get {
+          url = "https://raw.githubusercontent.com/sslarch/MapofComputationalArchaeology/main/data/teachingmaterial.tsv"
+          --url = "https://raw.githubusercontent.com/poseidon-framework/community-archive/07879ea4828b8e6d3b39cfbbf5bd51f6133a971f/2019_Jeong_InnerEurasia/POSEIDON.yml"
+        , expect = Http.expectString DataReceived
+        }
 
 updateMultiQuery : MultiQueryMsg String -> QueryModel -> ( QueryModel, Cmd (MultiQueryMsg String) )
 updateMultiQuery msg model =
@@ -306,7 +340,9 @@ view devel
         closestPoint,
         modalVisibility,
         selectedElement,
-        welcomeVisibility
+        welcomeVisibility,
+        errorVisibility,
+        errorMessage
     } =
     let
         -- general helper functions and settings
@@ -676,10 +712,27 @@ view devel
                                 text "Made by the "
                               , a [ href "https://sslarch.github.io" ] [ text "SIG for Scripting languages in Archaeology" ]
                               , text " - see the code on "
-                              , a [ href "https://github.com/sslarch/MapofComputationalArchaeology" ] [ text "Github" ]
+                              , a [ href "https://github.com/sslarch/MapofComputationalArchaeology" ] [ text "GitHub" ]
                               ] ]
                 |> Modal.view welcomeVisibility
 
+        -- error
+        detailsError =
+            Modal.config CloseError
+                |> Modal.large
+                |> Modal.hideOnBackdropClick False
+                |> Modal.scrollableBody True
+                |> Modal.h3 [] [ text "Error" ]
+                |> Modal.body [] [
+                    p [] [ H.pre [] [text errorMessage] ],
+                    p [] [
+                      text "Please report this error on "
+                    , a [ href "https://github.com/sslarch/MapofComputationalArchaeology" ] [ text "GitHub" ]
+                    , text "."
+                    ]
+                ]
+                |> Modal.view errorVisibility
+
     in
         -- main layout
         div [] [
@@ -743,7 +796,7 @@ view devel
                         ]
                     ]
                 ])
-            , detailsModal, detailsWelcome
+            , detailsModal, detailsWelcome, detailsError
             ]
 
 filterClosestPointToRealEntries : List (CI.One TeachingResource CI.Dot) -> List (CI.One TeachingResource CI.Dot)

src/TeachingMaterial.elm

@@ -1,9 +1,6 @@
 module TeachingMaterial exposing (
     TeachingResource, Difficulty (..),
-    makeDummyResource, teachingResources,
-    difficultyToString)
-
-import TeachingMaterialData exposing (teachingMaterialString)
+    makeDummyResource, difficultyToString, parseTeachingResources)
 
 import Csv.Decode as Decode exposing (Decoder)
 import String exposing (split, trim)
@@ -95,11 +92,8 @@ decodeTeachingResource =
             |> Decode.pipeline (Decode.field "Link" Decode.string)
             |> Decode.pipeline (Decode.field "Citation" Decode.string)
 
-teachingResources : List TeachingResource
-teachingResources =
+parseTeachingResources : String -> Result String (List TeachingResource)
+parseTeachingResources teachingMaterialString =
     case Decode.decodeCustom {fieldSeparator = '\t'} Decode.FieldNamesFromFirstRow decodeTeachingResource teachingMaterialString of
-        Err x -> -- for debugging of .tsv file and parsing
-                 --let _ = Debug.log "Parsing error" (Decode.errorToString x)
-                 --in []
-                 []
-        Ok x -> x
+        Err e -> Err (Decode.errorToString e)
+        Ok x ->  Ok x

tests/Tests.elm

@@ -0,0 +1,16 @@
+module Tests exposing (..)
+
+import TeachingMaterial exposing (parseTeachingResources)
+import TeachingMaterialData exposing (teachingMaterialString)
+
+import Expect
+import Test exposing (Test, describe, test)
+
+suite : Test
+suite =
+    describe "Pseudo-test for teaching material list validation"
+        [ test "if the .tsv table can be parsed" <|
+            \_ ->
+                let res = parseTeachingResources teachingMaterialString
+                in Expect.ok res
+        ]