proto-app-03-bulk-processor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "36a745d9-2003-496e-a394-290f63a970b7",
   "metadata": {},
   "source": [
    "# Speed Skydiving scoring bulk files processor\n",
    "Uses the `./data` directory as a data lake.  It finds all valid FlySight files in that directory, from 0 to as many as it finds and fit in memory, and process them all.  Summary of \"all files\" or \"all jumps\" at the bottom of the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7619c1b4-c552-4bec-8d38-ce895845d8df",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "!make local"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98c7a164-9492-4672-904b-5bf3abf61773",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "from collections import namedtuple\n",
    "from copy import deepcopy\n",
    "\n",
    "from ssscoring.calc import aggregateResults\n",
    "from ssscoring.calc import convertFlySight2SSScoring\n",
    "from ssscoring.calc import dropNonSkydiveDataFrom\n",
    "from ssscoring.calc import getSpeedSkydiveFrom\n",
    "from ssscoring.calc import isValidJump\n",
    "from ssscoring.calc import isValidMinimumAltitude\n",
    "from ssscoring.calc import jumpAnalysisTable\n",
    "from ssscoring.calc import processAllJumpFiles\n",
    "from ssscoring.calc import roundedAggregateResults\n",
    "from ssscoring.calc import totalResultsFrom\n",
    "from ssscoring.constants import BREAKOFF_ALTITUDE\n",
    "from ssscoring.constants import FT_IN_M\n",
    "from ssscoring.constants import PERFORMANCE_WINDOW_LENGTH\n",
    "from ssscoring.flysight import getAllSpeedJumpFilesFrom\n",
    "from ssscoring.flysight import validFlySightHeaderIn\n",
    "from ssscoring.notebook import SPEED_COLORS\n",
    "from ssscoring.notebook import graphAltitude\n",
    "from ssscoring.notebook import graphAngle\n",
    "from ssscoring.notebook import graphJumpResult\n",
    "from ssscoring.notebook import initializeExtraYRanges\n",
    "from ssscoring.notebook import initializePlot\n",
    "\n",
    "import csv\n",
    "import os\n",
    "import os.path as path\n",
    "\n",
    "import bokeh.plotting as bp\n",
    "import bokeh.models as bm\n",
    "import ipywidgets as widgets\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef1d2bf0-eaf3-4d78-b6c0-fb1bb4ed10f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_LAKE_ROOT = './data'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d52db03d-d83b-4af3-9137-3011537e5ed8",
   "metadata": {},
   "source": [
    "---\n",
    "## <span style=\"color: yellow\">IMPORTANT</span> - Set the drop zone altitude MSL\n",
    "\n",
    "Set the value in ft (even though SSScoring uses meters) out of convenience because all DZs in the US, many in Asia and Mexico, and many in Europe express the altitude in feet.\n",
    "\n",
    "### Some DZs\n",
    "\n",
    "| Drop zone | Alt (ft) |\n",
    "|-----------|----------|\n",
    "| Bay Area Skydiving | 23 |\n",
    "| Paraclete XP| 304 |\n",
    "| SkyDance SkyDiving | 100 |\n",
    "| Skydive Arizona | 1509 |\n",
    "| Skydive Chicago | 616 |\n",
    "| Thai Sky Adventures | 21 |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c95013a-4aa3-4a5e-9493-b945ead10ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "dropZoneAltMSL = 616\n",
    "ignoreBaseline = False # Set to False to evaluate any baseline tracks present in the data lake."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1f091813-03b0-45bc-a30e-704fac7f968a",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "dropZoneAltMSLMeters = dropZoneAltMSL/FT_IN_M\n",
    "display(widgets.HTML('<h2>DZ Altitude = <span style = \"color: green\">%7.2f ft</span> (%7.2f m)<h1>' % (dropZoneAltMSL, dropZoneAltMSLMeters)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1273b5c-44c1-46c2-a125-d01ad9268a1a",
   "metadata": {},
   "outputs": [],
   "source": [
    "jumpFiles = getAllSpeedJumpFilesFrom(DATA_LAKE_ROOT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53a5fe5a-dd39-40d3-bfbf-d7048374e4ea",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "%%capture warnings\n",
    "jumpResults = processAllJumpFiles(jumpFiles, altitudeDZMeters = dropZoneAltMSLMeters)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "797ab0ec-d5e0-4a3b-aa57-eee431bb9824",
   "metadata": {},
   "source": [
    "### Air density\n",
    "\n",
    "**<a href='https://aviex.goflexair.com/blog/density-of-air-air-density' target='_blank'>Pilot's Guide to Air Density</a>** - A useful guide with calculations, lots of information.\n",
    "\n",
    "![](https://ciurana.eu/personal/air-density-partial-table.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d80518ef-f197-4e40-a74e-8089eecb725a",
   "metadata": {},
   "source": [
    "---\n",
    "## Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27c12cb1-2f60-44c9-b8fa-998b3df9d878",
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregate = aggregateResults(jumpResults)\n",
    "if ignoreBaseline:\n",
    "    aggregate = aggregate[~aggregate.index.str.contains('baseline')]\n",
    "sumResults = totalResultsFrom(aggregate)\n",
    "display(aggregate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa8c207-b9cf-4275-ba37-5103e1b324a9",
   "metadata": {},
   "source": [
    "### Rounded results for training log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e2bfbc21-3121-436a-904f-547edb0f2d0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "roundedResults = roundedAggregateResults(aggregate)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faf1396d-1030-477e-9d87-5c167b61328f",
   "metadata": {},
   "source": [
    "### All jumps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a0571fd-5a22-4dda-94aa-56a4f2c68098",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def dumpSimple(table: pd.DataFrame):\n",
    "    fileName = os.path.join(DATA_LAKE_ROOT, resultRef+'.tsv')\n",
    "    simpleTable = table.copy()\n",
    "    simpleTable = simpleTable.drop('speedAngle', axis = 1)\n",
    "    simpleTable.to_csv(fileName, sep = '\\t', index = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ff50f6-4f29-4282-a55f-af524921af19",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "def displayJumpDataIn(resultsTable: pd.DataFrame):\n",
    "    table = resultsTable.copy()\n",
    "    # Experimental\n",
    "    # For more information on the `interpolate` method and its options, see the [pandas documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html).\n",
    "    # Additionally, you can also consider using other libraries like `scipy` which provides more advanced interpolation methods, such as `interp1d` or `griddata`. See the [scipy documentation](https://docs.scipy.org/doc/scipy/reference/interpolate.html) for more information.\n",
    "    table.vKMh = table.vKMh.apply(round)\n",
    "    table.hKMh = table.hKMh.apply(round)\n",
    "    table['altitude (ft)'] = table['altitude (ft)'].apply(round)\n",
    "    table.netVectorKMh = table.netVectorKMh.apply(round)\n",
    "    table.index = ['']*len(table)\n",
    "    display(table)\n",
    "    # dumpSimple(table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe6156be-7071-47a7-bb7f-9724f6d4a3ac",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "allJumpsPlot = initializePlot('All jumps in set')\n",
    "jumpNumber = 0\n",
    "mixColor = 0\n",
    "for resultRef in sorted(jumpResults.keys()):\n",
    "    if ignoreBaseline and 'baseline' in resultRef:\n",
    "        continue\n",
    "    jumpResult = jumpResults[resultRef]\n",
    "    if jumpResult.score > 0.0:\n",
    "        validJumpStatus = '<hr><h1><span style=\"color: %s\">%s jump - %s - score = %.02f km/h</span></h1>' % (jumpResult.color, resultRef, jumpResult.result, jumpResult.score)        \n",
    "    else:\n",
    "        validJumpStatus = '<hr><h1><span style=\"color: %s\">%s jump - %s</span></h1>' % (jumpResult.color, resultRef, jumpResult.result)\n",
    "    display(widgets.HTML(validJumpStatus))\n",
    "\n",
    "    maxSpeed = jumpResult.maxSpeed\n",
    "    window = jumpResult.window\n",
    "    mixColor = (mixColor+1)%len(SPEED_COLORS)\n",
    "    if jumpResult.score > 0.0:\n",
    "        display(widgets.HTML('<h3>Max speed = {0:,.0f}; '.format(maxSpeed)+('exit at %d m (%d ft), end scoring window at %d m (%d ft)'%(window.start, 3.2808*window.start, window.end, 3.2808*window.end))))\n",
    "        if not isValidMinimumAltitude(jumpResult.data.altitudeAGL.max()):\n",
    "            display(widgets.HTML('<span style=\"color: yellow\"><span style=\"font-weight: bold\">Warning:</span> exit altitude ASL was lower than the minimum scoring altitude according to IPC and USPA.'))\n",
    "        display(widgets.HTML('</h3>'))\n",
    "        # TODO: Move this to an in-notebook function or module.\n",
    "        displayJumpDataIn(jumpResult.table)\n",
    "        individualPlot = initializePlot(resultRef)\n",
    "        individualPlot = initializeExtraYRanges(individualPlot, startY = min(jumpResult.data.altitudeAGLFt)-500.0, endY = max(jumpResult.data.altitudeAGLFt)+500.0)\n",
    "        graphAltitude(individualPlot, jumpResult)\n",
    "        graphAngle(individualPlot, jumpResult)\n",
    "        hoverValue = bm.HoverTool(tooltips = [ ('Y-val', '@y{0.00}'), ])\n",
    "        individualPlot.add_tools(hoverValue)\n",
    "        graphJumpResult(individualPlot,\n",
    "                        jumpResult,\n",
    "                        lineColor = SPEED_COLORS[0])\n",
    "        graphJumpResult(allJumpsPlot,\n",
    "                        jumpResult,\n",
    "                        lineColor = SPEED_COLORS[mixColor],\n",
    "                        legend = '%s - %.2f' % (resultRef, jumpResult.score),\n",
    "                        showIt = False)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a5d31990-f5c2-43d9-817b-b50a35740834",
   "metadata": {},
   "source": [
    "---\n",
    "## Summary of all speed skydives in the data lake"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "491440ae-2efa-47fa-856c-25825a6fa4de",
   "metadata": {},
   "source": [
    "**<a href='https://intimescoring.com/Results/EventResults?dbid=100&mid=1074&smid=0&mdcid=1273&dseid=0' target='_blank'>InTime Scoring - 2024 USPA Speed Open results</a>**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17991fcf-e215-4a53-a6ff-d9a64cbafd94",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    }
   },
   "outputs": [],
   "source": [
    "display(roundedResults)\n",
    "display(sumResults)\n",
    "bp.show(allJumpsPlot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee49d352-aa6e-4ab6-88a0-941362f918e8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}