+post on documenting the file format

_posts/2024-06-27-documenting-the-file-format.markdown

@@ -0,0 +1,23 @@
+---
+title: Documenting the file format
+description: Understanding how the tile index works and how tiles are arranged in the file.
+image: /img/dsat_kaitai.png
+---
+
+Since [the first post on decoding the city
+database](/2005/03/26/decoding-the-city-database.html) I have learned
+so much more about the structure of the D-Sat file format: [where the
+image tiles are
+located](https://dsat.igada.de/2024/04/02/finding-the-tiles.html),
+[how they are
+indexed](https://dsat.igada.de/2024/05/11/visualising-entropy.html)
+and [which information the image header
+reveals](https://dsat.igada.de/2024/04/20/understanding-the-image-header.html),
+as well as [that the file also contains points describing borders and
+highways](https://dsat.igada.de/2024/05/06/finding-somehing-unexpected.html). It
+is time to document this knowledge and with [Kaitai
+Struct](http://kaitai.io/) I found a solution for that: I can describe
+the file format [in a simple YAML file](/src/dsat.ksy), automatically
+generate code to parse files, and also visualize the structure:
+
+![Visualization of the D-Sat file structure](/img/dsat_kaitai.png)

src/Searching_the_Index.ipynb

@@ -85,8 +85,8 @@
     "plt.rcParams['figure.figsize'] = (10, 10)\n",
     "plt.rcParams['figure.dpi'] = 140\n",
     "\n",
-    "#plt.plot(df.i1, df.i2, \"o\", label=\"\")\n",
-    "c(df, diag_kind=None)\n",
+    "plt.plot(df.i1, df.i2, \"o\", label=\"\")\n",
+    "#c(df, diag_kind=None)\n",
     "plt.show()"
    ]
   },
@@ -182,20 +182,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "    import statsmodels.tsa.stattools as smtsa\n",
-    "    import numpy as np\n",
+    "import statsmodels.tsa.stattools as smtsa\n",
+    "import numpy as np\n",
     "\n",
-    "    acf = smtsa.acf(df.ints, nlags=300, adjusted=False, fft=False)\n",
+    "acf = smtsa.acf(df.ints, nlags=50, adjusted=False, fft=False)\n",
     "\n",
-    "    lags = np.arange(len(acf))\n",
-    "    plt.rcParams['figure.figsize'] = (10, 5)\n",
+    "lags = np.arange(len(acf))\n",
+    "plt.rcParams['figure.figsize'] = (10, 5)\n",
     "\n",
-    "    plt.vlines([6, 10, 16], -0.2, 0.8, color=\"lightgrey\")\n",
-    "    plt.plot(lags[1:], acf[1:])\n",
-    "    plt.xlabel(\"bytes\")\n",
-    "    plt.xlim(xmin=0)\n",
-    "    plt.ylabel(\"correlation\")\n",
-    "    plt.show()"
+    "plt.vlines([6, 10, 16], -0.2, 0.8, color=\"lightgrey\")\n",
+    "plt.plot(lags[1:], acf[1:])\n",
+    "plt.xlabel(\"bytes\")\n",
+    "plt.xlim(xmin=0)\n",
+    "plt.ylabel(\"correlation\")\n",
+    "plt.show()"
    ]
   },
   {