Merge pull request #10 from kipcole9/master

Bug fixes

CHANGELOG.md

@@ -1,3 +1,21 @@
+# Changelog for Tz_World v0.5.0
+
+This is the changelog for Tz_World v0.5.0 released on May 23rd, 2020.  For older changelogs please consult the release tag on [GitHub](https://github.com/kimlai/tz_world/tags)
+
+### Bug Fixes
+
+* Move compile time configuration of the data directory to runtime and remove hard-coded default path
+
+* Start `:inets` and `:ssl` applications in the downloader mix task
+
+* Add certificate verification when downloading updates to the geo data
+
+### Enhancements
+
+* Document the `:data_dir` and `:cacertfile` configuration options it the README.md file
+
+* The backends `:dets` and `:dets_with_index_cache` now open the `:dets` file as `access: :read` which prevents errors if the file is abnormally closed.
+
 # Changelog for Tz_World v0.4.0
 
 This is the changelog for Tz_World v0.4.0 released on May 12th, 2020.  For older changelogs please consult the release tag on [GitHub](https://github.com/kimlai/tz_world/tags)
@@ -10,19 +28,19 @@ This is the changelog for Tz_World v0.4.0 released on May 12th, 2020.  For older
 
 ### Configurable backends
 
-* `TzWorld.Backend.Memory` which retains all data in memory for fastest performance at the expense of using approximately 1Gb of memory
+* `TzWorld.Backend.Memory` which retains all data in memory for fast (but *not* fastest) performance at the expense of using approximately 1Gb of memory
 * `TzWorld.Backend.Dets` which uses Erlang's `:dets` data store. This uses negligible memory at the expense of slow access times (approximaltey 500ms in testing)
 * `TzWorld.Backend.DetsWithIndexCache` which balances memory usage and performance. This backend is recommended in most situations since its performance is similar to `TzWorld.Backend.Memory` (about 5% slower in testing) and uses about 25Mb of memory
 * `TzWorld.Backend.Ets` which uses `:ets` for storage. With the default settings of `:compressed` for the `:ets` table its memory consumption is about 512Mb but with access that is over 20 times slower than `TzWorld.Backend.DetsWithIndexCache`
 * `TzWorld.Backend.EtsWithIndexCache` which uses `:ets` for storage with an additional in-memory cache of the bounding boxes. This still uses about 512Mb but is faster than any of the other backends by about 40%
 
 ### Enhancements
 
-* Add `TzWorld.all_timezones_at/2` to return all timezones for a given location.  In rare cases, usually disputed territory, multiple timezones may be declared for a overlapping regions. `TzWorld.all_timezones_at/2` returns a (potentially empty) list of all time zones knowns for a given point.  *Futher testing of this function is required and will be completed before version 1.0*.
+* Add `TzWorld.all_timezones_at/2` to return all timezones for a given location.  In rare cases, usually disputed territory, multiple timezones may be declared for overlapping regions. `TzWorld.all_timezones_at/2` returns a (potentially empty) list of all time zones known for a given point.  *Futher testing of this function is required and will be completed before version 1.0*.
 
 # Changelog for Tz_World v0.3.0
 
-This is the changelog for Tz_World v0.3.0 released on February 16th, 2020.  For older changelogs please consult the release tag on [GitHub](https://github.com/kimlai/tz_world/tags)
+This is the changelog for Tz_World v0.3.0 released on December 4th, 2019.  For older changelogs please consult the release tag on [GitHub](https://github.com/kimlai/tz_world/tags)
 
 ### Breaking Changes
 

README.md

@@ -18,10 +18,24 @@ After adding `TzWorld` as a dependency, run `mix deps.get` to install it. Then r
 
 **NOTE** No data is installed with the package and until the data is installed with `mix tz_world.update` all calls to `TzWorld.timezone_at/1` will return `{:error, :time_zone_not_found}`.
 
+### Configuration
+
+There is no mandatory configuration required however two options may be configured in `config.exs`:
+
+```elixir
+config :tz_world,
+	data_dir: "geodata/directory",        # The default is the `priv` directory of `:tz_world`
+	cacertfile: "path/to/ca_trust_store"  # The defaul is either the trust store included in the
+									                      # libraries `CAStore` or `certifi` or the platform
+																				# trust store.
+```
+
 ## Backend selection
 
 `TzWorld` provides alternative strategies for managing access to the backend data. Each backend is implemented as a `GenServer` that needs to be either manually started with `BackendModule.start_link/1` or preferably added to your application's supervision tree.
 
+The recommended backend is `TzWorld.Backend.EtsWithIndexCache`.
+
 For example:
 ```elixir
 defmodule MyApp.Application do
@@ -42,7 +56,7 @@ end
 ```
 The following backends are available:
 
-* `TzWorld.Backend.Memory` which retains all data in memory for fastest performance at the expense of using approximately 1Gb of memory
+* `TzWorld.Backend.Memory` which retains all data in memory for fast (but *not* fastest) performance at the expense of using approximately 1Gb of memory
 * `TzWorld.Backend.Dets` which uses Erlang's `:dets` data store. This uses negligible memory at the expense of slow access times (approximaltey 500ms in testing)
 * `TzWorld.Backend.DetsWithIndexCache` which balances memory usage and performance. This backend is recommended in most situations since its performance is similar to `TzWorld.Backend.Memory` (about 5% slower in testing) and uses about 25Mb of memory
 * `TzWorld.Backend.Ets` which uses `:ets` for storage. With the default settings of `:compressed` for the `:ets` table its memory consumption is about 512Mb  but with access that is over 20 times slower than `TzWorld.Backend.DetsWithIndexCache`

lib/mix/tasks/update_timezones_geojson.ex

@@ -10,6 +10,9 @@ defmodule Mix.Tasks.TzWorld.Update do
 
   def run(_args) do
     Application.ensure_all_started(:tz_world)
+    Application.ensure_all_started(:inets)
+    Application.ensure_all_started(:ssl)
+
     TzWorld.Backend.Memory.start_link
     TzWorld.Backend.Dets.start_link
 

lib/tz_world.ex

@@ -16,6 +16,13 @@ defmodule TzWorld do
     TzWorld.Backend.EtsWithIndexCache,
   ]
 
+  @doc """
+  Returns the OTP app name of :tz_world
+
+  """
+  def app_name do
+    :tz_world
+  end
 
   @doc """
   Returns the installed version of time

lib/tz_world/backend/dets_with_index_cache.ex

@@ -50,7 +50,7 @@ defmodule TzWorld.Backend.DetsWithIndexCache do
 
   @slots 800
   defp dets_options do
-    [file: filename(), estimated_no_objects: @slots]
+    [file: filename(), access: :read, estimated_no_objects: @slots]
   end
 
   @doc false
@@ -60,7 +60,9 @@ defmodule TzWorld.Backend.DetsWithIndexCache do
 
   @doc false
   def save_dets_geodata do
-    {:ok, t} = :dets.open_file(__MODULE__, dets_options())
+    dets_options = Keyword.put(dets_options(), :access, :read_write)
+
+    {:ok, t} = :dets.open_file(__MODULE__, dets_options)
     :ok = :dets.delete_all_objects(t)
 
     {:ok, geodata} = TzWorld.GeoData.load_compressed_data()

lib/tz_world/downloader.ex

@@ -93,7 +93,7 @@ defmodule TzWorld.Downloader do
   defp get_url(url) do
     require Logger
 
-    case :httpc.request(:get, {url, headers()}, [], []) do
+    case  :httpc.request(:get, {url, headers()}, https_opts(), []) do
       {:ok, {{_version, 200, 'OK'}, _headers, body}} ->
         {:ok, :erlang.list_to_binary(body)}
 
@@ -122,4 +122,89 @@ defmodule TzWorld.Downloader do
       {'User-Agent', 'httpc/22.0'}
     ]
   end
+
+  @certificate_locations [
+      # Configured cacertfile
+      Application.get_env(TzWorld.app_name(), :cacertfile),
+
+      # Populated if hex package CAStore is configured
+      if(Code.ensure_loaded?(CAStore), do: CAStore.file_path()),
+
+      # Populated if hex package certfi is configured
+      if(Code.ensure_loaded?(:certifi), do: :certifi.cacertfile() |> List.to_string),
+
+      # Debian/Ubuntu/Gentoo etc.
+      "/etc/ssl/certs/ca-certificates.crt",
+
+      # Fedora/RHEL 6
+      "/etc/pki/tls/certs/ca-bundle.crt",
+
+      # OpenSUSE
+      "/etc/ssl/ca-bundle.pem",
+
+      # OpenELEC
+      "/etc/pki/tls/cacert.pem",
+
+      # CentOS/RHEL 7
+      "/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
+
+      # Open SSL on MacOS
+      "/usr/local/etc/openssl/cert.pem",
+
+      # MacOS & Alpine Linux
+      "/etc/ssl/cert.pem"
+  ]
+  |> Enum.reject(&is_nil/1)
+
+  def certificate_store do
+    @certificate_locations
+    |> Enum.find(&File.exists?/1)
+    |> raise_if_no_cacertfile
+    |> :erlang.binary_to_list
+  end
+
+  defp raise_if_no_cacertfile(nil) do
+    raise RuntimeError, """
+      No certificate trust store was found.
+      Tried looking for: #{inspect @certificate_locations}
+
+      A certificate trust store is required in
+      order to download locales for your configuration.
+
+      Since ex_cldr could not detect a system
+      installed certificate trust store one of the
+      following actions may be taken:
+
+      1. Install the hex package `castore`. It will
+         be automatically detected after recompilation.
+
+      2. Install the hex package `certifi`. It will
+         be automatically detected after recomilation.
+
+      3. Specify the location of a certificate trust store
+         by configuring it in `config.exs`:
+
+         config :ex_cldr,
+           cacertfile: "/path/to/cacertfile",
+           ...
+
+      """
+  end
+
+  defp raise_if_no_cacertfile(file) do
+    file
+  end
+
+  defp https_opts do
+    [ssl:
+      [
+        verify: :verify_peer,
+        cacertfile: certificate_store(),
+        customize_hostname_check: [
+          match_fun: :public_key.pkix_verify_hostname_match_fun(:https)
+        ]
+      ]
+    ]
+  end
+
 end

lib/tz_world/geo_data.ex

@@ -3,23 +3,28 @@ defmodule TzWorld.GeoData do
 
   @compressed_data_file "timezones-geodata.etf.zip"
   @etf_data_file "timezones-geodata.etf"
-  @default_data_dir "./priv"
   @osm_srid 3857
 
   defdelegate version, to: TzWorld
 
   def default_data_dir do
-    @default_data_dir
+    TzWorld.app_name()
+    |> :code.priv_dir
+    |> List.to_string
+  end
+
+  def data_dir do
+    Application.get_env(TzWorld.app_name(), :data_dir, default_data_dir())
   end
 
   def compressed_data_path do
-    Application.get_env(:tz_world, :data_dir, default_data_dir())
+    data_dir()
     |> Path.join(@compressed_data_file)
     |> to_charlist
   end
 
   def etf_data_path do
-    Application.get_env(:tz_world, :data_dir, default_data_dir())
+    data_dir()
     |> Path.join(@etf_data_file)
     |> to_charlist
   end

mix.exs

@@ -1,7 +1,7 @@
 defmodule TzWorld.Mixfile do
   use Mix.Project
 
-  @version "0.4.0"
+  @version "0.5.0"
 
   def project do
     [
@@ -33,6 +33,8 @@ defmodule TzWorld.Mixfile do
     [
       {:geo, "~> 1.0 or ~> 2.0 or ~> 3.0"},
       {:jason, "~> 1.0"},
+      {:castore, "~> 0.1", optional: true},
+      {:certifi, "~> 2.5", optional: true},
       {:ex_doc, "~> 0.19", only: :dev, runtime: false},
       {:dialyxir, "~> 1.0.0-rc", only: :dev, runtime: false, optional: true},
       {:benchee, "~> 1.0", only: :dev, runtime: false}

mix.lock

@@ -1,5 +1,7 @@
 %{
   "benchee": {:hex, :benchee, "1.0.1", "66b211f9bfd84bd97e6d1beaddf8fc2312aaabe192f776e8931cb0c16f53a521", [:mix], [{:deep_merge, "~> 1.0", [hex: :deep_merge, repo: "hexpm", optional: false]}], "hexpm", "3ad58ae787e9c7c94dd7ceda3b587ec2c64604563e049b2a0e8baafae832addb"},
+  "castore": {:hex, :castore, "0.1.5", "591c763a637af2cc468a72f006878584bc6c306f8d111ef8ba1d4c10e0684010", [:mix], [], "hexpm", "6db356b2bc6cc22561e051ff545c20ad064af57647e436650aa24d7d06cd941a"},
+  "certifi": {:hex, :certifi, "2.5.2", "b7cfeae9d2ed395695dd8201c57a2d019c0c43ecaf8b8bcb9320b40d6662f340", [:rebar3], [{:parse_trans, "~>3.3", [hex: :parse_trans, repo: "hexpm", optional: false]}], "hexpm", "3b3b5f36493004ac3455966991eaf6e768ce9884693d9968055aeeeb1e575040"},
   "deep_merge": {:hex, :deep_merge, "1.0.0", "b4aa1a0d1acac393bdf38b2291af38cb1d4a52806cf7a4906f718e1feb5ee961", [:mix], [], "hexpm", "ce708e5f094b9cd4e8f2be4f00d2f4250c4095be93f8cd6d018c753894885430"},
   "dialyxir": {:hex, :dialyxir, "1.0.0", "6a1fa629f7881a9f5aaf3a78f094b2a51a0357c843871b8bc98824e7342d00a5", [:mix], [{:erlex, ">= 0.2.6", [hex: :erlex, repo: "hexpm", optional: false]}], "hexpm", "aeb06588145fac14ca08d8061a142d52753dbc2cf7f0d00fc1013f53f8654654"},
   "earmark": {:hex, :earmark, "1.4.3", "364ca2e9710f6bff494117dbbd53880d84bebb692dafc3a78eb50aa3183f2bfd", [:mix], [], "hexpm", "8cf8a291ebf1c7b9539e3cddb19e9cef066c2441b1640f13c34c1d3cfc825fec"},
@@ -11,4 +13,5 @@
   "makeup": {:hex, :makeup, "1.0.1", "82f332e461dc6c79dbd82fbe2a9c10d48ed07146f0a478286e590c83c52010b5", [:mix], [{:nimble_parsec, "~> 0.5.0", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "49736fe5b66a08d8575bf5321d716bac5da20c8e6b97714fec2bcd6febcfa1f8"},
   "makeup_elixir": {:hex, :makeup_elixir, "0.14.0", "cf8b7c66ad1cff4c14679698d532f0b5d45a3968ffbcbfd590339cb57742f1ae", [:mix], [{:makeup, "~> 1.0", [hex: :makeup, repo: "hexpm", optional: false]}], "hexpm", "d4b316c7222a85bbaa2fd7c6e90e37e953257ad196dc229505137c5e505e9eff"},
   "nimble_parsec": {:hex, :nimble_parsec, "0.5.3", "def21c10a9ed70ce22754fdeea0810dafd53c2db3219a0cd54cf5526377af1c6", [:mix], [], "hexpm", "589b5af56f4afca65217a1f3eb3fee7e79b09c40c742fddc1c312b3ac0b3399f"},
+  "parse_trans": {:hex, :parse_trans, "3.3.0", "09765507a3c7590a784615cfd421d101aec25098d50b89d7aa1d66646bc571c1", [:rebar3], [], "hexpm", "17ef63abde837ad30680ea7f857dd9e7ced9476cdd7b0394432af4bfc241b960"},
 }