Support for cache purging / seeding using a tile list

[ARolek]

The cache purge and cache seed commands should be able to accept a list of tiles in a file where each line has a z/x/y entry. Additionally, the commands should be able to watch a directory for new files and read the files as they're created.

[pnorman]

Just to note some information on an expire list

How does an expiry list look like?

The standard format is X Y Z separated by spaces, with each tile separated by newlines.

For what needs to be implemented, I'd look to render_list from renderd

$ render_list --help
Usage: render_list [OPTION] ...
  -a, --all            render all tiles in given zoom level range instead of reading from STDIN
  -f, --force          render tiles even if they seem current
  -m, --map=MAP        render tiles in this map (defaults to 'default')
  -l, --max-load=LOAD  sleep if load is this high (defaults to 16)
  -s, --socket=SOCKET  unix domain socket name for contacting renderd
  -n, --num-threads=N the number of parallel request threads (default 1)
  -t, --tile-dir       tile cache directory (defaults to '/var/lib/mod_tile')
  -z, --min-zoom=ZOOM  filter input to only render tiles greater or equal to this zoom level (default is 0)
  -Z, --max-zoom=ZOOM  filter input to only render tiles less than or equal to this zoom level (default is 20)
If you are using --all, you can restrict the tile range by adding these options:
  -x, --min-x=X        minimum X tile coordinate
  -X, --max-x=X        maximum X tile coordinate
  -y, --min-y=Y        minimum Y tile coordinate
  -Y, --max-y=Y        maximum Y tile coordinate
Without --all, send a list of tiles to be rendered from STDIN in the format:
  X Y Z
e.g.
  0 0 1
  0 1 1
  1 0 1
  1 1 1
The above would cause all 4 tiles at zoom 1 to be rendered

The difference with renderd is that it has "dirty" tiles, so it can have an outdated tile still in its store.

For an overview of the different methods, it's worth reading osm2pgsql-dev/osm2pgsql#709 and osm2pgsql-dev/osm2pgsql#747

[ARolek]

@pnorman thanks for the follow up on this. I'm actually about to start working on this and your comment spun up requirement to support a tile entry template. The imposm3 tile list is in z/x/y format. I will make sure to support a configurable template.

[sanfilippopablo]

How does this relate to using bounding box + max zoom + min zoom for cache seeding?

[ear7h]

@sanfilippopablo The changes made for this issue shouldn't have any effect on bounding box seeding/purging. It's mostly for adding support for a list of tiles to seed/purge.

You might also be interested in a new feature for using min zoom + max zoom with an individual tile. It uses the bounds of the specified tiles rather than explicit bounds.

If you're interested in the code you can check out my PR #372

[sanfilippopablo]

Oh, OK. But there isn't any seeding support right now, right?

[ear7h]

@sanfilippopablo the latest release has seed/purge support for individual tiles and bounding box

[sanfilippopablo]

@ear7h Great! But how? I didn't see anything in the docs about seeding

[erictheise]

@sanfilippopablo, you can sort of coax it out at the command line:

$ tegola cache --help
Use the cache command to seed or purge the tile cache

Usage:
  tegola cache [seed | purge] [flags]

Flags:
      --bounds string     lat / long bounds to seed the cache with in the format: minx, miny, maxx, maxy (default "-180,-85.0511,180,85.0511")
      --concurrency int   the amount of concurrency to use. defaults to the number of CPUs on the machine (default 8)
  -h, --help              help for cache
      --map string        map name as defined in the config
      --maxzoom uint      max zoom to seed cache to
      --minzoom uint      min zoom to seed cache from
      --overwrite         overwrite the cache if a tile already exists
      --zxy string        tile in z/x/y format

Global Flags:
      --config string   path to config file (default "config.toml")
$

I've used, e.g.,

$ tegola cache seed --minzoom 1 --maxzoom 7 --overwrite --config=./tegola-config/osm.toml

[sanfilippopablo]

@erictheise didn't know that! Thanks!