Modularization of export code #341

jywarren · 2019-02-14T00:37:36Z

Re: #298 just starting here!

jywarren · 2019-02-14T00:40:19Z

jywarren · 2019-02-16T22:20:30Z

This is looking really good but there are a lot of warnings in the log, and a couple of the steps seem to have a "no directory found" error. But it can start to be pulled into a separate utility now! @icarito should we make a gem out of /lib/exporter.rb, called mapknitter?

icarito · 2019-02-17T06:01:38Z

Great! About the Gem, we ourselves don't have an immediate need as the container will likely continue to pull from git directly.

jywarren · 2019-02-19T22:44:58Z

I guess i'm thinking of breaking this out as a separate Ruby library and independent repository so you don't have to include the whole MapKnitter container to use it, but we can start by just using this container, i guess. Can you try setting this up in a Google Cloud container so we can start trying to use it? Thanks!

jywarren · 2019-02-19T22:57:18Z

Looking to resolve a few issues before merging:


gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/saugus-landfill-incinerator/1-geo-unwarped.tif public/warps/saugus-landfill-incinerator/1-geo.tif
Output dataset public/warps/saugus-landfill-incinerator/1-geo.tif exists,
but some commandline options were provided indicating a new dataset
should be created.  Please delete existing dataset and run again.
- [#<BigDecimal:561a7ed604e0,'-0.1589606382 5039331034 5777777777 7777777777 7777777777 778E8',54(72)>, 10274177.588360637]
gdalwarp -te -71.3983854668 41.8351476452 -71.3916477578 41.840311368 public/warps/saugus-landfill-incinerator/1-geo.tif public/warps/saugus-landfill-incinerator/saugus-landfill-incinerator-geo.tif
Creating output file that is 1701P x 1304L.
Processing input file public/warps/saugus-landfill-incinerator/1-geo.tif.
Using band 4 of source image as alpha.
Using band 4 of destination image as alpha.
0...10...20...30...40...50...60...70...80...90...100 - done.
ERROR 6: EPSG PCS/GCS code 900913 not found in EPSG support files.  Is this a valid
EPSG coordinate system?
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Traceback (most recent call last):
  File "/usr/bin/gdal2tiles.py", line 2262, in <module>
    gdal2tiles.process()
  File "/usr/bin/gdal2tiles.py", line 475, in process
    self.generate_metadata()
  File "/usr/bin/gdal2tiles.py", line 1123, in generate_metadata
    f.write( self.generate_tilemapresource())
  File "/usr/bin/gdal2tiles.py", line 1519, in generate_tilemapresource
    for z in range(self.tminz, self.tmaxz+1):
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
nice: cd: No such file or directory
nice: cd: No such file or directory
convert.im6: unable to access configure file `colors.xml' @ warning/configure.c/GetConfigureOptions/591.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33922 (0x8482) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34736 (0x87b0) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34737 (0x87b1) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.

https://travis-ci.org/publiclab/mapknitter/builds/494340774#L2473

> generating tiles
ERROR 6: EPSG PCS/GCS code 900913 not found in EPSG support files.  Is this a valid
EPSG coordinate system?
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Traceback (most recent call last):
  File "/usr/bin/gdal2tiles.py", line 2262, in <module>
    gdal2tiles.process()
  File "/usr/bin/gdal2tiles.py", line 475, in process
    self.generate_metadata()
  File "/usr/bin/gdal2tiles.py", line 1123, in generate_metadata
    f.write( self.generate_tilemapresource())
  File "/usr/bin/gdal2tiles.py", line 1519, in generate_tilemapresource
    for z in range(self.tminz, self.tmaxz+1):
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
> zipping tiles
nice: cd: No such file or directory
nice: cd: No such file or directory
> generating jpg
convert.im6: unable to access configure file `colors.xml' @ warning/configure.c/GetConfigureOptions/591.
convert.im6: Unknown field with tag 33550 (0x830e) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 33922 (0x8482) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34735 (0x87af) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34736 (0x87b0) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.
convert.im6: Unknown field with tag 34737 (0x87b1) encountered. `TIFFReadDirectory' @ warning/tiff.c/TIFFWarnings/768.

https://travis-ci.org/publiclab/mapknitter/builds/494340774#L2542

I'm going to look for the 900913 error first as it might have cascaded to cause the others.

jywarren · 2019-02-19T23:18:17Z

So, i deleted the geotiff since we generate it twice in the tests.

Then i moved the 'delete directories' test to the end

I changed assert_not_nil to assert for a lot of assertions.

Finally, i added 3857 to try to bypass the 900913 error as per https://gis.stackexchange.com/questions/162767/gdal2tiles-py-gives-error-6-about-epsg900913-on-fresh-ubuntu-14-04-install

jywarren · 2019-02-20T01:17:33Z

Hmm,


gdalwarp -te -s EPSG:3857 -71.3983854668 41.8351476452 -71.3916477578 41.840311368 public/warps/saugus-landfill-incinerator/1-geo.tif public/warps/saugus-landfill-incinerator/saugus-landfill-incinerator-geo.tif
Usage: gdalwarp [--help-general] [--formats]
    [-s_srs srs_def] [-t_srs srs_def] [-to "NAME=VALUE"]
    [-order n | -tps | -rpc | -geoloc] [-et err_threshold]
    [-refine_gcps tolerance [minimum_gcps]]
    [-te xmin ymin xmax ymax] [-tr xres yres] [-tap] [-ts width height]
    [-wo "NAME=VALUE"] [-ot Byte/Int16/...] [-wt Byte/Int16]
    [-srcnodata "value [value...]"] [-dstnodata "value [value...]"] -dstalpha
    [-r resampling_method] [-wm memory_in_mb] [-multi] [-q]
    [-cutline datasource] [-cl layer] [-cwhere expression]
    [-csql statement] [-cblend dist_in_pixels] [-crop_to_cutline]
    [-of format] [-co "NAME=VALUE"]* [-overwrite]
    [-nomd] [-cvmd meta_conflict_value]
    srcfile* dstfile
Available resampling methods:
    near (default), bilinear, cubic, cubicspline, lanczos, average, mode.
FAILURE: Unknown option name '-71.3916477578'

ah i see... hang on

icarito · 2019-02-20T06:22:49Z

Okay, I see. I'm building a container with gdal 2.4.0 from the Debian repositories for the Testing distribution "buster". I'll put the Dockerfile in a pull request here when it works. I'll be adding Ruby and ImageMagick. I see buster has Ruby 2.5.1 - hope this is fine? Otherwise I can downgrade to Stretch (current stable, with Ruby 2.3.3 and GDAL 2.1.2).

icarito · 2019-02-20T07:24:22Z

Okay I've built a container including GDAL, Imagemagick and Ruby, and pushed it to the Docker Hub.
That means you can run it right now by using docker run icarito/gdal_ruby_imagemagick:buster - it'll pull directly from the Docker Hub. I've also pushed it to GCE but now it needs an interface to interact with the world.

@jywarren this should be ready for integrating, check out with docker run icarito/gdal_ruby_imagemagick:buster to begin integration with the export.rb script!

Regards,
Sebastian

jywarren · 2019-02-20T22:04:41Z

Thanks Sebastian - linking there now: #349 and I left some thoughts there too. Awesome!

jywarren · 2019-02-26T19:39:59Z

I believe everything but gdal2tiles.py is working and it just needed an API key, which I added.

jywarren · 2019-02-26T23:59:14Z

Great, only a couple small things left:

Error: test_isolated_exporter_lib(ExporterTest): Errno::ENOENT: No such file or directory @ rb_sysopen - /app/public/system/images/2/original/test.png
================================================================================
.> calculating scale
> calculating scale
rm: cannot remove '/app/public/tms/saugus-landfill-incinerator': No such file or directory

jywarren · 2019-02-28T23:15:38Z

:-( https://gis.stackexchange.com/questions/199915/unable-to-generate-tiles-using-gdal2tiles-py-for-a-single-band-gray-image

jywarren · 2019-02-28T23:15:38Z

:-( https://gis.stackexchange.com/questions/199915/unable-to-generate-tiles-using-gdal2tiles-py-for-a-single-band-gray-image

generating tiles
gdal2tiles.py -k --s_srs EPSG:3857 -t "saugus-landfill-incinerator" -g "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ" /app/public/warps/saugus-landfill-incinerator/saugus-landfill-incinerator-geo.tif /app/public/tms/saugus-landfill-incinerator/
Traceback (most recent call last):
File "/usr/bin/gdal2tiles.py", line 2436, in
main()
File "/usr/bin/gdal2tiles.py", line 2433, in main
gdal2tiles.process()
File "/usr/bin/gdal2tiles.py", line 478, in process
self.generate_metadata()
File "/usr/bin/gdal2tiles.py", line 1155, in generate_metadata
f.write( self.generate_tilemapresource())
File "/usr/bin/gdal2tiles.py", line 1553, in generate_tilemapresource
for z in range(self.tminz, self.tmaxz+1):
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

jywarren · 2019-02-28T23:30:47Z

I think this may have to do with the GDAL version; this is a post by me:

http://osgeo-org.1560.x6.nabble.com/gdal-dev-gdal2tiles-py-cannot-find-EPSG-900913-sees-only-GDAL-1-6-1-td3748343.html

jywarren · 2019-02-28T23:32:22Z

I'm running GDAL 1.11.3, released 2015/09/16, on my local. Not sure what's in Travis though!

jywarren · 2019-02-28T23:36:48Z

lib/exporter.rb

+  def self.generate_tiles(key, slug, root)
+    key = "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ" if key == "" # ugh, let's clean this up!
+    key = key || "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ"
+    gdal2tiles = 'gdal2tiles.py -k --s_srs EPSG:3857 -t "'+slug+'" -g "'+key+'" '+root+'/public/warps/'+slug+'/'+slug+'-geo.tif '+root+'/public/tms/'+slug+"/"


Hi @icarito if you have a moment, i'm a bit stuck here; i think we're down to the GDAL version on this last command. I've been trying to run it locally.

Here are the files that this should be runnable on: https://publiclab.org/i/29584.zip

gdal2tiles.py -k --s_srs EPSG:3857 -t "saugus-landfill-incinerator" -g "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ" /app/public/warps/saugus-landfill-incinerator/saugus-landfill-incinerator-geo.tif /app/public/tms/saugus-landfill-incinerator/

And Mapknitter.org uses:

$ gdal2tiles.py --version GDAL 2.1.2, released 2016/10/24

Travis was same: GDAL 2.1.2, released 2016/10/24

jywarren · 2019-03-15T22:00:43Z

@icarito would you mind trying to resolve the conflicts here?

* Try running Buster * Include binary GDal * Reorder Dockerfile * Install npm * Tweak apt command * Update Dockerfile * Simplfy * Rollback ruby upgrade and rather install 2.4.4 from rvm * Add login shell to support rvm * Mysql deps * Include git * Fix entry command * Move start command * Revert "Fix entry command" This reverts commit 1833f41. * Remove entrypoint from Dockerfile * Add -l parameter to bash * More deps * Bump mysql2 from 0.3.21 to 0.5.2 Bumps [mysql2](https://github.com/brianmario/mysql2) from 0.3.21 to 0.5.2. - [Release notes](https://github.com/brianmario/mysql2/releases) - [Changelog](https://github.com/brianmario/mysql2/blob/master/CHANGELOG.md) - [Commits](brianmario/mysql2@0.3.21...0.5.2) Signed-off-by: dependabot[bot] <[email protected]> * activerecord-mysql2-adapter * Return to mysql2 < 4 gem and add some grease * Update exporter_test.rb * Update exporter_test.rb * Update exporter.rb * Update exporter_test.rb

jywarren · 2019-03-16T19:11:46Z

Linking to #341, and continuing here.

jywarren · 2019-03-16T19:23:04Z

@icarito this is complete!

jywarren · 2019-03-16T20:51:40Z

I started working on a gem, but it's not all the way there. Still, closer to what we'd need for a completely externalized version of this! https://github.com/publiclab/mapknitter-exporter @icarito

icarito · 2019-03-17T06:21:55Z

Awesome I'll look at linking it to the Sinatra app for the API

sashadev-sky · 2019-03-28T11:34:29Z

@jywarren when I run mapknitter locally with rake test my output is the same as one of your earlier travis builds posts here: https://travis-ci.org/publiclab/mapknitter/builds/494340774#L2542. (I get the errors about removing files that don't exist, gdal2tiles.py doesn't exist). I made changes locally to fix these problems then realized your most recent PR https://travis-ci.org/publiclab/mapknitter/builds/507274672?utm_source=github_status&utm_medium=notification didn't have them in the first place. Any idea why?

Wanted to make sure I understand this PR before looking at @icarito new export API

@icarito

sashadev-sky · 2019-03-28T13:58:38Z

also am I remembering currently that the exports are supposed to export the images with the map background, not just the image overlays?

jywarren · 2019-03-28T21:20:37Z

Oh do we need to open a PR to solve the missing files errors as a distinct PR? And no, the export is just the overlays not the background. Thanks!!!!!

…

On Thu, Mar 28, 2019, 9:59 AM Sasha Boginsky ***@***.***> wrote: also am I remembering currently that the exports are supposed to export the images with the map background, not just the image overlays? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#341 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ1JhEC98TO2LceMK_ob052z2V4eRks5vbMqOgaJpZM4a6kkB> .

sashadev-sky · 2019-03-29T00:37:57Z

@jywarren hmm I don't know. Locally the tests wont pass if your root directory name has spaces in it, like for ex. on my computer I have the Mapknitter repo in a folder "Public Lab". Then it tries to run the commands 2 times: first on **/public then on lab/mapknitter/**. But if the root is https://mapknitter.org then it's fine.

To make it pass locally I just did root.gsub(" ", "\\ ") where appropriate. Would it make sense to add a PR with this change?

jywarren · 2019-03-29T01:14:42Z

ahhhhh, that makes sense. Or we could refactor this to use relative paths... what about that?

…

On Thu, Mar 28, 2019 at 8:38 PM Sasha Boginsky ***@***.***> wrote: @jywarren <https://github.com/jywarren> hmm I don't know. Locally the tests wont pass if your root directory name has spaces in it, like for ex. on my computer I have the Mapknitter repo in a folder "Public Lab". Then it tries to run the commands 2 times: first on **/public then on lab/mapknitter/**. But if the root is https://mapknitter.org then it's fine. To make it pass locally I just did root.gsub(" ", "\\ ") where appropriate. Would it make sense to add a PR with this change? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#341 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABfJ9nahwDiUA0hVWf71fYj3s2U5rMBks5vbWBlgaJpZM4a6kkB> .

sashadev-sky · 2019-03-29T01:25:20Z

@jywarren would relative paths still work if the root is https://mapknitter.org? These could get more confusing because the files locally are nested at varied levels 3 or 4 levels deep

jywarren · 2019-03-30T00:01:29Z

Hmm. Which PR didn't have them? I believe we were able to resolve them all by re-creating the directories in the tests at the right moments. Can you share your test changes? I do think relative paths may still work, like ../../ should work in URLs as well... see how https://mapknitter.org/maps/../maps/cranston-test just goes to https://mapknitter.org/maps/cranston-test?

jywarren · 2019-03-30T00:02:42Z

But i just started making some changes to the extracted code here, that'll have to be refactored back in this repo once it stabilizes: publiclab/mapknitter-exporter#6

I am taking notes on the changes in publiclab/mapknitter-exporter#5 but I think you don't have to worry too much at this point... we're hoping to make all of that run itself externally.

sashadev-sky · 2019-04-03T02:48:43Z

@jywarren I made changes to the code not the tests. Is it the tests I should have been changing?

* starting to move things into exporter lib, parameterize * additional mysql2 adapter change, sqlite compatibility * moved almost all of exporter code into ruby lib * Try to update GDAL from sources * Update Dockerfile * Try running Buster (publiclab#186) * Try running Buster * Include binary GDal * Reorder Dockerfile * Install npm * Tweak apt command * Update Dockerfile * Simplfy * Rollback ruby upgrade and rather install 2.4.4 from rvm * Add login shell to support rvm * Mysql deps * Include git * Fix entry command * Move start command * Revert "Fix entry command" This reverts commit 1833f41. * Remove entrypoint from Dockerfile * Add -l parameter to bash * More deps * Bump mysql2 from 0.3.21 to 0.5.2 Bumps [mysql2](https://github.com/brianmario/mysql2) from 0.3.21 to 0.5.2. - [Release notes](https://github.com/brianmario/mysql2/releases) - [Changelog](https://github.com/brianmario/mysql2/blob/master/CHANGELOG.md) - [Commits](brianmario/mysql2@0.3.21...0.5.2) Signed-off-by: dependabot[bot] <[email protected]> * activerecord-mysql2-adapter * Return to mysql2 < 4 gem and add some grease * Update exporter_test.rb * Update exporter_test.rb * Update exporter.rb * Update exporter_test.rb

jywarren force-pushed the modularization branch from 86a5f92 to 2fdb2d4 Compare February 14, 2019 00:39

jywarren mentioned this pull request Feb 18, 2019

MapKnitterExporter architecture discussion #298

Open

jywarren mentioned this pull request Feb 19, 2019

Reduce Ruby warnings in recent builds #345

Closed

jywarren mentioned this pull request Feb 20, 2019

GDAL export container #349

Closed

3 tasks

jywarren commented Feb 28, 2019

View reviewed changes

jywarren and others added 6 commits March 16, 2019 15:08

starting to move things into exporter lib, parameterize

e4ad4a9

additional mysql2 adapter change, sqlite compatibility

7fcf08b

moved almost all of exporter code into ruby lib

00b4060

Try to update GDAL from sources

621a281

Update Dockerfile

313650f

jywarren force-pushed the modularization branch from 71ec8bc to bbf1ec8 Compare March 16, 2019 19:08

jywarren merged commit 628b8b5 into publiclab:main Mar 16, 2019

jywarren mentioned this pull request Mar 29, 2019

Update .travis.yml, work on gdal errors publiclab/mapknitter-exporter#4

Merged

sashadev-sky mentioned this pull request Apr 18, 2019

fully externalize exporter; untested #530

Closed

sashadev-sky mentioned this pull request May 20, 2019

Upgrade from Rails 3.2 to Rails4.2 #578

Merged

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Modularization of export code #341

Modularization of export code #341

jywarren commented Feb 14, 2019

jywarren commented Feb 14, 2019

jywarren commented Feb 16, 2019

icarito commented Feb 17, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 20, 2019

icarito commented Feb 20, 2019

icarito commented Feb 20, 2019

jywarren commented Feb 20, 2019

jywarren commented Feb 26, 2019

jywarren commented Feb 26, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren Feb 28, 2019

jywarren Mar 1, 2019

jywarren Mar 1, 2019

jywarren Mar 15, 2019

jywarren commented Mar 15, 2019

jywarren commented Mar 16, 2019

jywarren commented Mar 16, 2019

jywarren commented Mar 16, 2019

icarito commented Mar 17, 2019

sashadev-sky commented Mar 28, 2019

sashadev-sky commented Mar 28, 2019

jywarren commented Mar 28, 2019 via email

sashadev-sky commented Mar 29, 2019

jywarren commented Mar 29, 2019 via email

sashadev-sky commented Mar 29, 2019

jywarren commented Mar 30, 2019

jywarren commented Mar 30, 2019

sashadev-sky commented Apr 3, 2019

Modularization of export code #341

Modularization of export code #341

Conversation

jywarren commented Feb 14, 2019

jywarren commented Feb 14, 2019

jywarren commented Feb 16, 2019

icarito commented Feb 17, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 19, 2019

jywarren commented Feb 20, 2019

icarito commented Feb 20, 2019

icarito commented Feb 20, 2019

jywarren commented Feb 20, 2019

jywarren commented Feb 26, 2019

jywarren commented Feb 26, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren commented Feb 28, 2019

jywarren Feb 28, 2019

Choose a reason for hiding this comment

jywarren Mar 1, 2019

Choose a reason for hiding this comment

jywarren Mar 1, 2019

Choose a reason for hiding this comment

jywarren Mar 15, 2019

Choose a reason for hiding this comment

jywarren commented Mar 15, 2019

jywarren commented Mar 16, 2019

jywarren commented Mar 16, 2019

jywarren commented Mar 16, 2019

icarito commented Mar 17, 2019

sashadev-sky commented Mar 28, 2019

sashadev-sky commented Mar 28, 2019

jywarren commented Mar 28, 2019 via email

sashadev-sky commented Mar 29, 2019

jywarren commented Mar 29, 2019 via email

sashadev-sky commented Mar 29, 2019

jywarren commented Mar 30, 2019

jywarren commented Mar 30, 2019

sashadev-sky commented Apr 3, 2019