Update .travis.yml, work on gdal errors #4

jywarren · 2019-03-17T02:20:23Z

fixes #3

jywarren · 2019-03-17T02:28:42Z

Ah, same issue here as we'd had in MapKnitter proper.

jywarren · 2019-03-17T02:29:11Z


gdal2tiles.py -k --s_srs EPSG:3857 -t "ten-forward" -g "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ" public/warps/ten-forward/ten-forward-geo.tif public/tms/ten-forward/
ERROR 6: EPSG PCS/GCS code 900913 not found in EPSG support files.  Is this a valid
EPSG coordinate system?
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Traceback (most recent call last):
  File "/usr/bin/gdal2tiles.py", line 2262, in <module>
    gdal2tiles.process()
  File "/usr/bin/gdal2tiles.py", line 475, in process
    self.generate_metadata()
  File "/usr/bin/gdal2tiles.py", line 1123, in generate_metadata
    f.write( self.generate_tilemapresource())
  File "/usr/bin/gdal2tiles.py", line 1519, in generate_tilemapresource
    for z in range(self.tminz, self.tmaxz+1):
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
F
Finished in 14.985569s, 0.0667 runs/s, 0.4671 assertions/s.
  1) Failure:
ExporterTest#test_all_functions [test/exporter_test.rb:92]:
Expected false to be truthy.
1 runs, 7 assertions, 1 failures, 0 errors, 0 skips
The command "ruby test/exporter_test.rb" exited with 1.

jywarren · 2019-03-17T02:29:17Z

https://travis-ci.com/publiclab/mapknitter-exporter/builds/104689658

jywarren · 2019-03-17T20:11:50Z

OK, this happens locally too. Verbose:

$ gdal2tiles.py --verbose -k --s_srs EPSG:3857 -t "ten-forward" -g "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXI
PpLQY8iQ" public/warps/ten-forward/ten-forward-geo.tif public/tms/ten-forward/
('Options:', <Values at 0xdfddab70: {'profile': 'mercator', 'kml': True, 's_srs': 'EPSG:3857', 'copyright': '', 'resume': False, 'url': '', 'googlekey': 'AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ', 'bingkey': 'INSERT_YOUR_KEY_HERE', 'webviewer': 'all', 'zoom': None, 'resampling': 'average', 'tmscompatible': None, 'title': 'ten-forward', 'srcnodata': None, 'verbose': True}>)
('Input:', 'public/warps/ten-forward/ten-forward-geo.tif')
('Output:', 'public/tms/ten-forward/')
Cache: 40 MB

('Input file:', '( 1701P x 1304L - 4 bands)')
NODATA: []
('Preprocessed file:', '( 1701P x 1304L - 4 bands)')
ERROR 6: EPSG PCS/GCS code 900913 not found in EPSG support files.  Is this a valid
EPSG coordinate system?
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Warping of the raster by AutoCreateWarpedVRT (result saved into 'tiles.vrt')
('Projected file:', 'tiles.vrt', '( 1701P x 1304L - 4 bands)')
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
('Bounds (output srs):', -71.3983854668186, 41.835146726815466, -71.39164846169663, 41.8403113680142)
('Bounds (latlong):', (0.0003758115171853941, -0.0006413826092630026), (0.00037585791193653264, -0.0006413220897163001))
('MinZoomLevel:', None)
Traceback (most recent call last):
  File "/usr/bin/gdal2tiles.py", line 2278, in <module>
    gdal2tiles.process()
  File "/usr/bin/gdal2tiles.py", line 482, in process
    self.open_input()
  File "/usr/bin/gdal2tiles.py", line 991, in open_input
    print("MaxZoomLevel:", self.tmaxz, "(", self.mercator.Resolution( self.tmaxz ),")")
  File "/usr/bin/gdal2tiles.py", line 286, in Resolution
    return self.initialResolution / (2**zoom)
TypeError: unsupported operand type(s) for ** or pow(): 'int' and 'NoneType'

jywarren · 2019-03-17T20:22:53Z

I was able to add zoom levels to gdal2tiles.py command and get a different error:

$ gdal2tiles.py -z '8-9' --verbose -k --s_srs EPSG:3857 -t "ten-forward" -g "AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ" public/warps/ten-forward/ten-forward-geo.tif public/tms/ten-forward/
('Options:', <Values at 0xe9359b70: {'profile': 'mercator', 'kml': True, 's_srs': 'EPSG:3857', 'copyright': '', 'resume': False, 'url': '', 'googlekey': 'AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ', 'bingkey': 'INSERT_YOUR_KEY_HERE', 'webviewer': 'all', 'zoom': '8-9', 'resampling': 'average', 'tmscompatible': None, 'title': 'ten-forward', 'srcnodata': None, 'verbose': True}>)
('Input:', 'public/warps/ten-forward/ten-forward-geo.tif')
('Output:', 'public/tms/ten-forward/')
Cache: 40 MB

('Input file:', '( 1701P x 1304L - 4 bands)')
NODATA: []
('Preprocessed file:', '( 1701P x 1304L - 4 bands)')
ERROR 6: EPSG PCS/GCS code 900913 not found in EPSG support files.  Is this a valid
EPSG coordinate system?
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
Warping of the raster by AutoCreateWarpedVRT (result saved into 'tiles.vrt')
('Projected file:', 'tiles.vrt', '( 1701P x 1304L - 4 bands)')
ERROR 6: No translation for an empty SRS to PROJ.4 format is known.
('Bounds (output srs):', -71.3983854668186, 41.835146726815466, -71.39164846169663, 41.8403113680142)
('Bounds (latlong):', (0.0003758115171853941, -0.0006413826092630026), (0.00037585791193653264, -0.0006413220897163001))
('MinZoomLevel:', 8)
('MaxZoomLevel:', 9, '(', 305.748113140705, ')')
Generating Base Tiles:

Tiles generated from the max zoom level:
----------------------------------------

('dataBandsCount: ', 3)
('tilebands: ', 4)
(1, '/', 1, 'public/tms/ten-forward/9/255/256.png')
('\tNative Extent (querysize', 19744442931L, '): ', (0, 0, 1701, 1304), (19744441230L, 19751904232L, 1701, 1304))
('\tReadRaster Extent: ', (0, 0, 1701, 1304), (1023, 1023, 0, 0))
ERROR 5: Illegal values for buffer size
ERROR 5: Illegal values for buffer size
Traceback (most recent call last):
  File "/usr/bin/gdal2tiles.py", line 2278, in <module>
    gdal2tiles.process()
  File "/usr/bin/gdal2tiles.py", line 488, in process
    self.generate_base_tiles()
  File "/usr/bin/gdal2tiles.py", line 1288, in generate_base_tiles
    dsquery.WriteRaster(wx, wy, wxsize, wysize, data, band_list=list(range(1,self.dataBandsCount+1)))
  File "/usr/lib/python2.7/dist-packages/osgeo/gdal.py", line 933, in WriteRaster
    buf_pixel_space, buf_line_space, buf_band_space )
TypeError: not a string

Hmm.

jywarren · 2019-03-17T20:25:34Z

That last might be because i'm using local GDAL v1.x, so could be worth it on the Travis version. I'll try that.

jywarren · 2019-03-29T20:35:11Z

Need to re-apply solution from publiclab/mapknitter#341

jywarren · 2019-03-29T20:40:52Z

OK! We need Dockerfile and maybe docker-compose.yml from publiclab/mapknitter@bbf1ec8

@icarito

icarito · 2019-04-10T05:32:23Z

@jywarren this is failing differently now, GDAL is working but looks like travis is not allowing https out?

icarito · 2019-04-10T05:34:20Z

Here's the current error:

Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100
rm: cannot remove 'ten-forward.zip': No such file or directory
convert-im6.q16: not authorized `HTTPS' @ error/delegate.c/InvokeDelegate/1717.
convert-im6.q16: unable to open file `': No such file or directory @ error/constitute.c/ReadImage/544.
convert-im6.q16: no images defined `https://mapknitter.org/public/warps/ten-forward/ten-forward.jpg' @ error/convert.c/ConvertImageCommand/3258.
F
Finished in 9.042163s, 0.1106 runs/s, 0.9953 assertions/s.
  1) Failure:
ExporterTest#test_all_functions [test/exporter_test.rb:96]:
Expected false to be truthy.
1 runs, 9 assertions, 1 failures, 0 errors, 0 skips

icarito · 2019-04-10T06:43:15Z

rm: cannot remove 'ten-forward.zip': No such file or directory
convert-im6.q16: no decode delegate for this image format `HTTPS' @ error/constitute.c/ReadImage/539.
convert-im6.q16: no images defined `https://mapknitter.org/public/warps/ten-forward/ten-forward.jpg' @ error/convert.c/ConvertImageCommand/3258.
F
Finished in 9.208966s, 0.1086 runs/s, 0.9773 assertions/s.

I overrode the ImageMagick policy about HTTPS only to find that the feature has been removed because of a security concern. We'll need to curl the image first then call ImageMagick locally!

icarito · 2019-04-10T06:44:15Z

Here's the commit from 2016 removing this:
ImageMagick/ImageMagick@e5ae849

Oh I remember this https://imagetragick.com/ - it was a little rush to patch...

jywarren · 2019-04-10T21:09:36Z

Ah! It shouldn't (at least in tests) be calling a remote image. I changed it to be a local path, and I think we can do the wget/curl stuff in the sinatra app instead!

jywarren · 2019-04-10T23:09:34Z

OK, I see a lot of things like this, where we haven't properly inserted a unique ID (formerly the slug):


convert -contrast-stretch 0 public/warps/ten-forward-working/-demo.png -crop 1543x1543+0+0! -flatten -distort Perspective '0,0 0,0  20,0 1500,171  20,20 1265,1543  0,20 47,764' -flatten -crop 1543x1543+0+0! +repage public/warps/ten-forward/.png
convert +antialias -size 1543x1543  xc:none -draw "fill black stroke red stroke-width 30 polyline 0,0 1500,171 1265,1543 47,764 0,0"  -alpha set -channel A -transparent red -blur 0x8 -channel R -evaluate set 0 +channel public/warps/ten-forward/-mask.png
composite public/warps/ten-forward/-mask.png public/warps/ten-forward/.png -compose DstIn -alpha Set public/warps/ten-forward/-masked.png
gdal_translate -of GTiff -a_srs EPSG:4326  -gcp 0.0, 0.0, -71.3983854668186, 41.8403113680142 -gcp 1500.0766798835248, 171.98331362381577, -71.3916477577732, 41.8397358653566 -gcp 1265.9882808048278, 1543.072508284822, -71.392699183707, 41.8351476451765 -gcp 47.773142671212554, 764.3702827505767, -71.3981708900974, 41.8377535388085  -co "TILED=NO" public/warps/ten-forward/-masked.png public/warps/ten-forward/-geo-unwarped.tif
Input file size is 1543, 1543
0...10...20...30...40...50...60...70...80...90...100 - done.
gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/ten-forward/-geo-unwarped.tif public/warps/ten-forward/-geo.tif
Using band 4 of source image as alpha.
Creating output file that is 1750P x 1304L.
Processing public/warps/ten-forward/-geo-unwarped.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
- [-15896063.825043472, 10274177.588364882]
gdalwarp -s_srs EPSG:3857 -te -71.3983854668186 41.8351476451765 -71.3916477577732 41.8403113680142 public/warps/ten-forward/-geo.tif public/warps/ten-forward/ten-forward-geo.tif
Using band 4 of source image as alpha.
Creating output file that is 1701P x 1304L.
Processing public/warps/ten-forward/-geo.tif [1/1] : 0...10...20...30...40...50...60...70...80...90...100 - done.
gdal2tiles.py -k -z 8-10 --s_srs EPSG:3857 -t ten-forward -g AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ public/warps/ten-forward/ten-forward-geo.tif public/tms/ten-forward/
Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100
Generating Overview Tiles:
0...10...20...30...40...50...60...70...80...90...100
rm: cannot remove 'ten-forward.zip': No such file or directory
convert-im6.q16: unable to open image `/public/warps/ten-forward/ten-forward-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/ten-forward/ten-forward.jpg' @ error/convert.c/ConvertImageCommand/3258.

jywarren · 2019-04-10T23:24:37Z

test/exporter_test.rb

@@ -33,6 +33,7 @@ def test_all_functions # break this into separate parts
    image = {
      height: 20,
      width: 20,
+      id: 1,


I think we needed this; we may need to refactor so it auto-adds the image ids here based on the order they're brought in, instead of relying on an external unique ID

jywarren · 2019-04-10T23:44:59Z

ExporterTest#test_all_functions:
NoMethodError: undefined method `user_id=' for #<MockExport:0x00005569b7ee6a30 @status="warping 1 of 1">
    /app/lib/mapknitterExporter.rb:317:in `run_export'
    test/exporter_test.rb:99:in `test_all_functions'

jywarren · 2019-04-10T23:50:48Z

NameError: uninitialized constant MapKnitterExporter::Open3
Did you mean?  OpenSSL
    /app/lib/mapknitterExporter.rb:326:in `run_export'

jywarren · 2019-04-10T23:51:58Z

I guess we don't have the right dependencies to use this? stdin, stdout, stderr = Open3.popen3('rm -r '+directory.to_s)

I think we need to add this to the Gemfile: https://ruby-doc.org/stdlib-2.4.1/libdoc/open3/rdoc/Open3.html

icarito · 2019-04-12T03:57:19Z

open3 is part of Ruby standard library so no need to add to Gemfile - it was just missing a require!

Finally a few attribute accessors were missing in MockExport class and passing all tests green!
🏄‍♂️ 🌴 🌊 🚀 🌚

jywarren · 2019-04-12T14:16:32Z

Amazing!!! Will move to the other PR now, and then let's get this rereleased and integrated into Sinatra. Thanks!!!

jywarren added 3 commits March 16, 2019 22:20

Update .travis.yml

f5bdd7e

Update .travis.yml

7d68d3e

Update .travis.yml

14aab91

Update mapknitterExporter.rb

73d20a6

jywarren changed the title ~~Update .travis.yml~~ Update .travis.yml, work on gdal errors Mar 17, 2019

jywarren mentioned this pull request Mar 29, 2019

Checklist of to-dos! publiclab/mapknitter-exporter-sinatra#2

Closed

5 tasks

icarito added 5 commits April 9, 2019 23:59

Try xenial dist in travis

b40e032

Add Docker to be able to use debian GDAL

7866d7e

Include Dockerfile

8936b0e

Reliable keyserver

369326f

Upgrade Gemfile to Ruby 2.4.6

1655961

icarito added 4 commits April 10, 2019 00:52

Allow HTTPS in ImageMagick

8c6d768

Fix imagemagick policy

67e7204

Remove HTTPS policy

1d55cf5

Again fix HTTPS

de2cd3c

set root to ""

58eb1e2

image needs an id attribute

5c3a51a

jywarren commented Apr 10, 2019

View reviewed changes

jywarren added 2 commits April 10, 2019 19:32

change for [image],

5165eb4

'.' instead of '' for root

e99e3fe

add user_id to export mock

758cfcc

jywarren mentioned this pull request Apr 10, 2019

simplify run_export #6

Merged

icarito added 2 commits April 11, 2019 22:06

Add missed require

841c280

Add missing accessors to MockExport

2eda079

icarito added the ready label Apr 12, 2019

jywarren merged commit 53dfe18 into main Apr 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update .travis.yml, work on gdal errors #4

Update .travis.yml, work on gdal errors #4

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 29, 2019

jywarren commented Mar 29, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019 •

edited

Loading

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

icarito commented Apr 12, 2019

jywarren commented Apr 12, 2019

Update .travis.yml, work on gdal errors #4

Update .travis.yml, work on gdal errors #4

Conversation

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 17, 2019

jywarren commented Mar 29, 2019

jywarren commented Mar 29, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019

icarito commented Apr 10, 2019 • edited Loading

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren Apr 10, 2019

Choose a reason for hiding this comment

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren commented Apr 10, 2019

icarito commented Apr 12, 2019

jywarren commented Apr 12, 2019

icarito commented Apr 10, 2019 •

edited

Loading