simplify run_export #6

jywarren · 2019-03-29T22:44:53Z

Merge once we resolve everything and after #4; also refactor https://github.com/publiclab/mapknitter/ to use this, and make downstream adjustments in https://github.com/publiclab/mapknitter-exporter-sinatra/

Changes:

turned slug into id (will need test fixes), removed average_scale

def self.run_export(user_id, resolution, export, id, slug, root, average_scale, placed_warpables, key) became
def self.run_export(user_id, resolution, export, id, root, placed_warpables, key)

def self.generate_perspectival_distort(pxperm, path, nodes_array, id, image_file_name, img_url, height, width, root = "https://mapknitter.org") became
def self.generate_perspectival_distort(pxperm, id, nodes_array, image_file_name, img_url, height, width, root = "https://mapknitter.org")

jywarren

Noting specific API changes

jywarren · 2019-04-10T20:55:33Z

lib/mapknitterExporter.rb

@@ -25,16 +25,16 @@ def self.delete_temp_files(path)
  ## Run on each image:

  # pixels per meter = pxperm 
-  def self.generate_perspectival_distort(pxperm, path, nodes_array, id, image_file_name, img_url, height, width, root = "https://mapknitter.org")
+  def self.generate_perspectival_distort(pxperm, id, nodes_array, image_file_name, img_url, height, width, root = "https://mapknitter.org")


jywarren · 2019-04-10T20:56:01Z

lib/mapknitterExporter.rb

    system(self.ulimit+imageMagick)
  end

  # runs the above map functions while maintaining a record of state in an Export model;
  # we'll be replacing the export model state with a flat status file
-  def self.run_export(user_id, resolution, export, id, slug, root, average_scale, placed_warpables, key)
+  def self.run_export(user_id, resolution, export, id, root, placed_warpables, key)


jywarren · 2019-04-10T20:56:58Z

test/exporter_test.rb

-      root,
-      scale,
-      [image], # TODO: these images need a nodes_array
+      [image],


Updated test

jywarren · 2019-04-10T21:02:27Z

test/exporter_test.rb


    coords = MapKnitterExporter.generate_perspectival_distort(
      scale, 
-      slug,
+      id,


Updated test

jywarren · 2019-04-10T23:44:11Z

Getting stuck where it can't find the output of the previous file...


gdal_translate -of GTiff -a_srs EPSG:4326  -gcp 0.0, 0.0, -71.3983854668186, 41.8403113680142 -gcp 1500.0766798835248, 171.98331362381577, -71.3916477577732, 41.8397358653566 -gcp 1265.9882808048278, 1543.072508284822, -71.392699183707, 41.8351476451765 -gcp 47.773142671212554, 764.3702827505767, -71.3981708900974, 41.8377535388085  -co "TILED=NO" public/warps/1/1-masked.png public/warps/1/1-geo-unwarped.tif
nice: gdal_translate: No such file or directory
gdalwarp -of GTiff -t_srs EPSG:4326 public/warps/1/1-geo-unwarped.tif public/warps/1/1-geo.tif
nice: gdalwarp: No such file or directory
rm: cannot remove ‘public/warps/1/1-geo.tif’: No such file or directory

I fixed a few but have to step away; @icarito would you be able to take this forward by looking at the output of each step as it goes to the next?

jywarren · 2019-04-12T14:21:01Z

Now let's rebase this and complete the simplification...

turned slug into id (will need test fixes), removed average_scale Simplified `run_export` to `def self.run_export(user_id, resolution, export, id, root, placed_warpables, key)`

jywarren · 2019-04-12T14:30:23Z

attempted a rebase at #9, but kept it separate just to compare to be sure i've done it right.

jywarren · 2019-04-12T14:48:58Z

OK, the rebase was good; force pushing it here and closing the other PR: #9

jywarren · 2019-04-12T14:53:09Z

Hmm, this looks like an issue with the .travis.yml


Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.KnIUpDVdm4/gpg.1.sh --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0
gpg: keyserver receive failed: No data
The command '/bin/sh -c apt-key adv --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0' returned a non-zero code: 2

jywarren · 2019-04-12T15:07:19Z

Getting there!

  1) Error:
ExporterTest#test_all_functions:
Errno::ENOENT: No such file or directory @ rb_sysopen - demo.png

jywarren · 2019-04-12T15:22:20Z

Somehow this is failing on line 85:

FileUtils.cp(img_url, local_location)

Hmm. I'll log out the vars.

icarito · 2019-04-12T15:27:19Z

Hmm, this looks like an issue with the .travis.yml


Warning: apt-key output should not be parsed (stdout is not a terminal)
Executing: /tmp/apt-key-gpghome.KnIUpDVdm4/gpg.1.sh --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0
gpg: keyserver receive failed: No data
The command '/bin/sh -c apt-key adv --keyserver fks.pgpkeys.eu --recv-keys BF26EE05EA6A68F0' returned a non-zero code: 2

There is a fix for this in main branch - using ipv4 pool (ipv4.pool.sks-keyservers.net) instead!

icarito · 2019-04-12T15:27:56Z

Ah sorry that fix is in mapknitter main repo

jywarren · 2019-04-12T16:18:19Z

😄


gdalwarp -s_srs EPSG:3857 -te -71.3983854668186 41.8351476451765 -71.3916477577732 41.8403113680142 public/warps/1/1-geo.tif public/warps/1/1-geo.tif
ERROR 1: Output dataset public/warps/1/1-geo.tif exists,
but some command line options were provided indicating a new dataset
should be created.  Please delete existing dataset and run again.
gdal2tiles.py -k --s_srs EPSG:3857 -t 1 -g AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ public/warps/1/1-geo.tif public/tms/1/
Generating Base Tiles:
0...10...20...30...40...50...60...70...80...90...100
rm: cannot remove '1.zip': No such file or directory
convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.
F

jywarren · 2019-04-12T16:28:50Z

Now:


gdalwarp -s_srs EPSG:3857 -te -71.3983854668186 41.8351476451765 -71.3916477577732 41.8403113680142 public/warps/1/1-geo.tif public/warps/1/1-geo.tif
ERROR 4: public/warps/1/1-geo.tif: No such file or directory
gdal2tiles.py -k --s_srs EPSG:3857 -t 1 -g AIzaSyAOLUQngEmJv0_zcG1xkGq-CXIPpLQY8iQ public/warps/1/1-geo.tif public/tms/1/
Usage: gdal2tiles.py [options] input_file [output]
gdal2tiles.py: error: The provided input file public/warps/1/1-geo.tif does not exist or is not a file

jywarren · 2019-04-12T16:39:59Z

test/exporter_test.rb


    # get rid of existing geotiff
-    system("rm -r public/warps/#{slug}/1-geo.tif")
+    system("rm -r public/warps/#{id}/1-geo.tif")


we should change all 1 to #{id} and set id to 2 for this run.

jywarren · 2019-04-12T16:41:00Z

Also noting identify-im6.q16: unknown image property "%[exif:Orientation]" @ warning/property.c/InterpretImageProperties/3831.

jywarren · 2019-04-12T16:46:57Z

Well, there we go. There are some issues to resolve, though:

convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.

And i'm not sure why the test doesn't catch that - any ideas, @icarito?

jywarren · 2019-04-12T16:48:35Z

Some of these warnings about not being able to delete things are because some of the methods try to delete anything in the location before trying to make something in that location, and I think that's OK to leave:

rm: cannot remove '/public/warps/1/': No such file or directory
rm: cannot remove '/public/tms/1': No such file or directory

jywarren · 2019-04-12T16:51:57Z

I think this is the last critical bug to resolve --

convert-im6.q16: unable to open image `/public/warps/1/1-geo.tif': No such file or directory @ error/blob.c/OpenBlob/2701.
convert-im6.q16: no images defined `/public/warps/1/1.jpg' @ error/convert.c/ConvertImageCommand/3258.

It's because the final command still runs against /public/... instead of .public/..., so i passed in '.' as root to try to fix it. I'd still like to be able to know that it failed, in a test, though.

jywarren · 2019-04-12T16:52:10Z

But, if this passes, we can merge it!

jywarren · 2019-04-12T17:50:41Z

I think we need to address and simplify the usage of "root" -- is it even relevant anymore for how it's structured now?

I think it originally was trying to use Rails.root but now we're mostly using it locally.

I fixed it again. We can do "root" cleanup in a follow-up PR.

This was referenced Mar 29, 2019

Ensure compatiblity with /lib/exporter.rb in MapKnitter #5

Open

Modularization of export code publiclab/mapknitter#341

Merged

icarito mentioned this pull request Apr 10, 2019

"Integration of modularized export code" publiclab/mapknitter-exporter-sinatra#1

Merged

jywarren commented Apr 10, 2019

View reviewed changes

jywarren added 11 commits April 12, 2019 09:26

simplify run_export

7225f16

turned slug into id (will need test fixes), removed average_scale Simplified `run_export` to `def self.run_export(user_id, resolution, export, id, root, placed_warpables, key)`

Update mapknitterExporter.rb

baa8fbe

Update exporter_test.rb

13935a9

Update exporter_test.rb

8aa2f5e

Update README.md

ae1df40

Update .travis.yml

a77526f

Update README.md

67b9dc0

#{path}

8f32e49

remove 'filename' and use only 'url' in generate_perspectival_distort

b01525a

img_url.slice(0,4) instead of img_url[0..3]

d3b4fe6

Update exporter_test.rb

e7ccbd5

jywarren mentioned this pull request Apr 12, 2019

Simplify run export rebased #9

Closed

jywarren force-pushed the simplify-run_export branch from 4ca193b to e7ccbd5 Compare April 12, 2019 14:50

removing ruby specific code...

7ef5960

jywarren added 2 commits April 12, 2019 10:08

Update exporter_test.rb

a67f868

Update exporter_test.rb

550e9dc

Update mapknitterExporter.rb

8c7e9ad

icarito and others added 4 commits April 12, 2019 10:28

Fix keyserver to IPV4

2f00069

Update mapknitterExporter.rb

e885d19

#{id} style string building

a4fcdd8

Update mapknitterExporter.rb

2b31f2b

jywarren added 3 commits April 12, 2019 11:24

Update exporter_test.rb

ae76894

Update .travis.yml

750fbcd

Update mapknitterExporter.rb

fe1b773

jywarren added 2 commits April 12, 2019 11:31

Update exporter_test.rb

da9ab43

Update exporter_test.rb

058872c

jywarren commented Apr 12, 2019

View reviewed changes

Update exporter_test.rb

8bd1d16

Update exporter_test.rb

6260d17

jywarren added 2 commits April 12, 2019 12:45

Update mapknitterExporter.rb

1d32448

Update exporter_test.rb

0b9cac8

jywarren merged commit ee160b3 into main Apr 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

simplify run_export #6

simplify run_export #6

jywarren commented Mar 29, 2019 •

edited

Loading

jywarren left a comment

jywarren Apr 10, 2019

jywarren Apr 10, 2019

jywarren Apr 10, 2019

jywarren Apr 10, 2019

jywarren commented Apr 10, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

icarito commented Apr 12, 2019

icarito commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

simplify run_export #6

simplify run_export #6

Conversation

jywarren commented Mar 29, 2019 • edited Loading

Changes:

jywarren left a comment

Choose a reason for hiding this comment

jywarren Apr 10, 2019

Choose a reason for hiding this comment

jywarren Apr 10, 2019

Choose a reason for hiding this comment

jywarren Apr 10, 2019

Choose a reason for hiding this comment

jywarren Apr 10, 2019

Choose a reason for hiding this comment

jywarren commented Apr 10, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

icarito commented Apr 12, 2019

icarito commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren Apr 12, 2019

Choose a reason for hiding this comment

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Apr 12, 2019

jywarren commented Mar 29, 2019 •

edited

Loading