MapKnitterExporter architecture discussion #298
Comments
|
@jywarren this looks really nice! Things immediately make a lot more sense!🎉 |
|
I've just spent some time deploying a learning project to Google Cloud Platform (App Engine) as a Docker container. I've got a better understanding now of what is required! Thanks! |
|
Awesome. I have a more system-wide planning issue drafted and will post
soon. But these container tests can start whenever you both are ready.
Sebastian do you think getting the gdal and imagemagick containers will be
pretty straightforward too?
Thanks again!
…On Tue, Jan 22, 2019, 4:12 AM Sebastian Silva ***@***.*** wrote:
I've just spent some time deploying a learning project to Google Cloud
Platform (App Engine) as a Docker container. I've got a better
understanding now of what is required! Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJxwWLCcBjSpGwbipwzLX6OgVhYoLks5vFrm-gaJpZM4aDgKo>
.
|
|
OK, i'd like to add in an overview of the export system step by step; i've left notes where we might make changes or improvements as well, and will link to lines of code where these things currently happen! Also -- a couple ideas:
Breaking down the export processSeparable steps
|
|
@SidharthBansal @tech4GT @icarito just so you see this additional note breaking down the export process. There are portions that could be accomplished with traditional ImageMagick/GDAL combo just breaking out the Ruby-controlled code in our codebase (see #296 but i'll copy in more here), but I am hoping we can accomplish a lot in stand-alone containers in a serverless or at least remote REST model. |
|
Starting work now! @icarito Can you please share some of the resources you have been going through, that would be a big help for me :) |
|
Okay a couple of things here
/* Request Body */
{
'url': <String>, // URL if input image
'sequence': <String> // The sequence string which will be imported into sequencer,
'upload': <Boolean> // Denotes whether to return the data uri or to upload to imgur and return that
} |
|
How about https://github.com/publiclab/image-sequencer-app, which we
created a while back?
Let's start with a dataurl, but we should also plan to have an abstract way
to "put" the image somewhere.
Great!
…On Thu, Feb 7, 2019 at 3:36 AM Varun Gupta ***@***.***> wrote:
Okay a couple of things here
- We should add a flag to the run config which allows us to disable
the progress logs(it'll unnecessarily slow down the server otherwise)
- I have some ideas in mind to speed up the pixelManipulation API
which will in turn speed up most modules
- Should we return the output as a data uri or us the imgur service
like we originally planned? Or maybe we can have a parameter in the request
which allows both options
/* Request Body */
{'url': String, // URL if input image'sequence': String // The sequence string which will be imported into sequencer,'upload': boolean // Denotes whether to return the data uri or to upload to imgur and return that
}
How does this sound @jywarren <https://github.com/jywarren> @icarito
<https://github.com/icarito> ??
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJw5cIYwIVHGe5TuxzTkK1mvYRb0Xks5vK-WFgaJpZM4aDgKo>
.
|
|
@jywarren Ok pushing the most basic setup now! |
|
@jywarren Can you please grant me push access to the repository 😅 |
|
doing so now, thanks!!!
…On Thu, Feb 7, 2019 at 2:46 PM Varun Gupta ***@***.***> wrote:
@jywarren <https://github.com/jywarren> Can you please grant me push
access to the repository 😅
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ6EgdE3JvF9S42CVQM3xjNkq2MuCks5vLIJcgaJpZM4aDgKo>
.
|
|
@jywarren One more thing, do you want me to get cracking on the optimizations for sequencer first or deploy the container first? |
|
Let's get the container working first -- but we can also encourage people
in IS to tackle some of the optimizations, and point at this container to
show why it'll be important!
…On Thu, Feb 7, 2019 at 3:22 PM Varun Gupta ***@***.***> wrote:
@jywarren <https://github.com/jywarren> One more thing, do you want me to
get cracking on the optimizations for sequencer first or deploy the
container first?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ8hcFGGQRrf4c7ercVIGG0UVva9dks5vLIrbgaJpZM4aDgKo>
.
|
|
Okay I'll try to deploy the container with a very basic setup tomorrow, and then I'll raise an issue for the optimizations, maybe I can document some of my ideas over there too! |
|
oh wow!!! very cool.
Check out the various "projects" several of which are optimization related:
https://github.com/publiclab/image-sequencer/labels/project
…On Thu, Feb 7, 2019 at 3:27 PM Varun Gupta ***@***.***> wrote:
Okay I'll try to deploy the container with a very basic setup tomorrow,
and then I'll raise an issue for the optimizations, maybe I can document
some of my ideas over there too!
Also on a different note I tried out the app locally and it works like a
charm ✌️
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJwekjjGujCRH36NHXMVawY1rYa5Rks5vLIv6gaJpZM4aDgKo>
.
|
|
One think I am concerned about though is, if we do switch to web assembly, what parts of the main code we would need to re-write or should we just switch to something like openCV entirely? |
|
Yeah i am not sure about it. I think we can follow multiple paths to
optimize and should probably discuss that in the IS repo. Switching several
modules to openCV would be powerful and flexible. So would webAssembly of
pixelManipulation.
…On Thu, Feb 7, 2019 at 3:33 PM Varun Gupta ***@***.***> wrote:
One think I am concerned about though is, if we do switch to web assembly,
what parts of the main code we would need to re-write or should we just
switch to something like openCV entirely?
I think we can start with making optimizations in javascript and then move
towards web-assembly if that gets unmanageable, what do you think?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ6gpOiK6JeljqGN9nA6zAp1HfQ67ks5vLI1xgaJpZM4aDgKo>
.
|
|
I think you are right, also please do have a look at the repository, I have pushed the basic file I wrote earlier today, will be extending this A LOT but I think this gives us a start. |
|
Just a note that Google Cloud Engine has Standard Environment and Flexible Environment and Ruby seems to only be supported on Flexible Environment which is significantly more expensive: https://cloud.google.com/appengine/docs/standard/appengine-generation |
|
Oh, but is-app will be pure node.js, so I guess thats not a problem! |
|
Also I was meaning to ask whether leaflet works in node? Actually I have no idea about that. Does it use gl? |
|
For the coordinate conversions? Hmm I'm not sure, but if it doesn't we can
probably use proj4.js.
We should make sure import-image also works in node, huh? But not as part
of this project unless it's the way to advance our compositing... I'd guess
it is, though... Why isn't import image using the same method as the
initial module? Probably for efficiency, I guess, if new Image is faster.
…On Tue, Mar 19, 2019, 3:14 AM Varun Gupta ***@***.***> wrote:
Also I was meaning to ask whether leaflet works in node? Actually I have
no idea about that. Does it use gl?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ4BW999cTnu3opzOG0n1Y1_PneWvks5vYI4-gaJpZM4aDgKo>
.
|
|
Okay so do you want me to update import image to work in node using method similar to load-image then? |
|
If that seems like the best way forward, what do you think?
…On Tue, Mar 19, 2019, 9:08 AM Varun Gupta ***@***.***> wrote:
Okay so do you want me to update import image to work in node using method
similar to lead-image then?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ6JpgiOVJp2VFoGmdGkTIY8L04Biks5vYOEfgaJpZM4aDgKo>
.
|
|
I mean without that there's no way of merging images anyway, so let's go with this for now! |
|
Okay and another thing is that we need a way to figure out where exactly to overlay the images onto one another, as in if some Image needs to be rotated first and such. Would it be possible for you to give me a simple sample to work this out, actually I tried doing this with the test dataset you mentioned but it's kind of confusing. I was wondering If I can get a stripped down version of that data on which I can figure this out. |
|
Well, rotation will be done as part of the warping, not as a separate step.
So I think best intermediate approach would be like just placing on the
first of the corner points provided. There should be code in the exporter
library to calculate relative positions -- let me see:
https://github.com/publiclab/mapknitter-exporter/blob/main/lib/mapknitterExporter.rb#L252
Actually this newly spun out library does the whole sequence in the
traditional ruby gdal toolchain. It's a good reference for us on the
Javascript version!
…On Tue, Mar 19, 2019, 9:22 AM Varun Gupta ***@***.***> wrote:
Okay and another thing is that we need a way to figure out where exactly
to overlay the images onto one another, as in if some Image needs to be
rotated first and such. Would it be possible for you to give me a simple
sample to work this out, actually I tried doing this with the test dataset
you mentioned but it's kind of confusing. I was wondering If I can get a
stripped down version of that data on which I can figure this out.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ0axA5e0gk1hrbCo6SdgitJbu092ks5vYORbgaJpZM4aDgKo>
.
|
|
I'll check this out! |
|
Alright, so we have import Image working now, I'll try a workflow of combining Images, right now overlay cuts the image off but we can fix that in a stitch module later. |
|
Yes or perhaps we can make a "resize-canvas" or "canvas-size" module; that might be useful later anyways. |
|
Portions of the above in the Ruby workflow are coming online:
Thanks @icarito !!! |
|
@jywarren Since divy is looking into puppeteer for the gl based implementation, I was wondering what should we pick next? |
|
Those are definitely critical, so let's dive into canvas-resize and
Geotiff? And if you can keep supporting Divy on webgl too, that would be
awesome. Thank you Varun!
…On Thu, Apr 4, 2019 at 10:18 AM Varun Gupta ***@***.***> wrote:
@jywarren <https://github.com/jywarren> Since divy is looking into
puppeteer for the gl based implementation, I was wondering what should we
pick next?
I mean on the js track we are left with canvas-resize, geotiff and JSON
conversion. Did I miss anything?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABfJ63-dddd-vP0YVVYXgLCpHfBXvhyks5vdgmpgaJpZM4aDgKo>
.
|
|
Sure! I’ll get started of cancas-resize then. 😄 |
|
Okay just to sum up, we have a basic exporter process running in gCloud based on is-app
|
|
This seems just right, although geotiff can wait, for sure! I think the key here is getting the canvas-resize, the overlay/compositing, and the whole thing running to generate a correct output for a given map, which need not include geotiff for now. And actually the optimization is important, but with more memory, we should be OK on that too, and can circle back to that later. For reference, here's where we're testing the output of map exports in the Ruby-based parallel exporter project: publiclab/mapknitter-exporter-sinatra#23 Those maps and the |
|
Update on the node.js exporter:
|
|
OK, awesome. Great to hear it!
…On Tue, May 28, 2019 at 12:23 PM Varun Gupta ***@***.***> wrote:
Update on the node.js exporter:
- The export process is now using multiSequencer parallelized workflow!
- The app is set up with docker, and is ready to be deployed and
tested.
cc @jywarren <https://github.com/jywarren>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298?email_source=notifications&email_token=AAAF6JYBBTKHYUWVMIKXRHDPXVL55A5CNFSM4GQOAKUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMVMUI#issuecomment-496588369>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAF6J6GAK2W35YSFWV4WADPXVL55ANCNFSM4GQOAKUA>
.
|
|
@jywarren I was also thinking of adding a benchmarking test like we have on main is repository. That'll tell us how each of the optimizations we make affects the server response times! One more thing.. what kind of workload can we expect on this service? I mean is it possible we would be dealing with multiple requests at the same time? |
|
And, we still need to address the array size issue in get-pixels and
save-pixels, right? I made an issue here to track all of that thoroughly:
publiclab/image-sequencer-app#12
…On Tue, May 28, 2019 at 12:25 PM Jeffrey Warren ***@***.***> wrote:
OK, awesome. Great to hear it!
On Tue, May 28, 2019 at 12:23 PM Varun Gupta ***@***.***>
wrote:
> Update on the node.js exporter:
>
> - The export process is now using multiSequencer parallelized
> workflow!
> - The app is set up with docker, and is ready to be deployed and
> tested.
> cc @jywarren <https://github.com/jywarren>
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#298?email_source=notifications&email_token=AAAF6JYBBTKHYUWVMIKXRHDPXVL55A5CNFSM4GQOAKUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWMVMUI#issuecomment-496588369>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAAF6J6GAK2W35YSFWV4WADPXVL55ANCNFSM4GQOAKUA>
> .
>
|
|
@jywarren Could you please also respond to the query I mentioned above! Thanks a ton! :) |
|
We should be able to load balance a cluster using the infrastructure Sebastian has made, yes! |
|
Oh! In that case I don't have to worry about it! Awesome!! Thanks a lot!! 😃 |
|
:-)
…On Tue, May 28, 2019 at 1:27 PM Varun Gupta ***@***.***> wrote:
Oh! In that case I don't have to worry about it! Awesome!! Thanks a lot!!
🤩
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#298?email_source=notifications&email_token=AAAF6J54224VYRGZODQXPYTPXVTNBA5CNFSM4GQOAKUKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWM3D7Y#issuecomment-496611839>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAAF6J2722BXAJSJ5S2YEYDPXVTNBANCNFSM4GQOAKUA>
.
|
|
Getting closer in #1192 |
We are exploring parallel tracks for cloud-based MapKnitter exporting, and one option is a JavaScript based process.
The base idea is to run the export process as a scalable web service, possibly "serverless" or REST, in Google Cloud and/or other cloud providers like Amazon AWS Lambda (primarily Google Cloud but compatible with others). Comments/suggestions/eurekas welcome! 🎉
Importantly, either track would ideally present the same API so that we could compare their performance.
JavaScript track
In this track, more experimentally, we'd use Image Sequencer, possibly with the webgl-distort library.
The major challenges here, I'd guess, would be:
For these last two, see #296 where there are some JS options to experiment with.
Also, we would try to develop this track in such a way as to make it possible to run locally in the browser, natively or in an Electron-style local JS app.
Ruby/ImageMagick/GDAL track
A more traditional approach is being explored here: #258, where we take the exporting sections currently featured in MapKnitter, and duplicate them in a minimal Ruby container that can be run on-demand.
Spec
To guide the development of both tracks, we're imagining a basic common behavior of:
Links and resources are being compiled here: #296
What have I missed? @tech4GT @icarito would you mind adding any questions, clarifications?
Update: diagrams
I've put together a diagram of the current exporter workflow, which I hope is helpful. It's also largely ported into a standalone Ruby library in #341 -- soon to potentially be a Gem:
Image Sequencer should allow us to parallelize this, and improve its speed, as illustrated in this diagram:
The text was updated successfully, but these errors were encountered: