Logical image() argument names & defaults to make images look like images

[IanButterworth]

Related to JuliaPlots/AbstractPlotting.jl#98, #295 and #205

With the caveat of not being someone with familiarity of the inner workings.. here's my suggestion for image() functionality, to make images appear like I think most people would expect them to. I've aimed for logical keywords that I might try before looking at the docs...

Defaults for image():

• transpose::Bool=true - Swaps the x & y axes. Happens before the options below
• flipy::Bool=true - Prevents images looking upside down
• flipx::Bool=false - A good option to have access to. i.e. front-facing cameras where flipx is common practice
• squarepixels::Bool=true - The equivalent of scale_plot = false.
• show_axis::Bool=false - I think most would expect image() to show without the axes & grid

@ssfrr @SimonDanisch

[c42f]

I think the suggested defaults for transpose, flipx, flipy are good. Having image() interpret the data as flipped and transposed is probably the best we can do, given we are stuck with an inconsistency between the way images are loaded and the desired consistency conditions in #205 (comment)

squarepixels strikes me as a confusing name (you can't change the shape of the pixels themselves, and moreover there may be displays which have non-square physical pixels). I'd prefer aspect_ratio for this.

show_axis seems like it could be fine one way or another (how do these settings compose with other plots in the same scene?)

[IanButterworth]

I like that aspect_ratio is a common term, but its interpretation is a little confusing. What would the default value be to create a non stretched image? Its value is dependent on w and h so I guess w/h (to conform to the standard 4:3, 16:9 etc), where h & w are calculated after transpose, meaning it would be tricky to make it an input argument.

In the matlab documentation for ‘axis image’ it states it:

Sets DataAspectRatio to [1 1 1]and sets the associated mode property to manual. Disables the “stretch-to-fill” behavior.

https://www.mathworks.com/help/matlab/ref/axis.html

So perhaps I’m thinking of aspect ratio in the wrong way, but I think there is a decent chance of confusion given the typical 4:3 etc interpretation. Perhaps something like DataAspectRatio is a little clearer?

[ssfrr]

From these matlab docs it seems that matlab gives both pbaspect to set the aspect ratio of the plot directly as well as daspect to set the aspect ratio as a scaling factor on the data size. I can definitely see use-cases for both.

Another consideration is that for 3D plots you might want to set the aspect of all three axes, so the option should work for both 2D and 3D plots (though auto-scaling the axes independently doesn't make sense in 3D, right?)

I think my favorite is:

• aspect (takes either :auto or a tuple specifying the aspect ratio terms. that way you can just do aspect=size(data) if you want the aspect ratio to match the data, or aspect=(4,3) if you want to explicitly set the ratio. It also generalizes nicely to 3 dimensions.)

I suppose aspect_ratio would be fine also, but it seems more wordy and I think if you know what an aspect ratio is then aspect would make sense to you.

I wish I could remember the result of some slack bikeshedding on this issue a while ago. Here are some other options that came to mind:

• maintain_aspect (true to keep aspect ratio of data)
• fixed_aspect (true to keep aspect ratio of data)
• lock_aspect (true to keep aspect ratio of data)
• data_aspect (true to keep aspect ratio of data)
• aspect (can be :equal, :auto, or a number. see matplotlib)
• autoscale (true for full autoscale, false for fixed-size, :lock_axes for fixed aspect ratio)

[mkborregaard]

I have the same favourite as Spencer

[IanButterworth]

Me too.

To be complete, while we're at it.. on the square pixels idea there is some precedent for pixel_aspect_ratio https://en.wikipedia.org/wiki/Pixel_aspect_ratio which would be 1.0 for standard image behaviour.

[c42f]

Right. The confusion seems to be whether an aspect ratio would apply to the axes or to the data. The point about 3D having more than one aspect ratio is a good one.

I think we'll gain some clarity by talking about this in terms of coordinate systems and transformations. There's a whole bunch of coordinate systems here. At the least the following coordinate systems are important in this discussion:

• image: Image sample coordinates (integer coordinates of the pixel samples, with (1,1) conventionally in the top left of the image)
• world: Global coordinate system of the scene. The image corners have some coordinates (2D or 3D) in the world, defining the quadrilateral geometry onto which the image is texture mapped.
• axis: the axis "model" coordinate system (I assume the axis->world transformation is the identity by definition in Makie but I'm not sure?)
• viewport: Logical (integer) pixel coordinates of the display device
• physical: Coordinates (in real world mm) of the physical display device

Setting the aspect ratio to 1.0 (or (1,1)) is a request that the overall image->physical mapping should be isotropic. Notice that by itself this doesn't specify the aspect ratios of the image->world, world->viewport, or viewport->physical transformations!

[c42f]

In light of the above set of coordinate systems, I think the current setup with scale_plot makes perfect sense: it defines the scaling of the world->physical transformation (including annoyances of the windowing system and physical properties of the screen).

The remaining question is how one defines the image->world transformation. This already has a solution: you just use the positional arguments x,y in image(x,y,img). If you want aspect=(4,3), that's simply image(0..4, 0..3, img). I don't think there's much onerous about that.

Side note: I'm not quite sure what scale_plot=false precisely means in the 3D case when the transformation reduces from 3 to 2 dimensions via a perspective projection. Not only is one dimension dropped, but a perspective projection has varying anisotropy across the viewport! Perhaps the best we can do in that case is that "the transformation is isotropic at the center of the viewport" or some such thing.

[c42f]

To summarize, I don't think we need aspect or alternatives, unless we had a great name which would be a replacement for scale_plot in specifying the scaling of the world->physical transformation. I feel that the desire to talk about the image->physical transformation alone by talking about aspect is somewhat incompatible with the idea of plotting multiple objects into a shared world coordinate system. Of course we can have good defaults which work when plotting single objects so that image does what you expect in isolation.

What we probably do need is more documentation for the coordinate systems, transformations and generally the rendering pipeline model used by Makie. Last I looked I remained confused about this.

[IanButterworth]

Perhaps stretch is a better word than scale?

How about:
stretch_plot::Union{Bool,Tuple}

• true: stretches image to fill axes
• false: shows image as per norm for images
• (4,3): displays image with image dimension aspect ratio 4:3

[ssfrr]

The confusion seems to be whether an aspect ratio would apply to the axes or to the data.

Yeah, that's a good point. The use-cases of specifying the aspect ratio of the output plot vs. the displayed image data are separate and I was conflating them.

I think we'll gain some clarity by talking about this in terms of coordinate systems and transformations. There's a whole bunch of coordinate systems here...

In light of the above set of coordinate systems, I think the current setup with scale_plot makes perfect sense: it defines the scaling of the world->physical transformation (including annoyances of the windowing system and physical properties of the screen).

I think Makie has a somewhat divided identity - in some sense it's this very flexible 2D/3D graphics package with a scene graph and nested coordinate systems. On the other hand for many (probably most) users it's also a plotting library and they don't want to think about coordinate systems. I think it's important to provide an API to plotting users that doesn't depend on them thinking in terms of nested coordinate systems. Of course this isn't easy and do-what-I-mean can be dangerous, but there is a lot of precedent for plotting libraries so hopefully that can help.

Setting the aspect ratio to 1.0 (or (1,1)) is a request that the overall image->physical mapping should be isotropic. Notice that by itself this doesn't specify the aspect ratios of the image->world, world->viewport, or viewport->physical transformations!

Not necessarily, setting aspect=(1,1) could be interpreted as setting the image->world transformation to make a square plot. This side-steps the issue of 3D perspective which you brought up, as well as the issues with multiple plot objects. As long as the downstream coordinate transformations are isotropic then the effect is the same.

If you want aspect=(4,3), that's simply image(0..4, 0..3, img). I don't think there's much onerous about that.

That has the side-effect of actually changing the tick labels, which seems independent of changing the shape of the plot

I also think scale_plot is a confusing name, because the plot is still scaled if scale_plot=false, it's just scaled uniformly rather than scaling the axes individually.

[mkborregaard]

I think Makie has a somewhat divided identity - in some sense it's this very flexible 2D/3D graphics package with a scene graph and nested coordinate systems. On the other hand for many (probably most) users it's also a plotting library and they don't want to think about coordinate systems.

This. I think the key selling point of Makie is that it (so far) manages to seamlessly bridge these two worlds.

[c42f]

On the other hand for many (probably most) users it's also a plotting library and they don't want to think about coordinate systems.

For sure, I'm not intending to suggest that understanding the coordinate transformation chain should be a requirement to use Makie. The reason I bring this up is clarify the development discussion.

The challenge is to have a keyword which, as you say, is friendly for incidental uses, but in the backend is precisely defined in terms of how it affects the coordinate transformation chain.

Not necessarily, setting aspect=(1,1) could be interpreted as setting the image->world transformation to make a square plot.

Yes, but it seems we've already got a way to do this: the x,y in image(x,y,img) are what define the image->world transformation. aspect could define a certain way to compute x,y from the image size, I suppose. But what would it mean for other plot types?

[mkborregaard]

I'm not actually following all the subtleties here, but to me an aspect ratio of 1 means that the scale of the y axis and the x axis is the same. Meaning that a 0,0 1,0 1,1 0,1 polygon will be square.

[c42f]

Meaning that a 0,0 1,0 1,1 0,1 polygon will be square.

By this, do you mean the following:

• Take a physical ruler, and measure the length of the interval [0,1] on the physical screen, in mm.
• This distance in millimeters should be the same for both the x and y axes. (And indeed also on the diagonal: measuring between the corners (0,0) and (1,1) of your polygon should have a length sqrt(2) times larger than the sides.)

This is what I mean when I say the world->physical coordinate transformation is isotropic: all directions are scaled the same when transforming from the virtual coordinate system in which the plot objects are embedded to the real coordinate system you measure with a physical ruler.

[mkborregaard]

Yes that is what I mean.

[IanButterworth]

As a path forward, I vote for converging on a replacement name for scale_plot

As I mentioned above, I vote for
stretch_plot::Union{Bool,Tuple}

• true: stretches image to fill axes
• false: shows image as per norm for images (default for image)
• (4,3) or (400,300) or (4/3,1): displays image with image dimension aspect ratio 4:3

I see this as a stretching of the plot canvas, but not the axes themself. To me, scaling a plot is a little ambiguous, but stretching a plot is clearer.

Plus, if were to use image() and the image appeared stretched or squashed, I'd probably google 'makie image plot stretched`.. but maybe others are better trained at going straight to docstrings/docs

[ssfrr]

Yes, but it seems we've already got a way to do this: the x,y in image(x,y,img) are what define the image->world transformation. aspect could define a certain way to compute x,y from the image size, I suppose. But what would it mean for other plot types?

But as I mentioned above that also changes the axis ticks. I may not be fully understanding the nested coordinate systems described in your message above. Should axis and world be flipped? I.e. I'd expect data to live inside an axis (you can have multiple data living in the same axis), and then axis to live inside world (you can have multiple axis coord systems inside the world). Which coordinate system gets modified when you resize the display window?

I'm not actually following all the subtleties here, but to me an aspect ratio of 1 means that the scale of the y axis and the x axis is the same. Meaning that a 0,0 1,0 1,1 0,1 polygon will be square.

This is not what I described above. My suggestion was that having an aspect ratio of 2 or (2,1) means that the displayed data is twice as wide as it is tall. So if you want it to match the data you'd use size(data), so e.g. if your data was 400x300, doing aspect=size(data) is equivalent to aspect=(400,300), which is the same as aspect=(4,3). That way it's easy both for the use-cases of setting the visualized aspect ratio or making it match your data.

As I mentioned above, I vote for stretch_plot::Union{Bool,Tuple}

stretch has the same problems as scale for me, in that the plot is still stretched/scaled as you resize the window, even though the option is false. It seems like this option should invoke the concept of aspect ratio in some sense. Maybe data_aspect?

[IanButterworth]

@ssfrr ok, how about:
data_aspect::Union{Symbol,Real}

• :isotropic: makes an image look like an image (in most cases) where data aspect is 1. Default for image()
• :stretchfill: auto stretches image to fill axes
• 1: same as :isotropic
• 4/3: would stretch a 100x100 image array to be 4/3 times wider than taller. Useful for cases where the image pixels aren't representing square areas of the source image they're representing (like the anamorphic format, for instance)

[c42f]

But as I mentioned above that also changes the axis ticks. I may not be fully understanding the nested coordinate systems described in your message above. Should axis and world be flipped? I.e. I'd expect data to live inside an axis (you can have multiple data living in the same axis), and then axis to live inside world (you can have multiple axis coord systems inside the world).

Right, the question is whether the axis and world coordinate systems are the same or not. At the moment it seems to me that they might be, but I haven't had time to read the code which implements axes. If the axes are different from the world, there would be a non-identity axes->world transformation and the data objects would naturally live in an axis coordinate system. Actually it's kinda required that these are different for certain types of plots, for example plots with two different y axes.

I was hoping @SimonDanisch would chime in and describe the coordinate systems in use and their preferred names, if any. There are somewhat standard names for the coordinate systems used by OpenGL but Makie might need more.

Which coordinate system gets modified when you resize the display window?

I'd prefer to think of the coordinate systems as unchanging, and the world->viewport transformation being modified in this case.

[logankilpatrick]

Let's re-ignite this thread and keep the conversation going!

[asinghvi17]

FWIW, all of these changes will (probably) need to be implemented individually in the backends, since image is a primitive.

[KronosTheLate]

I agree that it would feel right if image showed something that was flipped and tansposed (i.e. rotr90) compared to say a heatmat. I though that would be half the point of having both image and heatmap.

Are there anyone thinking that it a bad idea?

[SimonDanisch]

Nope, I think everyone is on board with that, I just need to find some time to implement it...

[ffreyer]

#1406 implemented some of the transformations in a roundabout way.