MIME based adaptor function

[ellisonbg]

Python objects can have special methods (_repr_mimebundle_, _repr_html, _repr_png_, etc.) that could be called to grab content that could be easily adapted to their VDOM based components. This would be a nice way of doing general purpose adaptation between other objects and the VDOM API. We could have a single function vdom_adapt that handles all this:

def vdom_adapt(o):
    if hasattr(o, '_repr_png'):
        velement = img(src=o._repr_png_()) # ok, put in the data uri base64 encoded
    elif ...
    return velement

Then you could do things like:

div(IPython.display.Image('coolthing.png'), my_matplotlib_figure, ...)

We might need a way of selecting which MIME type to use if an object had multiple.

[rgbkrk]

That is such a great idea! All the image reprs seem like easy choices.

_repr_html_ may be a dangerous one to handle for two reasons:

1. dangerouslySetInnerHTML does not actually run all scripts within since it's using innerHTML, which means it won't behave the same as how the current frontends handle html mimetypes (notebook uses $.html, nteract creates a document fragment)
2. I was really hoping we'd try to keep this as sanitary as possible to continue mitigating security concerns in the long term

That being said, it sure would be nice to be able to group up pandas dataframe html into these outputs. The main use case that I'm 💯 % likely to use this for is generating stylized reports. My current thinking was that I would iterate over df.to_dict(orient='row'), using the table schema to assist in choosing formatting.

We might need a way of selecting which MIME type to use if an object had multiple.

We could do our classic display order handling here, matching that of the classic notebook as a sane default.

[ellisonbg]

Yeah, for HTML we could parse it, walk the objects and emit JSON while screening allowed tags/attrs, but I don't think that is worth it. Is there anything else other than images this would work for?

…

On Fri, Sep 29, 2017 at 3:42 PM, Kyle Kelley ***@***.***> wrote: That is such a great idea! All the image reprs seem like easy choices. We could do our classic display order handling here. _repr_html_ may be a dangerous one to handle for two reasons: 1. dangerouslySetInnerHTML does not actually run all scripts within since it's using innerHTML, which means it won't behave the same as how the current frontends handle html mimetypes (notebook uses $.html, nteract creates a document fragment) 2. I was *really* hoping we'd try to keep this as sanitary as possible to continue mitigating security concerns in the long term That being said, it sure would be nice to be able to group up pandas dataframe html into these outputs. The main use case that I'm 💯 % likely to use this for is generating stylized reports. My current thinking was that I would iterate over df.to_dict(orient='row'), using the table schema to assist in choosing formatting. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0GEIyGpAToL6jiSZthYZWBegQscXks5snXJrgaJpZM4PoP32> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

[rgbkrk]

I feel like if we use the IPython formatter API, we can use text/plain. Technically we could turn application/javascript into script(code), though I'd much rather stick script on the blocklist for the frontend handling of vdom.

[ellisonbg]

yeah +1 on blocking `script`

…

On Tue, Oct 3, 2017 at 11:26 AM, Kyle Kelley ***@***.***> wrote: I feel like if we use the IPython formatter API, we can use text/plain. Technically we could turn application/javascript into script(code), though I'd much rather stick script on the blocklist for the frontend handling of vdom. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0Og9pTI4bhHUhRjPxxw73ZOV5xNjks5sonxMgaJpZM4PoP32> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

[rgbkrk]

Ok Brian, I tried something a little crazy...

Would it really be all that bad for us to parse some HTML or SVG 😉?

[ellisonbg]

Not too crazy - and probably going to be hard to stop folks from wanting this. Would allow us to handle the SVG and HTML MIME types.

[mpacer]

This approach could work with audio and video as easily as it does with images.

I think where it gets murky is when you have to do a lot of interpretation of the "type" of thing that it is and a lot of recursion once you know that that type is.

Images, audio and video are all direct data forms that have a conventional way of being displayed that requires relatively little interpretation and (at least as far as I know) there's no way to nest images, audio or video (and so no recursion is needed).

General html (on the other hand) requires interpretation (to determine the first element type, assuming it's actually a fragment and not a full-blown page) and then recursion, since it by definition is able to hold other objects.

SVGs initially require recursion, and the elements below it are not guaranteed to be of the svg type, so even though the top-level svg tag doesn't require interpretation, all of its elements would.

I feel like this risks becoming heavy weight in practice if you ever wanted to update deeply nested objects in this way.

The caveat to that heavy-weight-ness would be if we can intelligently figure out incremental updates to pass back up, but that seems like a really hard problem in the general case that we probably don't want to have to invent/reinvent a solution to.

[rgbkrk]

@mpacer is spot on in describing the potential complexity. Given my current stretch of maintained projects (which I admit I allow to grow), I'd prefer to keep this one simple. M is thinking ahead about if/when we handle updating nested objects (a virtual dom diff).

One way I'd probably think about this: React never takes arbitrary Objects -- you're expected to create a function that returns React elements or a class with a render function. That would keep it simple for us, which could mean we look for only _repr_vdom_ as if that was our render function. Note that we're probably going to end up going through multiple revisions of the vdom mimetype, so I don't know what that's going to look like...

[rgbkrk]

How about, for now, we create another repo/package, which we can toss any time we want, called vdomify. It would use the display_formatter to compute the mime bundle:

ip = get_ipython()
ip.display_formatter.format(obj)

Then we'd have our own "display order".

[ellisonbg]

I am fine putting the code to convert between other objects and and vdom elements in a separate package.

…

On Fri, Oct 6, 2017 at 8:10 PM, Kyle Kelley ***@***.***> wrote: How about, for now, we create another repo/package, which we can toss any time we want, called vdomify. It would use the display_formatter to compute the mime bundle: ip = get_ipython() ip.display_formatter.format(obj) Then we'd have our own "display order". — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0N0NqlcWUe7_UDEfkRgZUkFuwp2Aks5spuuSgaJpZM4PoP32> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

[rgbkrk]

Ok I made an initial stab at this in vdomify (which is also pip-installable -- run pip install vdomify).

I'm cheating with the HTML mimetype for now, I'll add image types shortly... The interface underneath pretty much matches https://github.com/nteract/nteract/tree/master/packages/transforms#standard-jupyter-transforms

Which... I suppose now that I see a clean path for overriding our opinionated defaults, I'm a little more ok with it.
¯\_(ツ)_/¯

[ellisonbg]

I like the core interface here. Would we provide some default transforms though?

…

On Sun, Oct 8, 2017 at 1:37 PM, Kyle Kelley ***@***.***> wrote: Ok I made an initial stab at this in vdomify <https://github.com/rgbkrk/vdomify> (which is also pip-installable -- run pip install vdomify). I'm cheating with the HTML mimetype for now, I'll add image types shortly... The interface underneath pretty much matches https://github.com/nteract/nteract/tree/master/packages/ transforms#standard-jupyter-transforms [image: screen shot 2017-10-08 at 1 36 27 pm] <https://user-images.githubusercontent.com/836375/31320698-b6b85ad2-ac2d-11e7-8fca-cbbaf233fa28.png> Which... I suppose now that I see a clean path for overriding our opinionated defaults, I'm a little more ok with it. ¯\_(ツ)_/¯ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0IgdYLHwfeqou5JTXY-LVnQ326Keks5sqTJ7gaJpZM4PoP32> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

[rgbkrk]

Yeah I made transforms for text/plain, text/html, and application/vdom.v1+json (which is a no-op). Open to the others.

It's kind of funny though, they mostly mirror the mimetype handling on the frontends (even the image/* ones will once implemented). Makes me kind of wish for either custom elements (web components) or a way to use a "namespace" of react components (note: we've been down the path before with the namespacing and was not pleased with the security ramifications if dynamically loaded).

[ellisonbg]

Cool, the main other ones would be for the bit mapped image types. I know there will be pressure from ourselves and others to allow custom components. I think it is reasonable to hold a hard line now and now go that way. Is easy to reconsider later, but once they are allowed,...you know how that goes...

…

On Sun, Oct 8, 2017 at 2:07 PM, Kyle Kelley ***@***.***> wrote: Yeah I made transforms for text/plain, text/html, and application/vdom.v1+json (which is a no-op). Open to the others. It's kind of funny though, they mostly mirror the mimetype handling on the frontend (even the image/* ones will). Makes me kind of wish for either custom elements (web components) or a way to use a "namespace" of react components (note: I've been down the path before with the namespacing and was not pleased with the security ramifications if dynamically loaded). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AABr0FmpryjYVbCzRrosKs-nBVyURtnaks5sqTmggaJpZM4PoP32> .

-- Brian E. Granger Associate Professor of Physics and Data Science Cal Poly State University, San Luis Obispo @ellisonbg on Twitter and GitHub [email protected] and [email protected]

[rgbkrk]

I know there will be pressure from ourselves and others to allow custom components. I think it is reasonable to hold a hard line now and now go that way. Is easy to reconsider later, but once they are allowed,...you know how that goes...

Yeah, I'd like to see the simple basics get integrated first. We evangelize how to work with vdom, show people how to create their own components, and how to do display updates with them.

[willingc]

Interesting discussion on gitter ipywidgets re: repr_html

From Matt Rocklin:

OK, so I have a semi-related question. Sometimes the data that I want to render has a _repr_html_ method defined. Sometimes it doesn't. In both cases I would like to render this thing with an HTML widget and get the same result as if the user had just rendered the value normally in the notebook.

[rgbkrk]

Something like this:

Or were you thinking of a way to do it with display updates?