Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Possible bug with ipywidgets or gmaps, not sure #3448

Open
ghost opened this issue May 1, 2022 · 4 comments · May be fixed by #3699
Open

Possible bug with ipywidgets or gmaps, not sure #3448

ghost opened this issue May 1, 2022 · 4 comments · May be fixed by #3699

Comments

@ghost
Copy link

ghost commented May 1, 2022 •
edited by ghost
Loading

During the generation of multiple heatmaps at the same location, I noticed the generated html files keep increasing in size without an equivalent increase in the volume at a given map. Here are the resulting file sizes when I generate the same map with the same data for 200 times:

import os
import shutil
import tempfile
from pathlib import Path

import gmaps
import numpy as np
from ipywidgets.embed import embed_minimal_html

tmp = tempfile.mkdtemp()
coordinates = np.random.randint(10, 50, (50, 2)).astype('float64')
for i in range(200):
    center = 20, 30
    fig = gmaps.figure(center=center, zoom_level=10)
    heatmap_layer = gmaps.heatmap_layer(coordinates)
    heatmap_layer.max_intensity = 100
    heatmap_layer.point_radius = 5
    fig.add_layer(heatmap_layer)
    p = Path(tmp) / f'{i}.html'
    embed_minimal_html(p.as_posix(), views=[fig])
    print(f'Current: {p.name}: {os.path.getsize(p.as_posix()) / 1024 ** 2} MB')
shutil.rmtree(tmp)

Result:

Current: 0.html: 0.01 MB
Current: 1.html: 0.01 MB
Current: 2.html: 0.02 MB
Current: 3.html: 0.02 MB
Current: 4.html: 0.03 MB
Current: 5.html: 0.04 MB
Current: 6.html: 0.04 MB
Current: 7.html: 0.05 MB
Current: 8.html: 0.05 MB
Current: 9.html: 0.06 MB
Current: 10.html: 0.07 MB
Current: 11.html: 0.07 MB
Current: 12.html: 0.08 MB
Current: 13.html: 0.08 MB
Current: 14.html: 0.09 MB
Current: 15.html: 0.1 MB
Current: 16.html: 0.1 MB
Current: 17.html: 0.11 MB
Current: 18.html: 0.11 MB
Current: 19.html: 0.12 MB
Current: 20.html: 0.13 MB
Current: 21.html: 0.13 MB
Current: 22.html: 0.14 MB
Current: 23.html: 0.14 MB
Current: 24.html: 0.15 MB
Current: 25.html: 0.16 MB
Current: 26.html: 0.16 MB
Current: 27.html: 0.17 MB
Current: 28.html: 0.17 MB
Current: 29.html: 0.18 MB
Current: 30.html: 0.19 MB
Current: 31.html: 0.19 MB
Current: 32.html: 0.2 MB
Current: 33.html: 0.2 MB
Current: 34.html: 0.21 MB
Current: 35.html: 0.22 MB
Current: 36.html: 0.22 MB
Current: 37.html: 0.23 MB
Current: 38.html: 0.23 MB
Current: 39.html: 0.24 MB
Current: 40.html: 0.25 MB
Current: 41.html: 0.25 MB
Current: 42.html: 0.26 MB
Current: 43.html: 0.26 MB
Current: 44.html: 0.27 MB
Current: 45.html: 0.28 MB
Current: 46.html: 0.28 MB
Current: 47.html: 0.29 MB
Current: 48.html: 0.29 MB
Current: 49.html: 0.3 MB
Current: 50.html: 0.31 MB
Current: 51.html: 0.31 MB
Current: 52.html: 0.32 MB
Current: 53.html: 0.32 MB
Current: 54.html: 0.33 MB
Current: 55.html: 0.34 MB
Current: 56.html: 0.34 MB
Current: 57.html: 0.35 MB
Current: 58.html: 0.35 MB
Current: 59.html: 0.36 MB
Current: 60.html: 0.37 MB
Current: 61.html: 0.37 MB
Current: 62.html: 0.38 MB
Current: 63.html: 0.38 MB
Current: 64.html: 0.39 MB
Current: 65.html: 0.39 MB
Current: 66.html: 0.4 MB
Current: 67.html: 0.41 MB
Current: 68.html: 0.41 MB
Current: 69.html: 0.42 MB
Current: 70.html: 0.42 MB
Current: 71.html: 0.43 MB
Current: 72.html: 0.44 MB
Current: 73.html: 0.44 MB
Current: 74.html: 0.45 MB
Current: 75.html: 0.45 MB
Current: 76.html: 0.46 MB
Current: 77.html: 0.47 MB
Current: 78.html: 0.47 MB
Current: 79.html: 0.48 MB
Current: 80.html: 0.48 MB
Current: 81.html: 0.49 MB
Current: 82.html: 0.5 MB
Current: 83.html: 0.5 MB
Current: 84.html: 0.51 MB
Current: 85.html: 0.51 MB
Current: 86.html: 0.52 MB
Current: 87.html: 0.53 MB
Current: 88.html: 0.53 MB
Current: 89.html: 0.54 MB
Current: 90.html: 0.54 MB
Current: 91.html: 0.55 MB
Current: 92.html: 0.56 MB
Current: 93.html: 0.56 MB
Current: 94.html: 0.57 MB
Current: 95.html: 0.57 MB
Current: 96.html: 0.58 MB
Current: 97.html: 0.59 MB
Current: 98.html: 0.59 MB
Current: 99.html: 0.6 MB
Current: 100.html: 0.6 MB
Current: 101.html: 0.61 MB
Current: 102.html: 0.62 MB
Current: 103.html: 0.62 MB
Current: 104.html: 0.63 MB
Current: 105.html: 0.63 MB
Current: 106.html: 0.64 MB
Current: 107.html: 0.65 MB
Current: 108.html: 0.65 MB
Current: 109.html: 0.66 MB
Current: 110.html: 0.66 MB
Current: 111.html: 0.67 MB
Current: 112.html: 0.68 MB
Current: 113.html: 0.68 MB
Current: 114.html: 0.69 MB
Current: 115.html: 0.69 MB
Current: 116.html: 0.7 MB
Current: 117.html: 0.71 MB
Current: 118.html: 0.71 MB
Current: 119.html: 0.72 MB
Current: 120.html: 0.72 MB
Current: 121.html: 0.73 MB
Current: 122.html: 0.74 MB
Current: 123.html: 0.74 MB
Current: 124.html: 0.75 MB
Current: 125.html: 0.75 MB
Current: 126.html: 0.76 MB
Current: 127.html: 0.77 MB
Current: 128.html: 0.77 MB
Current: 129.html: 0.78 MB
Current: 130.html: 0.78 MB
Current: 131.html: 0.79 MB
Current: 132.html: 0.8 MB
Current: 133.html: 0.8 MB
Current: 134.html: 0.81 MB
Current: 135.html: 0.81 MB
Current: 136.html: 0.82 MB
Current: 137.html: 0.82 MB
Current: 138.html: 0.83 MB
Current: 139.html: 0.84 MB
Current: 140.html: 0.84 MB
Current: 141.html: 0.85 MB
Current: 142.html: 0.85 MB
Current: 143.html: 0.86 MB
Current: 144.html: 0.87 MB
Current: 145.html: 0.87 MB
Current: 146.html: 0.88 MB
Current: 147.html: 0.88 MB
Current: 148.html: 0.89 MB
Current: 149.html: 0.9 MB
Current: 150.html: 0.9 MB
Current: 151.html: 0.91 MB
Current: 152.html: 0.91 MB
Current: 153.html: 0.92 MB
Current: 154.html: 0.93 MB
Current: 155.html: 0.93 MB
Current: 156.html: 0.94 MB
Current: 157.html: 0.94 MB
Current: 158.html: 0.95 MB
Current: 159.html: 0.96 MB
Current: 160.html: 0.96 MB
Current: 161.html: 0.97 MB
Current: 162.html: 0.97 MB
Current: 163.html: 0.98 MB
Current: 164.html: 0.99 MB
Current: 165.html: 0.99 MB
Current: 166.html: 1.0 MB
Current: 167.html: 1.0 MB
Current: 168.html: 1.01 MB
Current: 169.html: 1.02 MB
Current: 170.html: 1.02 MB
Current: 171.html: 1.03 MB
Current: 172.html: 1.03 MB
Current: 173.html: 1.04 MB
Current: 174.html: 1.05 MB
Current: 175.html: 1.05 MB
Current: 176.html: 1.06 MB
Current: 177.html: 1.06 MB
Current: 178.html: 1.07 MB
Current: 179.html: 1.08 MB
Current: 180.html: 1.08 MB
Current: 181.html: 1.09 MB
Current: 182.html: 1.09 MB
Current: 183.html: 1.1 MB
Current: 184.html: 1.11 MB
Current: 185.html: 1.11 MB
Current: 186.html: 1.12 MB
Current: 187.html: 1.12 MB
Current: 188.html: 1.13 MB
Current: 189.html: 1.14 MB
Current: 190.html: 1.14 MB
Current: 191.html: 1.15 MB
Current: 192.html: 1.15 MB
Current: 193.html: 1.16 MB
Current: 194.html: 1.17 MB
Current: 195.html: 1.17 MB
Current: 196.html: 1.18 MB
Current: 197.html: 1.18 MB
Current: 198.html: 1.19 MB
Current: 199.html: 1.2 MB

A temporary fix to the issue is to add the following to the code:

from ipywidgets.widgets.widget import Widget

And to include this line in the loop:

Widget.widgets.clear()

As gmaps.figure for some reason uses this object and keeps storing additional figure data inside it disregarding whether they are still needed. If this behavior is unintentional, I can issue a pr with the fix if there is one.

for i in range(200):
    Widget.widgets.clear()
    center = 20, 30
    fig = gmaps.figure(center=center, zoom_level=10)
    heatmap_layer = gmaps.heatmap_layer(coordinates)
    heatmap_layer.max_intensity = 100
    heatmap_layer.point_radius = 5
    fig.add_layer(heatmap_layer)
    p = Path(tmp) / f'{i}.html'
    embed_minimal_html(p.as_posix(), views=[fig])
    print(f'Current: {p.name}: {os.path.getsize(p.as_posix()) / 1024 ** 2} MB')
shutil.rmtree(tmp)

Result:

Current: 0.html: 0.0067386627197265625 MB
Current: 1.html: 0.0067386627197265625 MB
Current: 2.html: 0.0067386627197265625 MB
Current: 3.html: 0.0067386627197265625 MB
Current: 4.html: 0.0067386627197265625 MB
Current: 5.html: 0.0067386627197265625 MB
Current: 6.html: 0.0067386627197265625 MB
Current: 7.html: 0.0067386627197265625 MB
Current: 8.html: 0.0067386627197265625 MB
Current: 9.html: 0.0067386627197265625 MB
Current: 10.html: 0.0067386627197265625 MB
Current: 11.html: 0.0067386627197265625 MB
Current: 12.html: 0.0067386627197265625 MB
Current: 13.html: 0.0067386627197265625 MB
Current: 14.html: 0.0067386627197265625 MB
Current: 15.html: 0.0067386627197265625 MB
Current: 16.html: 0.0067386627197265625 MB
Current: 17.html: 0.0067386627197265625 MB
Current: 18.html: 0.0067386627197265625 MB
Current: 19.html: 0.0067386627197265625 MB
Current: 20.html: 0.0067386627197265625 MB
Current: 21.html: 0.0067386627197265625 MB
Current: 22.html: 0.0067386627197265625 MB
Current: 23.html: 0.0067386627197265625 MB
Current: 24.html: 0.0067386627197265625 MB
Current: 25.html: 0.0067386627197265625 MB
Current: 26.html: 0.0067386627197265625 MB
Current: 27.html: 0.0067386627197265625 MB
Current: 28.html: 0.0067386627197265625 MB
Current: 29.html: 0.0067386627197265625 MB
Current: 30.html: 0.0067386627197265625 MB
Current: 31.html: 0.0067386627197265625 MB
Current: 32.html: 0.0067386627197265625 MB
Current: 33.html: 0.0067386627197265625 MB
Current: 34.html: 0.0067386627197265625 MB
Current: 35.html: 0.0067386627197265625 MB
Current: 36.html: 0.0067386627197265625 MB
Current: 37.html: 0.0067386627197265625 MB
Current: 38.html: 0.0067386627197265625 MB
Current: 39.html: 0.0067386627197265625 MB
Current: 40.html: 0.0067386627197265625 MB
Current: 41.html: 0.0067386627197265625 MB
Current: 42.html: 0.0067386627197265625 MB
Current: 43.html: 0.0067386627197265625 MB
Current: 44.html: 0.0067386627197265625 MB
Current: 45.html: 0.0067386627197265625 MB
Current: 46.html: 0.0067386627197265625 MB
Current: 47.html: 0.0067386627197265625 MB
Current: 48.html: 0.0067386627197265625 MB
Current: 49.html: 0.0067386627197265625 MB
Current: 50.html: 0.0067386627197265625 MB
Current: 51.html: 0.0067386627197265625 MB
Current: 52.html: 0.0067386627197265625 MB
Current: 53.html: 0.0067386627197265625 MB
Current: 54.html: 0.0067386627197265625 MB
Current: 55.html: 0.0067386627197265625 MB
Current: 56.html: 0.0067386627197265625 MB
Current: 57.html: 0.0067386627197265625 MB
Current: 58.html: 0.0067386627197265625 MB
Current: 59.html: 0.0067386627197265625 MB
Current: 60.html: 0.0067386627197265625 MB
Current: 61.html: 0.0067386627197265625 MB
Current: 62.html: 0.0067386627197265625 MB
Current: 63.html: 0.0067386627197265625 MB
Current: 64.html: 0.0067386627197265625 MB
Current: 65.html: 0.0067386627197265625 MB
Current: 66.html: 0.0067386627197265625 MB
Current: 67.html: 0.0067386627197265625 MB
Current: 68.html: 0.0067386627197265625 MB
Current: 69.html: 0.0067386627197265625 MB
Current: 70.html: 0.0067386627197265625 MB
Current: 71.html: 0.0067386627197265625 MB
Current: 72.html: 0.0067386627197265625 MB
Current: 73.html: 0.0067386627197265625 MB
Current: 74.html: 0.0067386627197265625 MB
Current: 75.html: 0.0067386627197265625 MB
Current: 76.html: 0.0067386627197265625 MB
Current: 77.html: 0.0067386627197265625 MB
Current: 78.html: 0.0067386627197265625 MB
Current: 79.html: 0.0067386627197265625 MB
Current: 80.html: 0.0067386627197265625 MB
Current: 81.html: 0.0067386627197265625 MB
Current: 82.html: 0.0067386627197265625 MB
Current: 83.html: 0.0067386627197265625 MB
Current: 84.html: 0.0067386627197265625 MB
Current: 85.html: 0.0067386627197265625 MB
Current: 86.html: 0.0067386627197265625 MB
Current: 87.html: 0.0067386627197265625 MB
Current: 88.html: 0.0067386627197265625 MB
Current: 89.html: 0.0067386627197265625 MB
Current: 90.html: 0.0067386627197265625 MB
Current: 91.html: 0.0067386627197265625 MB
Current: 92.html: 0.0067386627197265625 MB
Current: 93.html: 0.0067386627197265625 MB
Current: 94.html: 0.0067386627197265625 MB
Current: 95.html: 0.0067386627197265625 MB
Current: 96.html: 0.0067386627197265625 MB
Current: 97.html: 0.0067386627197265625 MB
Current: 98.html: 0.0067386627197265625 MB
Current: 99.html: 0.0067386627197265625 MB
Current: 100.html: 0.0067386627197265625 MB
Current: 101.html: 0.0067386627197265625 MB
Current: 102.html: 0.0067386627197265625 MB
Current: 103.html: 0.0067386627197265625 MB
Current: 104.html: 0.0067386627197265625 MB
Current: 105.html: 0.0067386627197265625 MB
Current: 106.html: 0.0067386627197265625 MB
Current: 107.html: 0.0067386627197265625 MB
Current: 108.html: 0.0067386627197265625 MB
Current: 109.html: 0.0067386627197265625 MB
Current: 110.html: 0.0067386627197265625 MB
Current: 111.html: 0.0067386627197265625 MB
Current: 112.html: 0.0067386627197265625 MB
Current: 113.html: 0.0067386627197265625 MB
Current: 114.html: 0.0067386627197265625 MB
Current: 115.html: 0.0067386627197265625 MB
Current: 116.html: 0.0067386627197265625 MB
Current: 117.html: 0.0067386627197265625 MB
Current: 118.html: 0.0067386627197265625 MB
Current: 119.html: 0.0067386627197265625 MB
Current: 120.html: 0.0067386627197265625 MB
Current: 121.html: 0.0067386627197265625 MB
Current: 122.html: 0.0067386627197265625 MB
Current: 123.html: 0.0067386627197265625 MB
Current: 124.html: 0.0067386627197265625 MB
Current: 125.html: 0.0067386627197265625 MB
Current: 126.html: 0.0067386627197265625 MB
Current: 127.html: 0.0067386627197265625 MB
Current: 128.html: 0.0067386627197265625 MB
Current: 129.html: 0.0067386627197265625 MB
Current: 130.html: 0.0067386627197265625 MB
Current: 131.html: 0.0067386627197265625 MB
Current: 132.html: 0.0067386627197265625 MB
Current: 133.html: 0.0067386627197265625 MB
Current: 134.html: 0.0067386627197265625 MB
Current: 135.html: 0.0067386627197265625 MB
Current: 136.html: 0.0067386627197265625 MB
Current: 137.html: 0.0067386627197265625 MB
Current: 138.html: 0.0067386627197265625 MB
Current: 139.html: 0.0067386627197265625 MB
Current: 140.html: 0.0067386627197265625 MB
Current: 141.html: 0.0067386627197265625 MB
Current: 142.html: 0.0067386627197265625 MB
Current: 143.html: 0.0067386627197265625 MB
Current: 144.html: 0.0067386627197265625 MB
Current: 145.html: 0.0067386627197265625 MB
Current: 146.html: 0.0067386627197265625 MB
Current: 147.html: 0.0067386627197265625 MB
Current: 148.html: 0.0067386627197265625 MB
Current: 149.html: 0.0067386627197265625 MB
Current: 150.html: 0.0067386627197265625 MB
Current: 151.html: 0.0067386627197265625 MB
Current: 152.html: 0.0067386627197265625 MB
Current: 153.html: 0.0067386627197265625 MB
Current: 154.html: 0.0067386627197265625 MB
Current: 155.html: 0.0067386627197265625 MB
Current: 156.html: 0.0067386627197265625 MB
Current: 157.html: 0.0067386627197265625 MB
Current: 158.html: 0.0067386627197265625 MB
Current: 159.html: 0.0067386627197265625 MB
Current: 160.html: 0.0067386627197265625 MB
Current: 161.html: 0.0067386627197265625 MB
Current: 162.html: 0.0067386627197265625 MB
Current: 163.html: 0.0067386627197265625 MB
Current: 164.html: 0.0067386627197265625 MB
Current: 165.html: 0.0067386627197265625 MB
Current: 166.html: 0.0067386627197265625 MB
Current: 167.html: 0.0067386627197265625 MB
Current: 168.html: 0.0067386627197265625 MB
Current: 169.html: 0.0067386627197265625 MB
Current: 170.html: 0.0067386627197265625 MB
Current: 171.html: 0.0067386627197265625 MB
Current: 172.html: 0.0067386627197265625 MB
Current: 173.html: 0.0067386627197265625 MB
Current: 174.html: 0.0067386627197265625 MB
Current: 175.html: 0.0067386627197265625 MB
Current: 176.html: 0.0067386627197265625 MB
Current: 177.html: 0.0067386627197265625 MB
Current: 178.html: 0.0067386627197265625 MB
Current: 179.html: 0.0067386627197265625 MB
Current: 180.html: 0.0067386627197265625 MB
Current: 181.html: 0.0067386627197265625 MB
Current: 182.html: 0.0067386627197265625 MB
Current: 183.html: 0.0067386627197265625 MB
Current: 184.html: 0.0067386627197265625 MB
Current: 185.html: 0.0067386627197265625 MB
Current: 186.html: 0.0067386627197265625 MB
Current: 187.html: 0.0067386627197265625 MB
Current: 188.html: 0.0067386627197265625 MB
Current: 189.html: 0.0067386627197265625 MB
Current: 190.html: 0.0067386627197265625 MB
Current: 191.html: 0.0067386627197265625 MB
Current: 192.html: 0.0067386627197265625 MB
Current: 193.html: 0.0067386627197265625 MB
Current: 194.html: 0.0067386627197265625 MB
Current: 195.html: 0.0067386627197265625 MB
Current: 196.html: 0.0067386627197265625 MB
Current: 197.html: 0.0067386627197265625 MB
Current: 198.html: 0.0067386627197265625 MB
Current: 199.html: 0.0067386627197265625 MB
@filipre filipre mentioned this issue Feb 8, 2023
Open
@filipre
Copy link

filipre commented Feb 8, 2023 •
edited
Loading

The issue is with ipywidgets since I'm not using gmaps and still can reproduce this bug here jupyter-widgets/ipyleaflet#1098

The issue is that the states (widgets?) are copied and saved multiple times. Probably because new objects are generated with new model_ids. Take a look here:

ipywidgets/python/ipywidgets/ipywidgets/widgets/widget.py

Line 444 in 6f6156c

state[widget.model_id] = widget._get_embed_state(drop_defaults=drop_defaults)

For example, calling embed_minimal_html twice causes LeafletAttributionControlModel and all other widgets appearing twice in the state json, each with their own model_ids.

So, embed_minimal_html calls embed_snippet which calls embed_data where state is set to None by default. Because it is None, we run

ipywidgets/python/ipywidgets/ipywidgets/embed.py

Line 219 in 6f6156c

state = Widget.get_manager_state(drop_defaults=drop_defaults, widgets=None)['state']

Then, we fetch all widgets:

ipywidgets/python/ipywidgets/ipywidgets/widgets/widget.py

Line 442 in 6f6156c

widgets = _instances.values()

Something must create new widgets somewhere before or _instances.values() creates those duplicates. Maybe whenever comm is changed?

ipywidgets/python/ipywidgets/ipywidgets/widgets/widget.py

Line 531 in 6f6156c

def _comm_changed(self, change):

@filipre filipre mentioned this issue Feb 8, 2023
Open
@maartenbreddels
Copy link
Member

I'm slightly surprised by this. While it's true that we keep a dict of all widgets created (you need to manually close them, or use something like https://github.com/widgetti/reacton)
The embed code came from https://github.com/widgetti/ipyvolume which only stored in the HTML the models it needed. This led to

ipywidgets/python/ipywidgets/ipywidgets/embed.py

Line 138 in 7325e59

def dependency_state(widgets, drop_defaults=True):
which ipyvolume now uses.

However, the embed_minimal_html doesn't seem to use that.

The right fix would be for embed_minimal_html to use the dependency_state function.

@filipre filipre linked a pull request Feb 9, 2023 that will close this issue
Open
@filipre
Copy link

filipre commented Feb 9, 2023

Thank you for your insights! I created #3699 and it seems to solve the issue

@vidartf
Copy link
Member

vidartf commented Mar 8, 2023

The original intention was that including the full state was the safe assumption for ipywidgets to make. While dependency_state works in a high portion of the cases, it has some corner cases that we didn't want to expose users to by default (especially at the time when we wrote originally this, and wasn't sure if we knew about all the corner cases). Instead, we made it possible for users to call this as (in your case):

embed_minimal_html(p.as_posix(), views=[fig], state=dependency_state([fig]))

I think adding some enum-like values for the state flag as in your proposed PR ('complete', 'dependent') makes a lot of sense, but changing the default might be something to discuss more in depth, as it is strictly speaking a backwards incompatible change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add potential fix for increasing map size bug filipre/ipywidgets
3 participants
@vidartf @maartenbreddels @filipre