[RTM] New plotting! #887
[RTM] New plotting! #887
Conversation
There was a problem hiding this comment.
Great !
Reviewed 14 of 14 files at r1, all commit messages.
Reviewable status: all files reviewed, 37 unresolved discussions (waiting on @pariterre)
README.md line 794 at r1 (raw file):
One can refer to their respective solver's documentation to know which options exist. The `show_online_optim` parameter can be set to `True` so the graphs nicely update during the optimization. Please note that `ShowOnlineType.MULTIPROCESS` is not available on Windows. To see how to run the server on Windows, please refer to the `getting_started/pendulum.py` example.
ShowOnlineType.MULTITHREAD ?
README.md line 798 at r1 (raw file):
`show_options` can be also passed as a dict to the plotter to customize the plotter's behavior. The following keys are special options: - `type`: the type of plotter to use (default is `ShowOnlineType.MULTIPROCESS`)
idem
Code quote:
ShowOnlineType.MULTIPROCESSREADME.md line 1692 at r1 (raw file):
The accepted values are: MULTIPROCESS: The online plotter is in a separate process.
MULTITHREAD
bioptim/examples/getting_started/pendulum.py line 155 at r1 (raw file):
# --- Solve the ocp --- # if platform.system() == "Windows": # The default online type (ShowOnlineType.MULTIPROCESS) is not available on Windows
MULTITHREAD
bioptim/examples/getting_started/pendulum.py line 157 at r1 (raw file):
# The default online type (ShowOnlineType.MULTIPROCESS) is not available on Windows # Since `as_multiprocess` default value is True, it will automatically starts the server in the background solver = Solver.IPOPT(show_online_optim=True, show_options={"type": ShowOnlineType.SERVER})
OnlineOptim.None # False
OnlineOptim.MULTIPROCESS
OnlineOptim.DEFAULT # True, specific for each Plateform !
OnlineOptim.MULTIPROCESS_SERVER
OnlineOptim.SERVER
bioptim/gui/online_callback_multiprocess.py line 60 at r1 (raw file):
The multiprocessing queue to evaluate plot: PlotOcp The handler on all the figures
you can update
Code quote:
pipe: mp.Queue
The multiprocessing queue to evaluate
plot: PlotOcp
The handler on all the figuresbioptim/gui/online_callback_multiprocess.py line 118 at r1 (raw file):
for fig in self._plotter.all_figures: fig.canvas.draw() if [plt.fignum_exists(fig.number) for fig in self._plotter.all_figures].count(True) > 0:
has_at_least_one_active_figure =
Code quote:
[plt.fignum_exists(fig.number) for fig in self._plotter.all_figures].count(True)bioptim/gui/online_callback_server.py line 18 at r1 (raw file):
_default_host = "localhost"
_DEFAULT_HOST + _DEFAULT_PORT
bioptim/gui/online_callback_server.py line 42 at r1 (raw file):
class _ServerMessages(Enum):
EnumInt ?
bioptim/gui/online_callback_server.py line 95 at r1 (raw file):
@staticmethod def as_multiprocess(*args, **kwargs) -> None:
A class would abstract better the behavior. -> "PlottingMultiProcessServer"
bioptim/gui/online_callback_server.py line 105 at r1 (raw file):
from multiprocessing import Process thread = Process(target=_start_as_multiprocess_internal, args=args, kwargs=kwargs)
process = ...
bioptim/gui/online_callback_server.py line 110 at r1 (raw file):
def _run(self) -> None: """ Starts the server, this method can be called directly by the user to start a plot server
update doc
bioptim/gui/online_callback_server.py line 154 at r1 (raw file):
self._logger.warning("Client closed connexion") client_socket.close() return _ServerMessages.CLOSE_CONNEXION, None
get_data_length()
Code quote:
# Receive the actual data
try:
self._logger.debug("Waiting for data from client")
data = client_socket.recv(1024)
if not data:
return _ServerMessages.EMPTY, None
except:
self._logger.warning("Client closed connexion")
client_socket.close()
return _ServerMessages.CLOSE_CONNEXION, Nonebioptim/gui/online_callback_server.py line 168 at r1 (raw file):
data_out.append(client_socket.recv(len_data)) if len(data_out[-1]) != len_data: data_out[-1] += client_socket.recv(len_data - len(data_out[-1]))
Can go with the function above
Code quote:
if len(data_out[-1]) != len_data:
data_out[-1] += client_socket.recv(len_data - len(data_out[-1]))bioptim/gui/online_callback_server.py line 256 at r1 (raw file):
fig.canvas.draw() if [plt.fignum_exists(fig.number) for fig in self._plotter.all_figures].count(True) > 0:
has_at_least_one_active_plot =
Code quote:
[plt.fignum_exists(fig.number) for fig in self._plotter.all_figures].count(True) > 0bioptim/gui/online_callback_server.py line 291 at r1 (raw file):
return elif message_type == _ServerMessages.EMPTY or message_type == _ServerMessages.CLOSE_CONNEXION:
message_type in (Enum1, Enum2)
bioptim/gui/online_callback_server.py line 298 at r1 (raw file):
timer_get_data.start() def _update_data(self, data_raw: list) -> None:
jsonlike_data ?
bioptim/gui/online_callback_server.py line 305 at r1 (raw file):
---------- data_raw: list The raw data from the client
see Also ?
bioptim/gui/online_callback_server.py line 435 at r1 (raw file):
raise RuntimeError("The server did not acknowledge the connexion") # TODO ADD SHOW OPTIONS to the send
Done
bioptim/gui/online_callback_server.py line 458 at r1 (raw file):
self._socket.close() def eval(self, arg: list | tuple, force: bool = False) -> list:
Enforce, blocking ?
bioptim/gui/online_callback_server.py line 513 at r1 (raw file):
header += f",{y_steps.shape[0]}" y_steps_tp = y_steps.tolist() data_serialized += struct.pack("d" * len(y_steps_tp), *y_steps_tp)
Put it into a function called "xydata_encoder" with "xydata_decoder" up there ideal for testing !
Code quote:
args_dict = {}
for i, s in enumerate(nlpsol_out()):
args_dict[s] = arg[i]
xdata_raw, ydata_raw = self._plotter.parse_data(**args_dict)
header = f"{len(xdata_raw)}"
data_serialized = b""
for x_nodes in xdata_raw:
header += f",{len(x_nodes)}"
for x_steps in x_nodes:
header += f",{x_steps.shape[0]}"
x_steps_tp = np.array(x_steps)[:, 0].tolist()
data_serialized += struct.pack("d" * len(x_steps_tp), *x_steps_tp)
header += f",{len(ydata_raw)}"
for y_nodes_variable in ydata_raw:
if isinstance(y_nodes_variable, np.ndarray):
header += f",0"
y_nodes_variable = [y_nodes_variable]
else:
header += f",{len(y_nodes_variable)}"
for y_steps in y_nodes_variable:
header += f",{y_steps.shape[0]}"
y_steps_tp = y_steps.tolist()
data_serialized += struct.pack("d" * len(y_steps_tp), *y_steps_tp)bioptim/gui/plot.py line 294 at r1 (raw file):
fig.canvas.draw() if self.plot_options["general_options"]["use_tight_layout"]: fig.tight_layout()
def spread_the_figures_on_screen(...)
Code quote:
horz = 0
vert = 1 if len(self.all_figures) < self.n_vertical_windows * self.n_horizontal_windows else 0
for i, fig in enumerate(self.all_figures):
if self.automatically_organize:
try:
fig.canvas.manager.window.move(
int(vert * self.width_step), int(self.top_margin + horz * self.height_step)
)
vert += 1
if vert >= self.n_vertical_windows:
horz += 1
vert = 0
except AttributeError:
pass
fig.canvas.draw()
if self.plot_options["general_options"]["use_tight_layout"]:
fig.tight_layout()bioptim/gui/plot.py line 455 at r1 (raw file):
if not y_min_all[y_range_var_idx]: y_min_all[y_range_var_idx] = [np.inf] * nb_subplots y_max_all[y_range_var_idx] = [-np.inf] * nb_subplots
def initializes_all_axes(...)
Code quote:
if nlp.plot[variable].combine_to:
self.axes[variable] = self.axes[nlp.plot[variable].combine_to]
axes = self.axes[variable][1]
elif i > 0 and variable in self.axes:
axes = self.axes[variable][1]
else:
nb_subplots = max(
[
(
max(
len(nlp.plot[variable].phase_mappings.to_first.map_idx),
max(nlp.plot[variable].phase_mappings.to_first.map_idx) + 1,
)
if variable in nlp.plot
else 0
)
for nlp in self.ocp.nlp
]
)
# TODO: get rid of all_variables_in_one_subplot by fixing the mapping appropriately
if not nlp.plot[variable].all_variables_in_one_subplot:
n_cols, n_rows = PlotOcp._generate_windows_size(nb_subplots)
else:
n_cols = 1
n_rows = 1
axes = self._add_new_axis(variable, nb_subplots, n_rows, n_cols)
self.axes[variable] = [nlp.plot[variable], axes]
if not y_min_all[y_range_var_idx]:
y_min_all[y_range_var_idx] = [np.inf] * nb_subplots
y_max_all[y_range_var_idx] = [-np.inf] * nb_subplotsbioptim/gui/plot.py line 786 at r1 (raw file):
def update_data( self, xdata: dict,
list | np.ndarray
bioptim/gui/plot.py line 795 at r1 (raw file):
Parameters ---------- xdata: dict
list
bioptim/gui/serializable_class.py line 263 at r1 (raw file):
penalty = custom_plot.parameters[key] casadi_function = penalty.function[0] if penalty.function[0] is not None else penalty.function[-1]
emulated_casadi_function ?
bioptim/gui/serializable_class.py line 402 at r1 (raw file):
class OdeSolverSerializable:
todo: adjust base on if your plot is failing.
bioptim/gui/serializable_class.py line 417 at r1 (raw file):
return cls( polynomial_degree=5,
true value here ?
bioptim/interfaces/interface_utils.py line 27 at r1 (raw file):
show_options: dict The options to pass to PlotOcp, special options are: - type: ShowOnlineType.MULTIPROCESS or ShowOnlineType.SERVER
fix names
bioptim/interfaces/interface_utils.py line 30 at r1 (raw file):
- host: The host to connect to (only for ShowOnlineType.SERVER) - port: The port to connect to (only for ShowOnlineType.SERVER) - as_multiprocess: If the server should run as a multiprocess (only for ShowOnlineType.SERVER), if True,
as_multiprocess should be gone
bioptim/interfaces/interface_utils.py line 36 at r1 (raw file):
show_options = {} show_type = ShowOnlineType.MULTIPROCESS
idem
bioptim/interfaces/interface_utils.py line 42 at r1 (raw file):
if show_type == ShowOnlineType.MULTIPROCESS: if platform == "win32":
if not linux
bioptim/misc/enums.py line 109 at r1 (raw file):
MULTIPROCESS = 0 SERVER = 1
should have the all Enums
bioptim/optimization/optimization_vector.py line 359 at r1 (raw file):
A reference to the ocp data: np.ndarray | DM The solution in a vector, if no data is provided, dummy data is used (it can be useful getting the dimensions)
not true anymore
bioptim/gui/online_callback_abstract.py line 66 at r1 (raw file):
# There must be an option to add an if here from ..interfaces.ipopt_interface import IpoptInterface
Let's try once if it works outside.
bioptim/gui/online_callback_abstract.py line 158 at r1 (raw file):
@abstractmethod def eval(self, arg: list | tuple, force: bool = False) -> list:
list of int ?
bioptim/gui/online_callback_abstract.py line 166 at r1 (raw file):
arg: list | tuple The data to send
Add force in the doc string
There was a problem hiding this comment.
Reviewable status: all files reviewed, 37 unresolved discussions (waiting on @pariterre)
bioptim/gui/online_callback_server.py line 513 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Put it into a function called "xydata_encoder" with "xydata_decoder" up there ideal for testing !
Add a test for encoding decoding part.
There was a problem hiding this comment.
Reviewable status: all files reviewed, 37 unresolved discussions (waiting on @Ipuch)
README.md line 794 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
ShowOnlineType.MULTITHREAD ?
Done.
README.md line 798 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
idem
Done.
README.md line 1692 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
MULTITHREAD
Done.
bioptim/examples/getting_started/pendulum.py line 155 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
MULTITHREAD
Done.
bioptim/examples/getting_started/pendulum.py line 157 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
OnlineOptim.None # False
OnlineOptim.MULTIPROCESS
OnlineOptim.DEFAULT # True, specific for each Plateform !
OnlineOptim.MULTIPROCESS_SERVER
OnlineOptim.SERVER
Done.
bioptim/gui/online_callback_multiprocess.py line 60 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
you can update
Done.
bioptim/gui/online_callback_multiprocess.py line 118 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
has_at_least_one_active_figure =
Done.
bioptim/gui/online_callback_server.py line 18 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
_DEFAULT_HOST + _DEFAULT_PORT
Done.
bioptim/gui/online_callback_server.py line 42 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
EnumInt ?
Done.
bioptim/gui/online_callback_server.py line 95 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
A class would abstract better the behavior. -> "PlottingMultiProcessServer"
Done.
bioptim/gui/online_callback_server.py line 105 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
process = ...
Done.
bioptim/gui/online_callback_server.py line 110 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
update doc
Done.
bioptim/gui/online_callback_server.py line 256 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
has_at_least_one_active_plot =
Done.
bioptim/gui/online_callback_server.py line 291 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
message_type in (Enum1, Enum2)
Done.
bioptim/gui/online_callback_server.py line 298 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
jsonlike_data ?
serialized_raw_data
bioptim/gui/online_callback_server.py line 305 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
see Also ?
Done.
bioptim/gui/online_callback_server.py line 435 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Done
Done.
bioptim/gui/online_callback_server.py line 458 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Enforce, blocking ?
Done.
bioptim/gui/plot.py line 294 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
def spread_the_figures_on_screen(...)
Done.
bioptim/gui/plot.py line 786 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
list | np.ndarray
Done.
bioptim/gui/plot.py line 795 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
list
Done.
bioptim/gui/serializable_class.py line 263 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
emulated_casadi_function ?
My bad, at that point it is the actual casadi_function
bioptim/gui/serializable_class.py line 402 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
todo: adjust base on if your plot is failing.
Done.
bioptim/gui/serializable_class.py line 417 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
true value here ?
Done.
bioptim/interfaces/interface_utils.py line 27 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
fix names
Done.
bioptim/interfaces/interface_utils.py line 30 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
as_multiprocess should be gone
Done.
bioptim/interfaces/interface_utils.py line 36 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
idem
Done.
bioptim/interfaces/interface_utils.py line 42 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
if not linux
Done.
bioptim/misc/enums.py line 109 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
should have the all Enums
Done.
bioptim/optimization/optimization_vector.py line 359 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
not true anymore
Done.
bioptim/gui/online_callback_abstract.py line 66 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Let's try once if it works outside.
Nope
bioptim/gui/online_callback_abstract.py line 158 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
list of int ?
Done.
bioptim/gui/online_callback_abstract.py line 166 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Add force in the doc string
Done.
There was a problem hiding this comment.
Reviewable status: 1 of 30 files reviewed, 37 unresolved discussions (waiting on @Ipuch)
bioptim/gui/online_callback_server.py line 154 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
get_data_length()
Separated in two subfunctions
bioptim/gui/online_callback_server.py line 168 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Can go with the function above
Done.
bioptim/gui/online_callback_server.py line 513 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
Add a test for encoding decoding part.
Done.
bioptim/gui/plot.py line 455 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
def initializes_all_axes(...)
This part actually defines 4 different variables needed later. I feel unconfortable moving this part without proper testing
|
@Ipuch Ready to merge B-) |
There was a problem hiding this comment.
Reviewed 20 of 28 files at r2, 11 of 11 files at r3, all commit messages.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @pariterre)
bioptim/gui/online_callback_server.py line 435 at r1 (raw file):
Previously, pariterre (Pariterre) wrote…
Done.
The comment wasn't supposed to be gone ?
There was a problem hiding this comment.
Reviewable status: all files reviewed, 2 unresolved discussions (waiting on @Ipuch)
bioptim/gui/online_callback_server.py line 435 at r1 (raw file):
Previously, Ipuch (Pierre Puchaud) wrote…
The comment wasn't supposed to be gone ?
Done
There was a problem hiding this comment.
Reviewed 1 of 1 files at r4, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @pariterre)
New way of plotting using a server B-)
This change is