Optical Flow Data Missing

[wyddmw]

Hi, thanks for releasing this awesome work! I would like to know whether the optical flow data is missed in some scenarios. I unzipped the file and there is no optical flow data in the human_pose/PD_trex_Point_Cache_Final_2_17_2023/* folders.

[basilevh]

Yes, the published dataset does not contain all modalities because the total size would be ~17 TB otherwise, instead of 2.4 TB. I am generating the archive with the additional / missing ones right now and will post the download link when it is ready!

[basilevh]

Please see the update here:
https://gcd.cs.columbia.edu/#datasets
I clarified the description as follows:
The basic modalities are: RGB, depth, semantic segmentation, instance segmentation, and 2D bounding boxes. The additional modalities are: LiDAR point clouds, optical flow, scene flow, and surface normals. Direct download links can be found here:
The link for what you are looking for is under "Full dataset with additional modalities" (should be merged with the original extracted dataset; 3.2 TB archive). Hope this helps!

[wyddmw]

Please see the update here: https://gcd.cs.columbia.edu/#datasets I clarified the description as follows: The basic modalities are: RGB, depth, semantic segmentation, instance segmentation, and 2D bounding boxes. The additional modalities are: LiDAR point clouds, optical flow, scene flow, and surface normals. Direct download links can be found here: The link for what you are looking for is under "Full dataset with additional modalities" (should be merged with the original extracted dataset; 3.2 TB archive). Hope this helps!

Thanks for releasing the additional data! May I ask the format of optical flow or how am I supposed to load a .png optical flow.

[basilevh]

As far as I know from playing around with it myself, when you load an optical flow PNG frame as RGBA, the channels have the following meaning:

        # frame = (H, W, 4) array of float32 in [0, 1].
        dx = frame[..., 0] + frame[..., 1] * 256.0 - 128.0  # (H, W) array of float32 in [-128, 128].
        dy = frame[..., 2] + frame[..., 3] * 256.0 - 128.0  # (H, W) array of float32 in [-128, 128].
        angle = np.arctan2(dy, dx)  # (H, W) array of float32 in [-pi, pi].
        mag = np.sqrt(dx ** 2 + dy ** 2)  # (H, W) array of float32 in [0, 256].

The first two channels encode horizontal motion and the last two channels encode vertical motion. Unfortunately I do not know what the unit (i.e. normalization factor) is supposed to be, but please feel free to let us know here if anyone figures it out. Hope this helps!

[wyddmw]

As far as I know from playing around with it myself, when you load an optical flow PNG frame as RGBA, the channels have the following meaning:

        # frame = (H, W, 4) array of float32 in [0, 1].
        dx = frame[..., 0] + frame[..., 1] * 256.0 - 128.0  # (H, W) array of float32 in [-128, 128].
        dy = frame[..., 2] + frame[..., 3] * 256.0 - 128.0  # (H, W) array of float32 in [-128, 128].
        angle = np.arctan2(dy, dx)  # (H, W) array of float32 in [-pi, pi].
        mag = np.sqrt(dx ** 2 + dy ** 2)  # (H, W) array of float32 in [0, 256].

The first two channels encode horizontal motion and the last two channels encode vertical motion. Unfortunately I do not know what the unit (i.e. normalization factor) is supposed to be, but please feel free to let us know here if anyone figures it out. Hope this helps!

Thanks for your reply! I will give it a try following your instruction and see if I can figure it out.