Human Path Prediction

This repository contains the code for the papers:

It is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
Karttikeya Mangalam, Harshayu Girase, Shreyas Agarwal, Kuan-Hui Lee, Ehsan Adeli, Jitendra Malik, Adrien Gaidon
Accepted at ECCV 2020 (Oral)

From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting
Karttikeya Mangalam*, Yang An*, Harshayu Girase, Jitendra Malik
Accepted to ICCV 2021

This repository supports several state of the art pedestrian trajectory forecasting models on both short term (3.2 seconds input, 4.8 seconds ouput) and long term (upto a minute in future) prediction horizons. To train/test models, please visit the PECNet and Ynet folders for model-specific code.

Keywords: human path prediction, human trajectory prediction, human path forecasting, pedestrian location forecasting, location prediction, position forecasting, future path forecasting, long term prediction, instantaneous prediction, next second location, multi-agent forecasting, behavior prediction

Datasets

Stanford Drone Dataset:

- Both Short and Long Term prediction horizon dataloaders
- Hand Annotated Segmentation Maps 
- State of the art prediction trained prediction models as well as training/evaluation code 

ETH/UCY Dataset:

- Short term prediction dataloaders
- SOTA trajectory prediction trained prediction models for all five scenes 
- Training/evaluation code for SOTA model and the baselines

InD Dataset:

- Long term prediction dataloaders
- Hand Annotated Segmentation Maps 
- SOTA trajectory prediction trained prediction models as well as training/evaluation code 

We hope this allows easy benchmarking of several baselines as well as state-of-the-art path prediction models across several datasets and settings. If you find this repository or any code thereof useful in your work, kindly cite:

@inproceedings{mangalam2020pecnet,
  title={It is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction},
  author={Mangalam, Karttikeya and Girase, Harshayu and Agarwal, Shreyas and Lee, Kuan-Hui and Adeli, Ehsan and Malik, Jitendra and Gaidon, Adrien},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  month = {August},
  year={2020}
}

@inproceedings{mangalam2021goals,
 author = {Mangalam, Karttikeya and An, Yang and Girase, Harshayu and Malik, Jitendra},
 title = {From Goals, Waypoints \& Paths To Long Term Human Trajectory Forecasting},
 booktitle = {Proc. International Conference on Computer Vision (ICCV)},
 year = {2021},
 month = oct,
 month_numeric = {10}
}

Paper Summaries

It is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction
Published at ECCV 2020 (Oral)

Abstract: Human trajectory forecasting with multiple socially interacting agents is of critical importance for autonomous navigation in human environments, e.g., for self-driving cars and social robots. In this work, we present Predicted Endpoint Conditioned Network (PECNet) for flexible human trajectory prediction. PECNet infers distant trajectory endpoints to assist in long-range multi-modal trajectory prediction. A novel nonlocal social pooling layer enables PECNet to infer diverse yet socially compliant trajectories. Additionally, we present a simple “truncation trick” for improving diversity and multi-modal trajectory prediction performance.

Below is an example of pedestrian trajectories predicted by our model and the corresponding ground truth. Left pane shows future trajectories for 9.6 seconds predicted in a recurrent input fashion. Right pane shows the predicted trajectories for future 4.8 seconds at an intersection. Solid circles represent the past input & stars represent the future ground truth. Predicted multi-modal trajectories are shown as translucent circles jointly for all present pedestrians. Animation is best viewed in Adobe Acrobat Reader. More video visualizations available at project homepage: https://karttikeya.github.io/publication/htf/

From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting
Karttikeya Mangalam*, Yang An*, Harshayu Girase, Jitendra Malik
Accepted to ICCV 2021

Abstract: Human trajectory forecasting is an inherently multimodal problem. Uncertainty in future trajectories stems from two sources: (a) sources that are known to the agent but unknown to the model, such as long term goals and (b) sources that are unknown to both the agent & the model, such as intent of other agents & irreducible randomness in decisions. We propose to factorize this uncertainty into its epistemic & aleatoric sources. We model the epistemic uncertainty through multimodality in long term goals and the aleatoric uncertainty through multimodality in waypoints & paths. To exemplify this dichotomy, we also propose a novel long term trajectory forecasting setting, with prediction horizons upto a minute, upto an order of magnitude longer than prior works. Finally, we present Y-net, a scene compliant trajectory forecasting network that exploits the proposed epistemic & aleatoric structure for diverse trajectory predictions across long prediction horizons. Y-net significantly improves previous state-of-the-art performance on both (a) The short prediction horizon setting on the Stanford Drone (31.7% in FDE) & ETH/UCY datasets (7.4% in FDE) and (b) The proposed long horizon setting on the re-purposed Stanford Drone & Intersection Drone datasets.

Below is a GIF visualization demonstrating the goal, waypoint and path multimodality for long term human trajectory prediction (30 seconds horizon). Given the past 5 seconds input history (green), we predict diverse future trajectories (current location in orange, past in red).