Skip to content

Commit

Permalink
update
Browse files Browse the repository at this point in the history
  • Loading branch information
JianxGao committed Mar 21, 2024
1 parent a56c1e7 commit 62bcede
Show file tree
Hide file tree
Showing 6 changed files with 60 additions and 19 deletions.
Binary file added .DS_Store
Binary file not shown.
Binary file modified imgs/MinD-3D.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added imgs/fmri_shape.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified imgs/teaser.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added imgs/visualization.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
79 changes: 60 additions & 19 deletions index.html
Original file line number Diff line number Diff line change
Expand Up @@ -167,42 +167,66 @@ <h1 class="title is-1 publication-title">MinD-3D: Reconstruct High-quality 3D ob
</section>


<!-- <section class="hero teaser">
<section class="section">
<div class="container is-max-desktop">
<div class="hero-body">
<center>
<img src="./imgs/1_final.gif" width="90%"/>
<img src="./imgs/7_final.gif" width="90%"/>
<img src="./imgs/209_final.gif" width="90%"/>
</center>
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<div class="content has-text-justified">
<p>
In this paper, we introduce <b>Recon3DMind</b>, an innovative task aimed at reconstructing 3D visuals from Functional Magnetic Resonance Imaging (fMRI) signals, marking a significant advancement in the fields of cognitive neuroscience and computer vision.
To support this pioneering task, we present the <b>fMRI-Shape</b> dataset, which includes data from 14 participants and features 360-degree videos of 3D objects to enable comprehensive fMRI signal capture across various settings,
thereby laying a foundation for future research. Furthermore, we propose <b>MinD-3D</b>, a novel and effective three-stage framework specifically designed to decode the brain's 3D visual information from fMRI signals,
demonstrating the feasibility of this challenging task. The framework begins by extracting and aggregating features from fMRI frames through a neuro-fusion encoder, subsequently employs a feature bridge diffusion model to generate visual features,
and ultimately recovers the 3D object via a generative transformer decoder.
We assess the performance of MinD-3D using a suite of semantic and structural metrics and analyze the correlation between the features extracted by our model and the visual regions of interest (ROIs) in fMRI signals.
Our findings indicate that MinD-3D not only reconstructs 3D objects with high semantic relevance and spatial similarity but also significantly enhances our understanding of the human brain's capabilities in processing 3D visual information.
</p>
</div>
</div>
</div>
</div>
</section> -->
</section>

<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Abstract</h2>
<h2 class="title is-3">fMRI-Shape</h2>
<div class="content has-text-justified">
<center>
<img src="./imgs/fmri_shape.png" width="100%"/>
</center>
<p>
Here are some examples of the fMRI-Shape dataset across different subjects. You can download the dataset by this link: https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape.
</p>
</div>
</div>
</div>
</div>
</section>

<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
<div class="column is-four-fifths">
<h2 class="title is-3">Reconstruction</h2>
<div class="content has-text-justified">
<center>
<img src="./imgs/visualization.png" width="100%"/>
</center>
<p>
In this paper, we introduce <b>Recon3DMind</b>, a groundbreaking task focused on reconstructing 3D visuals from Functional Magnetic Resonance Imaging (fMRI) signals.
This represents a major step forward in cognitive neuroscience and computer vision.
To support this task, we present the <b>fMRI-Shape</b> dataset, utilizing 360-degree view videos of 3D objects for comprehensive fMRI signal capture.
Containing 55 categories of common objects from daily life, this dataset will bolster future research endeavors.
We also propose <b>MinD-3D</b>, a novel and effective three-stage framework that decodes and reconstructs the brain's 3D visual information from fMRI signals.
This method starts by extracting and aggregating features from fMRI frames using a neuro-fusion encoder,
then employs a feature bridge diffusion model to generate corresponding visual features,
and ultimately recovers the 3D object through a generative transformer decoder.
Our experiments demonstrate that this method effectively extracts features that are valid and highly correlated with visual regions of interest (ROIs) in fMRI signals.
Notably, it not only reconstructs 3D objects with high semantic relevance and spatial similarity but also significantly deepens our understanding of the human brain's 3D visual processing capabilities.
The qualitative results generated by LEA-3D, fMRI-PTE-3D, and our method
are presented. GT indicates the ground-truth 3D objects. All the objects have been
rendered into a 2D format
</p>
</div>
</div>
</div>
</div>
</section>


<section class="section">
<div class="container is-max-desktop">
<div class="columns is-centered has-text-centered">
Expand All @@ -225,6 +249,23 @@ <h2 class="title is-3">Framework</h2>




<section class="section" id="BibTeX">
<div class="container is-max-desktop content">
<h2 class="title">BibTeX</h2>
<pre><code>
@misc{gao2023mind3d,
title={MinD-3D: Reconstruct High-quality 3D objects in Human Brain},
author={Jianxiong Gao and Yuqian Fu and Yun Wang and Xuelin Qian and Jianfeng Feng and Yanwei Fu},
year={2023},
eprint={2312.07485},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
</code></pre>
</div>
</section>

<footer class="footer">
<div class="container">
<div class="content has-text-centered">
Expand Down

0 comments on commit 62bcede

Please sign in to comment.