tutorials aren't ready yet for primetime

facebookresearch · Sep 22, 2023 · 2e3ecfa · 2e3ecfa
1 parent ec9c22d
commit 2e3ecfa
Show file tree

Hide file tree

Showing 2 changed files with 55 additions and 8 deletions.
diff --git a/doc/index.rst b/doc/index.rst
@@ -8,14 +8,6 @@ fairseq2 is a sequence modeling toolkit that allows researchers and developers
 to train custom models for translation, summarization, language modeling, and
 other content generation tasks.
 
-.. toctree::
-   :maxdepth: 1
-   :caption: Tutorials
-
-   tutorials/cli/training
-   tutorials/cli/authoring
-   tutorials/data
-
 .. toctree::
    :caption: fairseq2 Reference
 

diff --git a/doc/reference/dataloader.rst b/doc/reference/dataloader.rst
@@ -0,0 +1,55 @@
+fairseq2.data
+=============
+.. body
+
+.. currentmodule:: fairseq2.data
+
+.. autosummary::
+    :toctree: generated/dataloader
+
+    DataPipeline
+    DataPipelineBuilder
+    CString
+
+    list_files
+    read_sequence
+    read_zipped_records
+
+    Collater
+    FileMapper
+
+    VocabularyInfo
+    ByteStreamError
+    CollateOptionsOverride
+    DataPipelineError
+    FileMapperOutput
+    RecordError
+    SequenceData
+    PathLike
+    StringLike
+    get_last_failed_example
+    is_string_like
+
+fairseq2.data.text
+~~~~~~~~~~~~~~~~~~
+
+.. currentmodule:: fairseq2.data.text
+
+.. autosummary::
+    :toctree: generated/data_text
+
+    TextTokenizer
+    MultilingualTextTokenizer
+    TextTokenDecoder
+    TextTokenEncoder
+
+    StrSplitter
+    StrToIntConverter
+    StrToTensorConverter
+
+    SentencePieceModel
+    SentencePieceEncoder
+    SentencePieceDecoder
+    vocabulary_from_sentencepiece
+    LineEnding
+    read_text