[PROPOSAL]: Build a modern & robust documentation system

[FrankLeeeee]

Proposal

Overview

Recently, I am working on improving our user experience from a non-technical perspective, i.e. installation, documentation and etc. I have dived into the documentation-build ecosystem in the Python community and found that there is no perfect solution so far. Therefore, this post will discuss how we should build a documentation system for our project and make it modern, beautiful, robust, easy to test and maintain.

Current tools

There are mainly two documentation-building tools so far in the open-source community, namely sphinx and docusaurus. Sphinx is the python-specific tool to generate html pages from docstring while docusaurus is a universal documentation framework for all languages. Their comparisons are listed below.

Criterion	Sphinx	Docusaurus
Extract documentation from docstring	yes	no
Support multi-language	complicated	easy
Support versioning	requires RTD	Easy
Support branch version	requires RTD	Requires extra efforts
Documentation format	rst, markdown is supported	markdown, MDX is supported
Theme	A bit old-fashioned	More modern and elegant

Requirements and Challenges

We want to allow the community to engage in the documentation update as well. Therefore, we want to make the whole process as simple as possible. It is rather difficult to write rst. Even if sphinx allows for markdown files, customizing the theme is still a troublesome task. For simplicity, we would rather choose docusaurus as the foundation to build up our documentation system.

However, the key problem is that we sometimes need to generate API documentation from our docstring. Docusaurus does not support this natively. Therefore, we have to implement docstring -> mdx conversion on our own.

Architecture

Workflow

Colossal-AI repository

All documentation markdown files are stored in the /docs directory. This directory contains two folders for internalization.

- docs
   - en
   - zh
   - README.md
   - version.js
   - sidebar.js

The sidebar.js is for docusaurus to define the table of contents on the sidebar. version.js tells docusaurus which versions to include on the website.

When a release PR is created, the Colossal-AI repository will trigger the Build Doc CI in the Colossal-AI Documentation repository. This part will be discussed below.

Colossal-AI Documentation repository

We used to keep all the markdown files in this repository. However, this makes it difficult to maintain as not all python developers know how to play with web stuff. Therefore, we docouple documentation and website and only keep the web-related stuff in this repository.

As Docusaurus does not extract documentation from docstring, so we cannot use something like automodule in sphinx. However, this can be implemented on our own. In fact, Huggingface has provided some code for autodoc. The problem is that it uses svelte which is not compatible with docusaurus, or it is troublesome to make them compatible. Therefore, we have to adapt this autodoc function to tailwind-based html elements.

Afterwords, we can embed the generated html in the documentation markdown file with this plugin.

Make documentation testable

We need to test our documentation regularly, so that we can be notified for any code break. One way to do this is to write our documentation as jupyter notebooks. In this way, we can convert this notebook to python file for testing and to markdown files for web rendering.

Plan

ID	Description	Issue Number
1	Migrate the markdown files to Colossal-AI	#2653
2	Create doc build CI in Colossal-AI Documentation repo	hpcaitech/ColossalAI-Documentation#164
3	Support github pages for Colossal-AI Documentation repo	hpcaitech/ColossalAI-Documentation#165
4	Re-write the landing page with tailwind	hpcaitech/ColossalAI-Documentation#166
5	Test doc build when docs changes	#2676
6	Trigger doc build when release PR is merged	#2679
7	Implement auto feature	hpcaitech/ColossalAI-Documentation#175
8	Add mdx support	hpcaitech/ColossalAI-Documentation#175
9	Clear outdated sphinx files	#2931
10	Support Doc Test	hpcaitech/ColossalAI-Documentation#177
11	Add README	#2745

Self-service

• I'd be willing to do some initial work on this proposal myself.

[FrankLeeeee]

All tasks listed in this discussion are completed! 🚩

[FrankLeeeee]

This project is managed by the kanban https://github.com/orgs/hpcaitech/projects/14/views/1

[FrankLeeeee]

Migrated from issue #2651

[yymmiinngg]

ding