Migrate docs.stackstorm.com to GH static pages

[arm4b]

Currently, https://docs.stackstorm.com/ are deployed to AWS S3 on every build.
Migrating to native GH static pages unties docs from AWS and would improve the transparency via git, security, and maintenance.
We had a good experience with Github Static pages as a great alternative so far where additional infra would be redundant.

Depends on #1095
Relevant story for apidocs: StackStorm/st2apidocs#16

[cognifloyd]

I think rtfd.io (readthedocs.org) would be ideal for docs.stackstorm.com as they already have the infra to deal with sphinx. Using a CNAME record we can continue using docs.stackstorm.com as well.

[arm4b]

readthedocs.org is really nice, but it's still 3rd party service that will require another new set of credentials and deployment.

With security in mind, what git + GH static pages provide us is native integration and transparency. The entire pipeline doesn't escape Github and everyone can track if the docs.stackstorm.com state is indeed in sync and not modified in the middle or something.

[cognifloyd]

readthedocs.org is really nice, but it's still 3rd party service that will require another new set of credentials and deployment.

Credentials management is easier than I thought it would be with ReadTheDocs. It supports login with github, so we don't need any new passwords to manage this.

As an experiment, I added @st2stanley as an admin of the st2docs project. I asked in the readthedocs/community chat about automatically giving a Github Team maintainer rights for a project, and they said:

for me, direct team access is not a problem. Team members can write the .readthedocs.yml file to update RTD config
I know is not the same thing...but for 99% of the changes, it works
this feature is available on our commercial hosting, as GitHub SSO, but we are planning on porting this back to our community platform -- likely starting sometime Q4 maybe?

In the meantime if having @st2stanley as an admin is not enough, we can also add whatever TSC member needs to work on the project. And if we need it to be automated, I can write a GHA workflow that will periodically sync a senior maintainer team (which I don't think we have) from github to RTD.

---

After adding @st2stanley, I tested building docs with the rtd branch of st2docs. In poking around I noticed that we are using an ancient version of the sphinx_rtd_theme: it seems to be somewhere between v0.1.5 and v0.2.0. The theme is at v1.0.0. So, I looked at updating and noticed this warning in the docs:

Installing directly from the repository source is deprecated and is not recommended. Static assets won’t be included in the repository starting in release 3.0.0.

So, I did the following:

• I updated the conf.py so that it would use the new version (1.0.0) of the theme, but only if the build is on RTD.
• I inspected the in-repo theme files to see what was customized for StackStorm's docs site.
• I added some css and js override files to apply most of those customizations on top of the upstream theme (this time without forking it)
• In copying over some of the colors I adjusted (or kept the upstream theme's default for) some of the colors to have a little more contrast.

Check out an example page: https://st2docs.readthedocs.io/en/rtd/install/u18.html
(we can use docs.stackstorm.com later if people like this)

I believe there are only three StackStorm customizations that I haven't ported:

• include the font css link (though I have updated the CSS to use the same fonts)

  • st2docs/docs/source/_themes/sphinx_rtd_theme/layout.html

    Line 35 in 828b2a2

    <link href='https://fonts.googleapis.com/css?family=Open+Sans:400,300,300italic,400italic,600,600italic,700,700italic|Inconsolata:400,700' rel='stylesheet' type='text/css'>

• sitemap.xml generation (which was actually broken before, because it did not generate the version-specific sitemaps correctly, so this needs more work, and probably needs an update based on the original https://github.com/guzzle/guzzle_sphinx_theme)
  • most of: https://github.com/StackStorm/st2docs/blob/master/docs/source/_themes/sphinx_rtd_theme/__init__.py
• add Google Tag Manager:

  • st2docs/docs/source/_themes/sphinx_rtd_theme/layout.html

    Lines 17 to 23 in 828b2a2

    	<!-- Google Tag Manager -->
    	<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
    	new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
    	j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
    	'https://www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
    	})(window,document,'script','dataLayer','GTM-NKZBMHZ');</script>
    	<!-- End Google Tag Manager -->

  • st2docs/docs/source/_themes/sphinx_rtd_theme/layout.html

    Lines 89 to 92 in 828b2a2

    	<!-- Google Tag Manager (noscript) -->
    	<noscript><iframe src="https://www.googletagmanager.com/ns.html?id=GTM-NKZBMHZ"
    	height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
    	<!-- End Google Tag Manager (noscript) -->

---

One gotcha is that this has the same issue as #1129 in that keeping the stable/ga docs at the top-level does not work. We can add some redirects in the RTD UI or via API, but it might will take some doing. Maybe we could have script that interacts with RTD API during the RTD docs build to add the appropriate redirects (if we think that's worth it).

_edit: RTD redirects are more powerful than I thought, so implementing the redirects is fairly simple. See #1096 (comment) _

---

So, please check out: https://st2docs.readthedocs.io/en/rtd/

What does everyone think? Can we go in this direction?

One of the features I really want is pull request builds:
https://docs.readthedocs.io/en/stable/pull-requests.html

[amanda11]

I took at the look at the generated pages. One thing I noticed is that the multi-level headings are handled differently, compare:
https://st2docs.readthedocs.io/en/rtd/rbac.html#how-it-works
and
https://docs.stackstorm.com/rbac.html#how-it-works

Looks like our current docs only go down a couple of levels in the tree, but the RTD goes down to more levels which is nice.

Looking at the history the main concern was introducing another 3rd party service, rather than hosting natively with GitHub. Though I read it's free hosting for open-source, so addresses the cost reduction exercise.

[cognifloyd]

Yes, the newer version of the theme has some new bells and whistles. Scrolling is also nicer because the menu can scroll separately from the main content.

I think we can use the newer version of the theme wherever we host the pages, but I've only tested it on RTD where some additional things get injected during the build (I don't think I'm relying on the injected bits, but we'd need to test).

[arm4b]

Thanks, @cognifloyd for trying it! Really helpful to have an example 👍

The diff key points to me:

1. URLs
   https://st2docs.readthedocs.io/en/rtd/actions.html
   vs
   https://docs.stackstorm.com/actions.html
   Migration-wise, all the links because of SEO, traffic and old references should be exactly the same as before. History accumulated quite a lot of link references and changing the structure is not desirable. In the future moving out of RDT will have URL structure change implications again.

2. Security
   GH Actions + GH Pages are already used across stackstorm, there is no learning or integration factor and it's transparent. You see the exact page byte-by-byte that already exists in git.
   With all the craze around the supply chain attacks and security, this is a really critical factor.
   Moving more out of AWS is not only about the cost reduction (S3 costs pennies), rather than being self-sufficient, less external services and transparent in the pipeline. There is an external mgmt, credentials management, user mgmt and access, settings, etc which all adds to the amount of moving pieces. With RDT, once the build happens the rest is unknown. Add here analytics or AD that happens on RTD side.

3. Preview
   As Jacob said, one of the great features of RTD is a preview mode. This is cool indeed 👍

With that, -2 on 3rd party service, security, mgmt, URLs, +1 on the preview.

[arm4b]

@cognifloyd Speaking of Preview & Security, why not have both?
We could just keep RDT integration for PR preview only on that development URL, while actually deploying the "production" https://docs.stackstorm.com/ via GH static pages as planned.

[arm4b]

The updated theme looks ok, just still need to preserve the original page width as before:

OLD

NEW

[cognifloyd]

fixed the width. I'll submit a PR that just updates to the new theme.

[cognifloyd]

1. URLs
   https://st2docs.readthedocs.io/en/rtd/actions.html
   vs
   https://docs.stackstorm.com/actions.html
   Migration-wise, all the links because of SEO, traffic and old references should be exactly the same as before. History accumulated quite a lot of link references and changing the structure is not desirable. In the future moving out of RDT will have URL structure change implications again.

Well that was easier to fix than I thought. RTD has a "redirects" feature that allows you to configure redirects that happens if a url would return a 404 (ie /bla.html instead of /en/3.7/bla.html).
https://docs.readthedocs.io/en/stable/user-defined-redirects.html

I just set up these redirects:

So all current urls should work on RTD.

Also, I found that we can have a version of the docs that RTD builds without getting included in the versions menu (active and hidden).

Whether we switch to RTD or gh-pages, we need a way to build old versions of the docs too. We currently have ALL old versions. So there could be old links we need to preserve.