Configuration for LSST the Docs

[jonathansick]

This PR is pursuing #251, following-up on the conversation in https://community.lsst.org/t/introducing-lsst-the-docs-for-continuous-documentation-delivery/781/7?u=jsick.

The PR does two things

1. Adds an empty Sphinx project to doc/
2. Enables publication to LSST the Docs (twinkles.lsst.io) via Travis.

I can't test a full publication workflow up LSST the Docs since this branch needs to be in the DarkEnergyScienceCollaboration/Twinkles repo itself, and not my fork. (Obviously I can't push to Twinkles).

Sorry this project kept slipping in my schedule. I'm not sure whether you want to continue pursuing this, or keep your gh-pages workflow. Likely Sphinx-LTD will work well for code documentation. Over time you may decide to switch the high-level docs you have in gh-pages over to LTD.

[coveralls]

Changes Unknown when pulling 35c2810 on jonathansick:issue/251/lsst-the-docs into * on DarkEnergyScienceCollaboration:master*.

[heather999]

This may be very timely indeed @jonathansick :)
Can you remind me what functionality is already available via LSST The Docs as far as searching content - the planned Data Management Documentation Index? This might also be an opportunity to point out the advantages of LSST The Docs over gh-pages.

As for testing, perhaps we can pull this into a branch in the Twinkles repository? We have travis set up to run on our branches - so would that work in this context?

[jonathansick]

Yeah I saw tweets about your Oxford meeting and it reminded me I owed DESC a bit of work :)

You can add this branch on jonathansick/Twinkles to the main DarkEnergyScienceCollaboration/Twinkles repo and you should see the publication go through to

https://twinkles.lsst.io/v/issue-251-lsst-the-docs/

The reason a fork like mine can't publish to the Twinkles LTD account is because the credentials can only be decrypted by a Travis job running from the main Trinkles repo... essentially it's a security feature.

The main advantage of LTD over gh-pages is that versioning is built-in. Each branch or tag is published independently off of the /v/ directory, while the 'main' version is published directly off of the root path. I describe the URL scheme here: https://sqr-006.lsst.io/#versioned-documentation-urls

For a software project like Twinkles, especially one that uses Sphinx, LTD might be more comparable to readthedocs.org. In that case, LTD is more flexible in the doc build environment than readthedocs.org. RTD has a few extra features at the moment (like version switcher UI elements), but we're catching up on those features.

I'll comment more on DM's overall documentation infrastructure roadmap over in LSSTDESC/ComputingInfrastructure#24

[jonathansick]

I should also mention that I haven't tested the .travis.yml file I've included here (because of the encryption detail mentioned above) so there may be some surprises. If you move my branch over to your repo we can work out any Travis issues.

Getting doc content into Sphinx is a separate issue as well :) That said, a lot of your content is already reST, so it's a matter of organizing those files and linking them from a toctree in index.rst.

[drphilmarshall]

This is so cool. Thanks Jonathan! We are going to hack on this tomorrow, one way or another - we'll keep you in the loop so you can see what we're up to, but for a start, you could subscribe to #284 :-)

[heather999]

@jonathansick I went ahead and merged your updates into the issue/251/try_lsst_the_docs branch and the travis build went well it seems:
https://travis-ci.org/DarkEnergyScienceCollaboration/Twinkles/builds/146443064
I tried visiting the site, but there seems to be a problem:
https://twinkles.lsst.io/v/issue-251-lsst-the-docs/
Is there a way to get some more information about what may be going wrong?
Would running ltd-mason-travis at the command line be helpful for debugging?

[jonathansick]

Looks like the issue is at line 1267 of the Travis log you linked: pip isn't working in the Conda-based env you're working in, hence the pip install -r requirements.txt isn't installing the dependencies needed to publish docs. I don't use conda regularly so I'll have to find a conda-native way of installing all the dependencies listed in doc/requirements.txt. If you know then go for it!

[heather999]

pip install is available - for fun I moved the pip install -r requirements.txt line before we set up to use the DMstack and that seemed to work. One more issue to sort out.. the sphinx build is resulting in a IOError: [Errno 28] No space left on device
I'll see if I can sort out how to fix that.

[drphilmarshall]

@jonathansick Returning to your nice PR here - it is a nice pull request but I'm not sure it does what we want yet. The Twinkles repo now contains a set of in-progress DESC Notes, in the doc/LSST_DESC_Notes folder. None have full LTD setups yet as I have not added the cookiecutter outputs. Moreover, some of the Notes are going to get moved to different repos, like the pserv one, for example. I think all of these notes should end up at https://desc.lsst.io. Is it possible? It seems desirable to me, but what do you think? Feel free to add whatever experimental LTD set up files you like, if you want to experiment with publishing our in-prep Notes!

[heather999]

I think we will need DESC project specific areas in LTD like (twinkles.lsst.io) for Twinkles specific documentation. The DESC notes being cross-project, would more rightly belong in a global DESC web area like desc.lsst.io. Perhaps the travis-ci configuration for each github repo would require the ability to split their documentation across two web-accessible areas.
I would still like to enable the LTD for twinkles.lsst.io as a proof of concept. We could use it for the weeklies ultimately.

[drphilmarshall]

Got it, thanks Heather. Your plan sounds good to me!

On Fri, Aug 5, 2016 at 11:53 AM, Heather Kelly [email protected]
wrote:

I think we will need DESC project specific areas in LTD like (
twinkles.lsst.io) for Twinkles specific documentation. The DESC notes
being cross-project, would more rightly belong in a global DESC web area
like desc.lsst.io. Perhaps the travis-ci configuration for each github
repo would require the ability to split their documentation across two
web-accessible areas.
I would still like to enable the LTD for twinkles.lsst.io as a proof of
concept. We could use it for the weeklies ultimately.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#283 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AArY93PFyvIJpQceS2VccfkOlYkVXmTbks5qc4a7gaJpZM4JQ4bz
.

[heather999]

@jonathansick Time to pick your brain! I have continued to play with your updates and have completely dumbed down our travis-ci setup as I am grappling with an "Out of Space" error. I turned off all of our other tests and just set up sphinx and let that bit go. I just want to see that bit work and I'll go back and deal with the space issue.
The sphinx build is successful and I can reproduce it on my laptop and see the resulting HTML files. Here is the successful travis-ci build from today:
https://travis-ci.org/DarkEnergyScienceCollaboration/Twinkles/builds/152750662
For now I am still working off a branch in github, and when I attempt to view the page, I receive an "access denied" error:
https://twinkles.lsst.io/v/issue-251-lsst-the-docs/

Do you have some ideas what might be going wrong?

[jonathansick]

@heather999 It ended up being published to https://twinkles.lsst.io/v/issue-251-try-lsst-the-docs 💥 So it works :)

What happened is that the branch you were working with is issue/251/try-lsst-the-docs in this repo, so the version ended up getting named after that. (I intend to build dashboards soon that it's clear what the available versions are).

It does highlight that only branches from the repo in the DarkEnergyScienceCollaboration GitHub org will be published, rather than PRs from forks like mine. It ends up being because of security issues (anyone can submit a PR, but you don't want anyone to necessarily publish to lsst.io). Now maybe there's a way around this we can find if fork-and-PR is central to DESC. DM uses its GitHub repos as shared origins that we push directly to, for example.

The out-of-space issues with Travis are a bit more problematic. I can't believe that installing a few extra Python dependencies and building docs is pushing you over the edge on Travis. I don't really have an answer for that yet. I wonder if a self-hosted CI like Jenkins is needed?