Adds packaging docs on readthedocs

.gitignore

@@ -0,0 +1,3 @@
+# Ignore everything in this directory
+.yardoc/*
+

.readthedocs.yaml

@@ -0,0 +1,15 @@
+version: 2
+
+build:
+  os: "ubuntu-22.04"
+  tools:
+    python: "3.11"
+
+# Build from the docs/ directory with Sphinx
+sphinx:
+  configuration: source/conf.py
+
+# Explicitly set the version of Python and its requirements
+python:
+  install:
+    - requirements: requirements.txt

Makefile

@@ -0,0 +1,28 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# New target that executes the bash script before calling the original target
+build-docs: pre-build   # Change 'html' to the target you want to execute
+
+pre-build:
+	# Execute your bash script here
+	bash api-reference.sh
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+html:
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+

requirements.in

@@ -0,0 +1,4 @@
+sphinxnotes-strike
+sphinx
+sphinx_rtd_theme
+readthedocs-sphinx-search

requirements.txt

@@ -0,0 +1,62 @@
+#
+# This file is autogenerated by pip-compile with Python 3.10
+# by the following command:
+#
+#    pip-compile
+#
+alabaster==0.7.13
+    # via sphinx
+babel==2.12.1
+    # via sphinx
+certifi==2023.5.7
+    # via requests
+charset-normalizer==3.1.0
+    # via requests
+docutils==0.18.1
+    # via
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.4
+    # via requests
+imagesize==1.4.1
+    # via sphinx
+jinja2==3.1.2
+    # via sphinx
+markupsafe==2.1.3
+    # via jinja2
+packaging==23.1
+    # via sphinx
+pygments==2.15.1
+    # via sphinx
+readthedocs-sphinx-search==0.3.1
+    # via -r requirements.in
+requests==2.31.0
+    # via sphinx
+snowballstemmer==2.2.0
+    # via sphinx
+sphinx==6.2.1
+    # via
+    #   -r requirements.in
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-jquery
+    #   sphinxnotes-strike
+sphinx-rtd-theme==1.2.2
+    # via -r requirements.in
+sphinxcontrib-applehelp==1.0.4
+    # via sphinx
+sphinxcontrib-devhelp==1.0.2
+    # via sphinx
+sphinxcontrib-htmlhelp==2.0.1
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==1.0.3
+    # via sphinx
+sphinxcontrib-serializinghtml==1.1.5
+    # via sphinx
+sphinxnotes-strike==1.2
+    # via -r requirements.in
+urllib3==2.0.3
+    # via requests

source/citus-docker-images.rst

@@ -0,0 +1,38 @@
+Citus Docker images
+====================
+
+General Structure
+------------------
+Docker images are part of the Citus release process. They are built and pushed after every release. Dockerfiles are stored in the `citusdata/docker <https://github.com/citusdata/docker>`_ repository. In each release, we change the below Dockerfiles to point to the new Citus release.
+Citus Docker images are published in DockerHub `citusdata/citus <https://hub.docker.com/repository/docker/citusdata/citus/>`_ 
+
+1. `Main Dockerfile <https://github.com/citusdata/docker/blob/master/Dockerfile>`_: Used to build the latest regular Citus image for the latest Postgres version.
+2. `alpine Dockerfile <https://github.com/citusdata/docker/tree/master/alpine>`_: Used to build the latest regular Citus image for the latest Postgres version with the alpine base Docker image.
+3. `postgres-xx` Dockerfiles (where `xx` represents the major Postgres versions) : Used to build the Citus images for the older Postgres versions. We normally support the last 3 major Postgres versions. For example, if the latest Postgres version is 15, we support 13, 14, and 15. Therefore, we have 2 Dockerfiles in total for supported Postgres major versions other than the latest one.
+
+Build Process
+------------------
+The Dockerfiles are built and pushed to Docker Hub by GH Actions pipelines. There are four pipelines used for the Citus Docker image build and publish process:
+
+1. `Update Citus versions <https://github.com/citusdata/docker/blob/master/.github/workflows/update_version.yml>`_: This pipeline updates the Citus versions in the Dockerfiles and pushes the changes to a newly created branch and creates a PR. It should be triggered manually after finishing the publishing of Citus packages to the packagecloud.
+
+2. `Build and publish Citus Docker images on push <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_on_push.yml>`_: This pipeline is triggered on every branch push to the repository. It builds the Docker images on every occasion and pushes them to Docker Hub if the branch is master.
+
+3. `Build and publish Citus Docker images on schedule <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_cron.yml>`_: This pipeline is triggered every day at 00:30 UTC and builds and publishes the Docker images for the main, alpine, and `postgres-xx` Docker images to Docker Hub.
+
+4. `Build and publish Citus Docker images manually <https://github.com/citusdata/docker/blob/master/.github/workflows/publish_docker_images_on_manual.yml>`_: This pipeline is triggered manually and builds and publishes the Docker images. It is useful when we want to build and publish the Docker images without waiting for the scheduled pipeline to run.
+
+New Postgres Version Support
+-----------------------------
+When a new Postgres version is released, we need to add support for the new Postgres version in the Dockerfiles. To add support for a new Postgres version:
+
+1. Create a new branch from the master branch.
+2. Create a new Dockerfile for the new Postgres version. For example, for Postgres 16, create `postgres-16/Dockerfile` by copying the `postgres-15/Dockerfile`.
+3. Change the Postgres version in the new Dockerfile to the new Postgres version.
+4. Test the new Dockerfile using the appropriate Docker build and run commands.
+5. Add new Postgres version support to the scheduled pipeline.
+6. If the tests are successful, create a pull request to merge the changes to the master branch.
+
+Adding new Postgres version support to the tools scripts
+---------------------------------------------------------
+When a new Postgres version is released, we also need to add support for the new Postgres version in the tools scripts. Mainly, the `update_docker.py <https://github.com/citusdata/tools/blob/develop/packaging_automation/update_docker.py>`_ script needs to be updated to add support for the new Postgres version. This script is used in the pipelines.

source/conf.py

@@ -0,0 +1,31 @@
+from datetime import date
+
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "Citus Packaging"
+copyright = f"{date.today().year} .Citus Data Licensed under the MIT license, see License for details. "
+author = "Citus Data"
+release = "2.2.0"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ["sphinxnotes.strike"]
+
+templates_path = ["_templates"]
+exclude_patterns = []
+
+language = "python"
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "sphinx_rtd_theme"
+html_static_path = ["_static"]
+

source/debian-packages.rst

@@ -0,0 +1,57 @@
+.. _debian-packages:
+
+Debian Packages
+===============
+
+Introduction to Debian packaging
+--------------------------------
+
+Debian packaging provides a standardized way to create software packages for Debian-based operating systems. It ensures easy installation, upgrade, and removal of software components. The packaging format used in Debian is known as the Debian package format or ``.deb``.
+
+Debian-specific tools and conventions
+-------------------------------------
+
+Debian packaging relies on a set of tools and conventions to streamline the packaging process. Familiarize yourself with the following:
+
+- **dpkg**: The Debian package manager (``dpkg``) is the core tool for installing, querying, and managing Debian packages. It operates at a lower level than higher-level package management tools like ``apt`` or ``apt-get``.
+
+- **Debian control file**: The ``debian/control`` file is a metadata file that provides information about the package, its dependencies, and other package-related details. It includes fields such as package name, version, maintainer, description, dependencies, and more. Maintaining an accurate and up-to-date control file is essential for proper package management.
+
+- **Debian changelog**: The ``debian/changelog`` file maintains a record of changes made to the package across different versions. It includes information about the package version, the person responsible for the change, the change description, and other relevant details. The changelog helps users and maintainers track the history of the package and understand the modifications introduced in each release.
+
+- **Debian rules file**: The ``debian/rules`` file is a makefile-like script that defines the build process and other package-specific tasks. It specifies how to compile the source code, configure the package, and install the files in the correct locations. The rules file allows for customization and automation of the packaging process.
+
+Other important Debian-specific tools and files include ``debuild``, ``lintian``, and ``pbuilder``, which aid in building, linting, and testing packages respectively.
+
+Additional Debian Files
+-----------------------
+
+In addition to the core Debian packaging files, the packaging process may involve other files with specific purposes. Here are some of the additional Debian files commonly used:
+
+- **Source/format**: This file specifies the format version of the Debian source package.
+
+- **Source/lintian-overrides**: This file is used to override specific lintian warnings or errors generated during the package build process.
+
+- **Tests/control**: This file contains information about the tests to be run on the package.
+
+- **Tests/installcheck**: This file specifies commands or scripts to be executed after the package is installed.
+
+- **Upstream/signing-key.asc**: This file contains the ASCII-armored representation of the upstream signing key used to verify the authenticity and integrity of the package.
+
+- **Changelog**: The ``debian/changelog`` file maintains a record of changes made to the package.
+
+- **Compat**: The ``debian/compat`` file specifies the compatibility level of the package's packaging format.
+
+- **Control.in**: The ``debian/control.in`` file is a template file used to generate the ``debian/control`` file.
+
+- **Copyright**: The ``debian/copyright`` file provides information about the copyright holders and licensing terms.
+
+- **Docs**: The ``debian/docs`` file lists the additional documentation files included with the package.
+
+- **NOTICE**: The ``debian/NOTICE`` file contains any legally required notices or attributions.
+
+- **Pgversion**: The ``debian/pgversion`` file specifies the PostgreSQL version for which the package is intended.
+
+- **Rules**: The ``debian/rules`` file is a makefile-like script that defines the build process.
+
+For more information about Debian packaging, see the `Debian New Maintainers' Guide <https://www.debian.org/doc/manuals/maint-guide/>`_.

source/index.rst

@@ -0,0 +1,75 @@
+
+Welcome to Citus Data Packaging documentation!
+======================================================
+
+|Citus Linux Packages Nightly Status| |Citus Docker Image Nightly Status|  |Latest Documentation Status|
+
+.. |Citus Linux Packages Nightly Status| image:: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml/badge.svg
+   :target: https://github.com/citusdata/packaging/actions/workflows/build-citus-community-nightlies.yml
+   :alt: Citus Linux Packages Nightly Status
+
+.. |Citus Docker Image Nightly Status| image:: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml/badge.svg
+   :target: https://github.com/citusdata/docker/actions/workflows/publish_docker_images_cron.yml
+   :alt: Citus Docker Image Nightly Status
+
+
+.. |Latest Documentation Status| image:: https://readthedocs.org/projects/django-multitenant/badge/?version=latest
+   :target: https://citusdata-packaging.readthedocs.io/en/latest//?badge=latest
+   :alt: Documentation Status
+
+
+Citus Data Packaging Documentation
+========================================
+
+Welcome to the official documentation for ``CitusData Packaging``.
+
+``CitusData Packaging`` is a collection of scripts and tools that are used to packaging Citus Data products.
+This documentation is intended to provide a detailed overview of the packaging process and the tools used to package Citus Data products.
+
+
+
+.Table of Contents
+===================
+
+
+.. toctree::
+   :maxdepth: 2
+
+   introduction
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Repository Management
+
+   package-build-environment
+   repository-installation-scripts
+   repository
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Tools and Configurations
+
+   tools-scripts
+   project-packaging-configurations
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Artifacts
+
+   debian-packages
+   rpm-packages
+   citus-docker-images
+   pgdg
+   pgxn
+
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Packaging Process
+
+   packaging-process
+
+

source/introduction.rst

@@ -0,0 +1,44 @@
+Introduction
+============
+
+Welcome to the documentation outlining the packaging process for Citus and its associated projects. Packaging plays a crucial role in the distribution and deployment of software, ensuring that it can be easily installed and utilized across various operating systems and environments.
+
+In this document, we will explore the steps and best practices involved in preparing packages for different operating systems and releases. We will focus on the Debian and RPM package formats, as they are widely used and supported in the industry. Additionally, we will cover the creation and management of Docker environments to facilitate packaging for specific OS/release/PostgreSQL version combinations.
+
+The supported operating systems and releases for packaging include:
+
+- CentOS 8
+- CentOS 7
+- Oracle Linux 8
+- Oracle Linux 7
+- AlmaLinux 9
+- Debian Buster
+- Debian Bullseye
+- Debian Bookworm
+- Ubuntu Bionic
+- Ubuntu Focal
+- Ubuntu Jammy
+- Ubuntu Kinetic
+
+The packaging process will involve preparing packages for various projects, including:
+
+* `Citus <https://github.com/citusdata/packaging/tree/all-citus>`_
+* `Pgazure <https://github.com/citusdata/packaging/tree/all-pg-azure-storage>`_
+* Cron
+    * `Cron Debian <https://github.com/citusdata/packaging/tree/debian-cron>`_
+    * `Cron RPM <https://github.com/citusdata/packaging/tree/redhat-cron>`_
+* Hll
+    * `Hll Debian <https://github.com/citusdata/packaging/tree/debian-hll>`_
+    * `Hll RPM <https://github.com/citusdata/packaging/tree/redhat-hll>`_
+* Topn
+    * `Topn Debian <https://github.com/citusdata/packaging/tree/debian-topn>`_
+    * `Topn RPM <https://github.com/citusdata/packaging/tree/redhat-topn>`_
+* `Azure-gdpr <https://github.com/citusdata/packaging/tree/all-azure_gdpr>`_
+
+Once the packages are ready, we will push them to PackageCloud, a package management and distribution service. Additionally, we will create Docker images for each OS/release/PostgreSQL version combination, which will be published on Docker Hub.
+
+Lastly, we will cover the publishing of the applications into PGXN (PostgreSQL Extension Network), a platform that hosts and distributes PostgreSQL extensions.
+
+By following the steps outlined in this document, you will gain a comprehensive understanding of the packaging process for Citus and related projects. This knowledge will empower you to efficiently distribute and deploy the software across a wide range of operating systems and environments.
+
+Now, let's dive into the detailed steps for package preparation, Docker environment management, Docker image creation, PackageCloud integration, and PGXN publishing.