adding pkg-sync tool and update docs to reflect changes in syncing

Signed-off-by: Matt Wrock <[email protected]>

README.md

@@ -50,11 +50,6 @@ Once installed, the following functionality will be available to users:
 * Package builds using the `hab` client and Chef Habitat Studio
 * Ability to import core packages from the upstream Chef Habitat Builder
 
-The following Chef Habitat Builder on-prem functionalities are *NOT* currently available:
-
-* Automated package builds using Chef Habitat Builder on-prem
-* Automated package exports using Chef Habitat Builder on-prem
-
 ## Documentation
 
 The documentation for Builder on-prem is located in the [on-prem-docs](on-prem-docs/getting-started.md) directory.

on-prem-docs/bootstrap-core.md

@@ -1,50 +1,46 @@
 # Bootstrap Core Packages
 
-## Create the Core Origin
-
-Create a `core` origin for an initial set of base packages. Uploads will fail unless you first populate your Chef Habitat Builder on-prem with the upstream `core` upstream origin.
-
-Once you are logged in to the Chef Habitat Builder on-prem UI, select the `New Origin` button and enter in `core` as the origin name.
-
 ## Generate a Personal Access Token
 
-Next, generate a Personal Access Token for bootstrapping the `core` packages, as well as for performing authenticated operations using the `hab` client.
+Generate a Personal Access Token for bootstrapping the `core` packages, as well as for performing authenticated operations using the `hab` client.
 
 Select your Gravatar icon on the top right corner of the Chef Habitat Builder on-prem web page, and then select **Profile**. This will take you to a page where you can generate your access token. Make sure to save it securely.
 
 ## Bootstrap Builder with Habitat Packages
 
 Chef Habitat Builder on-prem has no pre-installed package sets. You must populate your Builder instance by uploading packages.
-With Habitat *0.88.0*, two new commands were introduced to assist in bootstrapping an on-prem Builder instance with a set of stable packages:
+To assist in bootstrapping an on-prem Builder instance with a set of LTS packages, you can install the habitat/pkg-sync package which will download packages from the public [SaaS Builder](https://bldr.habitat.sh) followed by a bulkupload to your on-prem Builder instance(s).
+
+The following snippet illustrates how to bootstrap the on-prem Builder with a set of LTS-2024 core, chef, and chef-platform packages:
 
-1. *hab pkg download*
-1. *hab pkg bulkupload*
+    ```bash
+    hab pkg install habitat/pkg-sync
+    hab pkg exec habitat/pkg-sync pkg-sync --bldr-url https://your-builder.tld --channel LTS-2024 --auth <your_public_Builder_instance_token>
+    ```
+This performs a pre-flight check to ensure that you do not have local core, chef, or chef-platform packages in the LTS-2024 channel that are not in the same channel on bldr.habitat.sh. If there are, these local packages must be demoted. Once the pre-flight check passes, pkg-sync will build a list of all the latest LTS-2024 packages that you do not already have and then upload them to your on-prem builder instance.
 
-As you can see from the commands above, the package Bootstrap flow is comprised of two main phases: a download from the public [SaaS Builder](https://bldr.habitat.sh) followed by a bulkupload to your on-prem Builder instance(s). Historically, we bootstrapped on-prem-builders by downloading all the packages in 'core' for all targets. That amounted to ~15GB and was both too much and too little, in that many of the packages weren't needed, and for many patterns (effortless) other origins were needed.
+### Airgapped Environments
 
-The [new bootstrap process flow](https://forums.habitat.sh/t/populating-chef-habitat-builder-on-prem/1228) allows you to easily customize your Bootstrap package set or use pre-populated [Package Seed Lists](https://github.com/habitat-sh/on-prem-builder/blob/master/package_seed_lists/README.md) files.
+Airgapped builder instances must take an alternative approach because pkg-sync will not be able to transfer packages from the public internet to your instance. Instead you will use the `--generate-airgap-list` flag with pkg-sync to build a list of packages that need to be downloaded. Then you will use `hab pkg download` and `hab pkg upload` to download the packages from bldr.habitat.sh and upload them to your instance. Note that `pkg-sync` and `hab pkg download` must be used on a machine with access to the public internet. This will download a bundle you can archive and transfer to your instance. Finally you will use `hab pkg upload` locally on your builder instance to upload the packages into your instance.
 
-The following section illustrates the steps required to bootstrap the on-prem Builder with the [Effortless Linux](https://github.com/habitat-sh/on-prem-builder/blob/master/package_seed_lists/effortless_x86_64-linux_stable) package seed list. Simply repeat the following download/bulkupload flow for any other package seed lists you think you will need to have in your on-prem Builder, or even create your own custom package seed list file:
+The following section illustrates the steps required to bootstrap an airgapped on-prem Builder with a set of LTS-2024 core, chef, and chef-platform packages:
 
-1. Phase 1: download
+1. Phase 1: download from a machine with internet connectivity
 
     ```bash
-    export HAB_AUTH_TOKEN=<your_public_Builder_instance_token>
-    cd on-prem-builder
-    hab pkg download --target x86_64-linux --channel stable --file package_seed_lists/effortless_x86_64-linux_stable --download-directory builder_bootstrap
+    hab pkg install habitat/pkg-sync
+    hab pkg exec habitat/pkg-sync pkg-sync --generate-airgap-list --channel LTS-2024
+    hab pkg download --target x86_64-linux --channel LTS-2024 --file package_list_x86_64-linux.txt --download-directory builder_bootstrap
+    hab pkg download --target x86_64-windows --channel LTS-2024 --file package_list_x86_64-windows.txt --download-directory builder_bootstrap
     ```
 
-    > Note: If the on-prem Builder is Airgapped, this phase must be completed on a system with Internet connectivity. The downloaded content will need to be zipped and then transferred to the Builder system for phase 2.
-
-1. Phase 2: bulkupload
-
-     **Important**: Inspect the contents of the `builder_bootstrap/artifacts` directory created from the download command above. For each of the origins (`core`, `effortless`, etc),  create the origin name if one doesn't exist already in the on-prem Builder UI before starting the bulkupload.
+    Zip up the contents of `builder_bootstrap`. Copy and unzip to the builder instance
 
-    > Note: If your on-prem builder's SSL certificate was issued from an internal Public Key Infrastructure and not from a Publicly Trusted Certificate Authority, you will need to copy the SSL public key cert chain into `/hab/cache/ssl` locally on the system that is uploading packages to the on-prem Builder. This is described in more detail [here](https://blog.chef.io/chef-habitat-product-announcement-improved-tls-certificate-management/)
+1. Phase 2: bulkupload locally on the builder instance
 
     ```bash
     export HAB_AUTH_TOKEN=<your_on-prem_Builder_instance_token>
-    hab pkg bulkupload --url https://your-builder.tld --channel stable builder_bootstrap/
+    hab pkg bulkupload --url https://your-builder.tld --channel LTS-2024 --auto-create-origins builder_bootstrap/
     ```
 
 ## Configuring a user workstation

on-prem-docs/builder-oauth.md

@@ -83,22 +83,22 @@ In order to install the on-prem Chef Habitat Builder in an airgapped (no direct
     curl -LO https://github.com/habitat-sh/on-prem-builder/archive/master.zip
     ```
 
-1. Download the Chef Habitat [cli tool](https://api.bintray.com/content/habitat/stable/linux/x86_64/hab-%24latest-x86_64-linux.tar.gz?bt_package=hab-x86_64-linux)
+1. Download the Chef Habitat [cli tool](https://packages.chef.io/files/stable/habitat/latest/hab-x86_64-linux.tar.gz)
 
     ```bash
     curl -Lo hab.tar.gz https://packages.chef.io/files/stable/habitat/latest/hab-x86_64-linux.tar.gz
     ```
 
-1. Create the Habitat Builder package bundle from the [Builder Seed List](https://github.com/habitat-sh/on-prem-builder/blob/master/package_seed_lists/builder_x86_64-linux_stable) package seed list and download it
+1. Create the Habitat Builder package bundle from the Builder seed lists and download them
 
      ```bash
      git clone https://github.com/habitat-sh/on-prem-builder.git
      export DOWNLOAD_DIR=/some/base/download/directory
      cd on-prem-builder
+     hab pkg download --target x86_64-linux --channel LTS-2024 --file package_seed_lists/builder_x86_64-linux_lts_2024 --download-directory ${DOWNLOAD_DIR}/builder_packages
      hab pkg download --target x86_64-linux --channel stable --file package_seed_lists/builder_x86_64-linux_stable --download-directory ${DOWNLOAD_DIR}/builder_packages
      ```
 
-1. Create any additional package bundles to upload to Builder from package seed lists as documented in the [Bootstrap Builder](https://github.com/habitat-sh/on-prem-builder/blob/master/README.md#bootstrap-builder-with-habitat-packages-new) section of this README. You can specify `--download-directory ${DOWNLOAD_DIR}/builder_bootstrap` argument to the download command in order to consolidate all bootstrap packages in a single directory
 1. Zip up all the above content, transfer and unzip on the Linux system where Builder will be deployed in the Airgapped environment
 
 > Note: The following tasks are intended to be completed on the Airgapped system where Builder will be deployed, in advance of the [Installation](https://github.com/habitat-sh/on-prem-builder/blob/master/README.md#Installation).

on-prem-docs/getting-started.md

@@ -42,11 +42,6 @@ Once installed, the following functionality will be available to users:
 * Package builds using the `hab` client and Chef Habitat Studio
 * Ability to import core packages from the upstream Chef Habitat Builder
 
-The following Chef Habitat Builder on-prem functionalities are *NOT* currently available:
-
-* Automated package builds using Chef Habitat Builder on-prem
-* Automated package exports using Chef Habitat Builder on-prem
-
 ### Memory Filesystem Storage
 
 Preparing your filesystem (Optional)

package_seed_lists/README.md

@@ -1,23 +1,17 @@
 # Habitat seed lists
 
-Historically we bootstrapped on-prem Builders by downloading all the packages in 'core'
-for all targets. That amounted to about 15GB, and was both too much and too little, in that many of
-the packages weren't needed, and for many patterns (Effortless) other origins were needed.
+With the `hab pkg download` command, you can download a list of seed packages along with their transitive dependencies. Of course, the question now becomes 'what do I use for the seed'.
 
-With the creation of the `hab pkg download` command, a different approach is possible; you can start
-with a list of seed packages, and download them along with their transitive dependencies. Of course,
-the question now becomes 'what do I use for the seed'.
-
-This directory contains sample 'seed lists' of packages for bootstrapping and syncing on-prem Builder, for
+This directory contains sample 'seed lists' of packages for bootstrapping and syncing on-prem Builder for
 a number of different scenarios.
 
-The basic file naming pattern is TASK\_ARCH\_CHANNEL. The files are newline separated list of
+> Note: We are moving away from seed lists. See [Bootstrap Core Origin](../on-prem-docs/bootstrap-core.md) for instructions on using the `pkg-sync` tool to sync packages from public builder to an on-prem instance.
+
+The basic file naming pattern is TASK\_ARCH\_CHANNEL. The files are a newline separated list of
 package identifiers. The contents are specific to a particular architecture and channel and the
 correct architecture and channel must be provided on the command line when running the `hab pkg
 download` command. The simple file format used doesn't allow for specification of channel or target
-architecture inline, so we're using a file naming convention to represent that information. We plan
-to improve on that experience, but for now keep it mind when building your own lists.
-
+architecture inline, so we're using a file naming convention to represent that information.
 # Scenarios
 
 The current scenarios are:
@@ -26,21 +20,19 @@ The current scenarios are:
 * core_full (everything for a particular architecture)
 * effortless (starter set for the effortless pattern)
 
-Each is broken out by the architecture and channel required; to complete some two downloads, once
-from stable and once from unstable will be required.
+Each is broken out by the architecture and channel required.
 
-For example, to get the complete Effortless infrastructure for Linux,
+For example, to get the complete Builder infrastructure for Linux,
 ```
-hab pkg download --download-directory download_pkgs --channel=unstable --target x86\_64-linux  --file  package_seed_lists/effortless_x86_64-linux_unstable
-hab pkg download --download-directory download_pkgs --channel=stable --target x86\_64-linux  --file package_seed_lists/effortless_x86_64-linux_stable
+hab pkg download --download-directory download_pkgs --channel=LTS-2024 --target x86\_64-linux  --file package_seed_lists/builder_x86_64-linux_lts_2024
+hab pkg download --download-directory download_pkgs --channel=stable --target x86\_64-linux  --file package_seed_lists/builder_x86_64-linux_stable
 ```
 
 Current scenarios include:
 
 ## Effortless (effortless_ARCH_CHANNEL)
 
-These should provide all required packages for the various Effortless patterns. They're broken out
-by architecture, and both stable and unstable are required for a complete Effortless infrastructure.
+These should provide all required packages for the various Effortless patterns, broken down by architecture.
 
 See https://github.com/chef/effortless for more details on the Effortless infrastructure pattern.
 
@@ -57,7 +49,6 @@ Linux kernel2 are about 3.5GB and 1GB respectively.
 These are the packages listed as a transitive dep or build dep of another package in core. It is
 intended as good starting point for building packages.
 
-## Builder (builder_x86_64-linux_stable)
+## Builder (builder_x86_64-linux_CHANNEL)
 
-This should be just enough packages to get Builder installed on your system. Currently Builder only
-supports the x86_64-linux platform.
+This should be just enough packages to get Builder installed on your system. Currently Builder only supports the x86_64-linux platform. You will need both the LTS-2024 channel list and also the stable channel list.

package_seed_lists/builder_x86_64-linux_lts_2024

@@ -0,0 +1,8 @@
+core/hab
+core/hab-sup
+core/hab-launcher
+habitat/builder-api
+habitat/builder-api-proxy
+habitat/builder-datastore
+habitat/builder-minio
+habitat/builder-memcached

package_seed_lists/builder_x86_64-linux_stable

@@ -1,8 +1,3 @@
 core/hab
 core/hab-sup
 core/hab-launcher
-habitat/builder-api
-habitat/builder-api-proxy
-habitat/builder-datastore
-habitat/builder-minio
-habitat/builder-memcached

pkg-sync/README.md

@@ -0,0 +1,45 @@
+# pkg-sync
+
+A tool for syncing packages in an on-prem builder instance with the packages in public builder.
+
+Given a channel (defaults to `LTS-2024`) and an on-prem builder url and token, this tool can download all core, chef, or chef-platform packages in the channel from the piblic Chef Habitat builder that the on-prem instance does not already have and upload them to the on-prem builder.
+
+This performs a pre-flight check to ensure that you do not have local core, chef, or chef-platform packages in the channel that are not in the same channel on bldr.habitat.sh. If there are, these local packages must be demoted.
+
+This tool can also be used to generate a list of packages without actually syncing them.
+
+## Usage
+
+Install this package with:
+
+```
+hab pkg install habitat/pkg-sync
+```
+
+Examples:
+
+Sync all the latest core, chef, or chef-platform LTS-2024 packages that you do not already have from the public builder and upload them to your on-prem builder instance.
+
+```
+hab pkg exec habitat/pkg-sync pkg-sync --bldr-url https://your-builder.tld --channel LTS-2024 --auth <your_public_Builder_instance_token>
+```
+
+Generate a list of all LTS-2024 packages in the core, chef, and chef-platform origins.
+
+```
+hab pkg exec habitat/pkg-sync pkg-sync --generate-airgap-list --channel LTS-2024
+```
+
+These lists can be used with `hab pkg download` to download the hart files from builder:
+
+```
+hab pkg download --target x86_64-linux --channel LTS-2024 --file package_list_x86_64-linux.txt --download-directory builder_bootstrap
+hab pkg download --target x86_64-windows --channel LTS-2024 --file package_list_x86_64-windows.txt --download-directory builder_bootstrap
+```
+
+You could then copy the `--download-directory` contents to an airgapped builder and use `hab pkg bulkupload` to upload them:
+
+```
+export HAB_AUTH_TOKEN=<your_on-prem_Builder_instance_token>
+hab pkg bulkupload --url https://your-builder.tld --channel LTS-2024 --auto-create-origins builder_bootstrap/
+```