Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update explaining syncing and bi-directional syncing #329

Merged
merged 3 commits into from
Nov 6, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 45 additions & 10 deletions vcluster/_fragments/how-does-syncing-work.mdx
Original file line number Diff line number Diff line change
@@ -1,9 +1,37 @@
Syncing is the process where vCluster copies a resource between the virtual cluster and the host cluster. It's the basic principle vCluster uses to emulate a fully functional Kubernetes cluster.
Since a virtual cluster does not have actual worker nodes or a network, syncing is the process where vCluster copies a resource between the virtual cluster and the host cluster.
It's the basic principle that vCluster uses to emulate a fully functional Kubernetes cluster.

By default, vCluster only syncs low-level resources, such as pods, secrets, configmaps, or services, but a user can
enable many other resources for all use cases. vCluster control plane runs with a minimal set of RBAC permissions to allow execution in restricted environments. Certain resources require
extra permissions to sync, which are automatically given to the vCluster ServiceAccount if you enable a resource that requires additional
permissions.
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved

vCluster only syncs low-level resources by default, such as pods, secrets, configmaps or services. As soon as one of these resources are created in the virtual cluster, vCluster will sync it to the other side. As soon as a resource was copied by vCluster, it will start watching for changes to the resource on either side, so when a controller changes something on the host cluster side, the vCluster will sync that change to the virtual cluster or when a change on the virtual cluster side happens, it will sync it to the host cluster, to make sure the resource is in sync.
All fields of a resources are synced from one direction to the other based on what section the resource is part of in the `vcluster.yaml`. There
are a few exceptions for each resource where specific fields are kept in sync in both directions (i.e. bi-directional sync).
As soon as a resource is copied by vCluster, the syncer starts watching for changes to the resource on either side. When the resource
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
is updated in the origin cluster, the syncer updates and copies those changes in the destination cluster.

When a resource is defined in the `toHost` section, as soon as that resource is created in the virtual cluster, vCluster copies it to the host cluster.
Any changes to the resource in the virtual cluster are automatically applied to the resource in the host cluster.
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved

When a resource is defined in the `fromHost` section, as soon as that resource is created in the host cluster, vCluster copies it to the virtual cluster.
Any changes to the resource in the host cluster are automatically applied to the resource in the virtual cluster.


When syncing resources, vCluster stores the mapping of the `VIRTUAL NAME <-> HOST NAME` for the actual resource, and the references of the other resources used by this resource.
Tracking references helps the vCluster to understand what resources are still needed in the host cluster. This important for secrets and configmaps because by default only the secrets and configmaps used are actually synced to the host cluster.
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved

Assume you've started your vCluster with ingress syncing to the host enabled.

``` vcluster.yaml
sync:
toHost:
ingresses:
enabled: true
```

After the virtual cluster is up, the user creates an ingress in the virtual cluster.

Let's take `Ingress` as an example. A user that creates the following ingress in the virtual cluster:
```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
Expand All @@ -26,7 +54,14 @@ spec:
number: 80
```

When copying to the host cluster, the vCluster will change the ingress to:
When vCluster copies the ingress to the host cluster, the vCluster creates an ingress in the namespace of the virtual cluster with the following changes:

* **Name Rewritten**: vCluster rewrites the name of the resource to avoid conflicts in the host cluster. The name is rewritten in the form of `NAME-x-NAMESPACE-x-VCLUSTER_NAME`
* **Namespace Rewritten**: vCluster rewrites the namespace of the resource to sync everything into the namespace of the deployed virtual cluster control plane.
* **Annotations / Labels Added**: vCluster adds annotations and labels to identify the resource in the host cluster, which are pre-fixed with `vcluster.loft.sh/`.
* **References Rewritten**: vCluster rewrites the `spec.rules[*].http.paths[*].backend.service.name` and `spec.tls[*].secretName` to match the rewritten names of the resources in the host cluster.


```yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
Expand Down Expand Up @@ -60,11 +95,11 @@ spec:
secretName: testsecret-tls-x-default-x-vcluster
```

There is a couple of important changes to be noticed:
* **Name Rewritten**: vCluster rewrites the name of the resource to avoid conflicts in the host cluster. The name is rewritten in the form of `NAME-x-NAMESPACE-x-VCLUSTER_NAME`
* **Namespace Rewritten**: vCluster rewrites the namespace of the resource to sync everything into the vcluster's namespace.
* **Annotations / Labels Added**: vCluster adds annotations and labels to identify the resource in the host cluster.
* **References Rewritten**: vCluster rewrites the `spec.rules[*].http.paths[*].backend.service.name` and `spec.tls[*].secretName` to match the rewritten names of the resources in the host cluster.
### Syncing labels to the host cluster

vCluster will also keep track from now of the resource within its internal name cache, that tracks the `VIRTUAL NAME <-> HOST NAME` as well as all references used by this resource. In the ingress case, this would be the service `nginx` and the secret `testsecret-tls`. Tracking references helps the vCluster to understand what resources are still needed in the host cluster. For secrets and configmaps this is important, because by default only the needed ones are actually synced to the host cluster.
When syncing labels to the host cluster, vCluster rewrites the following labels to avoid conflicts:

* `release` (Label is needed to avoid conflicts with the vCluster pods themselves). This is only excluded in single-namespace mode as multi-namespace mode does sync all pods into separate namespaces.
* `vcluster.loft.sh/namespace`
* `vcluster.loft.sh/managed-by`
* `vcluster.loft.sh/controlled-by`
25 changes: 21 additions & 4 deletions vcluster/_fragments/sync-from-host-resources.mdx
Original file line number Diff line number Diff line change
@@ -1,17 +1,34 @@
vCluster can sync certain resources purely from host to make them available inside the vCluster. This is useful if you only want to view certain resources in read-only mode, but not want the users inside the vCluster to change them.
vCluster can sync certain resources from the host cluster to make them available inside the virtual cluster, but when these resources are
synced, they are only synced in read-only mode. No changes to the resource in the virtual cluster syncs back to the host cluster as the
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
resources are shared across the host cluster.

A good example would be nodes, which are nice to view inside the vCluster and can be also used to enabled certain features such as [scheduling inside the vCluster](/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.mdx), but you wouldn't want your vCluster to change the node itself. Another benefit of purely syncing from host is that vCluster itself will only require read-only RBAC permissions.
A good example would be nodes, which are nice to view inside the virtual cluster and can be also used to enabled certain features such as [scheduling inside the vCluster](/vcluster/configure/vcluster-yaml/control-plane/other/advanced/virtual-scheduler.mdx),
but you wouldn't want your virtual cluster to change the node itself. Another benefit of only syncing from host is that vCluster itself only requires read-only RBAC permissions.

vCluster also allows to sync custom resources via the [custom resource definitions syncer](../configure/vcluster-yaml/sync/from-host/custom-resources.mdx)

### Enabled by Default
There are a couple of labels that are created on the host cluster by vCluster that never get synced to the virtual cluster resource. These labels are:
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved

* `release` (Label is needed to avoid conflicts with the vCluster pods themselves). This is only excluded in single-namespace mode as multi-namespace mode does sync all pods into separate namespaces.
* `vcluster.loft.sh/namespace`
* `vcluster.loft.sh/managed-by`
* `vcluster.loft.sh/controlled-by`

### Enabled by default

* [Events](../configure/vcluster-yaml/sync/from-host/events.mdx)

### Disabled by Default
### Disabled by default
* [CustomResources](../configure/vcluster-yaml/sync/from-host/custom-resources.mdx)
* [Nodes](../configure/vcluster-yaml/sync/from-host/nodes.mdx)
* [IngressClasses](../configure/vcluster-yaml/sync/from-host/ingress-classes.mdx)
* [StorageClasses](../configure/vcluster-yaml/sync/from-host/storage-classes.mdx)
* [CSINodes](../configure/vcluster-yaml/sync/from-host/csi-nodes.mdx)
* [CSIDrivers](../configure/vcluster-yaml/sync/from-host/csi-drivers.mdx)
* [CSIStorageCapacities](../configure/vcluster-yaml/sync/from-host/csi-storage-capacities.mdx)

### No bi-directional syncing

Since syncing resources from the host cluster is in read only mode and changes in the virtual cluster do not
get applied to the resource in the host cluster, bi-directional syncing does not exist across these resources.

36 changes: 31 additions & 5 deletions vcluster/_fragments/sync-to-host-resources.mdx
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
vCluster can sync certain resources from the virtual cluster to the host cluster and back. These resources are typically created within the virtual cluster and are expected to be changeable within the virtual cluster. These are typically namespace-scoped resources (although exceptions exist) such as Pods, Secrets, ConfigMaps, Services etc.
vCluster syncs specific Kubernetes resources from the virtual cluster to the host cluster. vCluster
expects these resources to be created in the virtual cluster and updated in the virtual cluster. Besides native
Kubernetes resources, custom resources can also be synced.

vCluster also allows to sync custom resources via the [custom resources syncer](../configure/vcluster-yaml/sync/to-host/advanced/custom-resources.mdx)

### Enabled by Default
### Enabled by default
* [Pods](../configure/vcluster-yaml/sync/to-host/core/pods.mdx)
* [Secrets](../configure/vcluster-yaml/sync/to-host/core/secrets.mdx)
* [ConfigMaps](../configure/vcluster-yaml/sync/to-host/core/config-maps.mdx)
* [Services](../configure/vcluster-yaml/sync/to-host/networking/services.mdx)
* [Endpoints](../configure/vcluster-yaml/sync/to-host/networking/endpoints.mdx)
* [PersistentVolumeClaims](../configure/vcluster-yaml/sync/to-host/storage/persistent-volume-claims.mdx)

### Disabled by Default
### Disabled by default
* [CustomResources](../configure/vcluster-yaml/sync/to-host/advanced/custom-resources.mdx)
* [Ingresses](../configure/vcluster-yaml/sync/to-host/networking/ingresses.mdx)
* [NetworkPolicies](../configure/vcluster-yaml/sync/to-host/networking/network-policies.mdx)
Expand All @@ -20,3 +20,29 @@ vCluster also allows to sync custom resources via the [custom resources syncer](
* [ServiceAccounts](../configure/vcluster-yaml/sync/to-host/advanced/service-accounts.mdx)
* [PodDisruptionBudgets](../configure/vcluster-yaml/sync/to-host/advanced/pod-disruption-budgets.mdx)
* [PriorityClasses](../configure/vcluster-yaml/sync/to-host/advanced/priority-classes.mdx)
* [Custom Resources](../configure/vcluster-yaml/sync/to-host/advanced/custom-resources.mdx)

### Bi-directional syncing

For all resources synced from the virtual cluster to the host cluster, metadata labels and metadata annotations are synced in both directions. This means that not only changes in the
virtual cluster are reflected in the host cluster, but changes in host cluster are reflected in the virtual cluster.

:::warning vCluster specific Labels
Any labels pre-fixed with `vcluster.loft.sh` on the host resources should not be updated by the user. These labels are used by the vCluster control plane to manage the resources and to avoid conflicts.
:::

#### Additional Fields that support bi-directionally sync

All `metadata.labels` and `metadata.annotations` support bi-directional sync for any resources being synced to the host cluster.
In addition to those fields, there are several other fields that vCluster supports bi-directional sync.

| Resource | Fields |
| --- | --- |
| Pods | <ul><li> status.conditions </li></ul> |
| Secrets | <ul><li> spec.data </li><li> type </li></ul> |
| ConfigMaps | <ul><li> spec.data </li><li> spec.binaryData </li></ul> |
| Services | <ul><li> spec.externalIPs </li><li> spec.loadBalancerIP </li><li> spec.ports </li><li> spec.publishNotReadyAddresses </li><li> spec.type </li><li> spec.externalName </li><li> spec.externalTrafficPolicy </li><li> spec.sessionAffinity </li><li> spec.sessionAffinityConfig </li><li> spec.loadBalancerSourceRanges </li><li> spec.healthCheckNodePort </li><li> spec.selector </li></ul>
| Ingresses | <ul><li> spec.ingressClassName </li></ul> |
| PersistentVolumes | <ul><li>spec.persistentVolumeSource </li><li>spec.capacity </li><li> spec.accessModes </li><li> spec.persistentVolumeReclaimPolicy </li><li> spec.nodeAffinity </li><li> spec.volumeMode </li><li> spec.mountOptions </li></ul> |
| StorageClass | <ul><li> spec.provisioner </li><li> spec.parameters</li><li> spec.reclaimPolicy </li><li> spec.mountOptions </li><li> spec.allowVolumeExpansion </li><li> spec.volumeBindingMode </li><li> spec.allowedTopologies </li></ul> |

33 changes: 6 additions & 27 deletions vcluster/configure/vcluster-yaml/sync/README.mdx
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
deniseschannon marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -6,40 +6,19 @@ description: Configure syncing.
---

import Sync from '../../../_partials/config/sync.mdx'
import SyncToHostResources from '@site/vcluster/_fragments/sync-to-host-resources.mdx'
import SyncFromHostResources from '@site/vcluster/_fragments/sync-from-host-resources.mdx'
import SyncToHostResources from '../../../_fragments/sync-to-host-resources.mdx'
import SyncFromHostResources from '../../../_fragments/sync-from-host-resources.mdx'
import HowDoesSyncingWork from '../../../_fragments/how-does-syncing-work.mdx'

{/*
Links to previous docs:
https://www.vcluster.com/docs/architecture/syncer/

TODO: Determine if we should remove this from the new docs since it's not multi-namespace mode has additional management overhead for users
https://www.vcluster.com/docs/architecture/syncer/single_vs_multins
*/}

A virtual cluster does not have actual worker nodes or a network. Instead, the vCluster syncer component replicates the pods that are created within the virtual cluster to the host cluster. Subsequently, the host cluster schedules the pods, and the syncer keeps the virtual cluster pods and host cluster pods in sync.

By default, vCluster runs with a minimal set of RBAC permissions to allow execution in restricted environments. Certain resources require extra permissions to sync, which will automatically be given to the vCluster ServiceAccount if you enable the resource sync with the associated config value.

## Sync from Virtual Cluster to Host Cluster
<SyncToHostResources/>

## Sync from Host Cluster to Virtual Cluster
<SyncFromHostResources/>


## How does syncing work?

<HowDoesSyncingWork />

## Label Syncing
## Resources available to sync from virtual cluster to host cluster
<SyncToHostResources/>

When syncing metadata, vCluster will rewrite the following labels to avoid conflicts:
* `release` (Label is needed to avoid conflicts with the vCluster pods themselves). This is only excluded in single-namespace mode as multi-namespace mode does sync all pods into separate namespaces.
* `vcluster.loft.sh/namespace`
* `vcluster.loft.sh/managed-by`
* `vcluster.loft.sh/controlled-by`
## Resources available to sync from host cluster to virtual cluster
<SyncFromHostResources/>

## Config reference

Expand Down
14 changes: 3 additions & 11 deletions vcluster/configure/vcluster-yaml/sync/from-host/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,19 +6,11 @@ description: Configuration for ...
---

import SyncFromHost from '../../../../_partials/config/sync/fromHost.mdx'
import SyncFromHostResources from '@site/vcluster/_fragments/sync-from-host-resources.mdx'
import SyncFromHostResources from '../../../../_fragments/sync-from-host-resources.mdx'


Declare which host resources you want to sync to the virtual cluster.

{/*
This page covers all synced resources, but does not document which direction the syncing occurs:
https://www.vcluster.com/docs/syncer/core_resources

General instructions for enabling / disabling synced resources:
https://www.vcluster.com/docs/syncer/config#enable-or-disable-synced-resources
*/}

Read more about [how syncing works](../) before deciding which resources to
sync from the host cluster to the virtual cluster.

<SyncFromHostResources/>

Expand Down
13 changes: 3 additions & 10 deletions vcluster/configure/vcluster-yaml/sync/to-host/README.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,10 @@ description: Configuration for ...
---

import SyncToHost from '../../../../_partials/config/sync/toHost.mdx'
import SyncToHostResources from '@site/vcluster/_fragments/sync-to-host-resources.mdx'
import SyncToHostResources from '../../../../_fragments/sync-to-host-resources.mdx'

Declare which virtual cluster resources you want to sync to the host cluster.

{/*
This page covers all synced resources, but does not document which direction the syncing occurs:
https://www.vcluster.com/docs/syncer/core_resources

General instructions for enabling / disabling synced resources:
https://www.vcluster.com/docs/syncer/config#enable-or-disable-synced-resources
*/}
Read more about [how syncing works](../) before deciding which resources to
sync from the virtual cluster to the host cluster.

<SyncToHostResources/>

Expand Down