Add design for only restore volume data.

[blackpiglet]

Thank you for contributing to Velero!

Please add a summary of your change

Does your change fix a particular issue?

Fixes #(issue)
This is the design document for issue #7345.

Please indicate you've done the following:

• Accepted the DCO. Commits without the DCO will delay acceptance.
• Created a changelog file or added /kind changelog-not-required as a comment on this pull request.
• Updated the corresponding documentation in site/content/docs/main.

[draghuram]

@blackpiglet, Can you please rebase the PR? It is currently showing what appears to be many unrelated commits.

Few comments/questions:

• How are the new PVC names generated? I doesn't look like users can specify the target names so I guess Velero will rename them in a certain way?
• Since "pods" is not included in the resource type, I guess this restore is not supported for FSB?
• Does Snapshot data mover restore work for PVCs without specifying pods as resource type? I will verify it but it can work I guess because a mover pod is started?

I personally think that in-place restore is going to be more useful, especially if files can be selected for restore. If the whole PVC is being created any way, users will need to restart pods to attach the PVCs so it is almost like restoring pods themselves.

[anshulahuja98]

Can #6354 be used in a generic way to achieve this use case?
CC: @kaovilai

[kaovilai]

Can #6354 be used in a generic way to achieve this use case?
CC: @kaovilai

So restore limiting resources to a labeled pv (or namespaced PVC) with recreate flag?

[blackpiglet]

@anshulahuja98
If the in-place restore is required, the recreating Existing Resource Policy cannot do that.

[blackpiglet]

@draghuram
Thanks. I rebased the branch, but there is still some problem with GPRC generating.

First, could you share some thoughts about why the in-place restore is more useful? I recently also heard some requirements about the GitOps and DevOps pipeline scenario. In those cases, some tools guarantee that the workload in k8s always running as expected. The in-place restore is needed in those cases, but I don't know whether those are common cases for k8s usage.

Second, to answer your question.

• How are the new PVC names generated? I doesn't look like users can specify the target names so I guess Velero will rename them in a certain way? - User cannot specify the generated name. I haven't decided how to build the name yet. Maybe something like rename-pvc-<uuid>.
  Since "pods" is not included in the resource type, I guess this restore is not supported for FSB? - FSB restore is supported, because the Generate Backup/Restore will first create an intermediate pod and PVC to mount the PV. After the backup/restore, the intermediate pod and PVC are deleted.
  Does Snapshot data mover restore work for PVCs without specifying pods as resource type? I will verify it but it can work I guess because a mover pod is started? - Snapshot data mover can work without the pod, the reason is the same as the second item.

[kaovilai]

Heard from US community meeting that this will be moved into Goals.

Would have to discuss how we're handling volumes that are already mounted.. or do we say, you have to pre-delete/scaledown so volumes are not mounted by restore time.

[sseago]

@kaovilai For in-place, we can't scale down unless we create a dummy pod to mount the PVC, since there must be a mounting pod for kopia to have access to it from the node agent.

[kaovilai]

We already have dummy pod pattern from data mover work to follow

[kaovilai]

Does this work via CSI only? file system backup only? or both?

[kaovilai]

We could use the data mover like approach where we write to a new pvc using file system backup through a dummy pod.

[blackpiglet]

If we go with the in-place restore, only filesystem backup is supported.

[kaovilai]

An advanced CSI in-place could auto patch deployments to a new pvc restored from snapshots.

[kaovilai]

It should be possible to add labelSelector to all resources that user expect to be swapping to restored PVC name, and having velero do the swapping on labeled (core k8s only for now) resources in the namespace.

kubectl label deployment deployName velero.io/sync-restored-pvc-name=<original-pvc-in-backup-name>

Any PVC restored using this method with a new name will have annotation velero.io/restored-pvc-original-name=<original name> that velero can use as reference for subsequent backup/restore pvc name syncing.

[blackpiglet]

As we go with the in-place way, the name-mapping logic is unnecessary.

[kaovilai]

right.. for FSB with deleting PVC prior to restoring to the same name.

[draghuram]

@blackpiglet, gitops is certainly one reason for in place restore. Another use case is the need to periodically update data in an alternate standby cluster. Finally, if some application files are deleted or corrupted, user may only want to restore those files.

[blackpiglet]

@blackpiglet, gitops is certainly one reason for in place restore. Another use case is the need to periodically update data in an alternate standby cluster. Finally, if some application files are deleted or corrupted, user may only want to restore those files.

@draghuram
Thanks for the feedback.
If we go for the in-place data restore way, there will be some limitations to the using scenario. Only the filesystem uploader backup is supported. For the snapshot-based backup, Achieving the in-place restore result is impossible.

Is this acceptable? Data-only restore cannot support snapshot-based backups.
Involve more maintainers in this discussion.
@anshulahuja98 @kaovilai @sseago

[kaovilai]

FSB only is acceptable.

Tho I think restore from CSI snapshot to new PVC and then patching to use new PVC could be considered in-place with the caveat that it requires pod restart.

[blackpiglet]

However I think restore from CSI snapshot to new PVC and then patching to use new PVC could be considered in-place with the caveat that it requires pod restart.

Thanks for the quick response, then I will follow the filesystem-only way for now.
I got your point. There was a similar discussion in the Velero team. It does work, although it seems a bit inefficient. We can continue the discussion.
The idea of the filesystem data-only restore will also cause the pod to restart because the PodVolumeRestore way is preferred. That will introduce a new InitContainer in the pod. The advantage is that the pod's data operation will not interrupt the data restore process.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 61.71%. Comparing base (2518824) to head (156685a).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #7481   +/-   ##
=======================================
  Coverage   61.71%   61.71%           
=======================================
  Files         263      263           
  Lines       28869    28869           
=======================================
  Hits        17816    17816           
  Misses       9793     9793           
  Partials     1260     1260           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[anshulahuja98]

@blackpiglet, gitops is certainly one reason for in place restore. Another use case is the need to periodically update data in an alternate standby cluster. Finally, if some application files are deleted or corrupted, user may only want to restore those files.

@draghuram Thanks for the feedback. If we go for the in-place data restore way, there will be some limitations to the using scenario. Only the filesystem uploader backup is supported. For the snapshot-based backup, Achieving the in-place restore result is impossible.

Is this acceptable? Data-only restore cannot support snapshot-based backups. Involve more maintainers in this discussion. @anshulahuja98 @kaovilai @sseago

I went through further over the discussion
I understand a pure in-place restore can't be achieved without pod downtime.

With that caveat, I want to push for CSI snapshot based in-place restore. Here in-place restore refers mainly to Detaching PVC from workload, deleting PVC and re-creating PVC.

This is useful for Disaster recovery scenarios.

[blackpiglet]

With that caveat, I want to push for CSI snapshot based in-place restore. Here in-place restore refers mainly to Detaching PVC from workload, deleting PVC and re-creating PVC.

This proposal also cannot avoid pod downtime, so I suppose it aims to make the data-only restore can support more types of backups, right?

I understand a pure in-place restore can't be achieved without pod downtime.

If no downtime is a must-have feature, we can achieve that by ignoring the PodVolumeRestore's related pod InitContainer check logic, although that will compromise the data integrity, and data write could fail due to conflict.

[Lyndon-Li]

I just noticed that this design doesn't cover the consideration for WaitForFirstConsumer volumes, some discussion see here.
According to the design, we will create the PVC/PV for users.
If the volume to be restored is with WaitForFirstConsumer mode, there is no way to create the volume appropriately without having the information which possible nodes the pod (that is going to consume the restored volume) will be scheduled.
For data only restore, we cannot assume that users could create the pod beforehand, but users must know which nodes they want to schedule their pod if they want to make some constraints.
Therefore, we need to add a new parameter for users to deliver the candidate nodes for a specific volume. These is a must have parameter.