docs: clarify volume binding modes

The comment about `allowedTopologies` was incorrect and referred to an
experiment that didn't work out. What we have to use instead are
volume binding modes and scheduler extensions.

(cherry picked from commit ce4ca6e)

docs/design.md

@@ -169,13 +169,25 @@ provisioning"](https://github.com/kubernetes-csi/external-provisioner/tree/v2.1.
   for first consumer"), a volume is tentatively assigned to a node
   before creating it, in which case the `external-provisioner` running
   on that node can tell that it is responsible for provisioning.
+- The scheduler extensions help the scheduler with picking nodes where
+  volumes can be created. Without them, the risk of choosing nodes
+  without PMEM may be too high and manual pod scheduling may be needed
+  to avoid long delays when starting pods. In the future with
+  Kubernetes >= 1.21, [storage capacity
+  tracking](https://kubernetes.io/docs/concepts/storage/storage-capacity/)
+  will be another solution for that problem.
 - For volumes with storage classes that use immediate binding, the
   different `external-provisioner` instances compete with each for
   ownership of the volume by setting the "selected node"
   annotation. Delays are used to avoid the thundering herd problem.
   Once a node has been selected, provisioning continues as with late
   binding. This is less efficient and therefore "late binding" is the
-  recommended binding mode.
+  recommended binding mode. The advantage is that this mode does not
+  depend on scheduler extensions to put pods onto nodes with PMEM
+  because once a volume has been created, the pod will automatically
+  run on the node of the volume. The downside is that a volume might
+  have been created on a node which has insufficient RAM and CPU
+  resources for a pod.
 
 PMEM-CSI also has a central component which implements the [scheduler
 extender](#scheduler-extender) webhook. That component needs to know

docs/install.md

@@ -58,15 +58,21 @@ installation by name, which indirectly determines the device mode. A
 storage class also chooses which filesystem is used (xfs or ext4) and
 enables [Kata Containers support](#kata-containers-support).
 
-It is recommended that storage classes use `allowedTopologies` as in
-the [`pmem-storageclass.yaml`](/deploy/kustomize/storageclass/pmem-storageclass.yaml)
-to ensure that pods with volumes that use late binding land on a node
-where the driver is available.
-
 Optionally, the administrator can enable [the scheduler
 extensions](#enable-scheduler-extensions) (recommended) and monitoring
 of resource usage via the [metrics support](#metrics-support).
 
+It is [recommended](./design.md#dynamic-provisioning-of-local-volumes)
+to enable the scheduler extensions in combination with
+`volumeBindingMode: WaitForFirstConsumer` as in the
+[`pmem-storageclass-late-binding.yaml`](/deploy/common/pmem-storageclass-late-binding.yaml)
+example. This ensures that pods get scheduled onto nodes that have
+sufficient RAM, CPU and PMEM. Without the scheduler extensions, it is
+random whether the scheduler picks a node that has PMEM available and
+immediate binding (the default volume binding mode) might work
+better. However, then pods might not be able to run when the node
+where volumes were created are overloaded.
+
 Optionally, the log output format can be changed from the default
 "text" format (= the traditional glog format) to "json" (= output via
 [zap](https://github.com/uber-go/zap/blob/master/README.md)) for easier