Merge pull request ceph#60821 from Matan-B/wip-matanb-crimson-seastor…

…e-deafult

crimson: switch to SeaStore as default object store

Reviewed-by: Samuel Just <[email protected]>
Reviewed-by: Yingxin Cheng <[email protected]>

doc/dev/crimson/crimson.rst

@@ -43,82 +43,6 @@ use a Crimson build:
 You'll likely need to supply the ``--allow-mismatched-release`` flag to
 use a non-release branch.
 
-Configure Crimson with Bluestore
-================================
-
-As Bluestore is not a Crimson native `object store backend`_,
-deploying Crimson with Bluestore as the back end requires setting
-one of the two following configuration options:
-
-.. note::
-
-   #. These two options, along with ``crimson_alien_op_num_threads``,
-      can't be changed after deployment.
-   #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag.
-
-
-1) ``crimson_seastar_num_threads``
-
-   In order to allow easier cluster deployments, this option can be used
-   instead of setting the CPU mask manually for each OSD.
-
-   It's recommended to let the **number of OSDs on each host** multiplied by
-   ``crimson_seastar_num_threads`` to be less than the node's number of CPU
-   cores (``nproc``).
-
-   For example, for deploying two nodes with eight CPU cores and two OSDs each:
-
-   .. code-block:: yaml
-
-      conf:
-        # Global to all OSDs
-        osd:
-          crimson seastar num threads: 3
-
-   .. note::
-
-      #. For optimal performance ``crimson_seastar_cpu_cores`` should be set instead.
-
-2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``.
-
-   Explicitly set the CPU core allocation for each ``crimson-osd``
-   and for the BlueStore back end. It's recommended for each set to be mutually exclusive.
-
-   For example, for deploying two nodes with eight CPU cores and two OSDs each:
-
-   .. code-block:: yaml
-
-      conf:
-        # Both nodes
-        osd:
-          crimson alien thread cpu cores: 6-7
-
-        # First node
-        osd.0:
-          crimson seastar cpu cores: 0-2
-        osd.1:
-          crimson seastar cpu cores: 3-5
-
-        # Second node
-        osd.2:
-          crimson seastar cpu cores: 0-2
-        osd.3:
-          crimson seastar cpu cores: 3-5
-
-   For a single node with eight node and three OSDs:
-
-   .. code-block:: yaml
-
-        conf:
-          osd:
-            crimson alien thread cpu cores: 6-7
-          osd.0:
-            crimson seastar cpu cores: 0-1
-          osd.1:
-            crimson seastar cpu cores: 2-3
-          osd.2:
-            crimson seastar cpu cores: 4-5
-
 Running Crimson
 ===============
 
@@ -182,7 +106,7 @@ The following options can be used with ``vstart.sh``.
     (as determined by `nproc`) will be assigned to the object store.
 
 ``--bluestore``
-    Use the alienized BlueStore as the object store backend. This is the default (see below section on the `object store backend`_ for more details)
+    Use alienized BlueStore as the object store backend.
 
 ``--cyanstore``
     Use CyanStore as the object store backend.
@@ -191,7 +115,7 @@ The following options can be used with ``vstart.sh``.
     Use the alienized MemStore as the object store backend.
 
 ``--seastore``
-    Use SeaStore as the back end object store.
+    Use SeaStore as the back end object store. This is the default (see below section on the `object store backend`_ for more details)
 
 ``--seastore-devs``
     Specify the block device used by SeaStore.
@@ -207,20 +131,11 @@ The following options can be used with ``vstart.sh``.
     Valid types include ``HDD``, ``SSD``(default), ``ZNS``, and ``RANDOM_BLOCK_SSD``
     Note secondary devices should not be faster than the main device.
 
-To start a cluster with a single Crimson node, run::
-
-  $  MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh \
-    --without-dashboard --bluestore --crimson \
-    --redirect-output
 
-Another SeaStore example::
+To start a simple cluster with a single core Crimson OSD, run::
 
-  $  MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x \
-    --without-dashboard --seastore \
-    --crimson --redirect-output \
-    --seastore-devs /dev/sda \
-    --seastore-secondary-devs /dev/sdb \
-    --seastore-secondary-devs-type HDD
+  $  MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n \
+    --without-dashboard --seastore --crimson
 
 Stop this ``vstart`` cluster by running::
 
@@ -239,7 +154,7 @@ They are:
 
 .. describe:: seastore
 
-   Seastore is still under active development.
+   Seastore is the default Crimson backend and is still under active development.
 
 The alienized object store backends are backed by a thread pool, which
 is a proxy of the alienstore adaptor running in Seastar. The proxy issues
@@ -254,6 +169,82 @@ managed by the Seastar framework. They are:
 
    The object store used by the classic ``ceph-osd``
 
+Configure Crimson with Bluestore
+================================
+
+As Bluestore is not a Crimson native `object store backend`_,
+deploying Crimson with Bluestore as the back end requires setting
+one of the two following configuration options:
+
+.. note::
+
+   #. These two options, along with ``crimson_alien_op_num_threads``,
+      can't be changed after deployment.
+   #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag.
+
+
+1) ``crimson_seastar_num_threads``
+
+   In order to allow easier cluster deployments, this option can be used
+   instead of setting the CPU mask manually for each OSD.
+
+   It's recommended to set the **number of OSDs on each host** multiplied by
+   ``crimson_seastar_num_threads`` to be less than the node's number of CPU
+   cores (``nproc``).
+
+   For example, for deploying two nodes with eight CPU cores and two OSDs each:
+
+   .. code-block:: yaml
+
+      conf:
+        # Global to all OSDs
+        osd:
+          crimson seastar num threads: 3
+
+   .. note::
+
+      #. For optimal performance ``crimson_seastar_cpu_cores`` should be set instead.
+
+2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``.
+
+   Explicitly set the CPU core allocation for each ``crimson-osd``
+   and for the BlueStore back end. It's recommended for each set to be mutually exclusive.
+
+   For example, for deploying two nodes with eight CPU cores and two OSDs each:
+
+   .. code-block:: yaml
+
+      conf:
+        # Both nodes
+        osd:
+          crimson alien thread cpu cores: 6-7
+
+        # First node
+        osd.0:
+          crimson seastar cpu cores: 0-2
+        osd.1:
+          crimson seastar cpu cores: 3-5
+
+        # Second node
+        osd.2:
+          crimson seastar cpu cores: 0-2
+        osd.3:
+          crimson seastar cpu cores: 3-5
+
+   For a single node with eight node and three OSDs:
+
+   .. code-block:: yaml
+
+        conf:
+          osd:
+            crimson alien thread cpu cores: 6-7
+          osd.0:
+            crimson seastar cpu cores: 0-1
+          osd.1:
+            crimson seastar cpu cores: 2-3
+          osd.2:
+            crimson seastar cpu cores: 4-5
+
 daemonize
 ---------
 

qa/config/crimson_bluestore.yaml

@@ -0,0 +1,25 @@
+overrides:
+  ceph:
+    fs: xfs
+    conf:
+      osd:
+        # crimson's osd objectstore option
+        crimson osd objectstore: bluestore
+        debug alienstore: 20
+        bluestore block size: 96636764160
+        debug bluestore: 20
+        debug bluefs: 20
+        debug rocksdb: 10
+        bluestore compression mode: aggressive
+        bluestore fsck on mount: true
+        bluestore compression algorithm: snappy
+        # lower the full ratios since we can fill up a 100gb osd so quickly
+        mon osd full ratio: .9
+        mon osd backfillfull_ratio: .85
+        mon osd nearfull ratio: .8
+        osd failsafe full ratio: .95
+        bluestore rocksdb cf: false
+        log to stderr: true
+        err to stderr: true
+        log flush on exit: true
+        log to file: false

qa/config/crimson_qa_overrides.yaml

@@ -9,7 +9,6 @@ overrides:
         osd pool default crimson: true
       osd:
         crimson osd obc lru size: 10
-        debug alienstore: 20
         debug ms: 20
     flavor: crimson
   workunit:

qa/config/seastore.yaml → qa/config/crimson_seastore.yaml

@@ -1,13 +1,13 @@
 overrides:
   ceph:
-    fs: xfs
     conf:
       osd:
-        osd objectstore: seastore
+        # crimson's osd objectstore option
+        crimson osd objectstore: seastore
         debug seastore: 20
         debug seastore onode: 20
         debug seastore odata: 20
-        debug seastore ompap: 20
+        debug seastore omap: 20
         debug seastore tm: 20
         debug seastore t: 20
         debug seastore cleaner: 20

qa/suites/crimson-rados-experimental/seastore/basic/objectstore/crimson_seastore.yaml

@@ -0,0 +1 @@
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/basic/objectstore/bluestore.yaml

@@ -1 +1 @@
-.qa/config/bluestore.yaml
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/basic/objectstore/seastore.yaml

@@ -1 +1 @@
-.qa/config/seastore.yaml
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/perf/objectstore/bluestore.yaml

@@ -1 +1 @@
-.qa/config/bluestore.yaml
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/perf/objectstore/seastore.yaml

@@ -1 +1 @@
-.qa/config/seastore.yaml
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml

@@ -1 +1 @@
-.qa/config/bluestore.yaml
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/rbd/objectstore/seastore.yaml

@@ -1 +1 @@
-.qa/config/seastore.yaml
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/singleton/objectstore/.qa

@@ -0,0 +1 @@
+../.qa/

qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml

@@ -0,0 +1 @@
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/singleton/objectstore/seastore.yaml

@@ -0,0 +1 @@
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml

@@ -1 +1 @@
-.qa/config/bluestore.yaml
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/thrash/objectstore/seastore.yaml

@@ -0,0 +1 @@
+.qa/config/crimson_seastore.yaml

qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml

@@ -1 +1 @@
-.qa/config/bluestore.yaml
+.qa/config/crimson_bluestore.yaml

qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml

@@ -1 +1 @@
-.qa/config/seastore.yaml
+.qa/config/crimson_seastore.yaml

src/common/options/crimson.yaml.in

@@ -2,6 +2,17 @@
 ---
 
 options:
+- name: crimson_osd_objectstore
+  type: str
+  level: advanced
+  desc: backend type for a Crimson OSD (e.g seastore or bluestore)
+  default: seastore
+  enum_values:
+  - bluestore
+  - seastore
+  - cyanstore
+  flags:
+  - create
 - name: crimson_osd_obc_lru_size
   type: uint
   level: advanced

src/crimson/osd/main.cc

@@ -202,7 +202,7 @@ int main(int argc, const char* argv[])
                                                    true);
           }
           auto store = crimson::os::FuturizedStore::create(
-            local_conf().get_val<std::string>("osd_objectstore"),
+            local_conf().get_val<std::string>("crimson_osd_objectstore"),
             local_conf().get_val<std::string>("osd_data"),
             local_conf().get_config_values());