diff --git a/doc/dev/crimson/crimson.rst b/doc/dev/crimson/crimson.rst index f6d59a057ff85..f9582ec6c84b4 100644 --- a/doc/dev/crimson/crimson.rst +++ b/doc/dev/crimson/crimson.rst @@ -43,82 +43,6 @@ use a Crimson build: You'll likely need to supply the ``--allow-mismatched-release`` flag to use a non-release branch. -Configure Crimson with Bluestore -================================ - -As Bluestore is not a Crimson native `object store backend`_, -deploying Crimson with Bluestore as the back end requires setting -one of the two following configuration options: - -.. note:: - - #. These two options, along with ``crimson_alien_op_num_threads``, - can't be changed after deployment. - #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag. - - -1) ``crimson_seastar_num_threads`` - - In order to allow easier cluster deployments, this option can be used - instead of setting the CPU mask manually for each OSD. - - It's recommended to let the **number of OSDs on each host** multiplied by - ``crimson_seastar_num_threads`` to be less than the node's number of CPU - cores (``nproc``). - - For example, for deploying two nodes with eight CPU cores and two OSDs each: - - .. code-block:: yaml - - conf: - # Global to all OSDs - osd: - crimson seastar num threads: 3 - - .. note:: - - #. For optimal performance ``crimson_seastar_cpu_cores`` should be set instead. - -2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``. - - Explicitly set the CPU core allocation for each ``crimson-osd`` - and for the BlueStore back end. It's recommended for each set to be mutually exclusive. - - For example, for deploying two nodes with eight CPU cores and two OSDs each: - - .. code-block:: yaml - - conf: - # Both nodes - osd: - crimson alien thread cpu cores: 6-7 - - # First node - osd.0: - crimson seastar cpu cores: 0-2 - osd.1: - crimson seastar cpu cores: 3-5 - - # Second node - osd.2: - crimson seastar cpu cores: 0-2 - osd.3: - crimson seastar cpu cores: 3-5 - - For a single node with eight node and three OSDs: - - .. code-block:: yaml - - conf: - osd: - crimson alien thread cpu cores: 6-7 - osd.0: - crimson seastar cpu cores: 0-1 - osd.1: - crimson seastar cpu cores: 2-3 - osd.2: - crimson seastar cpu cores: 4-5 - Running Crimson =============== @@ -182,7 +106,7 @@ The following options can be used with ``vstart.sh``. (as determined by `nproc`) will be assigned to the object store. ``--bluestore`` - Use the alienized BlueStore as the object store backend. This is the default (see below section on the `object store backend`_ for more details) + Use alienized BlueStore as the object store backend. ``--cyanstore`` Use CyanStore as the object store backend. @@ -191,7 +115,7 @@ The following options can be used with ``vstart.sh``. Use the alienized MemStore as the object store backend. ``--seastore`` - Use SeaStore as the back end object store. + Use SeaStore as the back end object store. This is the default (see below section on the `object store backend`_ for more details) ``--seastore-devs`` Specify the block device used by SeaStore. @@ -207,20 +131,11 @@ The following options can be used with ``vstart.sh``. Valid types include ``HDD``, ``SSD``(default), ``ZNS``, and ``RANDOM_BLOCK_SSD`` Note secondary devices should not be faster than the main device. -To start a cluster with a single Crimson node, run:: - - $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh \ - --without-dashboard --bluestore --crimson \ - --redirect-output -Another SeaStore example:: +To start a simple cluster with a single core Crimson OSD, run:: - $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n -x \ - --without-dashboard --seastore \ - --crimson --redirect-output \ - --seastore-devs /dev/sda \ - --seastore-secondary-devs /dev/sdb \ - --seastore-secondary-devs-type HDD + $ MGR=1 MON=1 OSD=1 MDS=0 RGW=0 ../src/vstart.sh -n \ + --without-dashboard --seastore --crimson Stop this ``vstart`` cluster by running:: @@ -239,7 +154,7 @@ They are: .. describe:: seastore - Seastore is still under active development. + Seastore is the default Crimson backend and is still under active development. The alienized object store backends are backed by a thread pool, which is a proxy of the alienstore adaptor running in Seastar. The proxy issues @@ -254,6 +169,82 @@ managed by the Seastar framework. They are: The object store used by the classic ``ceph-osd`` +Configure Crimson with Bluestore +================================ + +As Bluestore is not a Crimson native `object store backend`_, +deploying Crimson with Bluestore as the back end requires setting +one of the two following configuration options: + +.. note:: + + #. These two options, along with ``crimson_alien_op_num_threads``, + can't be changed after deployment. + #. `vstart.sh`_ sets these options using the ``--crimson-smp`` flag. + + +1) ``crimson_seastar_num_threads`` + + In order to allow easier cluster deployments, this option can be used + instead of setting the CPU mask manually for each OSD. + + It's recommended to set the **number of OSDs on each host** multiplied by + ``crimson_seastar_num_threads`` to be less than the node's number of CPU + cores (``nproc``). + + For example, for deploying two nodes with eight CPU cores and two OSDs each: + + .. code-block:: yaml + + conf: + # Global to all OSDs + osd: + crimson seastar num threads: 3 + + .. note:: + + #. For optimal performance ``crimson_seastar_cpu_cores`` should be set instead. + +2) ``crimson_seastar_cpu_cores`` and ``crimson_alien_thread_cpu_cores``. + + Explicitly set the CPU core allocation for each ``crimson-osd`` + and for the BlueStore back end. It's recommended for each set to be mutually exclusive. + + For example, for deploying two nodes with eight CPU cores and two OSDs each: + + .. code-block:: yaml + + conf: + # Both nodes + osd: + crimson alien thread cpu cores: 6-7 + + # First node + osd.0: + crimson seastar cpu cores: 0-2 + osd.1: + crimson seastar cpu cores: 3-5 + + # Second node + osd.2: + crimson seastar cpu cores: 0-2 + osd.3: + crimson seastar cpu cores: 3-5 + + For a single node with eight node and three OSDs: + + .. code-block:: yaml + + conf: + osd: + crimson alien thread cpu cores: 6-7 + osd.0: + crimson seastar cpu cores: 0-1 + osd.1: + crimson seastar cpu cores: 2-3 + osd.2: + crimson seastar cpu cores: 4-5 + daemonize --------- diff --git a/qa/config/crimson_bluestore.yaml b/qa/config/crimson_bluestore.yaml new file mode 100644 index 0000000000000..d5ba487b9bfd2 --- /dev/null +++ b/qa/config/crimson_bluestore.yaml @@ -0,0 +1,25 @@ +overrides: + ceph: + fs: xfs + conf: + osd: + # crimson's osd objectstore option + crimson osd objectstore: bluestore + debug alienstore: 20 + bluestore block size: 96636764160 + debug bluestore: 20 + debug bluefs: 20 + debug rocksdb: 10 + bluestore compression mode: aggressive + bluestore fsck on mount: true + bluestore compression algorithm: snappy + # lower the full ratios since we can fill up a 100gb osd so quickly + mon osd full ratio: .9 + mon osd backfillfull_ratio: .85 + mon osd nearfull ratio: .8 + osd failsafe full ratio: .95 + bluestore rocksdb cf: false + log to stderr: true + err to stderr: true + log flush on exit: true + log to file: false diff --git a/qa/config/crimson_qa_overrides.yaml b/qa/config/crimson_qa_overrides.yaml index 8cf98f3800167..a10c59d77cc01 100644 --- a/qa/config/crimson_qa_overrides.yaml +++ b/qa/config/crimson_qa_overrides.yaml @@ -9,7 +9,6 @@ overrides: osd pool default crimson: true osd: crimson osd obc lru size: 10 - debug alienstore: 20 debug ms: 20 flavor: crimson workunit: diff --git a/qa/config/seastore.yaml b/qa/config/crimson_seastore.yaml similarity index 79% rename from qa/config/seastore.yaml rename to qa/config/crimson_seastore.yaml index 6158563eedfdb..d1919456ab1c7 100644 --- a/qa/config/seastore.yaml +++ b/qa/config/crimson_seastore.yaml @@ -1,13 +1,13 @@ overrides: ceph: - fs: xfs conf: osd: - osd objectstore: seastore + # crimson's osd objectstore option + crimson osd objectstore: seastore debug seastore: 20 debug seastore onode: 20 debug seastore odata: 20 - debug seastore ompap: 20 + debug seastore omap: 20 debug seastore tm: 20 debug seastore t: 20 debug seastore cleaner: 20 diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/crimson_seastore.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/crimson_seastore.yaml new file mode 120000 index 0000000000000..61e26e7acf824 --- /dev/null +++ b/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/crimson_seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml deleted file mode 120000 index 6a70c38170939..0000000000000 --- a/qa/suites/crimson-rados-experimental/seastore/basic/objectstore/seastore.yaml +++ /dev/null @@ -1 +0,0 @@ -.qa/config/seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml index e84f396e4b2b2..481e393be4a3e 120000 --- a/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml \ No newline at end of file +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml index 6a70c38170939..61e26e7acf824 120000 --- a/qa/suites/crimson-rados/basic/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/basic/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml \ No newline at end of file +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml index e84f396e4b2b2..481e393be4a3e 120000 --- a/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml \ No newline at end of file +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml index 6a70c38170939..61e26e7acf824 120000 --- a/qa/suites/crimson-rados/perf/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/perf/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml \ No newline at end of file +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml index e84f396e4b2b2..481e393be4a3e 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml \ No newline at end of file +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml index 6a70c38170939..61e26e7acf824 120000 --- a/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/rbd/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml \ No newline at end of file +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore b/qa/suites/crimson-rados/singleton/objectstore deleted file mode 120000 index dbccf5ad92894..0000000000000 --- a/qa/suites/crimson-rados/singleton/objectstore +++ /dev/null @@ -1 +0,0 @@ -../thrash/objectstore \ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/.qa b/qa/suites/crimson-rados/singleton/objectstore/.qa new file mode 120000 index 0000000000000..a602a0353e751 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/.qa @@ -0,0 +1 @@ +../.qa/ \ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml new file mode 120000 index 0000000000000..481e393be4a3e --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/bluestore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml new file mode 120000 index 0000000000000..61e26e7acf824 --- /dev/null +++ b/qa/suites/crimson-rados/singleton/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml index e84f396e4b2b2..481e393be4a3e 120000 --- a/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml \ No newline at end of file +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml new file mode 120000 index 0000000000000..61e26e7acf824 --- /dev/null +++ b/qa/suites/crimson-rados/thrash/objectstore/seastore.yaml @@ -0,0 +1 @@ +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml index e84f396e4b2b2..481e393be4a3e 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/bluestore.yaml @@ -1 +1 @@ -.qa/config/bluestore.yaml \ No newline at end of file +.qa/config/crimson_bluestore.yaml \ No newline at end of file diff --git a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml index 6a70c38170939..61e26e7acf824 120000 --- a/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml +++ b/qa/suites/crimson-rados/thrash_simple/objectstore/seastore.yaml @@ -1 +1 @@ -.qa/config/seastore.yaml \ No newline at end of file +.qa/config/crimson_seastore.yaml \ No newline at end of file diff --git a/src/common/options/crimson.yaml.in b/src/common/options/crimson.yaml.in index 69b3a61557688..abef7483bcfce 100644 --- a/src/common/options/crimson.yaml.in +++ b/src/common/options/crimson.yaml.in @@ -2,6 +2,17 @@ --- options: +- name: crimson_osd_objectstore + type: str + level: advanced + desc: backend type for a Crimson OSD (e.g seastore or bluestore) + default: seastore + enum_values: + - bluestore + - seastore + - cyanstore + flags: + - create - name: crimson_osd_obc_lru_size type: uint level: advanced diff --git a/src/crimson/osd/main.cc b/src/crimson/osd/main.cc index 7ed3dc0e3fe40..0bfd3e2266b04 100644 --- a/src/crimson/osd/main.cc +++ b/src/crimson/osd/main.cc @@ -202,7 +202,7 @@ int main(int argc, const char* argv[]) true); } auto store = crimson::os::FuturizedStore::create( - local_conf().get_val("osd_objectstore"), + local_conf().get_val("crimson_osd_objectstore"), local_conf().get_val("osd_data"), local_conf().get_config_values()); diff --git a/src/vstart.sh b/src/vstart.sh index 41a8310891bf6..b445e250a0006 100755 --- a/src/vstart.sh +++ b/src/vstart.sh @@ -963,7 +963,17 @@ $BLUESTORE_OPTS ; kstore kstore fsck on mount = true +EOF + if [ "$crimson" -eq 1 ]; then + wconf <