Skip to content

Commit

Permalink
Allow OSD nodes to quiesce after a deleting pool
Browse files Browse the repository at this point in the history 
When deleting a pool, it may take a while for the OSD nodes to delete the
objects in the pool. This change makes CBT wait until the OSD nodes quiesce
in order to ensure they are idle before starting the next test run.

Quiescing is done by waiting until the maximum disk utilization for any
disk falls below 3% across a 30 second window, and waiting until the maximum
CPU utilization for any ceph-osd process falls below 3%.

Closes ceph#117
  • Loading branch information
asbishop committed Sep 27, 2016
1 parent 637c430 commit c45a356
Showing 1 changed file with 11 additions and 0 deletions.
11 changes: 11 additions & 0 deletions cluster/ceph.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -562,6 +562,17 @@ def rmpool(self, name, profile_name):
common.pdsh(settings.getnodes('head'), 'sudo %s -c %s osd pool delete %s %s --yes-i-really-really-mean-it' % (self.ceph_cmd, self.tmp_conf, name, name),
continue_if_error=False).communicate()

logger.info('Waiting for OSD disk utilization to settle...')
disk_util_max = 3
window_size = 30
wait_cmd = 'while [ $(iostat -dxyz ALL %s 1 | awk \'BEGIN {m=0} {v=int($NF); if(v>m){m=v}} END {print m}\') -gt %s ]; do true; done' % (window_size, disk_util_max)
common.pdsh(settings.getnodes('osds'), wait_cmd).communicate()

logger.info('Waiting for OSD CPU utilization to settle...')
osd_cpu_max = 3
wait_cmd = 'while [ $(top -bn1 | awk \'$NF == "ceph-osd" {print int($9) ; exit}\') -gt %s ]; do sleep 5; done' % (osd_cpu_max)
common.pdsh(settings.getnodes('osds'), wait_cmd).communicate()

def rbd_unmount(self):
common.pdsh(settings.getnodes('clients'), 'sudo find /dev/rbd* -maxdepth 0 -type b -exec umount \'{}\' \;').communicate()
# common.pdsh(settings.getnodes('clients'), 'sudo find /dev/rbd* -maxdepth 0 -type b -exec rbd -c %s unmap \'{}\' \;' % self.tmp_conf).communicate()
Expand Down

2 comments on commit c45a356

@bengland2
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I do "iostat -dxyz ALL 3 1" I get nothing, so I don't see how this particular command works, but I'm using RHEL7.2, maybe it's some newer and sexier Linux distro ;-)

Linux 3.10.0-327.28.2.el7.x86_64 (c07-h01-6048r.rdu.openstack.engineering.redhat.com)   10/19/2016  _x86_64_    (56 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util

However, this command works on RHEL7.2:

(root@c07-h01-6048r) - (20:02) - (~)
-=>>iostat -dxyz /dev/sd[a-z] /dev/sd[a-z][a-z] 10 1
Linux 3.10.0-327.28.2.el7.x86_64 (c07-h01-6048r.rdu.openstack.engineering.redhat.com)   10/19/2016  _x86_64_    (56 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00     0.00    0.00    0.80     0.00     1.70     4.25     0.01   10.75    0.00   10.75   8.75   0.70
sdb               0.00     0.00    0.00    0.80     0.00     1.70     4.25     0.01   10.62    0.00   10.62   8.50   0.68

@bengland2
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This command too is problematic at large scale - any Ceph cluster that is doing scrubbing may take a long time to get to the point where this command exits:

 while [ $(top -bn1 | awk '$NF == "ceph-osd" {print int($9) ; exit}') -gt 3 ]; do sleep 3; done

I do not want to disable scrubbing on my cluster because I want to understand how it will perform in the real world, at least as a baseline. I'm watching ceph with "ceph -w" and it wasn't doing anything at all except scrubbing. So I guess I'd use a higher threshold than 5% of 1 core. Maybe 15%? When I deleted a pool, they all got real busy real fast.

Please sign in to comment.