No result from subprocess.check_output(...)

[xavier-calland]

The commands like "subprocess.check_output(['ceph', 'df', '-f', 'json'])" return a empty string when it's called from collectd.
Other commands in a collectd python plugin are working (like 'ls' or 'pwd').
A simple python script executing the same command ("subprocess.check_output(['ceph', 'df', '-f', 'json'])") works well (out of collectd).
Where is the incompatibility between python/collectd/ceph (user rights, subprocess, ...) ?
Any suggestions ?

OS: ubuntu 1204
Collectd: 5.1.0
Python: 2.7.3

[rochaporto]

Hi.

Can you try with this packaged collectd?
https://launchpad.net/~catalystit/+archive/ubuntu/collectd5-precise

It's a build for ubuntu precise which we're using in the production boxes.

[xavier-calland]

Thx.
I'll try to upgrade collectd.

[xavier-calland]

Still the same problem.
How is your ceph cluster configured ? authentication, client keyring ?

[xavier-calland]

It seems python failed loading some SO libraries using "ldconfig -p" (_findSoname_ldconfig)

Traceback (most recent call last):
  File "/usr/bin/ceph", line 830, in 
    sys.exit(main())
  File "/usr/bin/ceph", line 590, in main
    conffile=conffile)#012  File "/usr/lib/python2.7/dist-packages/rados.py", line 198, in __init__
    librados_path = find_library('rados')
  File "/usr/lib/python2.7/ctypes/util.py", line 224, in find_library
    return _findSoname_ldconfig(name) or _get_soname(_findLib_gcc(name))
  File "/usr/lib/python2.7/ctypes/util.py", line 217, in _findSoname_ldconfig
    f.close()
IOError: [Errno 10] No child processes

[rochaporto]

Hmm that looks weird it seems to fail to find the rados libraries. I'm using it with python 3.4, ceph authenticated (cephx keyrings). When you execute it in a python script, is it also python 2.7?

[xavier-calland]

Yes, the following script works with python 2.7

import subprocess
stats_output = subprocess.check_output('ceph osd pool stats -f json', shell=True)
print "{}".format(stats_output)

[xavier-calland]

Replacing

f = os.popen('/sbin/ldconfig -p 2>/dev/null')

by

f = subprocess.Popen('/sbin/ldconfig -p', shell=True, stdout=subprocess.PIPE).stdout

seems to fix the problem.
But I'm still wondering why it only fails in collectd.

[xavier-calland]

Definitely a problem with python and collectd.
Thx for your help.

[Babbleshack]

I am experiencing the same issue, did you ever find out why this was happening?

[ksingh7]

@xavier-calland @rochaporto

This issue is slightly similar to what i am facing #31 , in my case we are using python2.6.

Do you know some hack , that i can use to make this plugin work against my python2.6 Ceph nodes ?

[Babbleshack]

I ended up with
subprocess.check_output('ceph -s --format json', shell=True)

The python docs mention some vulnerabilities with shell=True, but only in circumstances where the script user would be prompt for the shell commands.

[benh57]

Same problem here, hmm. And i have shell=True on, on python 2.7 with Centos 7.2. Also with ceph-collectd plugin after updating collectd to 5.6.

[benh57]

Ah yes, i recall the other fix for this:

#28

Just have to 'Import sigchld' in the plugin config. My fix had gotten reverted.