PNDA-4780 CDH: orchestrate-pnda-knox fails on centos in openstack #582
Conversation
pillar/hadoop/CDH.sls
Outdated
| - '"/var/log/pnda/hue/*.log"' | ||
| hadoop_service_name: |
There was a problem hiding this comment.
It's unusual to specify things in the Hadoop section but only specify them for one Hadoop version. Please look at how these pillar sections are used elsewhere in the code.
If you need a generic service name abstractor for Hadoop, that's fine - but please use it consistently, I still see e.g. 'oozie01' in the code even though this abstraction has been placed in the pillar.
There was a problem hiding this comment.
working for the New fix will update once it's completed.
salt/_modules/pnda.py
Outdated
| @@ -150,6 +150,7 @@ def cloudera_get_hosts_by_hadoop_role(service, role_type): | |||
| password = hadoop_manager_password() | |||
| endpoint = hadoop_manager_ip() + ':7180' | |||
| cluster = cluster_name() | |||
| service = __salt__['pillar.get']('hadoop_service_name:'+service) | |||
There was a problem hiding this comment.
Please see above comment about pillar usage
salt/knox/init.sls
Outdated
| {% set mr2_history_server_host = salt['pnda.get_hosts_by_hadoop_role']('MAPREDUCE2', 'HISTORYSERVER')[0] %} | ||
| {% set spark_history_server_host = salt['pnda.get_hosts_by_hadoop_role']('SPARK', 'SPARK_JOBHISTORYSERVER')[0] %} | ||
| {% set ambari_server_host = salt['pnda.get_hosts_for_role']('hadoop_manager')[0] %} | ||
| {% set hadoop_manager_host = salt['pnda.get_hosts_for_role']('hadoop_manager')[0] %} |
There was a problem hiding this comment.
This change is incorrect. This isn't a generic Hadoop manager, it's Ambari. The knox config for Ambari will not work for anything else.
salt/knox/init.sls
Outdated
| @@ -26,6 +23,23 @@ | |||
| {% set opentsdb_version = pillar['opentsdb']['version'] %} | |||
| {% set helper_directory = knox_home_directory + '/helper' %} | |||
|
|
|||
| {%- if grains['hadoop.distro'] == 'CDH' -%} | |||
There was a problem hiding this comment.
Please see above comment about Hadoop usage and avoid replication like this, it leads to bugs (see DRY)
|
Updated the changes, |
Issue: Knox deployment failed in CDH
Solution:
Added Hadoop manager based condition to handle both CDH and HDP.