Skip to content

Latest commit

 

History

History
174 lines (146 loc) · 7.41 KB

README.md

File metadata and controls

174 lines (146 loc) · 7.41 KB
Raw

init_ec2

This project helps users to initialize a new cluster.

For example, you can watch a use case vedio on Youtube: https://youtu.be/OvVaib5xllo

It is suitable for:

  • a real cluster, which consists of PCs, small servers or other physical machines you own.
  • cloud environment, such as Amazon EC2, aliyun.

The only conditions you should have are:

  • you have installed ubuntu (tested on 14.04) OS on all the nodes. (it is always eligibly in cloud environments, e.g., EC2, aliyun)
  • you know or have set the ip address on each node.
  • python 2.7 has been installed on your laptop/PC. (unless you use a Windows OS, otherwise you are eligibly. on Windows, you need to install Python yourself)
  • fabric (version >=1.10, but not compatible with 2.0 version) has been installed on your laptop/PC

    • it seems a little harder than above steps. But believe me, you deserve it!

    • on Mac, install easy_install and then fabric:

    • on Linux (Ubuntu), the same way (notice, do not use pip to install fabric, you will be disappointed; besides, apt-get install fabric is ok, but the version is 1.8 now (2016.4))

      • sudo apt-get install python-setuptools python-dev build-essential
      • sudo easy_install "six==1.6.0" # I am not sure that whether it is necessary,
      • sudo easy_install "fabric == 1.10.5"
      • notice that, if there is something wrong, google it: e.g., lack of some .h file, just use apt-get install

    • on Windows,... Good luck, guy. I have not tried how to install fabric on Windows. But it should not be a probelm.
      init EC2 cluster, for free-password-login(ubuntu and root). for hostname, for hosts file.

##how to use

  1. modify passwd.ini

    this is an example

     [default]
 #which section is actived
 activeSession=
 #the sudo user, only used for sudo tasks
 root =
 #passwd of sudoer
 passwd =
 
 #example of section
 [server]
 #uesr name, if use createUser task, this user name will be the new user
 newuser =
 #password of newuser
 passwd =
 #cluster ips, use comma to split
 hosts=
 #hostnames of cluster , the number of hostnames must equal with hosts. the hostnames is used for changeHostname tass
 hostnames=
 #locally jdk file. the related path is ./
 jdk_source_file=
 # when the jdk.tar.gz is unziped, the folder in the tar file.
 jdk_folder=
 # the ntp server address you  want to sync
 ntp_server=
 # the allowed net that can sync with your ntp server
 ntp_net=
 # the allowed net mask that can sync with your ntp server
 ntp_net_mask=

    this is an instance:

     [default]
 activeSession=server
 root = fit
 passwd =111
 
 [server]
 newuser = hxd
 passwd = 1112
 hosts=192.168.130.3, 192.168.130.5		hostnames=s3, s5, 
 jdk_source_file=files/jdk1.8.77.tar.gz
 jdk_folder=jdk1.8.0_77
 ntp_server=192.168.130.2
 [client]
 newuser = hxd
 passwd = 1112
 hosts = 192.168.130.2
 hostnames = s2
 jdk_source_file=files/jdk1.8.77.tar.gz
 jdk_folder=jdk1.8.0_77
 ntp_net=192.168.130.0
 ntp_net_mask=255.255.255.0
 ntp_server=192.168.130.2
 [all]
 newuser = hxd
 passwd = 1112
 hosts=192.168.130.2,192.168.130.3, 192.168.130.5
 hostnames=s2,s3, s5
 jdk_source_file=files/jdk1.8.77.tar.gz
 jdk_folder=jdk1.8.0_77

  2. you can use this script like this

    fab task_name

    you can use fab help to show how many tasks supportted.

    current supportted tasks:

    • createUser
      • create a new user (username is 'newuser' in the configurations, password is 'passwd' in the active configurations).
    • removeUser
      • delete the user we created
    • addIntoHostFile
      • add all the <ip,hostname> into /etc/hosts
    • changeHostname
      • change all the hostnames as the values in the actived conf.
    • installnptserver
      • install ntp server on nodes. We suggest you only install ntp server on one node in the cluster, instead of all the nodes.(how to do: enable a configruation which has only one ip in 'hosts')
    • distributeJDK
      • after you manual download a jdk file, you can use this task to push it to all the nodes and modify the $JAVA_HOME and $PATH
    • setnpttaks
      • add a scheduled task every day for ntpdate time with the ntp server
    • password-free ssh
      • setting password-free ssh. how to use:
        • fab ssh1 (this can generate pem files)
        • fab ssh2 (this finish the password-free)
        • fab ssh3 (cleanup)
    • restart
      • reboot the cluster
    • correct_bashrc
      • normally, we use ssh hxd@node1 "jps", the terminal says: jps command not found, though we have set $JAVA_HOME and $PATH in .bashrc file. The reason is that .bashrc is ignored when we run a non-interactive login command. To solve that, use this task to repair the bashrc file.
    • modifyAtpSource
      • modify the apt sources.list as files/sources.list.
    • addDisk
      • when you add a new disk, you should format it and then mount it. this task help you. use fab addDisk:device,location. e.g., fab addDisk:/dev/sdb,/datab
    • TODO: we will add new tasks to help user add new nodes in an existed cluster

  3. some bigdata system and monitor system. you can use it like fab -f fab_bigdata taskname

    current supported task:

    • installCollectd

      • you need add a section in passwd.ini

        [collectd] #server ip tell the node that where to send the collected data #the receiver should have a receive process. e.g. influxdb and others #the most easy receiver is collected itself. In this way, you should modify the receiver's collectd configuration file yourself: /etc/collectd/collectd.conf #in network plugin: replace "Server" by "Listen" server_ip= server_port= #collect interval unit: second interval=5

    • runCollectd

      • fab -f fab_bigdata runCollectd:stop or fab -f fab_bigdata runCollectd:start

    • installGangliaClient

      • ganglia is a master-slave architecture. now you can only install the client, ganglia-monitor. The master processes, gmetad and ganglia-webfrontend, are not included.
      • if you want to install the master processes, see https://www.digitalocean.com/community/tutorials/introduction-to-ganglia-on-ubuntu-14-04.
      • to install the client process, you need a gmond.conf in files folder. Notice, modify the ip address in the file as your master ip, and modify the cluster name as the same as your master configuration.

    • installCouch

      • you can install CouchDB.
      • normally, couchdb is a single-node. therefore, your active section.hosts should have only one ip. But if you exactly want to install couchdb on all the nodes, use fab -f fab_bigdata installCouch:y
      • in current version, we do not modify the bind-address of CouchDB, the default is localhost.

    • runCouch

      • you can start or stop CouchDB
      • how to use: fab -f fab_bigdata runCouch:'start' or fab -f fab_bigdata runCouch:'stop' or fab -f fab_bigdata runCouch:'start','y' or fab -f fab_bigdata runCouch:'stop','y'. 'y' means you have couchdb on each node

##Amazon Ec2 (or other pem based ssh) in Amazon Ec2, we use a pem file to login instead of password.

To use the script on Amazon EC2. you need modify the script now. But do not worry, just a little thing you need to do:

  • comment "env.password=..." in rootUser()
  • add env.key_filename=['filepath'] in rootUser()
  • Cheers!