forked from jtriley/StarCluster
-
Notifications
You must be signed in to change notification settings - Fork 2
/
Changelog
422 lines (317 loc) · 15.1 KB
/
Changelog
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
===============
Version History
===============
.. contents::
:local:
.. _version-0.93.3:
0.93.3
======
.. _v0933-news:
News
----
Version 0.93.3 of StarCluster is a patch release that fixes several bugs. See
the "Bug Fixes" section below for details.
* Bumped boto dep to 2.3.0
* Updated and improved default config template
.. _v0933-fixes:
Bug Fixes
---------
* sge: fix bug where master could never be an exec host even when toggling the
``master_is_exec_host`` plugin setting (issue 89)
* fix bug when setting custom SSH permission without ``cidr_ip`` (issue 91)
* start-cmd: fix bug where ``refresh_interval`` is always ``None`` if
``[global]`` is not defined in the config (issue 90)
* dont allow ``cluster_user=root`` when validating cluster settings (issue 92)
* fix another textwrap error with python 2.5
.. _version-0.93.2:
0.93.2
======
.. _v0932-news:
News
----
Version 0.93.2 of StarCluster fixes several bugs and adds a couple new
features. See the "Bug Fixes" section below for details.
* Amazon recently announced a new instance type ``m1.medium`` which is now
supported. Amazon also enabled the m1.small and c1.medium instance types to
be compatible with both 32bit *and* 64bit AMIs now. StarCluster now supports
the new ``m1.medium`` instance type as well as the new changes to
``m1.small`` and ``c1.medium``.
* Switched from paramiko to actively maintained fork called `ssh` (1.7.13)
* loadbalancer: ``--kill-master`` option renamed to ``--kill-cluster``
* Bumped boto dep to 2.2.2
* Bumped jinja dep to 2.6
.. _v0932-features:
Features
--------
* SGE setup code has been moved to a plugin that is executed by default (**no
configuration changes required**). This plugin supports an optional setting
``master_is_exec_host`` which controls whether or not the master node
executes jobs. This is set to ``True`` by default. To change this you will
need to set ``disable_queue=True`` in your cluster config and then manually
define and add the SGE plugin in order to configure the new plugin's
settings. See :ref:`sge-plugin` for more details.
* Added the following new options to ``addnode`` command: ``--image-id``,
``--instance-type``, ``--availability-zone``, ``--bid``. Thanks to Dan Yamins
for his contributions. (issue 76)
* New ``CreateUsers`` plugin that creates multiple cluster users and configures
passwordless SSH. Either number of users or a list of usernames can be
specified. Also has options to tar and download all SSH keys for generated
users. See :ref:`users-plugin` for more details.
* Added -X option to ``sshmaster`` and ``sshnode`` commands for forwarding X11.
Can be used both when logging in interactively and when executing remote
commands.
.. _v0932-fixes:
Bug Fixes
---------
* catch `Garbage packet received` exception and warn users about incompatible
AMI
* ``createvolume``: support new /dev/xvd* device names
* fix textwrap error with python 2.5 (issue 85)
* ``sshmaster``, ``sshnode``: load /etc/profile by default when executing
remote cmds
* ``terminate``: handle Ctrl-D gracefully when prompting for input (issue 80)
* ``terminate``: ignore keys when terminating manually
* validate key_location is readable (issue 77)
* relax validation constraints for running instances when using ``starcluster
start -x`` (issue 78)
* fix cluster validation for `t1.micro` AMIs/instance types
* ``loadbalance``: fix _get_stats when no exec hosts
* ``loadbalance``: respect min_nodes when removing nodes (issue 75)
* ``loadbalance``: fix -K option to kill cluster when queue is empty (issue 86)
* ``loadbalance``: retry fetching stats 5x before failing (issue 65)
* skip NFS exports setup when there are no worker nodes
* sge: configure SGE profile on all nodes concurrently
* sge: use qconf's -\*attr commands to update cluster config (replaced a bunch
of ugly code that parsed SGE's console output)
* add Changelog to MANIFEST.in in order to include Changelog in future release
dists
.. _version-0.93.1:
0.93.1
======
.. _v0931-news:
News
----
.. warning:: It is recommended that you upgrade as soon as possible.
Version 0.93.1 of StarCluster is largely a patch release that fixes several
bugs. See the "Bug Fixes" section below for details.
Also, the ``put`` and ``get`` commands have been marked stable which means they
no longer require ``ENABLE_EXPERIMENTAL=True`` in the config in order to use
them.
.. _v0931-features:
Features
--------
* Added sys.argv to crash reports for better details when things go wrong
* Implemented on_add_node and on_remove_node for TMUX plugin to add/remove
nodes from the TMUX control-center sessions for both ``root`` and
``CLUSTER_USER``
* Implemented on_add_node for the Ubuntu package installer plugin
(``pkginstaller``) to ensure new nodes also install the required packages
* Implemented on_add_node for the MPICH2 plugin to ensure new nodes are
configured for MPICH2 and also added to the hosts file. Moved the MPICH2
hosts file to an NFS-shared location (``/home/mpich2.hosts``) for faster
configuration
* Implemented on_add_node for the Xvfb plugin that installs and runs Xvfb on
newly added nodes
.. _v0931-fixes:
Bug Fixes
---------
* Fixed major NFS bug in ``addnode`` command where ``/etc/exports`` was being
wiped each time a new node was added. This caused the last-to-be-added node
to be the only node correctly configured for NFS in the entire cluster.
* Updated all plugins which subclass DefaultClusterSetup to implement
on_add_node and on_remove_node regardless of whether or not they actually use
them to avoid the 'default' plugin's implementation being incorrectly called
multiple times by plugins when adding/removing nodes using the ``addnode``
and ``removenode`` commands
* Fixed string formatting bug in Condor plugin
.. _version-0.93:
0.93
====
.. _v093-news:
News
----
New StarCluster Ubuntu 11.10 EBS AMIs available including a new HVM AMI for use
with CC/CC2/CG instance types! Includes OpenGridScheduler (to replace SGE),
Condor, Hadoop/Dumbo, MPICH2, OpenMPI, IPython 0.12 (parallel+notebook), and
more. Run ``dpkg -l`` on the AMIs for the full list of packages.
The new StarCluster Ubuntu 11.10 EBS HVM AMI additionally comes with NVIDIA
driver, CUDA Toolkit/SDK, PyCuda, PyOpenCL, and Magma for GPU computing.
These new AMIs are available in all AWS regions. The new ``us-east-1`` 11.10
EBS AMIs are now the defaults in the StarCluster config template. Use the
``listpublic`` command to obtain a list of available StarCluster AMIs in other
regions::
$ starcluster -r sa-east-1 listpublic
.. note::
The **HVM** AMI is currently only available in the ``us-east-1`` region
until other regions support HVM.
.. _v093-features:
Features
--------
* Added support for new CC2 instance types
* Added support for specifying a proxy host to use when connecting to AWS::
[aws info]
aws_proxy = your.proxy.com
aws_proxy_port = 8080
aws_proxy_user = yourproxyuser
aws_proxy_pass = yourproxypass
See :ref:`proxy-config` for more details
* Updated IPCluster plugin to support IPython 0.12 with :ref:`parallel
<using-ipython-cluster>` and :ref:`notebook <ipython-notebook>`
support (SSL+password auth). See :ref:`ipcluster-plugin` for more details
* Keypairs are now validated against EC2 fingerprint which notifies users ahead
of time if the file they specified in KEY_LOCATION doesn't match the EC2
keypair specified
* Support for Windows (tested on 32bit Windows 7). See :ref:`windows-install`
for more details
New Plugins
^^^^^^^^^^^
* **Condor** - experimental support for creating a Condor pool on StarCluster.
See :ref:`condor-plugin` for more details
* **Hadoop** - support for creating a Hadoop cluster with StarCluster
See :ref:`hadoop-plugin` for more details
* **MPICH2** - configure cluster to use MPICH2 instead of OpenMPI
See :ref:`mpich2-plugin` for more details
* **TMUX** - configures cluster SSH "dashboard" in tmux
See :ref:`tmux-plugin` for more details
* **Package installer** - install Ubuntu packages on all nodes concurrently
See :ref:`pkginstaller-plugin` for more details
* **Xvfb** - install and configure an Xvfb session on all nodes (``sets
DISPLAY=:1``) See :ref:`xvfb-plugin` for more details
* **MySQL** - install and configure a MySQL cluster on StarCluster. See
:ref:`mysql-plugin` for more details
.. _v093-fixes:
Bug Fixes
---------
* Fix bug in ``put`` command where in random cases the last few bytes of the
file weren't being transferred
* Raise an error if placement group creation fails
* Fix bug in terminate command in the case that nodes are already
``shutting-down`` when the terminate command is executed
* Fix bug in s3image when destination S3 bucket already exists
* Fix known_hosts warnings when ssh'ing from node to node internally
* Fix trivial logging bug in createvolume command
.. _version-0.92.1:
0.92.1
======
:release-date: 11-05-2011
.. _v0921-features:
Features
--------
* Support for splitting the config into an arbitrary set of files::
[global]
include=~/.starcluster/awscreds, ~/.starcluster/myconf
See :ref:`splitting-the-config` for more details
* createvolume: support naming/tagging newly created volumes::
$ starcluster createvolume --name mynewvol 30 us-east-1d
See :ref:`create-and-format-ebs-volumes` for more details
* listvolumes: add support for filtering by tags::
$ starcluster listvolumes --name mynewvol
$ starcluster listvolumes --tag mykey=myvalue
See :ref:`managing-ebs-volumes` for more details
* sshmaster, sshnode, sshinstance: support for running remote
commands from command line::
$ starcluster sshmaster mycluster 'cat /etc/fstab'
$ starcluster sshnode mycluster node001 'cat /etc/fstab'
$ starcluster sshinstance i-99999999 'cat /etc/hosts'
See :doc:`manual/runcommands` for more details
.. _v0921-fixes:
Bug Fixes
---------
The following bugs were fixed in this release:
**spothistory command**
* add package_data to sdist in order to include the necessary web media and
templates needed for the ``--plot`` feature. The previous 0.92 version left
these out and thus the ``--plot`` feature was broken. This should be fixed.
* fix bug when launching default browser on mac
**start command**
* fix bug in option completion when using the start command's
``--cluster-template`` option
**terminate command**
* fix bug in terminate cmd when region != us-east-1
**listkeypairs command**
* fix bug in list_keypairs when no keys exist
Improvements
------------
* listinstances: add 'state_reason' msg to output if available
* add system info, Python info, and package versions to crash-report
* listregions: sort regions by name
* improved bash/zsh completion support. completion will read from the correct
config file, if possible, in the case that the global -c option is specified
while completing.
* separate the timing of cluster setup into time spent on waiting for EC2
instances to come up and time spent configuring the cluster after all
instances are up and running. this is useful when profiling StarCluster's
performance on large (100+ node) clusters.
.. _version-0.92:
0.92
====
:release-date: 10-17-2011
.. _v092-news:
News
----
.. note::
Before upgrading please be aware that rc1 does not support clusters created
with previous versions and will print a warning along with instructions on
how to proceed if it finds old clusters. With that said, it's best not to
upgrade until you've terminated any currently running clusters created with
an old version. Also, the @sc-masters group is no longer needed and can be
removed after upgrading by running "starcluster terminate masters"
You can find the new docs for 0.92rc1 here:
http://web.mit.edu/stardev/cluster/docs/0.92rc1/
NOTE: these docs are still very much a work in progress. If anyone is
interested in helping to improve the docs, please fork the project on github
and submit pull requests. Here's a guide to get you started:
http://web.mit.edu/stardev/cluster/docs/0.92rc1/contribute.html
Please upgrade at your leisure, test the new release candidate, and submit any
bug reports preferably on StarCluster's github issue tracker or this mailing
list.
.. _v092-features:
Features
--------
* Cluster Compute/GPU Instances- Added support for the new Cluster
Compute/GPU instance types. Thanks to Fred Rotbart for his contributions
* EBS-Backed Clusters- Added support for starting/stopping EBS-backed
clusters on EC2. The ``stop`` command now stops (instead of terminate)
an EBS-backed cluster and terminates an S3-backed cluster. Added a
terminate command that terminates *any* cluster
* Improved performance - using workerpool library
(http://pypi.python.org/pypi/StarCluster) to configure nodes concurrently
* New ``ebsimage`` and ``s3image`` commands for easily creating new EBS-backed
AMIs. Each command supports creating a new AMI from S3 and EBS-backed image
hosts respectively. (NOTE: ``createimage`` has been renamed to s3image. you can
still call ``createimage`` but this will go away in future releases)
* Add/Remove Nodes - added new ``addnode`` and ``removenode`` commands for
adding/removing nodes to a cluster and removing existing nodes from a cluster
* Restart command - Added new restart command that reboots the cluster and
reconfigures the cluster
* Create Keypairs - Added ability to add/list/remove keypairs
* Elastic Load Balancing - Support for shrinking/expanding clusters based on Sun
Grid Engine queue statistics. This allow the user to start a single-node
cluster (or larger) and scale the number of instances needed to meet the
current SGE queue load. For example, a single-node cluster can be launched and
as the queue load increases new EC2 instances are launched, added to the
cluster, used for computation, and then removed when they're idle. This
minimizes the cost of using EC2 for an unknown and on-demand workload. Thanks
to Rajat Banerjee (rqbanerjee)
* Security Group Permissions - Added ability to specify permission settings to
be applied automatically to a cluster's security group after it's been started
* Multiple Instance Types - Added support for specifying instance types on a
per-node basis. Thanks to Dan Yamins for his contributions
* Unpartitioned Volumes- StarCluster now supports both partitioned and
unpartitioned EBS volumes
* New Plugin Hooks - Plugins can now play a part when adding or removing a node as
well as when restarting or shutting down the entire cluster by implementing the
``on_remove_node``, ``on_add_node``, ``on_shutdown``, and ``on_reboot``
methods respectively
* Added a ``runplugin`` command and fixed the run_plugin() method in the
development shell. You can now run a plugin in the dev shell like so::
~[1]> cm.run_plugin('myplugin', 'mycluster')
* Added support for easily switching regions (without config changes)
using a global -z option. For example::
$ starcluster -r eu-west-1 listclusters
StarCluster - (http://web.mit.edu/starcluster) (v. 0.92rc1)
Software Tools for Academics and Researchers (STAR)
Please submit bug reports to [email protected]
>>> No clusters found...
* Added new ``resizevolume`` command for easily resizing existing EBS volumes
* Added listregions/listzones commands