forked from ooici/coi-services
-
Notifications
You must be signed in to change notification settings - Fork 0
/
README
346 lines (245 loc) · 12.7 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
===========================================================
Ocean Observatories Initiative Cyberinfrastructure (OOI CI)
Integrated Observatory Network (ION)
coi-services - ION Release 2 system and services
(C) UC Regents, 2010-2013
===========================================================
This is the repository that defines the COI services.
Note: Initially, all ION services are defined here and later moved to subsystem repositories.
COI services are intended to be deployed as part of the ION system launch.
Services are deployed in pyon capability containers.
INSTALL
=======
This assumes basic development environment setup (git, directory structure). Please follow the
"New Developers Tutorial" for basic steps.
Pyon: The main dependency of this repository is the pyon Capability Container. Follow the listed
steps to install the minimal needed dependencies to run pyon on a Mac. For more details and Linux
install instructions, check out the pyon README: https://github.com/ooici/pyon/blob/master/README
Install the following if not yet present:
* OS Packages and package management:
For Mac, use homebrew
> /usr/bin/ruby <(curl -fsSkL raw.github.com/mxcl/homebrew/go)
- python 2.7.2
- git 1.7.7: Use brew or download the Mac or Linux installer and run it
- couchdb 1.1.0 (optional if memory mockdb is used)
You will need to upgrade couchdb to at least 1.1.0.
- rabbitmq 2.6.1 or later (recommended, but can use rabbitmq on amoeba)
Use brew.
Alternative: download generic Linux version and unpack into a suitable directory.
- Install libevent, libyaml, zeromq, couchdb, python, rabbitmq, and pkg-config with Homebrew
> brew install libevent libyaml zeromq couchdb python rabbitmq hdf5 pkg-config netcdf udunits
You can even reinstall git using brew to clean up your /usr/local directory
Be sure to read the pyon README for platform specific guidance to installing
dependent libraries and packages.
Linux: Note that many installs have much older versions installed by default.
(If you have trouble with brew 'MD5 mismatch' errors, try running 'brew update'
and try the install again)
- Install libgswteos-10 for marine science calculations
> brew tap lukecampbell/homebrew-libgswteos
> brew install libgswteos-10
> brew test -v libgswteos-10
- Install swig and gfortran (Needed by udunits and scipy)
> brew install swig (Make sure brew installed swig is used. `which swig`)
> brew install gfortran
installation
- Install
* python packages and environment management:
- pip
> easy_install pip
- virtualenv and virtualenvwrapper modules for your python 2.7 installation
> easy_install --upgrade virtualenv
> easy_install --upgrade virtualenvwrapper
Note: This may require Mac's XCode (use XCode 3.3 free version)
* Setup a virtualenv to run COI-services (use any name you like):
> mkvirtualenv --no-site-packages --python=python2.7 coi
Note: Do not use the pyon virtualenv if you are a pyon developer
* Pin setuptools in your virtualenv
> pip install -U setuptools==0.8
* Install numpy to the virtualenv
> pip install numpy==1.7.1
PACKAGE DOWNLOAD
================
Clone coi-services (if you haven't already) in your code directory:
> cd code # Whatever your code repository directory is
Clone coi-services (this repository, if you haven't already):
> git clone [email protected]:ooici/coi-services.git
Initialize submodules::
> cd coi-services
> git submodule update --init
(see Pyon README for hints with submodule anonymous checkout and password-less push)
PACKAGE DEPENDENCIES
====================
Use Buildout to install the python package dependencies (including pyon itself) with:
> python bootstrap.py -v 2.2.0
> bin/buildout
** Add following to your .bashrc profile on Macs if buildout fails to
compile for packages such as gevent, pyml, etc.
export C_INCLUDE_PATH=/usr/local/include
** If matplot lib fails to build due to not finding the free type headers, try:
if /usr/X11/lib/pkgconfig has a file named freetype2.pc, then add the following line to your .bash_profile
export PKG_CONFIG_PATH=/usr/X11/lib/pkgconfig
** Mac should have libpng and libfreetype headers, which are required
for matplotlib python build. If for some reason your buildout fails on
matplotlib due to missing libpng libraries, try install from here:
http://ethan.tira-thompson.com/Mac_OS_X_Ports.html
For additional support to install matplotlib on MacOS X 10.7 (Lion) please see the following:
http://www.thisisthegreenroom.com/2011/installing-python-numpy-scipy-matplotlib-and-ipython-on-lion/
Generate service definition interfaces and abstract base classes files:
> bin/generate_interfaces
Note: These will be placed in the interfaces/ subdirectory (and not in GIT nor in the egg).
RUN-TIME DEPENDENCIES
=====================
COUCHDB (OBJECT DATA STORE):
Run couchdb in second terminal before starting the container or running tests:
Note: optional. An in-memory mockdb is used by default and couchdb unit-tests are skipped.
> couchdb
RABBITMQ (MESSAGE BROKER):
Run RabbitMQ in second terminal before starting the container or running tests:
Note: optional. RabbitMQ on amoeba us used by default for new developers.
> cd rabbitmq_server-2.x.x
> sudo sbin/rabbitmq-server
ELASTICSEARCH (SEARCH ENGINE):
------------
Installation
------------
Installation is performed with a script which downloads the file from
github, extracts the contents to the INSTALL_DIR and installs the correct
version of the plugins.
> bash scripts/install_es.sh
Note: Works on both Mac and (Untested) Linux. This installs ElasticSearch
to /usr/local/elasticsearch-$(VERSION)/ by default and can be overridden
with the INSTALL_DIR environment variable
e.g. > INSTALL_DIR=/opt/elasticsearch bash scripts/install_es.sh
---------------------
Running ElasticSearch
---------------------
After ElasticSearch has been installed it can be started in foreground mode
or daemon mode. Assuming that the install directory was /usr/local
Foreground mode:
> /usr/local/elasticsearch-0.18.7/bin/elasticsearch -f
Daemon mode:
> /usr/local/elasticsearch-0.18.7/bin/elasticsearch
ElasticSearch is an instance of the JVM with the ElasticSearch cluster node,
each node is equivalent to one JVM instance running ElasticSearch.
Additional nodes means more instances of the JVM running ElasticSearch. The
JVM minimum heap and maximum heap size and performance flags can be changed
in the script /usr/local/elasticsearch-0.18.7/bin/elasticsearch which is
strongly recommended for production.
To install the ElasticSearch-Head plugin which provides a graphical
interface for browsing and viewing the contents
of the engine run:
> bash scripts/install_es.sh -h
---------------------------------
Viewing ElasticSearch Graphically
---------------------------------
If the ElasticSearch-Head plugin is installed (see above) then access to the
plugin is located at:
http://localhost:9200/_plugin/head/
SOURCE PATH:
Add the current directory to your PYTHONPATH for an easier time running scripts. Add the following
line to your ~/.profile (create it if it doesn't exist, or use bash_profile if you already have that):
> export PYTHONPATH=.
USE
===
Provide local configuration overrides (e.g. broker host, debug levels) in:
> res/config/pyon.local.yml
> res/config/logging.local.yml
Note: DO NOT EDIT the standard configuration and logging config files.
List supported Pyon Capability Container arguments:
> bin/pycc --help
Run unit tests:
> bin/nosetests -v -a UNIT
Run smoke tests to verify sanity of system:
> bin/nosetests -v -a SMOKE
Run full integration tests - this will take a long time so may be optional:
> bin/nosetests -v -a INT
Start an example (bank) locally:
> bin/pycc --rel res/deploy/examples/bank_complete.yml
Note: It does not do anything but start by itself. You need to use a client to trigger actions.
Start development instance of the ION system with all services:
> bin/pycc --rel res/deploy/r2deploy.yml
TROUBLESHOOTING
===============
Outdated *.pyc files can causes your
tests to fail unexpectedly. Do an ant clean and try again.
> ant clean
> bin/generate_interface
If your tests still fail, you might have some incorrect values in
pyon.local.yml or logging.local.yml. Check with other developers.
If your tests fail unexpectedly when you have tried everything else, your build
might be corrupted.
> deactivate from your existing virtualenv
> rm and create the virtualenv again
> re-run bootstrap.py and buildout
> If you suspect an egg is bad, Remove the suspected egg from the eggs/ dir. Re-reun buildout.
DEBUG
=====
USE Pycharm or any other Python IDE to debug; breakpoints and step-by-step are possible.
Set the executable to the bin/pycc and set the working directory to the git repository root.
Use the -n option to avoid the interactive ipython shell. It does not work property with the debugger.
Set your debugger to start the bin/pycc or bin/nosetests scripts, with the arguments you like.
Later versions of PyCharm may need to be enabled to run with gevent. You can google how to do this.
Remote debugging of services in a running container is possible too. Setup a remote debugging
configuration to use the localhost and port 8585. First start the remote debuging server, then
start the container from the command line below using the pyccd script ( the -n seems to be required )
> bin/pyccd -n --rel res/deploy/xxx.yml
DEVELOPMENT
===========
You can develop services locally in this repository. Use this repository until subsystem
specific repositories are available.
Please follow the following steps as long as you are new:
Get the latest code before you start editing, or anytime you want:
> git pull --rebase
(fix any issues, git add, git rebase --continue)
> git submodule update # Do NOT forget. This does not happen automatically
See below for an automated approach.
As extern pointers change, service interfaces can change and old pyc files
can be left around unintentinally. enerate interfaces frequently (especially in case of error):
> ant clean
> bin/generate-interfaces
*See below for an automated approach to git-submodules*
Note: The res/ and obj/ dirs are symlinks to a subdirectory in a git submodule. Beware of the pitfalls
of git submodule. You need to treat it as a separate GIT module. In case of changes, both GIT modules
must be pushed, submodule first:
> cd extern/ion-definitions
> git checkout master
> do edits...
> git status # Just to see what's going on
> git commit -am "Something smart"
> git push origin master
> cd ../.. # To the root of coi-services
> git commit -am "Something smarter"
> git push
Put your services in ion/services/<subsystem>/... (subdirectories are allowed).
The ion-definition submodules approach above apply to other submodules as
well.
GIT SUBMODULE HOOKS
===================
A git hook is a script that executes during various points of using git. Some simple hooks have been written
to help automate dealing with submodules for most people. See the steps here:
http://blog.chaitanyagupta.com/2009/08/couple-of-hooks-to-make-life-easy-with.html
They do require an initial setup. Simple instructions:
Clone this repository:
> cd /some/tmp/directory
> git clone https://github.com/chaitanyagupta/gitutils.git
Use the provided install script:
> sh gitutils/submodule-hooks/install.sh /path/to/your/pyon/or/coi-services/dir
The install script does the following (you can also do it manually):
> cd /your/pyon/.git/hooks
> cp /your/tmp/gitutils/submodule-hooks/pre-commit pre-commit
> cp /your/tmp/gitutils/submodule-hooks/post-merge-commit post-merge
> ln -s post-merge post-checkout
> chmod +x post-merge post-checkout pre-commit
Now, when checking out a branch, pulling, merging etc, git will prompt you to automatically update
if it notices a change to the commit that your supermodule points to.
The pre-commit script is so you don't forget to push changes to the submodule *BEFORE* you push changes
to the supermodule.
Instrument Development Kit (IDK)
================================
There are now 4 configuration options in a new config file res/config/idk.yml
The only option you are likely to need to override is the driver_path, currently set to ion/code. Override it in idk.local.yml to the parrent directory of coi-services, pyon, and marine-integrations if they are located elsewhere.
Use driver_path: OOI
If you have the following structure.
~/OOI/coi-services
~/OOI/pyon
~/OOI/marine-integrations