-
Notifications
You must be signed in to change notification settings - Fork 0
/
imagetransfer-software-subscriber.xml
775 lines (731 loc) · 43.2 KB
/
imagetransfer-software-subscriber.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE part PUBLIC "-//OASIS//DTD DocBook XML V4.3//EN"
"http://www.oasis-open.org/docbook/xml/4.3/docbookx.dtd" [
<!ENTITY % sharedents SYSTEM "shared-entities.xml" >
%sharedents;
]>
<chapter id="imagetransfer-software-subscriber">
<info>
<author>
<firstname>Owen</firstname>
<surname>Synge</surname>
</author>
</info>
<title>&vmil; Subscribing Software</title>
<section id="imagetransfer-software-vmcatcher">
<title>vmcatcher</title>
<para>This virtual machine &vmil; subscriber implementation is intended to be a production grade reference implementation.</para>
<para>The software makes use of a database to store subscriptions in a similar way to a podcast reader or a Linux package manager.
The tested Database is SqlLight, but it is based upon SQLalchamy so should support many databases. Sdllight has proved more than
adequate for the low transaction rate of a image list subscriber and so deployment issues are just backing up a databasefile.</para>
<para>Since the software is made with the Grid in mind it is only natural that the &x509; certificate model is used. </para>
<section id="imagetransfer-software-vmcatcher-introduction">
<title>Introduction</title>
<para>This application allows users to subscribe to &vmil;s, cache
the images referenced to in the &vmil;, validate the images list with &x509;
based public key cryptography, and validate the images against sha512 hashes in
the images lists and provide events for further applications to process update
or expire changes of virtual machine images without having to further validate the
images.</para>
<para>This software is available at:</para>
<code>https://github.com/hepix-virtualisation/vmcatcher</code>
<para>The software is based upon a simple database that stores subscriptions to &vmil;s,
who can sign the &vmil;, and which images belong to which subscriptions. It allows
images to selected for subscription.</para>
<para>Subscribed images can be downloaded verified and cached. Cached images can be verified,
and if invalid or expired they are moved to an expiry directory.</para>
<section id="magetransfer-software-imagelist-consumer-status-features">
<title>Features</title>
<itemizedlist>
<listitem>Add and delete multiple subscriptions to &vmil;s.</listitem>
<listitem>Update subscriptions checking authenticity of the message using &x509; based signatures.</listitem>
<listitem>Automation as a cron script.</listitem>
<listitem>Subscribe and unsubscribe to images from &vmil;s.</listitem>
<listitem>Download verify images into a local cache.</listitem>
<listitem>Expire images to an archive when no longer endorsed or corrupt.</listitem>
<listitem>Open Stack intgration by Mattieu Puel CC-IN2P3.fr.</listitem>
<listitem>OpenNebula intgration by Roberto Rosende Dopazo CESGA.es.</listitem>
</itemizedlist>
</section>
<para> This set of applications are designed to provide a similar work flow
from each area of control to the &vmil; archive.</para>
<itemizedlist>
<listitem>&vmcatcher_endorser; - Endorsers of &vmil; subscriptions.</listitem>
<listitem>&vmcatcher_subscribe; - Subscription list details.</listitem>
<listitem>&vmcatcher_image; - Image details.</listitem>
<listitem>&vmcatcher_cache; - Cache images and update events.</listitem>
</itemizedlist>
<para>They work in conjunction with a database to ease navigation, a local cache of
&vmil; subscriptions. The database is message format agnostic, but it
authenticates and validates all messages in import. Because these are just
caches of &vmil;s they are meant to be used the majority of the time without
intervention.</para>
<para>If you are signing a list using the &hepix; &vmil; signer, you should also
install this application and subscribe to your current image.</para>
<para>It is intended to with a couple of cron scripts to be informed at any time if
your local images are matching signatures in the &vmil;.</para>
<para>Anyone curious about this application should consider this application a
software application similar to Debian's 'aptitude' or Redhats 'yum', but rather
for virtual machines, authenticated by the &x509; signatures.</para>
<para>Like yum and apt this tool can be extended. The following handlers have been developed</para>
<table frame='all'><title>Cloud Integration with vmcatcher</title>
<tgroup cols='5' align='left' colsep='1' rowsep='1'>
<colspec colname='c1'/>
<colspec colname='c2'/>
<colspec colname='c3'/>
<colspec colname='c4'/>
<colspec colname='c5'/>
<thead>
<row>
<entry>Cloud</entry>
<entry>Name</entry>
<entry>Author</entry>
<entry>Employer</entry>
<entry>URI</entry>
</row>
</thead>
<tbody>
<row>
<entry>Open Stack</entry>
<entry>glancepush-vmcatcher</entry>
<entry>Mattieu Puel</entry>
<entry>CC-IN2P3.fr</entry>
<entry>https://github.com/EGI-FCTF/glancepush-vmcatcher</entry>
</row>
<row>
<entry>Open Stack</entry>
<entry>glancepush</entry>
<entry>Mattieu Puel</entry>
<entry>CC-IN2P3.fr</entry>
<entry>https://github.com/EGI-FCTF/glancepush</entry>
</row>
<row>
<entry>OpenNebula</entry>
<entry>Cesga cloud tools</entry>
<entry>Roberto Rosende Dopazo</entry>
<entry>Cesga.es</entry>
<entry>https://github.com/grid-admin/cloud</entry>
</row>
</tbody>
</tgroup>
</table>
</section>
<section id="imagetransfer-software-vmcatcher-quickstart">
<title>Quick start use of vmcatcher</title>
<para>First make sure that all the Certificate Revocation Lists (CRL) are upto date.</para>
<programlisting>&prompt-root; <userinput>fetch-crl</userinput></programlisting>
<para>This suit of applications can use either environment variable or command line to set most parameters.
If neither environment variables or command line parameters are not set for critical variables the application
will provide defaults and show warnings.</para>
<para>The most important setting is the location of the database. This is read from VMCATCHER_RDBMS,</para>
<programlisting>&prompt-user; <userinput>export VMCATCHER_RDBMS="sqlite:////var/lib/vmcatcher/vmcatcher.db"</userinput></programlisting>
<para>The above line instructs the SQLalchamy interface to databases to use sqlite and path "/var/lib/vmcatcher/vmcatcher.db"
on a UNIX system. This is the only important file and stores the older signed image lists. SQL is used to enforce most of the rules
such as unique nature of &uuid;'s and the URLS for the subscriptions, this should be backed up. Other environment variables
are documented later.</para>
<para>To add a subscription,</para>
<programlisting>&prompt-user; <userinput>wget --no-check-certificate &blobUrl;</userinput></programlisting>
<para>Now you can check the &vmil; by visual inspection.</para>
<programlisting>&prompt-user; <userinput>grep 'hv:[cd][an]' hepix_signed_image_list</userinput>
"hv:ca": "&blobIssuer;",
"hv:dn": "&blobDn;", </programlisting>
<para>Now create this endorser. The endorser_uuid can be any string but its recommended this is a
short string possibly following the &uuid; standard:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --create \
--endorser_uuid='&blobEndorserName;' \
--subject='&blobDn;' \
--issuer='&blobIssuer;'</userinput></programlisting>
<para>Now we can add the subscription, this will automatically link the endorser
with this subscription.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser -l</userinput>
Ian '&blobDn;' '&blobIssuer;'</programlisting>
<para>The above command will show you the endorsers. Note the first column is the
identifier. In this case its shorter than a &uuid;. The second column is the
users certificate subject (some times know as distinguished name) while the
third column is the subject of the issuing certificate authority.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -s file:////`pwd`/hepix_signed_image_list</userinput>
INFO:main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
WARNING:db_actions:list hv:uri does not match subscription uri</programlisting>
<para>Although less secure it is also possible to add the option '--auto-endorse'
to the command line so that both the endorsers, and endorsers issuer's certificate subjects are added
to the database automatically. This is particularly useful for testing.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe --auto-endorse -s file:////`pwd`/hepix_signed_image_list</userinput>
INFO:main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
WARNING:db_actions:list hv:uri does not match subscription uri</programlisting>
<para>List the registered Images.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -l</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
327016b0-6508-41d2-bce0-c1724cb3d3e2 0 &blobSubscriptionUuid;
&blobImageIdExampleSubscribed; 0 &blobSubscriptionUuid;
da42ca85-179b-4873-b12e-32d549bf02b6 0 &blobSubscriptionUuid;</programlisting>
<para>The results show the &uuid; of the image, the availability state and the
subscription &uuid;. The state value is a bitmap;</para>
<itemizedlist>
<listitem>1 Image is subscribed</listitem>
<listitem>2 Image is available from a valid &vmil;s.</listitem>
</itemizedlist>
<para>Now we will select an image for local caching.</para>
<para>Next update the subscriptions.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -U</userinput>
INFO:main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
INFO:db_actions:Updating:&blobSubscriptionUuid;</programlisting>
<para>Now the data base contains the latest version of the &vmil;. To list the
available images referenced in the local database:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -l</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
327016b0-6508-41d2-bce0-c1724cb3d3e2 2 &blobSubscriptionUuid;
&blobImageIdExampleSubscribed; 2 &blobSubscriptionUuid;
da42ca85-179b-4873-b12e-32d549bf02b6 2 &blobSubscriptionUuid;</programlisting>
<para>This now shows the images are available in the latest &vmil;.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -a -u &blobImageIdExampleSubscribed;</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'</programlisting>
<para>The &vmil; state is now changed to</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -l</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
327016b0-6508-41d2-bce0-c1724cb3d3e2 2 &blobSubscriptionUuid;
&blobImageIdExampleSubscribed; 3 &blobSubscriptionUuid;
da42ca85-179b-4873-b12e-32d549bf02b6 2 &blobSubscriptionUuid;</programlisting>
<para>Clearly showing that the image '&blobImageIdExampleSubscribed;' is subscribed.</para>
<para>Make the directories for caching the images.</para>
<programlisting>&prompt-user; <userinput>mkdir cache cache/partial cache/expired</userinput></programlisting>
<para>Now cache the images.</para>
<programlisting>&prompt-user; <userinput> vmcatcher_cache</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
INFO:DownloadDir:Downloading '&blobImageIdExampleSubscribed;'.
INFO:CacheMan:moved file &blobImageIdExampleSubscribed;</programlisting>
<para>Once this is complete the image from the &vmil; will be cached.</para>
<programlisting>&prompt-user; <userinput> find cache/</userinput>
cache/
cache/partial
cache/partial/cache.index
cache/expired
cache/expired/cache.index
cache/&blobImageIdExampleSubscribed;
cache/cache.index</programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-installation">
<title>Installation</title>
<para>The latest build system artefacts are published here http://www.yokel.org/pub/software/yokel.org/release/</para>.
<section id="imagetransfer-software-vmcatcher-repo">
<title>Package Repositories.</title>
<para>The intra site tools are tested on every release for &rhel; 6 and are developed
on the &debian; platform. They are available as src and binary RPM packages in the following
repository sporting.</para>
<itemizedlist>
<listitem>&repoUriRpmYokel_stable_SL6;</listitem>
<listitem>&repoUriRpmYokel_stable_SL5;</listitem>
</itemizedlist>
<para>Prebuilt Debian packages is currently work in progress, but prebuilt tar balls are available.</para>
<para>Deployment instructions are provided in the README included in the source code and
the RPM.</para>
</section>
<section id="imagetransfer-software-vmcatcher-installation-rhel7">
<title>Installation on Redhat Enterprise Linux 7</title>
<para>Install EPEL for dependencies.</para>
<programlisting>&prompt-root; <userinput>rpm -i https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm</userinput></programlisting>
<para>Install Yokel yum repository.</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/vmcasting.repo</userinput>
[vmcasting]
name=vmcasting
baseurl=&repoUriRpmYokel_stable_SL7;
enabled=1
gpgcheck=0</programlisting>
<para>Install the Grid CA repository for details please see https://wiki.egi.eu/wiki/EGI_IGTF_Release</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/egi-trust-anchor.repo</userinput>
[EGI-trustanchors]
name=EGI-trustanchors
baseurl=http://repository.egi.eu/sw/production/cas/1/current/
gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3
gpgcheck=1
enabled=1</programlisting>
<para>install the ca-policy-egi-core</para>
<programlisting>&prompt-root; <userinput>yum install ca-policy-egi-core</userinput></programlisting>
<para>install fetch-crl</para>
<programlisting>&prompt-root; <userinput>yum install fetch-crl</userinput></programlisting>
<para></para>
<programlisting>&prompt-root; <userinput>yum install vmcatcher</userinput></programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-installation-rhel6">
<title>Installation on Redhat Enterprise Linux 6</title>
<para>Install EPEL for dependencies.</para>
<programlisting>&prompt-root; <userinput>rpm -i https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm</userinput></programlisting>
<para>Install Yokel yum repository.</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/vmcasting.repo</userinput>
[vmcasting]
name=vmcasting
baseurl=&repoUriRpmYokel_stable_SL6;
enabled=1
gpgcheck=0</programlisting>
<para>Install the Grid CA repository for details please see https://wiki.egi.eu/wiki/EGI_IGTF_Release</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/egi-trust-anchor.repo</userinput>
[EGI-trustanchors]
name=EGI-trustanchors
baseurl=http://repository.egi.eu/sw/production/cas/1/current/
gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3
gpgcheck=1
enabled=1</programlisting>
<para>install the ca-policy-egi-core</para>
<programlisting>&prompt-root; <userinput>yum install ca-policy-egi-core</userinput></programlisting>
<para>install fetch-crl</para>
<programlisting>&prompt-root; <userinput>yum install fetch-crl</userinput></programlisting>
<para></para>
<programlisting>&prompt-root; <userinput>yum install vmcatcher</userinput></programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-installation-rhel5">
<title>Installation on Redhat Enterprise Linux 5</title>
<para>Redhat Enterprise Linux 5 is no longer updated please report a bug if you still need this platform.</para>
<para>Install EPEL for dependencies.</para>
<programlisting>&prompt-root; <userinput>rpm -i https://dl.fedoraproject.org/pub/epel/epel-release-latest-5.noarch.rpm</userinput></programlisting>
<para>Install DESY yum repository.</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/vmcasting.repo</userinput>
[vmcasting]
name=vmcasting
baseurl=&repoUriRpmYokel_stable_SL5;
enabled=1
gpgcheck=0</programlisting>
<para>Install the Grid CA repository for details please see
https://wiki.egi.eu/wiki/EGI_IGTF_Release</para>
<programlisting>&prompt-root; <userinput>cat /etc/yum.repos.d/egi-trust-anchor.repo</userinput>
[EGI-trustanchors]
name=EGI-trustanchors
baseurl=http://repository.egi.eu/sw/production/cas/1/current/
gpgkey=http://repository.egi.eu/sw/production/cas/1/GPG-KEY-EUGridPMA-RPM-3
gpgcheck=1
enabled=1</programlisting>
<para>Install the lcg-CA</para>
<programlisting>&prompt-root; <userinput>yum install lcg-CA</userinput></programlisting>
<para>install fetch-crl</para>
<programlisting>&prompt-root; <userinput>yum install fetch-crl</userinput></programlisting>
<para>Install the &hepix; &vmil; subscriber.</para>
<programlisting>&prompt-root; <userinput>yum install vmcatcher</userinput></programlisting>
<para>This may fail due to a dependency of m2crypto that cannot be satisfied. This is
due to known bugs in m2crypto in the version shipped in RHEL5. If this is a
problem please download the following</para>
<programlisting>http://ftp.informatik.uni-frankfurt.de/fedora-archive/fedora/linux/releases/8/Everything/source/SRPMS/m2crypto-0.18-2.src.rpm</programlisting>
<para>And build a native RPM.</para>
</section>
<section id="imagetransfer-software-vmcatcher-installation-debian-wheezy">
<title>Installation on &wheezy; or later,</title>
<para>Do not install this on Debian 6.0 as the included version of python-m2crypto is not stable.</para>
<para>These instructions are for &wheezy; or later.</para>
<para>Unfortunately at this moment the code is not packaged, but they will be soon. All the dependencies are available
in the Debian repository. </para>
<para>For Grid scientific use you can get a trust store easily using the egi.eu repository.</para>
<programlisting>&prompt-root; <userinput>wget -q -O - \
https://dist.eugridpma.info/distribution/igtf/current/GPG-KEY-EUGridPMA-RPM-3 \
| apt-key add -</userinput></programlisting>
<para>Add the following line to your sources.list file for APT:</para>
<programlisting>#### EGI Trust Anchor Distribution ####
deb http://repository.egi.eu/sw/production/cas/1/current egi-igtf core</programlisting>
<para>for example:</para>
<programlisting>&prompt-root; <userinput>echo '#### EGI Trust Anchor Distribution ####' >> \
/etc/apt/sources.list</userinput>
&prompt-root; <userinput>echo 'deb http://repository.egi.eu/sw/production/cas/1/current egi-igtf core' >> \
/etc/apt/sources.list</userinput></programlisting>
<para>Now install the &ca; for the grid (Other &ca; can be substituted), install a tool to download and cache the &crl; </para>
<programlisting>&prompt-root; <userinput>aptitude update</userinput>
&prompt-root; <userinput>aptitude install ca-policy-egi-core</userinput>
&prompt-root; <userinput>aptitude install fetch-crl</userinput>
&prompt-root; <userinput>fetch-crl </userinput></programlisting>
<para>Now install the code from git.</para>
<programlisting>&repoUriTarYokel_stable_SL6;</programlisting>
<para>The latest version of hepixvmitrust-X.X.XX.src.tar.gz should be downloaded extracted and installed.</para>
<programlisting>&prompt-root; <userinput>wget &repoUriTarYokel_stable_SL6;&hepixvmitrust-latest;.src.tar.gz</userinput>
Resolving grid.desy.de (grid.desy.de)... 131.169.180.46
Connecting to grid.desy.de (grid.desy.de)|131.169.180.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19922 (19K) [application/x-tar]
Saving to: `&hepixvmitrust-latest;.src.tar.gz'
100%[======================================>] 19,922 --.-K/s in 0.05s
2012-05-28 19:45:45 (413 KB/s) - `&hepixvmitrust-latest;.src.tar.gz' saved [19922/19922]
&prompt-root; <userinput>tar -zxf &hepixvmitrust-latest;.src.tar.gz </userinput>
&prompt-root; <userinput>cd &hepixvmitrust-latest;</userinput>
&prompt-root; <userinput>python setup install</userinput>
&prompt-root; <userinput>echo $?</userinput>
&prompt-root; <userinput>cd ..</userinput></programlisting>
<para>The latest version of&smimeX509validation-latest;.src.tar.gz -X.X.XX.src.tar.gz should be downloaded extracted and installed.</para>
<programlisting>&prompt-root; <userinput></userinput>
&prompt-root; <userinput>wget &repoUriTarYokel_stable_SL6;&smimeX509validation-latest;.src.tar.gz</userinput>
Resolving grid.desy.de (grid.desy.de)... 131.169.180.46
Connecting to grid.desy.de (grid.desy.de)|131.169.180.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19922 (19K) [application/x-tar]
Saving to: `&smimeX509validation-latest;.src.tar.gz'
100%[======================================>] 19,922 --.-K/s in 0.05s
2012-05-28 19:45:45 (413 KB/s) - `&smimeX509validation-latest;.src.tar.gz' saved [19922/19922]
&prompt-root; <userinput>tar -zxf &smimeX509validation-latest;.src.tar.gz </userinput>
&prompt-root; <userinput>cd &smimeX509validation-latest;</userinput>
&prompt-root; <userinput>python setup install</userinput>
&prompt-root; <userinput>echo $?</userinput>
&prompt-root; <userinput>cd ..</userinput></programlisting>
<para>The latest version of vmcatcher-X.X.XX.src.tar.gz should be downloaded extracted and installed.</para>
<programlisting>&prompt-root; <userinput></userinput>
&prompt-root; <userinput>wget &repoUriTarYokel_stable_SL6;&vmcatcher-latest;.src.tar.gz</userinput>
Resolving grid.desy.de (grid.desy.de)... 131.169.180.46
Connecting to grid.desy.de (grid.desy.de)|131.169.180.46|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 19922 (19K) [application/x-tar]
Saving to: `&vmcatcher-latest;.src.tar.gz'
100%[======================================>] 19,922 --.-K/s in 0.05s
2012-05-28 19:45:45 (413 KB/s) - `&vmcatcher-latest;.src.tar.gz' saved [19922/19922]
&prompt-root; <userinput>tar -zxf &vmcatcher-latest;.src.tar.gz </userinput>
&prompt-root; <userinput>cd &vmcatcher-latest;</userinput>
&prompt-root; <userinput>python setup install</userinput>
&prompt-root; <userinput>echo $?</userinput>
&prompt-root; <userinput>cd ..</userinput></programlisting>
</section>
</section>
<section id="imagetransfer-software-vmcatcher-config-env">
<title>Environment Variables</title>
<para>Environment variables can be used to set default values but the command line
options will override any set environment options.</para>
<section>
<title>VMCATCHER_RDBMS </title>
<para>Sets the path to the database. For example "sqlite:///vmcatcher.db"</para>
</section>
<section>
<title>VMCATCHER_CACHE_EVENT</title>
<para>Sets the executions string. Command line options can be set as environment
variables just like the command line interface. Users of the "sh shell" must
protect the environment variables from being substituted by their shell.</para>
<programlisting>&prompt-user; <userinput> export VMCATCHER_CACHE_EVENT="./myEventProcessor \$VMCATCHER_EVENT_TYPE"</userinput></programlisting>
<para>An example of how to execute a command with an action command line.</para>
</section>
<section>
<title>VMCATCHER_LOG_CONF</title>
<para>Sets the path to the logging configuration file.</para>
</section>
<section>
<title>VMCATCHER_DIR_CERT</title>
<para>Sets the Path to the certificate authorities public keys, certificate revocation
lists and certificate name spaces.</para>
</section>
<section>
<title>VMCATCHER_CACHE_DIR_CACHE</title>
<para>Path used by '&vmcatcher_endorser;' to store verified VM images.</para>
</section>
<section>
<title>VMCATCHER_CACHE_DIR_DOWNLOAD</title>
<para>Path used by '&vmcatcher_endorser;' to download VM images before VM image integrity
is checked.</para>
</section>
<section>
<title>VMCATCHER_CACHE_DIR_EXPIRE</title>
<para>Path used by '&vmcatcher_endorser;' to store VM images when they are no longer
endorsed.</para>
</section>
<section>
<title>VMCATCHER_CACHE_ACTION_DOWNLOAD</title>
<para>Instructs '&vmcatcher_endorser;' to download the latest VM images and check
integrity.</para>
</section>
<section>
<title>VMCATCHER_CACHE_ACTION_CHECK</title>
<para>Instructs '&vmcatcher_endorser;' check integrity for all currently stored VM images.</para>
</section>
<section>
<title>VMCATCHER_CACHE_ACTION_EXPIRE</title>
<para>Instructs '&vmcatcher_endorser;' to expire stored VM images that are no longer
endorsed.</para>
</section>
</section>
<section id="imagetransfer-software-vmcatcher-vmcatcher_endorser">
<title>&vmcatcher_endorser;</title>
<para>This application is for managing who the subscriber trusts to update image
lists. Since individuals are identified with &x509; certificates, Each certificate
has an issuing certificate and a unique string called a 'subject' to identify
the certificate. The 'subject' of a certificate and the 'subject' of the issuing
certificate combined are called 'credentials', and will be globally unique.</para>
<para>Individuals on rare occasions will need more than one certificate, for this
reason they are given a unique identifier under this system and allowed to
have more than one set of credentials. </para>
<para>Adding a individual to the &vmcatcher; database.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --create \
--endorser_uuid=&blobEndorserUuid; \
--subject='&blobDn;' \
--issuer='&blobIssuer;'</userinput></programlisting>
<para>Deleting and individual from a &vmcatcher; database.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --delete \
--endorser_uuid=&blobEndorserUuid;</userinput></programlisting>
<para>Allowing an individual to update a subscription.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --link \
--endorser_uuid=&blobEndorserUuid; \
--subscription_uuid=&blobSubscriptionUuid;</userinput></programlisting>
<para>Removing an individuals right to update a subscription.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --unlink \
--endorser_uuid=&blobEndorserUuid; \
--subscription_uuid=&blobSubscriptionUuid;</userinput></programlisting>
<para>Each endorser_uuid must be unique or they will be assumed to be the same item.
The endorser_uuid could be a more human name:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_endorser --create \
--endorser_uuid='&blobEndorserName;' \
--subject='&blobDn;' \
--issuer='&blobIssuer;'</userinput></programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-vmcatcher_subscribe">
<title>&vmcatcher_subscribe;</title>
<para>This application manages your subscriptions and their update:</para>
<para>To add a subscription</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -s &blobUrl;</userinput></programlisting>
<para>Or alternatively you can download a file visually inspect it and subscribe to the local file.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -s file:////`pwd`/hepix_signed_image_list</userinput></programlisting>
<para>To update your subscriptions</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -U</userinput></programlisting>
<para>To list subscriptions</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -l</userinput>
&blobSubscriptionUuid; True &blobUrl;</programlisting>
<para>Getting Information on a subscription:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -i --uuid=&blobSubscriptionUuid;</userinput>
dc:identifier=&blobSubscriptionUuid;
subscription.dc:description=&blobSubscriptionDcdescription;
subscription.sl:authorised=True
subscription.hv:uri=&blobUrl;
subscription.dc:date:updated=2011-04-16T19:23:19Z
imagelist.dc:date:imported=2011-04-16T19:23:18Z
imagelist.dc:date:created=2011-03-16T00:15:07Z
imagelist.dc:date:expires=2011-04-13T00:15:07Z</programlisting>
<para>you can also select on the basis of url:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -i -r &blobUrl;</userinput>
dc:identifier=&blobSubscriptionUuid;
subscription.dc:description=&blobSubscriptionDcdescription;
subscription.sl:authorised=True
subscription.hv:uri=&blobUrl;
subscription.dc:date:updated=2011-04-17T19:04:35Z
imagelist.dc:date:imported=2011-04-17T19:04:34Z
imagelist.dc:date:created=2011-03-16T00:15:07Z
imagelist.dc:date:expires=2011-04-13T00:15:07Z</programlisting>
<para>Change the output format to get the original message without the security wrapper, or in original form:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -i --uuid=&blobSubscriptionUuid; -f message</userinput></programlisting>
<para>Three formats exist SMIME, message, lines. </para>
<itemizedlist>
<listitem>SMIME for applications that wish to process the signature as if from the endorser directly.</listitem>
<listitem>message for applications that have no interest in processing the SMIME signature.</listitem>
<listitem>lines for human users of this application.</listitem>
</itemizedlist>
<para>To delete a subscription</para>
<programlisting>&prompt-user; <userinput>vmcatcher_subscribe -D --uuid=&blobSubscriptionUuid;</userinput></programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-vmcatcher_image">
<title> &vmcatcher_image;</title>
<para> This application manages images within your subscription. </para>
<para>List the available images</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -l</userinput>
327016b0-6508-41d2-bce0-c1724cb3d3e2 2 &blobSubscriptionUuid;
&blobImageIdExampleSubscribed; 3 &blobSubscriptionUuid;
da42ca85-179b-4873-b12e-32d549bf02b6 2 &blobSubscriptionUuid;</programlisting>
<para>The results show the &uuid; of the image, the availability state and the
subscription &uuid;. The state value is a bitmap, 1 is subscribed, 2 means its
available in the current &vmil;s. Now we will select an image for local caching.</para>
<section>
<title>Selecting Images</title>
<para>Images can be selected by either &uuid; or Sha512 hash. This allows explicit
selection of images or by the sha512 from an old image.</para>
<para>Delete the subscription by image.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -D -u 327016b0-6508-41d2-bce0-c1724cb3d3e2</userinput></programlisting>
<para>Subscribe to an image.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -a -u 327016b0-6508-41d2-bce0-c1724cb3d3e2</userinput></programlisting>
<para>Unsubscribe an image</para>
<programlisting>&prompt-user; <userinput>vmcatcher_image -r -u 327016b0-6508-41d2-bce0-c1724cb3d3e2</userinput></programlisting>
</section>
</section>
<section id="imagetransfer-software-vmcatcher-vmcatcher_cache">
<title>&vmcatcher_cache; </title>
<para> This application downloads images. By default it will download images, check the sha512
hash of cached images and expire images from old &vmil;s.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_cache</userinput>
INFO:vmcatcher_subscribe.main:Defaulting DB connection to 'sqlite:///vmcatcher.db'
INFO:vmcatcher_subscribe.main:Defaulting actions as 'expire', 'sha512' and 'download'.
INFO:vmcatcher_subscribe.main:Defaulting cache-dir to 'cache'.
INFO:vmcatcher_subscribe.main:Defaulting partial-dir to 'cache/partial'.
INFO:vmcatcher_subscribe.main:Defaulting expired-dir to 'cache/expired'.
INFO:DownloadDir:Downloading '&blobImageIdExampleSubscribed;'.
INFO:CacheMan:moved file &blobImageIdExampleSubscribed;
</programlisting>
<section>
<title>&vmcatcher_cache; Event interface</title>
<para>Since this application suite is intended to be embedded in a larger application
and concerned with downloading and managing updates of VM images into a cloud
infrastructure, it is some times beneficial to have an event interface so that
applications may embed these applications in larger systems.</para>
<programlisting>&prompt-user; <userinput>vmcatcher_cache -x "/usr/bin/VmImageUpdateProcessor \$VMCATCHER_EVENT_TYPE"</userinput></programlisting>
<para>The events interface launches a shell with a series of environment variables. The
event must process its command within 10 seconds or else it will be sent a
termination signal. See the following example:</para>
<programlisting>&prompt-user; <userinput>vmcatcher_cache -x 'env ; exit 1'</userinput></programlisting>
<para>All Events have a type. This is given to the event handler by setting the
variable, &VMCATCHER_EVENT_TYPE; with the following values "AvailablePrefix",
"AvailablePostfix", "ExpirePrefix" and "ExpirePosfix".</para>
<para>"Available" events happen when a new image is validated, while "Expire" events
occur when an image i no longer the validated image. The "Prefix" events occur
before the file changes state, and the "Posfix" events occur after the state
change.</para>
<para>The following environment variables may be set by events:</para>
<itemizedlist>
<listitem>VMCATCHER_EVENT_TYPE</listitem>
<listitem>VMCATCHER_EVENT_DC_DESCRIPTION</listitem>
<listitem>VMCATCHER_EVENT_DC_IDENTIFIER</listitem>
<listitem>VMCATCHER_EVENT_DC_TITLE</listitem>
<listitem>VMCATCHER_EVENT_HV_HYPERVISOR</listitem>
<listitem>VMCATCHER_EVENT_HV_SIZE</listitem>
<listitem>VMCATCHER_EVENT_HV_URI</listitem>
<listitem>VMCATCHER_EVENT_SL_ARCH</listitem>
<listitem>VMCATCHER_EVENT_SL_CHECKSUM_SHA512</listitem>
<listitem>VMCATCHER_EVENT_SL_COMMENTS</listitem>
<listitem>VMCATCHER_EVENT_SL_OS</listitem>
<listitem>VMCATCHER_EVENT_SL_OSVERSION</listitem>
<listitem>VMCATCHER_EVENT_TYPE</listitem>
<listitem>VMCATCHER_EVENT_FILENAME</listitem>
<listitem>VMCATCHER_EVENT_IL_DC_IDENTIFIER</listitem>
</itemizedlist>
<para>These correspond to the variables within the &vmil;.</para>
<section>
<title>&vmcatcher_cache; Event Environment variables</title>
<section>
<title>VMCATCHER_EVENT_TYPE</title>
<itemizedlist>
<listitem>AvailablePrefix<para>An image will be available soon as it is being attempted to be retrieved.</para></listitem>
<listitem>AvailablePostfix<para>An image was successfully validated as being available and placed in the cache
directory.</para></listitem>
<listitem>ExpirePrefix<para>This image is will no longer be available in the cache directory.</para></listitem>
<listitem>ExpirePosfix<para>This image is no longer in the cache directory.</para></listitem>
</itemizedlist>
</section>
<section>
<title>VMCATCHER_EVENT_DC_DESCRIPTION</title>
<para>The description text in the image.</para>
</section>
<section>
<title>VMCATCHER_EVENT_DC_IDENTIFIER</title>
<para>Unique identifier of the image. Its suggested that image producers use RFC 4122
&uuid; for &vmil; this allows updating the list, and uniqueness.</para>
</section>
<section>
<title>VMCATCHER_EVENT_DC_TITLE</title>
<para>Image Title.</para>
</section>
<section>
<title>VMCATCHER_EVENT_HV_HYPERVISOR</title>
<para>Typically set to reflect the Virtualization technology values such as "xen", "kvm".</para>
</section>
<section>
<title>VMCATCHER_EVENT_HV_SIZE</title>
<para>The Image Size</para>
</section>
<section>
<title>VMCATCHER_EVENT_HV_URI</title>
<para>The Original URI for the image</para>
</section>
<section>
<title>VMCATCHER_EVENT_SL_ARCH</title>
<para>The images architecture.</para>
</section>
<section>
<title>VMCATCHER_EVENT_SL_CHECKSUM_SHA512</title>
<para>The Images sha512 checksum.</para>
</section>
<section>
<title>VMCATCHER_EVENT_SL_COMMENTS</title>
<para>Comments added by the image author</para>
</section>
<section>
<title>VMCATCHER_EVENT_SL_OS</title>
<para>The Operating System the VM image contains</para>
</section>
<section>
<title>VMCATCHER_EVENT_SL_OSVERSION</title>
<para>The Operating System version</para>
</section>
<section>
<title>VMCATCHER_EVENT_FILENAME</title>
<para>The Image file name.</para>
</section>
<section>
<title>VMCATCHER_EVENT_IL_DC_IDENTIFIER</title>
<para>The image list the image comes from.</para>
</section>
<section>
<title>VMCATCHER_EVENT_HV_FORMAT</title>
<para>The format of the image. This is only available if the image list contains the format metadata.</para>
</section>
</section>
</section>
</section>
<section id="imagetransfer-software-vmcatcher-config">
<section id="imagetransfer-software-vmcatcher-config-cron">
<title>Set up for Production using Cron</title>
<para>Then the by hand configuration for your master DB</para>
<programlisting>&prompt-root; <userinput>useradd vmcatcher</userinput></programlisting>
<programlisting>&prompt-root; <userinput>mkdir -p /var/lib/vmcatcher /var/cache/vmimages/endorsed \
/var/cache/vmimages/partial /var/cache/vmimages/expired</userinput></programlisting>
<programlisting>&prompt-root; <userinput>touch /var/log/vmcatcher.log</userinput></programlisting>
<programlisting>&prompt-root; <userinput>chown vmcatcher:vmcatcher /var/lib/vmcatcher /var/cache/vmimages/endorsed \
/var/cache/vmimages/partial /var/cache/vmimages/expired \
/var/log/vmcatcher.log</userinput></programlisting>
<programlisting>&prompt-root; <userinput>sudo -u vmcatcher /usr/bin/vmcatcher_subscribe \
-s &blobUrl; \
-d sqlite:////var/lib/vmcatcher/vmcatcher.db</userinput></programlisting>
<para>make a cron job</para>
<programlisting>&prompt-root; <userinput> cat /etc/cron.d/vmcatcher</userinput>
export VMCATCHER_RDBMS="sqlite:////var/lib/vmcatcher/vmcatcher.db"
export VMCATCHER_CACHE_DIR_CACHE="/var/cache/vmimages/endorsed/"
export VMCATCHER_CACHE_DIR_DOWNLOAD="/var/cache/vmimages/partial/"
export VMCATCHER_CACHE_DIR_EXPIRE="/var/cache/vmimages/expired/"
export VMCATCHER_CACHE_EVENT="python /usr/share/doc/&vmcatcher-latest;/vmcatcher_eventHndlExpl --output_file=/tmp/foo --datetime"
50 */6 * * * vmcatcher (/usr/bin/vmcatcher_subscribe -U; /usr/bin/vmcatcher_cache ) >> /var/log/vmcatcher.log 2>&1</programlisting>
<para>So the script is executed every 6 hours shortly after fetch CRL.</para>
<para>If a new &vmi; is downloaded, or an old &vmi; is expired the event will trigger &VMCATCHER_CACHE_EVENT; and
the application &vmcatcher_eventHndlExplscript; will append the data to /tmp/foo</para>
<para>Now at any time users with file permissions can get a list of valid images.</para>
<programlisting>&prompt-user; <userinput>VMCATCHER_RDBMS="sqlite:////var/lib/vmcatcher/vmcatcher.db" vmcatcher_image -l</userinput></programlisting>
</section>
<section id="imagetransfer-software-vmcatcher-events">
<title>Replacing the event handler</title>
<para>&vmcatcher_cache; produces "events" in the form of launching an application when
a new image is downloaded or expired, this application is then launched with
environment variables that include the image ID,date, etc etc. Since all the
information about current images is available from other vmcatcher commands
you don't need to handle events but it does make it simpler for some setups.</para>
<para>So with a cron job like:</para>
<programlisting>&prompt-root; <userinput> cat /etc/cron.d/vmcatcher</userinput>
export VMCATCHER_RDBMS="sqlite:////var/lib/vmcatcher/vmcatcher.db"
export VMCATCHER_CACHE_DIR_CACHE="/var/cache/vmimages/endorsed/"
export VMCATCHER_CACHE_DIR_DOWNLOAD="/var/cache/vmimages/partial/"
export VMCATCHER_CACHE_DIR_EXPIRE="/var/cache/vmimages/expired/"
export VMCATCHER_CACHE_EVENT="python /usr/share/doc/&vmcatcher-latest;/vmcatcher_eventHndlExpl --output_file=/tmp/foo --datetime"
50 */6 * * * vmcatcher (/usr/bin/vmcatcher_subscribe -U; /usr/bin/vmcatcher_cache ) >> /var/log/vmcatcher.log 2>&1</programlisting>
<para>The &VMCATCHER_CACHE_EVENT; environment variable specifies the even handler.</para>
<para>It is expected that &vmcatcher_eventHndlExplscript; will be replaced by sites wanting to
load images into image catalogues of popular clouds. "vmcatcher_eventHndlExplscript" is an example
"event" handler, which takes a path parameter. It reads the environment variables generates a
simple JSON output line, and appends it to the file described in the path parameter.</para>
<para>It is recommended the replacement &vmcatcher_eventHndlExplscript; copies the images
from the &VMCATCHER_CACHE_DIR_CACHE; directory to the cloud, as &vmcatcher_cache; assumes
it can delete and update its local cache. A second recommendation for site replacement
is that &vmcatcher_eventHndlExplscript; should do very little and end quickly as then
&vmcatcher_cache; can process the next download without blocking. Since events are not
resent so error handling is more complex, it may be wise to use a message queue, or
storing the event and processing after, rather than just using a simple fork.</para>
</section>
<section id="imagetransfer-software-vmcatcher-config-logging">
<title>Logging configuration</title>
<para>All scripts have a logging option. This is used to configure pythons logging
library. An example is shown below.</para>
<programlisting>&prompt-user; <userinput> vmcatcher_image -L /usr/share/doc/vmcatcher/logger.conf -l</userinput></programlisting>
<para>Logging can be independently set up for each object to multiple locations, and
with different log levels.</para>
</section>
</section>
<section id="imagetransfer-software-imagelist-consumer-status-todo">
<title>To Do (16-05-2012)</title>
<itemizedlist>
<listitem>Only message authenticity is checked, does not yet check authenticity of transport.</listitem>
<listitem>PGP signatures.</listitem>
<listitem>Support encrypted messages.</listitem>
</itemizedlist>
<para>While it does check the authenticity of the message using &x509;, at the moment the
authenticity of the host is unchecked. For the ease of programing it would be far simpler to
use &x509; certificates to check the host server. In terms of deployment it would be far
easier just to check any host key mechanism, as this is sufficient.</para>
</section>
</section>
</chapter >