Cyphertite configuration

Table of Contents Linux/BSD single machine initial setup running the first backup restoring data multiple machines differential backups metadata operations COMING SOON - Mac OSX: IPv6 support

Linux/BSD

single machine

Setting up cyphertite on a single machine should take less than 10 minutes and only requires a few dials to be set. Now that cyphertite is installed the following files must be put in place on your machine. We assume the user has username testuser for the purposes of the instructions below:

initial setup

The name 'testuser' is an example username for the purposes of this example. Please replace 'testuser' throughout with your actual username. User input is shown in bold.

1. run the cyphertite configuration file generator
   $ cyphertitectl config generate
   Use expert setup mode? [no]: <return>
   login username: testuser
   login password: your account password
   Downloading certificates...
   Generating crypto secrets file...
   Uploading secrets file to server...
   Configuration file created.
   WARNING: It is highly recommended that you store your '/home/testuser/.cyphertite/cyphertite.conf' file to an offline location, preferrably offsite, as your crypto passphrase CANNOT be recovered.
   Examples:
   - Copy it to a USB memory drive
   - Print a copy of it and store it in a fire proof safe
   
   In this example, the user chose not to use expert mode. The result is the following config file:
   username = testuser
   password = supersikritpassword!!
   crypto_passphrase = hcZe5ngC4cRcZFo+s7YDyfT62IQ0YekgTcHhRrWmbj95714OSbGab9mtYu555HurjNIx6BILA0lFptDOaQq37w==
   cache_db = /home/testuser/.cyphertite/ct_db
   session_compression = lzo
   crypto_secrets = /home/testuser/.cyphertite/ct_crypto
   ca_cert = /home/testuser/.cyphertite/ct_certs/ct_ca.crt
   cert = /home/testuser/.cyphertite/ct_certs/ct_testuser.crt
   key = /home/testuser/.cyphertite/ct_certs/private/ct_testuser.key
   ctfile_mode = remote
   ctfile_cachedir = /home/testuser/.cyphertite/ct_cachedir
   ctfile_remote_auto_differential = 0
   upload_crypto_secrets = 1

WARNING: you must keep track of your crypto passphrase, if it is lost you cannot restore your backups. This passphrase cannot be reset since it is not shared with us. We recommend saving cyphertite.conf and ct_crypto in a safe offline location.

running the first backup

At this point you should have everything setup correctly for running your first backup. To start a backup, one must choose (1) the filename for local metadata storage and (2) choose a directory to backup. We use <md_path></md_path> and <backup_dir></backup_dir> for (1) and (2) in the example below

cyphertite &#45&#59;cvRf &lt&#59;md_path&gt&#59; &lt&#59;backup_dir&gt&#59;

The first time you run cyphertite it will create a crypto secrets file in the location specified in cyphertite.conf. Once cyphertite starts running it first maps all the files and subdirectories it will be backing up. When everything is mapped the backup begins and data is sent to the server.

$ cyphertite &#45&#59;cvRf 110522_usr_local_bin_1.ct /usr/local/bin
usr
usr/local
usr/local/bin
usr/local/bin/expectk
usr/local/bin/multixterm
...

Note that if the user running cyphertite lacks sufficient permissions on any files, they cannot backup those files. You should see something like the following when the backup completes:

...
usr/local/bin/MagickCore&#45&#59;config
usr/local/bin/Wand&#45&#59;config
usr/local/bin/GraphicsMagick++&#45&#59;config
Files scanned                         1244
Total bytes                       99235002      (237KB/sec)
Total chunks                          1394
Bytes read                        99200269      (237KB/sec)
Bytes written                            0
Bytes compressed                  50801470      (121KB/sec)
Bytes crypted                     50812192
Bytes dbexists                      112368
Bytes sent                        40779904
Chunks completed                      1394
Bytes sha                         99200269
Bytes crypt                              0
Bytes csha                        50812192
Files completed                          0
Scan Time                              1.370s 
Total Time                           409.409s 
ssl bytes written 40848308
ssl writes        5088
avg write len     8028
ssl bytes read    40704
ssl reads         5088
avg read len      8

When running cyphertite in e.g. cron it is best to remove the -v option because it generates a large amount of unused output. It is instructive to see exactly what cyphertite is doing the first time you run a backup. The -R option generates performance statistics and should likely be turned off when running via cron or in batch mode.

restoring data

Whenever setting up a backup system it is important to test your backups to ensure that you can actually recover your data. It is assumed that a backup completed successfully as above. To restore files or directories cyphertite requires (1) the metadata file that contains the files or directories to restore, (2) a glob or regex pattern for the files to restore and (3) a directory into which the files or directories will be restored. In the absence of (2), a glob or regex, the entire contents of the metadata file, (1), will be restored to the current directory.

Before restoring data it is frequently helpful to list the desired files or directories, (2). This is achieved by using the -t option on the metadata of interest, (1).

$ cyphertite &#45&#59;tf 110522_usr_local_bin_1.ct &#42&#59;expect&#42&#59;  
usr/local/bin/expectk
usr/local/bin/autoexpect
usr/local/bin/expect

NOTE: We have prepended a * because the glob matching is on the full path of the file or directory being searched for. Without the first * cyphertite would not match the usr/local/bin/ portion of the full path.

Now that it is clear that there are indeed files or directories that match the glob pattern, *expect*, we can restore them.

$ cyphertite &#45&#59;C /tmp &#45&#59;vxf 110522_usr_local_bin_1.ct &#42&#59;expect&#42&#59;
usr/local/bin/expectk
usr/local/bin/autoexpect
usr/local/bin/expect
$ ls &#45&#59;al /tmp/usr/local/bin/
total 68
drwxr&#45&#59;xr&#45&#59;x  2 root  wheel    512 May 22 19&#58&#59;56 .
drwxr&#45&#59;xr&#45&#59;x  3 root  wheel    512 May 22 19&#58&#59;56 ..
&#45&#59;r&#45&#59;xr&#45&#59;xr&#45&#59;x  1 root  wheel   7625 May 22 19&#58&#59;56 autoexpect
&#45&#59;r&#45&#59;xr&#45&#59;xr&#45&#59;x  1 root  wheel   7584 May 22 19&#58&#59;56 expect
&#45&#59;r&#45&#59;xr&#45&#59;xr&#45&#59;x  1 root  wheel  13632 May 22 19&#58&#59;56 expectk

In this example (1) is 110522_usr_local_bin_1.ct, (2) is *expect* and (3) is /tmp. Notice that the files have been restored to their full path, usr/local/bin, prepended by the restore directory, /tmp.

As per the initial backup, we have used options that are not required or advised for batch execution. It is usually not necessary to use the -v option, one can also use the -R option to get performance statistics on the restore process.

multiple machines

Having configured and tested cyphertite running on a single machine it is straightforward to setup additional machines that backup using the same account. The main utility of this is to obtain the benefit of doing realm deduplication, wherein multiple machines can share a pool of chunks and receive the corresponding space savings from this process. An example would be that file A is stored on 10 machines, named M0 to M9: only a single copy of A would need to be stored for all the ten machines M0 to M9.

In what follows the configured machine is called host1 and the new machine is called host2.

1. install cyphertite on host2
2. copy the contents of /etc/cyphertite from host1 to the new machine, host2. this can be done a number of ways, one example is
   (on host1)
   sudo scp -r /etc/cyphertite host2:/etc/

All the commands used on the single machine configuration above should work on host2 just as they did on host1. The only difference is that host2 does not share its local cache database with host1. The entire process should take a few minutes to perform manually.

Having multiple machines sharing a single account is a good idea in several cases

• workstations and laptops that share a lot of data e.g. music and video for a family
• group of machines at a single site e.g. the accounting department machines
• group of machines spread across sites e.g. corporate firewalls at multiple locations

cyphertite is built so that users do not pay per machine protected but rather by the amount of data that is stored.

differential backups

In order to keep backup windows short after the initial backup, cyphertite performs differential backups using the modification time (mtime) of the files on the filesystem. Files or directories that have been modified after the time at which the previous backup started are backed up. Doing this requires that there be (1) an existing metadata file upon which (2) the current differential metadata is based and (3) the same backup target. To build on the examples from above

$ cyphertite &#45&#59;B 110522_usr_local_bin_1.ct &#45&#59;cvf 110523_usr_local_bin_2.ct /usr/local/bin
prev backup time Sun May 22 19&#58&#59;24&#58&#59;35 2011
 110522_usr_local_bin_1.ct
usr
usr/local
usr/local/bin
usr/local/bin/avahi&#45&#59;browse&#45&#59;domains
usr/local/bin/ldapadd
usr/local/bin/unlzma
...
$ ls &#45&#59;al &#42&#59;_usr_local_bin&#42&#59;                                                            
&#45&#59;rw&#45&#59;r&#45&#59;&#45&#59;r&#45&#59;&#45&#59;  1 root  wheel  244588 May 22 19&#58&#59;31 110522_usr_local_bin_1.ct
&#45&#59;rw&#45&#59;r&#45&#59;&#45&#59;r&#45&#59;&#45&#59;  1 root  wheel   16400 May 23 11&#58&#59;17 110523_usr_local_bin_2.ct

The differential metadata file is much smaller than the first one and takes much less time to create. When restoring from differential metadata, everything works as in the usual restore case, except that the files are restored recursively from the prior metadata up to the differential metadata file specified.

usr/local/bin/pnmtotiff
usr/local/bin/pnmtotiffcmyk
should start restoring &#91&#59;110523_usr_local_bin_2.ct&#93&#59;
usr/local/bin/pnmtoxwd
usr/local/bin/zeisstopnm
...

The restores of files with multiple versions are done "on top of each other", leaving only the most recent copy in the latest metadata. This behavior will be changed in the future and there will be an interface that allows users to select which version(s) of such files to restore.

metadata operations

By default metadata is created locally and should be kept offsite along with the rest of the data, which is stored as chunks on our remote servers. There are currently two cyphertite.conf settings that are used for automatically uploading metadata to our servers, md_mode and md_cachedir. md_mode can be set to either 'local' or 'remote', and when set to 'remote' md_cachedir must be set to a local directory where metadata will be cached.

To build on the same testuser scenario, one would add the following lines to cyphertite.conf:

md_mode &#61&#59; remote
md_cachedir &#61&#59; /home/testuser/.cyphertite/ct_cachedir

The metadata file referenced by the next cyphertite command will then have a date prepended to it and be uploaded to our servers automatically, and a copy will be stored locally in md_cachedir:

$ cyphertite &#45&#59;cvRf host.domain.com_full.ct /

This would result in a metadata file being created with name YYYYMMDD-hhmmss-host.domain.com_full.ct, where the full date is prepended as per the earlier comments, both locally in ct_cachedir and an encrypted copy stored remotely on our servers. One can check that the metadata uploaded successfully with

$ cyphertite &#45&#59;mt

This will list the metadata files stored on our servers under your account.

COMING SOON - Mac OSX:

work-in-progress

IPv6 support

All cyphertite infrastructure runs on both IPv4 and IPv6. Everything "just works" over IPv6, there are no dials to set to have this work on your end as a client besides using IPv6.

If you have any questions or problems configuring Cyphertite, please visit our Cyphertite support page and feel free to contact us!

You can also return to the Cyphertite main wiki page.