README File for "sph2pipe"

1. Introduction

The "sph2pipe" program was created by the Linguistic Data Consortium to provide greater flexibility and ease of use for SPHERE-formatted digital audio data. It is equivalent in most respects to the related utility "sph_convert", but each of these tools provides some abilities that the other does not. Here is a brief summary of the similarities and differences.

Both sph_convert and sph2pipe will:

• work on all Microsoft Windows systems, via the "MS-DOS" command line prompt
• read any SPHERE-formatted data file and convert it to Microsoft RIFF ("WAV") format, Sun/Java AU format, MAC AIFF format or raw (headerless) format
• automatically uncompress SPHERE files that have been compressed using the "shorten" algorithm (often used in LDC speech corpora)
• allow demultiplexing of two-channel waveform data, to output one or the other channel alone
• allow conversion of the sample data to 16-bit linear PCM or to 8-bit mu-law encoding, regardless of the input sample encoding

Only sph_convert can:

• run on older (pre-OSX) Macintosh systems, via the old Mac-style GUI
• do multiple file conversions in a single run (sph2pipe only does one file at a time); there are two methods for doing "batches":
  • treat all files in a chosen directory that match a user-specified file-name pattern, or
  • treat all files in all subdirectories under a chosen base directory
• in either case, convert all SPHERE files and copy (or bypass) all non-SPHERE files

Only sph2pipe can:

• run on UNIX systems (should also work on MacOS X, via its unix shell/command-line interface, using the "Terminal" utility)
• provide SPHERE-formatted output as well as RIFF, AU, AIFF and raw
• handle raw sample data as input, using a SPHERE header stored in a separate file.
• trim off the beginning and/or end of the input data, to output just a user-specified segment based on either time or sample offsets (sph_convert always outputs the entire file)
• write the output data to stdout, for redirection to any named file, or to a pipeline process (sph_convert always writes the data to a new file, with a name derived automatically from the input file)
• support input and output of A-law speech data

When installed on MS Windows or MacOS, these tools will produce RIFF output files by default; when compiled for UNIX systems (Linux, Solaris, etc), sph2pipe will output SPHERE format by default. In any case, the user has the option to specify what format is desired -- any machine can be used to generate any kind of output. (Well, a Mac that is running OS-9 or older cannot produce SPHERE output, but we haven't heard any requests for that...)

Sph2pipe will not work on older Mac systems because the notion of a pipeline command did not exist on Macs prior to OS X. Of course, it is possible to create custom-edited RIFF/AIFF/AU/raw files using sph2pipe on unix or wintel, then copy those files to an older Mac; but the combination of sph_convert and any of several waveform editing tools for Macs can provide all the functionality of sph2pipe, and then some.

The "shorten" speech compression technique, used in the LDC's publication of many speech corpora, was developed by Tony Robinson, originally at Cambridge University; "shorten" is available from SoftSound, Inc. (http://www.softsound.com/Shorten.html). The algorithm and source code for uncompressing "shortened" speech data are included here by permission of Tony Robinson and SoftSound, Inc.

People who have used the original "shorten" package (dating from the mid-1990's) will find that sph2pipe is more much flexible, because of the range of options available for controlling output. UNIX users who are familiar with the NIST SPHERE utilities "w_decode" and "w_edit" will find that sph2pipe runs faster and is easier to use, especially when extracting a subset of data from a compressed file: in this case sph2pipe alone handles a job that would require both w_decode and w_edit, and works a lot quicker (and also avoids a nasty bug in the sphere_2.6a package that can arise when you try to run w_decode and w_edit together in a pipeline).

Note that sph2pipe and sph_convert are NOT able to do sample-rate conversion. If you have a need for this, try the "SoX" package -- see under "Licensing" below for more information about SoX.

2. Installation

Wintel users can simply download the executable file (sph2pipe.exe) that has been precompiled for MS Windows/DOS systems, and start using it. (You can download the source files too, if you have your own C compiler and want to customize the program for your needs.) UNIX and MacOS X users are advised to compile the program from the source code.

To build from sources, download "sph2pipe_v2.4.tgz", and do this:

-- if you have the Gnu version of tar (standard on linux):

 tar xzf sph2pipe_v2.4.tgz

-- otherwise (with Wintel systems or non-Gnu versions of tar):

 gzip -c -d sph2pipe_v2.4.tgz | tar xf -

-- then:

 cd sph2pipe_v2.4

 gcc -o sph2pipe *.c -lm     ## on unix

or gcc -o sph2pipe.exe *.c -lm ## on wintel, using the djgpp compiler

That's it -- no configuration scripts, makefiles or special libraries are needed (the source code consists of just 3 *.c files, and 3 *.h files; the standard math library is needed for compilation). Put the resulting "sph2pipe" executable in your path and start using it. If you don't have gcc, try whatever C compiler you do have; you might need to change a few details in sph_convert.h, but we hope the code is generic enough (POSIX compliant) to work anywhere.

3. Usage

The command line syntax is:

sph2pipe [-h hdr] [-t|-s b:e] [-c 1|2] [-p|-u|-a] [-f typ] infile [outfile]

-h hdr -- treat the input file as raw (headerless) sample data, and read header information from a separate file, given as the "hdr" argument; the "hdr" must contain a valid SPHERE header that correctly describes the nature of the input sample data ("hdr" may contain actual sample data as well, which will be ignored). If the output format is "sph", the SPHERE header in "hdr" will be written first, with appropriate adjustments where needed. (When this option is not used, "input" must begin with a valid SPHERE header.)

-t b:e -- output only the portion of waveform data that lies between the stated beginning and ending points, given in seconds, as positive real numbers; "b" defaults to start-of-file, "e" defaults to end-of-file -- so the following usages are valid:

     "-t :10.05"  (output first 10.05 sec, skip the rest)
     "-t 4:"      (skip first 4 sec, output the rest)
     "-t 4:10.05  (output 6.05 sec, starting at 4 sec in)

-s b:e -- output only the portion of waveform data that lies between the stated beginning and ending points, given in samples as positive integers; "b" defaults to start-of-file, "e" defaults to end-of-file -- so the following usages are valid:

     "-s :32000"    (output first 32K samples, skip the rest)
     "-s 8000:"     (skip first 8K samples, output the rest)
     "-s 8000:32000 (output 24K samples, starting at 8K in)

-c 1 or -c 2 -- output only the first or second channel, in case input is two-channel (has no effect if input is single channel); default is to output all channels

-p -- force 16-bit PCM output, in case input is something else (has no effect if input is already 16-bit PCM)

-u -- force 8-bit mu-law output, in case input is 16-bit pcm (has no effect if input is already mu-law)

-a -- force 8-bit a-law output, in case input is 16-bit pcm (has no effect if input is already a-law)

 The -p, -u and -a options are ignored if "-f aif" is used,
 because AIFF only supports PCM samples.  When none of these
 three is specified, the default behavior is to leave original
 sample format "as is" (or to force PCM if using "-f aif")

-f fmt -- selects the output header format; "fmt" can be: rif (or wav) -- default for Wintel & Mac systems aif (or mac) -- similar to rif, but more Mac-ish... sph -- SPHERE format, default on unix systems au -- common on Sun/Java/Next raw -- i.e. headerless

If only one file name is given on the command line, output is written to stdout (i.e. for redirection via "> output.file", or for input to a pipeline). If a second file name is given, output is written directly to a file with this name, and not to stdout; if the named output file already exists and contains data, its contents will be overwritten (replaced) by the sph2pipe output.

If the output format is RIFF, AU, AIFF or SPH, a fully specified and correct file header is written first (*). When writing via stdout to a pipeline, a downstream process can behave exactly as it would for a valid disk file in the target format (except that "seek()" does not work on stdin, of course).

(*) Note: for SPHERE-formatted output, sph2pipe will eliminate the "sample_checksum" field, since this cannot be given a correct value prior to processing and writing the output data. Also, when converting PCM input to mu-law or a-law, sph2pipe removes the "sample_byte_format" header field, which defines the byte order for 16-bit sample data. Apart from these two circumstances, the output sphere header retains all information in the original input header, along with appropriate changes, where necessary, to the sample_count, channel_count, sample_coding, sample_n_bytes, sample_byte_format and sample_sig_bits fields, making the header information consistent with the data being written.

A useful benefit provided by pipeline operation is the ability to "compose" a single output file by concatenating any number of input files, or pieces of one or more input files. For instance, to combine all the speech data in one directory into a single file for signal analysis (using bash as the command-line shell, which is available for wintel systems as well as for unix):

$ for i in *.sph; do

sph2pipe -f raw $i >> allsph.raw done

Or, to put together a set of excerpts that you want to play back during your next PowerPoint presentation:

sph2pipe -f raw -t 0:1 empty.sph > silence.raw sph2pipe -f sph -t 0:1 empty.sph > slideshow.sph sph2pipe -f raw -t 15.5:18.2 example1.sph >> slideshow.sph cat silence.raw >> slideshow.sph sph2pipe -f raw -t 300:305.5 example2.sph >> slideshow.sph cat silence.raw >> slideshow.sph sph2pipe -f raw -t 1832:1838 example3.sph >> slideshow.sph cat silence.raw >> slideshow.sph ... sph2pipe -f wav slideshow.sph > slideshow.wav

Note the use of "raw" format to concatenate waveform data (we don't want file headers to be interspersed with the speech). Also, in the second example, the sphere header that is initially created for "slideshow.sph" will be "numerically" correct only in reference to the initial one-second chunk; as more segments are appended to this file, the "sample_count" field in the header will be further and further from the truth. But this doesn't matter -- at the final stage, when this file is converted to RIFF, sph2pipe will notice the discrepancy between the "sample_count" value in the header and the actual size of the file, and will automatically correct the sample_count to be consistent with the file size.

There are important rules to follow when combining segments from multiple files. If you happen to violate any of these rules, the resulting output will certainly come out sounding wrong (sometimes painfully so):

(1) be sure that all the input files have the same sampling rate. (2) be sure to append data using a consistent number of channels, always a single channel, or always two channels (3) it's a good idea to specify "-p" on all runs -- or "-u" or "-a" on all runs -- to guarantee that the output file will have the same sample coding throughout, no matter what the original sample codings may have been in the source files

When combining data from files in any single LDC corpus, these issues normally won't pose any problem: within a given corpus, all files tend to have the same properties.

4. Version specific information

This version will only convert one sphere file in one run, and must read that file directly from disk or cdrom (it does not accept input via stdin, because it must be able to do "fseek()" on the input file). Handling bunches of files is easily done on both unix and wintel systems using generic tools like the unix "bash" shell, the unix "find" utility, and/or the Perl or Python scripting languages; fully capable ports of all these tools are available for wintel systems.

Version History:

• Version 2.0 was the first "public" release; it did not support a-law sample coding, AU or AIFF output formats, the "-h hdrfile" option, or the "-s|-t bgn:end" options. It contained a significant bug that arose when converting some 16-bit PCM sphere files to ulaw output.

• Version 2.1 provided a fix for the pcm-to-ulaw bug.

• Version 2.2 added the options for AU and AIFF output formats.

• Version 2.3 added the "-s|-t" options to select regions for output based on sample or time offsets, and also added the "-h" option for using "stand-off" sphere headers with raw sample data files.

• Version 2.4 added support for a-law sample coding, and added a thorough test suite, allowing end users to verify their installation; there were some minor bug fixes involving the "-h" option; the README file has also been revised to bring various URL's up to date.

• Version 2.5 added the ability to include an output file name as a command line argument; this was done to avoid concerns on MS-Windows systems about some command-line shells that impose "text-mode" alterations to data when running commands with redirection or pipes.

5. License

Various portions of source code from Tony Robinson's "shorten-2.0" package are used here by permission of Tony Robinson and SoftSound, Inc. http://www.softsound.com -- these portions are found in the file "shorten_x.c"; please note the copyright information in that file. By agreement with Tony Robinson and SoftSound, Inc, the Linguistic Data Consortium (LDC) grants permission to copy and use this software for the purpose of reading "shorten"-compressed speech data provided in NIST SPHERE file format by the LDC or others. SoftSound provides useful tools for audio compression and other signal processing tasks.

Other portions of source code (in particular the "writeRIFFHeader" and "writeAIFFHeader" functions in "file_headers.c", and the "alaw2pcm" conversion function) were adapted from the "SoX" package, a valuable open-source tool maintained primarily by Chris Bagwell, with assistance from many others (http://sox.sourceforge.net/). We gratefully acknowledge the value provided by all contributors to SoX; sph2pipe would have been much harder to write without this resource. We recommend that you use SoX if you need to do sample-rate conversion on audio data.