Get rid of intermediate files (.o's in .a's for example)

[bdbaddog]

This issue was originally created at: 2003-07-18 19:56:35.
This issue was reported by: issues@scons.
issues@scons said at 2003-07-18 19:56:35

env = Environment()
env.StaticLibrary(target = 'foo', source = ['a.F', 'b.F', 'c.F','d.F'])

makes a libfoo.a, and leaves a.o, b.o, c.o, and d.o.

I would like for the "intermediate" object files to be deleted. Can this be done ?

Just FYI, here is an example of make handling intermediate objects.

bdavis@rh ~]$ cat Makefile
 
libxyz.a: libxyz(a.o) libxyz(b.o)
c:libxyz.a c.c
   cc -o c c.c libxyz.a
 
[bdavis@rh ~]$ make
 
cc    -c -o a.o a.c
ar rv libxyz a.o
a - a.o
cc    -c -o b.o b.c
ar rv libxyz b.o
a - b.o
rm a.o b.o
 
[bdavis@rh ~]$ touch a.c
[bdavis@rh ~]$ make
 
cc    -c -o a.o a.c
ar rv libxyz a.o
r - a.o
rm a.o

stevenknight said at 2003-09-01 05:29:30

From Sergey Fomel, 31 August 2003:

1. Question:

Suppose I have target B built from source A and target C built from source B. I would like to have the following behavior:

If asked to build target C, don't do anything if C is newer then A and if B does not exist. In other words, don't rebuild the intermediate target B only because it does not exist (got cleaned).

In GNU make, one can achieve this behavior with the .SECONDARY keyword. See http://www.gnu.org/manual/make/html_node/make_103.html (The .SECONDARY feature got broken in certain versions of gmake but is fixed in the latest release.)

Question: Is it possible to enforce this sort of behavior in SCons? If not, can I submit a feature request (a Default-like Secondary function to list all intermediate targets considered "secondary")?

2. Explanation:

According to Kernighan, "make" is a program "debugged into existence". The .SECONDARY feature was commissioned from Richard Stallman by Stanford Exploration Project (SEP), a scientific group, where I got my Ph.D.

SEP uses GNU make in a creative way to maintain the discipline of scientific research named "reproducible research". This tradition is described in the paper "Making scientific computations reproducible" (CiSE, 2000, v.2, p.61-67 and on the web at http://sepwww.stanford.edu/research/redoc/cip.html )

The main idea is the following: The results of research in computational geophysics (and possibly many other scientific and engineering disciplines) are often presented in the form of pictures. Generating a picture usually requires a long complicated chain of commands, which can be preserved in a batch form. Capturing the full flow of information from the input sources (real data and scientific software) to the final results with a dependency tool like make or scons gives us an ability to easily reproduce and verify the results.

This is a very powerful idea. However, its practical implementation resulted in a horrible mess of gmake rules and specialized Perl scripts (for things that were too complicated for make to handle). The package works at Stanford, but it confuses the hell out of unexperienced users (students and scientists), is prone to errors of different kinds (including the infamous make's tabulation), and is a nightmare to port to a different environment. That is one of the reasons why "reproducible research" did not spread outside of SEP.

SCons solves all these problems brilliantly and could bring new life to the practice of reproducible research. Now, if we only could get the "secondary" feature... This feature is important if you want to clean the intermediate files while preserving the sources and the final results and while having an ability to verify that the results are up-to-date without rebuilding the intermediate (secondary) files.

I would appreciate any other suggestions on using SCons in reproducible scientific computations.

stevenknight said at 2003-09-01 08:39:23

Followup from Sergey Fomel, 1 September 2003:

Note that, in GNU make, there is a difference between .INTERMEDIATE and .SECONDARY targets. .SECONDARY files are NOT removed after the main targets get created. In other ways, they behave like .INTERMEDIATE files.

Here is an example:

linux> make -v
GNU Make 3.80
Copyright (C) 2002  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or
FITNESS FOR A
PARTICULAR PURPOSE.
linux> cat Makefile
b1: a
        cp a b1
 
c1: b1
        cp b1 c1
 
.INTERMEDIATE: b2
 
b2: a
        cp a b2
 
c2: b2
        cp b2 c2
 
.SECONDARY: b3
 
b3: a
        cp a b3
 
c3: b3
        cp b3 c3
linux> make c1
cp a b1
cp b1 c1
linux> rm b1
linux> make c1
cp a b1
cp b1 c1
linux> make c2
cp a b2
cp b2 c2
rm b2
linux> make b2
cp a b2
linux> make c2
cp b2 c2
linux> rm b2
linux> make c2
make: `c2' is up to date.
linux> make c3
cp a b3
cp b3 c3
linux> rm b3
linux> make c3
make: `c3' is up to date.

issues@scons said at 2003-09-01 08:39:23

Converted from SourceForge tracker item 774033

garyo said at 2008-03-17 20:28:17

Milestone: 2.x.

garyo said at 2008-03-17 20:31:28

Greg said he'd look into this for 2.x.

gregnoel said at 2008-04-18 18:47:38

We shouldn't have UNCONFIRMED issues; changing them to NEW.

gregnoel said at 2008-12-26 13:20:25

Adjust triage of issues.

garyo said at 2012-09-01 10:04:01

de-assigning all tickets assigned to Greg Noel (no longer working on SCons)