rjust · rjust · Jul 3, 2020 · Mar 11, 2020 · Mar 11, 2020 · Mar 11, 2020
diff --git a/README.md b/README.md
@@ -31,10 +31,18 @@ Defects4J contains 835 bugs from the following open-source projects:
 | Time            | joda-time                  |       26       | 1-20,22-27          | 21                      |
 
 \* Due to behavioral changes introduced under Java 8, some bugs are no longer
-reproducible. These bugs have been removed from the commit-db, but their
-metadata is still available in the project directory. As publications using
-Defects4J artifacts refer to bugs by their specific bug id, we do not re-number
-active bug ids of existing bugs.
+reproducible. Hence, Defects4J distinguishes between active and deprecated bugs:
+
+- Active bugs can be accessed through `active-bugs.csv`.
+
+- Deprecated bugs are removed from `active-bugs.csv`, but their metadata is
+  retained in the project directory.
+
+- Deprecated bugs can be accessed through `deprecated-bugs.csv`, which also
+  details when and why a bug was deprecated.
+
+We do not re-enumerate active bugs because publications using Defects4J artifacts
+usually refer to bugs by their specific bug id.
 
 The bugs
 ---------------

diff --git a/framework/bug-mining/Patch-Minimization-Guide.md b/framework/bug-mining/Patch-Minimization-Guide.md
@@ -27,8 +27,8 @@ Keep in mind that each patch in `framework/projects/PROJECTNAME/patches/` is a r
 
 Proper minimization requires an understanding of *what* the fault in the code means, and how it affects the system. 
 Three key pieces of information should guide your minimization:
-* The bug report: The `commit-db` includes a link to the bug report (or, in some cases, a pull request). 
-* The commit message: The `commit-db` includes the hash of the commit that fixes the bug. The commit message often explains what was changed and *why*. 
+* The bug report: The `active-bugs.csv` includes a link to the bug report (or, in some cases, a pull request). 
+* The commit message: The `active-bugs.csv` includes the hash of the commit that fixes the bug. The commit message often explains what was changed and *why*. 
 * The trigger tests: Each fault has a set of trigger tests -- tests that fail on the buggy version and pass on the fixed version. The trigger test file lists
 the failing tests and, for each, includes the stack trace of the failing test case.
 
@@ -77,7 +77,7 @@ Code refactoring may consist of one or more of the following:
     * Example 2: Collections-71 contains several tab changes (refactorings) that result in an unnecessarily large patch. See [Collections-71 non-minimized](https://github.com/rjust/defects4j/blob/master/framework/bug-mining/code-example/col.71.preminimized.patch) vs. [Collections-71 minimized](https://github.com/rjust/defects4j/blob/master/framework/bug-mining/code-example/col.71.minimized.patch)
 
 2. __Comments__  
-     Comments could be considered as part of the bug fix: a developer may want to associate a comment with a bug fix and therefore include it in the pure bug-fixing patch. However, a researcher may want to ignore comments when reasoning about or analyzing a bug-fixing patch. Therefore, we remove all changes to comments or documentation from the patch. This information can be seen by directly examining the commits, as offered in the `commit-db`.
+     Comments could be considered as part of the bug fix: a developer may want to associate a comment with a bug fix and therefore include it in the pure bug-fixing patch. However, a researcher may want to ignore comments when reasoning about or analyzing a bug-fixing patch. Therefore, we remove all changes to comments or documentation from the patch. This information can be seen by directly examining the commits, as offered in the `active-bugs.csv`.
 
 3. __Sementically-equivalent changes__  
     Changes that do not alter the visible program behavor -- those that are semantically equivalent across program versions -- can be removed. These changes will have no effect on the bug as they produce the same output before and after the change.

diff --git a/framework/bug-mining/README.md b/framework/bug-mining/README.md
@@ -73,7 +73,7 @@ BUG_FIX_REGEX="/(CODEC-\d+)/mi"
 
 The following table reports the issue trackers, issue tracker project IDs, and
 regular expressions previously used in `Defects4J` (note that we manually built
-the `commit-db` for Chart):
+the `active-bugs.csv` for Chart):
 
 | Project ID | Issue tracker | Issue tracker project ID | Regexp                    |
 |------------|---------------|--------------------------|---------------------------|
@@ -115,7 +115,7 @@ This script performs 3 tasks:
    `$WORK_DIR/issues.txt`.
 
 3. Performs a cross-reference of commit log and the issue ids, and creates a
-   `commit-db` with all commits hashes for all issues ids that have been
+   `active-bugs.csv` with all commits hashes for all issues ids that have been
    reported in the issue tracker.
 
 
@@ -124,7 +124,7 @@ This script performs 3 tasks:
 
 1. Initialize all project revisions with `initialize-revisions.pl`. This script
    will identify the various directory layouts and run a sanity check on each
-   candidate revision in `commit-db`:
+   candidate revision in `active-bugs.csv`:
 
 ```bash
 ./initialize-revisions.pl -p $PROJECT_ID -w $WORK_DIR
@@ -203,8 +203,8 @@ a revision fails.
       `-b <bug_id>`.
 
 4. If particular revisions cannot be built, often due to dependencies that no
-   longer exist, then they may be removed from the `commit-db`. It is
-   recommended to keep a backup of the commit-db until the entire bug mining
+   longer exist, then they may be removed from the `active-bugs.csv`. It is
+   recommended to keep a backup of the active-bugs.csv until the entire bug mining
    process is complete.
 
 5. Upon completion of this stage, inspect all stack traces in the files that are
@@ -218,7 +218,7 @@ a revision fails.
 ## Reproducing bugs
 
 1. Determine triggering tests with the `get-trigger.pl` script. This will
-   determine the revisions in `commit-db` that have a test that can reproduce a
+   determine the revisions in `active-bugs.csv` that have a test that can reproduce a
    fault:
 
 ```bash
@@ -320,7 +320,7 @@ Terms commonly used in Defects4J
 - `VCS`: Version control system (e.g., git, mercurial, or subversion; all VCS
   abstractions in Defects4J inherit from Vcs.pm).
 - `Rev ID`: A VCS-specific revision id (e.g., a git commit hash).
-- `commit-db`: A csv file, per project, that maps each BID to the revision ids
+- `active-bugs.csv`: A csv file, per project, that maps each BID to the revision ids
   of the pre-fix and post-fix revision.
 
 

diff --git a/framework/bug-mining/analyze-project.pl b/framework/bug-mining/analyze-project.pl
@@ -26,7 +26,7 @@
 
 =head1 NAME
 
-analyze-project.pl -- Determine all suitable candidates listed in the commit-db.
+analyze-project.pl -- Determine all suitable candidates listed in the active-bugs csv.
 
 =head1 SYNOPSIS
 
@@ -57,13 +57,13 @@ =head1 OPTIONS
 =item B<-b C<bug_id>>
 
 Only analyze this bug id. The bug_id has to follow the format B<(\d+)(:(\d+))?>.
-Per default all bug ids, listed in the commit-db, are considered.
+Per default all bug ids, listed in the active-bugs csv, are considered.
 
 =back
 
 =head1 DESCRIPTION
 
-Runs the following worflow for all candidate bugs in the project's C<commit-db>,
+Runs the following worflow for all candidate bugs in the project's C<active-bugs.csv>,
 or (if -b is specified) for a subset of candidates:
 
 =over 4

diff --git a/framework/bug-mining/create-project.pl b/framework/bug-mining/create-project.pl
@@ -101,7 +101,7 @@ =head1 DESCRIPTION
 # Directory to which the remote repository is cloned.
 my $repo_dir    = "$WORK_DIR/project_repos";
 
-# Initialize working directory and create empty commit-db
+# Initialize working directory and create empty active-bugs csv
 my $project_dir = "$WORK_DIR/framework/projects/$PID";
 
 my $ISSUES_DIR = "$WORK_DIR/issues";
@@ -118,7 +118,14 @@ =head1 DESCRIPTION
 my $core_dir = "$WORK_DIR/framework/core/Project";
 
 system("mkdir -p $project_dir $core_dir $ISSUES_DIR $PATCH_DIR $FAILING_DIR $TRIGGER_DIR $RELEVANT_DIR $MOD_CLASSES $REL_CLASSES");
-system("touch $project_dir/commit-db");
+
+# Create active-bugs csv and print header
+my $active_header = $BUGS_CSV_BUGID.",".$BUGS_CSV_COMMIT_BUGGY.",".$BUGS_CSV_COMMIT_FIXED.",".$BUGS_CSV_ISSUE_ID.",".$BUGS_CSV_ISSUE_URL;
+system("echo $active_header > $project_dir/$BUGS_CSV_ACTIVE");
+
+# Create deprecated-bugs csv and print header
+my $deprecated_header = $active_header.",".$BUGS_CSV_DEPRECATED_WHEN.",".$BUGS_CSV_DEPRECATED_WHY;
+system("echo $deprecated_header > $project_dir/$BUGS_CSV_DEPRECATED");
 
 # Copy module template and set project id and name
 open(IN, "<$module_template") or die "Cannot open template file: $!";

diff --git a/framework/bug-mining/get-metadata.pl b/framework/bug-mining/get-metadata.pl
@@ -48,7 +48,7 @@ =head1 OPTIONS
 =item B<-b C<bug_id>>
 
 Only analyze this bug id. The bug_id has to follow the format B<(\d+)(:(\d+))?>.
-Per default all bug ids, listed in the commit-db, are considered.
+Per default all bug ids, listed in the active-bugs csv, are considered.
 
 =back
 

diff --git a/framework/bug-mining/get-trigger.pl b/framework/bug-mining/get-trigger.pl
@@ -48,7 +48,7 @@ =head1 OPTIONS
 =item B<-b C<bug_id>>
 
 Only analyze this bug id. The bug_id has to follow the format B<(\d+)(:(\d+))?>.
-Per default all bug ids, listed in the commit-db, are considered.
+Per default all bug ids, listed in the active-bugs csv, are considered.
 
 =head1 DESCRIPTION
 

diff --git a/framework/bug-mining/initialize-project-and-collect-issues.pl b/framework/bug-mining/initialize-project-and-collect-issues.pl
@@ -132,7 +132,7 @@ =head1 OPTIONS
 my $REGEXP = $cmd_opts{e};
 my $GIT_LOG_FILE = "$WORK_DIR/gitlog";
 my $REPOSITORY_DIR = "$WORK_DIR/project_repos/$NAME.git";
-my $COMMIT_DB_FILE = "$WORK_DIR/framework/projects/$PID/commit-db";
+my $COMMIT_DB_FILE = "$WORK_DIR/framework/projects/$PID/$BUGS_CSV_ACTIVE";
 my $VCS_TYPE = $cmd_opts{v} // "git";
 
 # Configure project for Defects4J
@@ -179,9 +179,12 @@ =head1 OPTIONS
 # have already been mined.
 if (-e "$CORE_DIR/Project/$PID.pm") {
     # Remove exiting ids
-    system("cut -f 2- -d',' $PROJECTS_DIR/$PID/commit-db > $COMMIT_DB_FILE.orig");
+    system("tail -n +2 $PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE | cut -f 2- -d',' > $COMMIT_DB_FILE.orig");
     # Find all versions that have not been mined
     system("grep -vFf $COMMIT_DB_FILE.orig $COMMIT_DB_FILE > $COMMIT_DB_FILE.filter && mv $COMMIT_DB_FILE.filter $COMMIT_DB_FILE");
+    # Print header to the active bugs csv
+    my $active_header = $BUGS_CSV_BUGID.",".$BUGS_CSV_COMMIT_BUGGY.",".$BUGS_CSV_COMMIT_FIXED.",".$BUGS_CSV_ISSUE_ID.",".$BUGS_CSV_ISSUE_URL;
+    system("echo $active_header > $COMMIT_DB_FILE.new && cat $COMMIT_DB_FILE >> $COMMIT_DB_FILE.new && mv $COMMIT_DB_FILE.new $COMMIT_DB_FILE");
 }
 
 print("Project $PID has been successfully initialized!\n");
diff --git a/framework/bug-mining/initialize-revisions.pl b/framework/bug-mining/initialize-revisions.pl
@@ -52,7 +52,7 @@ =head1 OPTIONS
 =item B<-b C<bug_id>>
 
 Only analyze this bug id. The bug_id has to follow the format B<(\d+)(:(\d+))?>.
-Per default all bug ids, listed in the commit-db, are considered.
+Per default all bug ids, listed in the active-bugs csv, are considered.
 
 =back
 

diff --git a/framework/bug-mining/promote-to-db.pl b/framework/bug-mining/promote-to-db.pl
@@ -40,7 +40,8 @@ =head1 NAME
   - framework/projects/<PROJECT_ID>/trigger_tests
   - framework/projects/<PROJECT_ID>/build.xml.patch
   - framework/projects/<PROJECT_ID>/<PROJECT_ID>.build.xml
-  - framework/projects/<PROJECT_ID>/commit-db
+  - framework/projects/<PROJECT_ID>/$BUGS_CSV_ACTIVE
+  - framework/projects/<PROJECT_ID>/$BUGS_CSV_DEPRECATED	
   - framework/projects/<PROJECT_ID>/dir-layout.csv
   - project_repos/<PROJECT_NAME>.git
 and updates the project_repos/README file with information of when the project
@@ -61,7 +62,7 @@ =head1 OPTIONS
 =item B<-b C<bug_id>>
 
 Only analyze this bug id. The bug_id has to follow the format B<(\d+)(:(\d+))?>.
-Per default all bug ids, listed in the commit-db, are considered.
+Per default all bug ids, listed in the $BUGS_CSV_ACTIVE, are considered.
 
 =item B<-w C<work_dir>>
 
@@ -125,7 +126,7 @@ =head1 OPTIONS
                             "modified_classes/<id>.src", "modified_classes/<id>.test",
                             "patches/<id>.src.patch", "patches/<id>.test.patch",
                             "trigger_tests/<id>", "relevant_tests/<id>");
-my @generic_files_and_directories_to_replace = ("build.xml.patch", "${PID}.build.xml", "lib");
+my @generic_files_and_directories_to_replace = ("build.xml.patch", "${PID}.build.xml", "lib", $BUGS_CSV_DEPRECATED);
 my @generic_files_to_append = ("dependent_tests", "dir-layout.csv");
 
 my @ids = _get_bug_ids($BID);
@@ -140,10 +141,11 @@ =head1 OPTIONS
 
     # find number
     my $max_number = 0;
-    my $output_commit_db = "$OUTPUT_DIR/$PID/commit-db";
+    my $output_commit_db = "$OUTPUT_DIR/$PID/$BUGS_CSV_ACTIVE";
     if (-e $output_commit_db) {
-        open FH, $output_commit_db or die "could not open output commit-db";
+        open FH, $output_commit_db or die "could not open output active-bugs csv";
         my $exists_line = 0;
+        my $header = <FH>;
         while (my $line = <FH>) {
             chomp $line;
             $line =~ /^(\d+),(.*),(.*),(.*),(.*)$/ or die "could not parse line";
@@ -162,7 +164,13 @@ =head1 OPTIONS
     ++$max_number;
     print "\t... adding as new commit-id $max_number\n";
 
-    open FH, ">>$output_commit_db" or die "could not open output commit-db for writing";
+    open FH, ">>$output_commit_db" or die "could not open output active-bugs csv for writing";
+
+    # If this is the first bug to be promoted, print the header to the active-bugs csv file.
+    if ($max_number == 1) {
+        print FH $BUGS_CSV_BUGID.",".$BUGS_CSV_COMMIT_BUGGY.",".$BUGS_CSV_COMMIT_FIXED.",".$BUGS_CSV_ISSUE_ID.",".$BUGS_CSV_ISSUE_URL."\n";
+    }
+
     print FH "$max_number,$v1,$v2,$issue_id,$issue_url\n";
     close FH;
     for my $rev ($v1, $v2) {

diff --git a/framework/bug-mining/vcs-log-xref.pl b/framework/bug-mining/vcs-log-xref.pl
@@ -188,7 +188,7 @@ =head1 OPTIONS
 if (scalar keys %commits eq 0) {
     print("Warning, no commit that matches the regex expression provided has been found\n");
 }
-open $fh, ">$OUTPUT_FILE" or die "Cannot open ${OUTPUT_FILE}!";
+open $fh, ">>$OUTPUT_FILE" or die "Cannot open ${OUTPUT_FILE}!";
 for my $commit_id (sort { $a <=> $b} keys %commits) {
     my $row = $commits{$commit_id};
     print $fh "$commit_id,$row->{'p'},$row->{'c'},$row->{'issue_id'},$row->{'issue_url'}\n";

diff --git a/framework/core/Constants.pm b/framework/core/Constants.pm
@@ -256,6 +256,22 @@ our $TAG_FIXED            = "FIXED_VERSION";
 our $TAG_BUGGY            = "BUGGY_VERSION";
 our $TAG_PRE_FIX          = "PRE_FIX_REVISION";
 
+# Filenames for bugs csv files
+our $BUGS_CSV_ACTIVE = "active-bugs.csv";
+our $BUGS_CSV_DEPRECATED = "deprecated-bugs.csv";
+
+# Columns in active-bugs and deprecated-bugs csvs
+our $BUGS_CSV_BUGID = "bug_id";
+our $BUGS_CSV_COMMIT_BUGGY = "revision_id_buggy";
+our $BUGS_CSV_COMMIT_FIXED = "revision_id_fixed";
+our $BUGS_CSV_ISSUE_ID = "bug_report_id";
+our $BUGS_CSV_ISSUE_URL = "bug_report_url";
+our $BUGS_CSV_DEPRECATED_WHEN = "deprecated_version";
+our $BUGS_CSV_DEPRECATED_WHY = "deprecated_reason";
+
+# Reasons for deprecation
+our $DEPRECATED_JVM8 = "JVM8-NonReproducable";
+
 # Filenames for test results
 our $FILE_ALL_TESTS     = "all_tests";
 our $FILE_FAILING_TESTS = "failing_tests";
@@ -303,6 +319,19 @@ $TAG_FIXED
 $TAG_BUGGY
 $TAG_PRE_FIX
 
+$BUGS_CSV_ACTIVE
+$BUGS_CSV_DEPRECATED
+
+$BUGS_CSV_BUGID
+$BUGS_CSV_COMMIT_BUGGY
+$BUGS_CSV_COMMIT_FIXED
+$BUGS_CSV_ISSUE_ID
+$BUGS_CSV_ISSUE_URL
+$BUGS_CSV_DEPRECATED_WHEN
+$BUGS_CSV_DEPRECATED_WHY
+
+$DEPRECATED_JVM8
+
 $FILE_ALL_TESTS
 $FILE_FAILING_TESTS
 

diff --git a/framework/core/Project.pm b/framework/core/Project.pm
@@ -51,7 +51,7 @@ A specific project instance can be created with C<create_project(project_id)>.
     my $name  = "my-project-name";
     my $vcs   = Vcs::Git->new($PID,
                               "$REPO_DIR/$name.git",
-                              "$PROJECTS_DIR/$PID/commit-db");
+                              "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE");
 
     return $class->SUPER::new($PID, $name, $vcs);
 }

diff --git a/framework/core/Project/Chart.pm b/framework/core/Project/Chart.pm
@@ -50,7 +50,7 @@ sub new {
     my $name = "jfreechart";
     my $vcs = Vcs::Svn->new($PID,
                             "file://$REPO_DIR/$name/trunk",
-                            "$PROJECTS_DIR/$PID/commit-db",
+                            "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                             \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Cli.pm b/framework/core/Project/Cli.pm
@@ -51,7 +51,7 @@ sub new {
     my $name = "commons-cli";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Closure.pm b/framework/core/Project/Closure.pm
@@ -50,7 +50,7 @@ sub new {
     my $name = "closure-compiler";
     my $vcs = Vcs::Git->new($PID,
                             "$REPO_DIR/$name.git",
-                            "$PROJECTS_DIR/$PID/commit-db",
+                            "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                     \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Codec.pm b/framework/core/Project/Codec.pm
@@ -51,7 +51,7 @@ sub new {
     my $name = "commons-codec";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Collections.pm b/framework/core/Project/Collections.pm
@@ -51,7 +51,7 @@ sub new {
     my $name = "commons-collections";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Compress.pm b/framework/core/Project/Compress.pm
@@ -50,7 +50,7 @@ sub new {
     my $name = "commons-compress";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Csv.pm b/framework/core/Project/Csv.pm
@@ -51,7 +51,7 @@ sub new {
     my $name = "commons-csv";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/Gson.pm b/framework/core/Project/Gson.pm
@@ -50,7 +50,7 @@ sub new {
     my $name = "gson";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);

diff --git a/framework/core/Project/JacksonCore.pm b/framework/core/Project/JacksonCore.pm
@@ -51,7 +51,7 @@ sub new {
     my $name = "jackson-core";
     my $vcs  = Vcs::Git->new($PID,
                              "$REPO_DIR/$name.git",
-                             "$PROJECTS_DIR/$PID/commit-db",
+                             "$PROJECTS_DIR/$PID/$BUGS_CSV_ACTIVE",
                              \&_post_checkout);
 
     return $class->SUPER::new($PID, $name, $vcs);