Inconsistent highly negative results

[BrandonLMorris]

The simulation will occasionally compile and run with very negative results (depicted in detail below). This happens inconsistently, and across different compiler versions and compilation options (always executed on the CPU).

Compiler results:

• GCC 4.9: Never shows issue
• GCC 6(.1): Always shows issue
• PGI 16.5: Sometimes shows issue

Normal execution

[oxa110@pegasus MCGPU-062016]$ time bin/metrosim ./meoh256.config -n 1000 -i 100 -k -s
Using seed: 12345
Simulation Name: meoh256
Running 1000 time steps
VERBOSE: No energy calculation strategy specified, defaulting to brute force
VERBOSE: Using original system energy calculation
Inter: -569.955 Intra: 40771.3
Bond: 0.401477
Angle: 39906.5
Non-Bond: 864.433

VERBOSE: Duration of system energy calculation function: 0.05 seconds.
VERBOSE: 
Running 1000 steps

VERBOSE: Step 0:
--Current Energy: 40177.9

VERBOSE: Step 100:
--Current Energy: 39769.6

VERBOSE: Step 200:
--Current Energy: 39455.7

VERBOSE: Step 300:
--Current Energy: 39077.2

VERBOSE: Step 400:
--Current Energy: 38770.6

VERBOSE: Step 500:
--Current Energy: 38527.5

VERBOSE: Step 600:
--Current Energy: 38036.6

VERBOSE: Step 700:
--Current Energy: 37646.7

VERBOSE: Step 800:
--Current Energy: 37432.4

VERBOSE: Step 900:
--Current Energy: 36988.2

VERBOSE: Step 1000:
--Current Energy: 36582
VERBOSE: Saving state file meoh256_1000.state

Finished running 1000 steps
LJ-Energy Subtotal: 0.000
Charge Energy Subtotal: 0.000
Energy Long-range Correction: -23.524
Final Energy: 36581.975
Run Time: 0.380 seconds
Accepted Moves: 120
Rejected Moves: 880
Acceptance Ratio: 12.00%
Finishing simulation...

Incorrect execution

[oxa110@pegasus MCGPU-062016]$ time bin/metrosim ./meoh256.config -n 1000 -i 100 -k -s
Using seed: 12345
Simulation Name: meoh256
Running 1000 time steps
VERBOSE: No energy calculation strategy specified, defaulting to brute force
VERBOSE: Using original system energy calculation
Inter: -569.955 Intra: -9.52918e+06
Bond: 0.401477
Angle: -9.53536e+06
Non-Bond: 6184.98

VERBOSE: Duration of system energy calculation function: 0.04 seconds.
VERBOSE: 
Running 1000 steps

VERBOSE: Step 0:
--Current Energy: -9.52977e+06

VERBOSE: Step 100:
--Current Energy: -9.52977e+06

VERBOSE: Step 200:
--Current Energy: -9.52977e+06

VERBOSE: Step 300:
--Current Energy: -9.52977e+06

VERBOSE: Step 400:
--Current Energy: -9.52977e+06

VERBOSE: Step 500:
--Current Energy: -9.52977e+06

VERBOSE: Step 600:
--Current Energy: -9.52977e+06

VERBOSE: Step 700:
--Current Energy: -9.52977e+06

VERBOSE: Step 800:
--Current Energy: -9.52977e+06

VERBOSE: Step 900:
--Current Energy: -9.52977e+06

VERBOSE: Step 1000:
--Current Energy: -9.52977e+06
VERBOSE: Saving state file meoh256_1000.state

Finished running 1000 steps
LJ-Energy Subtotal: 0.000
Charge Energy Subtotal: 0.000
Energy Long-range Correction: -23.524
Final Energy: -9529772.203
Run Time: 0.210 seconds
Accepted Moves: 0
Rejected Moves: 1000
Acceptance Ratio: 0.00%
Finishing simulation...

Very possible that the issue deals with the simulation initialization start-up code, though not sure why it only occasionally presents itself. Could also be some compiler optimization bug.