Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add driver script for running benchmarks #6

Merged
merged 10 commits into from
Sep 16, 2024
Merged

Add driver script for running benchmarks #6

merged 10 commits into from
Sep 16, 2024

Conversation

odow
Copy link
Member

@odow odow commented Sep 12, 2024

Now I understand why the default settings are changed #5

Default is dual simplex

(base) oscar@Oscars-MBP open-energy-modeling-benchmarks % julia --project=benchmark benchmark/main.jl --instance=09d8dfdaa3dff578cb9f3af8ecefe9dab893a0ada31310c6b4010972415e8b44-GenX_8_three_zones_w_colocated_VRE_storage_electrolyzers.mps.gz
Running HiGHS 1.7.2 (git hash: 5ce7a2753): Copyright (c) 2024 HiGHS under MIT licence terms
Coefficient ranges:
  Matrix [5e-03, 5e+02]
  Cost   [5e-01, 2e+07]
  Bound  [0e+00, 0e+00]
  RHS    [2e+00, 3e+08]
Presolving model
193417 rows, 166571 cols, 718423 nonzeros  0s
186080 rows, 159234 cols, 722167 nonzeros  0s
Presolve : Reductions: rows 186080(-99569); columns 159234(-91175); elements 722167(-244524)
Solving the presolved LP
Using EKK dual simplex solver - serial
  Iteration        Objective     Infeasibilities num(sum)
          0    -7.1670893480e+05 Ph1: 20161(20852.8); Du: 10080(716709) 1s
       6309    -1.2499719557e+02 Ph1: 6(3.93322); Du: 1(124.997) 6s
       9603    -1.2499682869e+02 Ph1: 6(2.30736); Du: 1(124.997) 11s
      12437    -1.2499648326e+02 Ph1: 5(3.61473); Du: 1(124.996) 16s
      15032    -1.2499611570e+02 Ph1: 8(3.29623); Du: 2(124.996) 21s
      17513    -1.2499568718e+02 Ph1: 18(8.02397); Du: 3(124.996) 27s
      21527    -1.2396651214e+02 Ph1: 12(15.3808); Du: 1(123.967) 32s
      23949    -1.2396631111e+02 Ph1: 10(9.36891); Du: 1(123.966) 38s

IPM makes better progress

(base) oscar@Oscars-MBP open-energy-modeling-benchmarks % julia --project=benchmark benchmark/main.jl --instance=09d8dfdaa3dff578cb9f3af8ecefe9dab893a0ada31310c6b4010972415e8b44-GenX_8_three_zones_w_colocated_VRE_storage_electrolyzers.mps.gz --solver=ipm
Running HiGHS 1.7.2 (git hash: 5ce7a2753): Copyright (c) 2024 HiGHS under MIT licence terms
Coefficient ranges:
  Matrix [5e-03, 5e+02]
  Cost   [5e-01, 2e+07]
  Bound  [0e+00, 0e+00]
  RHS    [2e+00, 3e+08]
Presolving model
193417 rows, 166571 cols, 718423 nonzeros  0s
186080 rows, 159234 cols, 722167 nonzeros  0s
Presolve : Reductions: rows 186080(-99569); columns 159234(-91175); elements 722167(-244524)
Solving the presolved LP
IPX model has 186080 rows, 159234 columns and 722167 nonzeros
Input
    Number of variables:                                159234
    Number of free variables:                           0
    Number of constraints:                              186080
    Number of equality constraints:                     44743
    Number of matrix entries:                           722167
    Matrix range:                                       [6e-03, 3e+02]
    RHS range:                                          [5e+02, 3e+08]
    Objective range:                                    [5e-01, 2e+07]
    Bounds range:                                       [2e+00, 3e+08]
Preprocessing
    Dualized model:                                     no
    Number of dense columns:                            40
    Range of scaling factors:                           [1.25e-01, 4.00e+00]
IPX version 1.0
Interior Point Solve
 Iter     P.res    D.res            P.obj           D.obj        mu     Time
   0   1.32e+08 1.21e+05   3.85666634e+14 -8.40523648e+15  1.61e+13       0s
   1   1.19e+08 1.15e+05   1.19042833e+15 -8.53131850e+15  1.58e+13       1s
   2   1.14e+08 9.62e+04   1.66132234e+15 -9.09616356e+15  1.55e+13       2s
   3   1.03e+08 8.47e+04   2.85865874e+15 -9.57439700e+15  1.51e+13       3s
   4   2.89e+07 3.79e+04   1.10443687e+16 -1.01441902e+16  7.54e+12       3s
   5   6.99e+06 9.02e+03   1.34900404e+16 -5.80578221e+15  1.90e+12       5s
   6   2.22e+06 7.78e+02   1.07758823e+16 -1.79678742e+15  3.33e+11       7s
 Constructing starting basis...
   7   8.50e+04 6.70e+01   2.72737663e+15 -6.72996361e+14  2.50e+10      13s
   8   7.99e+03 1.16e+01   8.17031984e+14 -1.75132307e+14  4.57e+09      16s
   9   4.30e+03 2.88e+00   5.47384299e+14 -6.27080935e+13  2.32e+09      41s
  10   2.19e+03 1.39e+00   3.37338088e+14 -3.50396207e+13  1.34e+09      50s

And they run with

(base) oscar@Oscars-MBP open-energy-modeling-benchmarks % julia --project=benchmark benchmark/main.jl --instance=09d8dfdaa3dff578cb9f3af8ecefe9dab893a0ada31310c6b4010972415e8b44-GenX_8_three_zones_w_colocated_VRE_storage_electrolyzers.mps.gz --solver=ipm --ipm_optimality_tolerance=1e-4 --primal_feasibility_tolerance=1e-4 --run_crossover=off
Running HiGHS 1.7.2 (git hash: 5ce7a2753): Copyright (c) 2024 HiGHS under MIT licence terms
Coefficient ranges:
  Matrix [5e-03, 5e+02]
  Cost   [5e-01, 2e+07]
  Bound  [0e+00, 0e+00]
  RHS    [2e+00, 3e+08]
Presolving model
193417 rows, 166571 cols, 718423 nonzeros  0s
186080 rows, 159234 cols, 713431 nonzeros  0s
Presolve : Reductions: rows 186080(-99569); columns 159234(-91175); elements 713431(-253260)
Solving the presolved LP
IPX model has 186080 rows, 159234 columns and 713431 nonzeros
Input
    Number of variables:                                159234
    Number of free variables:                           0
    Number of constraints:                              186080
    Number of equality constraints:                     44743
    Number of matrix entries:                           713431
    Matrix range:                                       [1e-02, 3e+02]
    RHS range:                                          [5e+02, 3e+08]
    Objective range:                                    [5e-01, 2e+07]
    Bounds range:                                       [2e+00, 3e+08]
Preprocessing
    Dualized model:                                     no
    Number of dense columns:                            40
    Range of scaling factors:                           [1.25e-01, 4.00e+00]
IPX version 1.0
Interior Point Solve
 Iter     P.res    D.res            P.obj           D.obj        mu     Time
   0   1.32e+08 1.21e+05   3.86393282e+14 -8.40355954e+15  1.61e+13       0s
   1   1.19e+08 1.15e+05   1.02485436e+15 -8.52922599e+15  1.58e+13       2s
   2   1.14e+08 9.69e+04   1.43685477e+15 -9.06843832e+15  1.55e+13       3s
   3   1.03e+08 8.47e+04   2.49158742e+15 -9.56747357e+15  1.51e+13       3s
   4   2.93e+07 3.80e+04   1.04726505e+16 -1.00995635e+16  7.56e+12       4s
   5   5.82e+06 7.43e+03   1.36103022e+16 -5.53138279e+15  1.56e+12       6s
   6   2.37e+06 8.58e+02   1.08252572e+16 -1.81125889e+15  3.61e+11       8s
 Constructing starting basis...
   7   8.73e+04 7.12e+01   2.90611737e+15 -6.80994893e+14  2.66e+10      18s
   8   8.66e+03 2.81e+01   9.15318057e+14 -3.17153750e+14  7.16e+09      23s
   9   4.01e+03 4.71e+00   6.21964396e+14 -8.66311830e+13  2.83e+09      38s
  10   1.91e+03 1.99e+00   3.76414524e+14 -4.58304135e+13  1.55e+09      52s
  11   8.81e+02 1.00e+00   2.53927232e+14 -2.91909351e+13  9.99e+08      64s

odow
odow commented Sep 12, 2024
View reviewed changes
benchmark/main.jl Outdated
"highs_run_time" => HiGHS.Highs_getRunTime(highs),
"highs_objective_value" => HiGHS.Highs_getObjectiveValue(highs),
"run_status" => run_status,
"model_status" => HiGHS.Highs_getModelStatus(highs),
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the other useful things here would be the various feasibility metrics

@odow odow changed the title WIP: add driver script for running benchmarks Add driver script for running benchmarks Sep 16, 2024
@odow
Copy link
Member Author

odow commented Sep 16, 2024

Latest run

(base) oscar@Oscars-MBP open-energy-modeling-benchmarks % make analyze
julia --project=benchmark benchmark/main.jl --analyze
[ Info: SGM(sh=10) for :julia_total_time
4×2 DataFrame
 Row │ solver_version  julia_total_time 
     │ String          Float64          
─────┼──────────────────────────────────
   1 │ v1.7.2-default            122.5
   2 │ v1.7.2-ipm                109.09
   3 │ v1.5.1-ipm                115.84
   4 │ v1.5.1-default            121.75
[ Info: run_status
12×5 DataFrame
 Row │ uuid                               v1.7.2-default  v1.7.2-ipm  v1.5.1-ipm  v1.5.1-default 
     │ String                             Int64?          Int64?      Int64?      Int64?         
─────┼───────────────────────────────────────────────────────────────────────────────────────────
   1 │ 09d8dfdaa3dff578cb9f3af8ecefe9da…               0           0           0               0
   2 │ 19ae5afee2b17471262aad2b1a73543a…               0           0           0               0
   3 │ 3457d585bff0f977e229e67cbfbe7c16…               0           0           0               0
   4 │ 37dfc6b70590080b0bde4f26ad406bfe…               0           0           0               0
   5 │ 5b5e177fc71e5f42c895e002aca810a8…               0           0           0               0
   6 │ 6f0d4f53f340ecba3780a31e9941b1f1…               0           0           0               0
   7 │ 70d2edf33db6d750d2aa23411e9372c3…               0           0           0               0
   8 │ a3e6151efdccdbf99633a72828a97064…               0           0           0               0
   9 │ c29bbab5e2280eed636507b9f5bfc508…               0           0           0               0
  10 │ cb3b18586d17e6f73810f4bea8fd9fa2…               0           0           0               0
  11 │ d53e1bdfc4f5cf7a3da3a28ca87e9943…               0           0           0               0
  12 │ de3cc3c5090b33543b9766827f6b7f20…               0           0           0               0
[ Info: model_status
12×5 DataFrame
 Row │ uuid                               v1.7.2-default  v1.7.2-ipm  v1.5.1-ipm  v1.5.1-default 
     │ String                             Int64?          Int64?      Int64?      Int64?         
─────┼───────────────────────────────────────────────────────────────────────────────────────────
   1 │ 09d8dfdaa3dff578cb9f3af8ecefe9da…               7           7           7               7
   2 │ 19ae5afee2b17471262aad2b1a73543a…               7           7           7               7
   3 │ 3457d585bff0f977e229e67cbfbe7c16…               7           7           7               7
   4 │ 37dfc6b70590080b0bde4f26ad406bfe…               7           7           7               7
   5 │ 5b5e177fc71e5f42c895e002aca810a8…               7           7           7               7
   6 │ 6f0d4f53f340ecba3780a31e9941b1f1…               7           7           7               7
   7 │ 70d2edf33db6d750d2aa23411e9372c3…               7           7           7               7
   8 │ a3e6151efdccdbf99633a72828a97064…               7           7           7               7
   9 │ c29bbab5e2280eed636507b9f5bfc508…               7           7           7               7
  10 │ cb3b18586d17e6f73810f4bea8fd9fa2…               7           7           7               7
  11 │ d53e1bdfc4f5cf7a3da3a28ca87e9943…               7           7           7               7
  12 │ de3cc3c5090b33543b9766827f6b7f20…               7           7           7               7
[ Info: highs_objective_value
12×5 DataFrame
 Row │ uuid                               v1.7.2-default  v1.7.2-ipm      v1.5.1-ipm      v1.5.1-default 
     │ String                             Float64?        Float64?        Float64?        Float64?       
─────┼───────────────────────────────────────────────────────────────────────────────────────────────────
   1 │ 09d8dfdaa3dff578cb9f3af8ecefe9da…      2.12042e13      2.12042e13      2.12042e13      2.12042e13
   2 │ 19ae5afee2b17471262aad2b1a73543a…   9146.47         9146.47         9146.47         9146.47
   3 │ 3457d585bff0f977e229e67cbfbe7c16…  11040.4         11040.4         11040.4         11040.4
   4 │ 37dfc6b70590080b0bde4f26ad406bfe…      4.43326e5       4.43326e5       4.43326e5       4.43326e5
   5 │ 5b5e177fc71e5f42c895e002aca810a8…  19779.8         19779.8         19779.8         19779.8
   6 │ 6f0d4f53f340ecba3780a31e9941b1f1…   3260.34         3260.34         3260.34         3260.34
   7 │ 70d2edf33db6d750d2aa23411e9372c3…   4602.47         4602.47         4602.47         4602.47
   8 │ a3e6151efdccdbf99633a72828a97064…      1.15096e6       1.15096e6       1.15096e6       1.15096e6
   9 │ c29bbab5e2280eed636507b9f5bfc508…   3115.76         3115.76         3115.76         3115.76
  10 │ cb3b18586d17e6f73810f4bea8fd9fa2…   6063.12         6063.12         6063.12         6063.12
  11 │ d53e1bdfc4f5cf7a3da3a28ca87e9943…   9391.61         9391.61         9391.61         9391.61
  12 │ de3cc3c5090b33543b9766827f6b7f20…   8762.96         8762.96         8762.96         8762.96

@odow odow merged commit e318dc4 into main Sep 16, 2024
@odow odow deleted the od/benchmark branch September 16, 2024 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@odow