Performance of Oceananigans v0.90

[Yixiao-Zhang]

I have some old simulations that use Oceananigans v0.63 and Julia v1.6.2. Now, I need to add topography to these simulations. So, I set up a new simulation with Oceananigans v0.90 and Julia v1.9.3 and used an immersed boundary condition. However, I found the new simulation is 2.3 times slow as the old ones. I do not know whether it is because of the immersed boundary condition, so I tested the performance of the two versions of Oceananigans for a standard non-hydrostatic simulation.

I did not use the benchmark scripts in Oceananigans, because some of them are broken (it seems to me that it is due to changes in the API of PrettyTables.). The script for Oceananigans v0.63 is

using Printf
using Oceananigans
using Oceananigans.Utils: oceananigans_versioninfo, versioninfo_with_gpu

CUDA = Oceananigans.CUDA

function print_system_info()
    println()
    println(oceananigans_versioninfo())
    println(versioninfo_with_gpu())
    CUDA.has_cuda_gpu() && println(CUDA.versioninfo())
    println()
    return nothing
end

print_system_info()

# ---------------------------------------------------------------------- #
# Define Parameters

# Numerical Technic
const arch = GPU()
const time_stepper = :RungeKutta3
const advection = WENO()

# Grid
const Nx = 340
const Ny = 2640
const Nz = 100
const Lx = 1.0
const Ly = 1.0
const Lz = 1.0

@info "Size" (Nx, Ny, Nz)

# Ouput
const n_output_steps = 10
const previous_time_ref = Ref(time())

# ---------------------------------------------------------------------- #
# Define the Simulation

# Grid
grid = begin
    underlying_grid = RectilinearGrid(
        arch,
        size = (Nx, Ny, Nz),
        x = (0.0, Lx),
        y = (0.0, Ly),
        z = (0.0, Lz),
        topology = (Periodic, Bounded, Bounded),
        halo = (4, 4, 4),
    )
end

# Model
model = NonhydrostaticModel(;
    grid = grid,
    timestepper = time_stepper,
    advection = advection,
    tracers = (:T, :S),
)

# Simulation
simulation = Simulation(model; Δt = 1.0, stop_iteration = 99)

# Set Progress Function
function print_simulation_progress(simulation)

    i = model.clock.iteration
    time_now = time()

    @info Printf.@sprintf("Iteration: %d, Time Used: %.2f Seconds", i, time_now - previous_time_ref[])
    previous_time_ref[] = time_now

    return nothing
end

simulation.callbacks[:progress] = Callback(
    print_simulation_progress,
    IterationInterval(n_output_steps),
)

# ---------------------------------------------------------------------- #
# Run the Simulation
run!(simulation)

I tested on MIT Satori, using a Tesla V100 GPU. For this simulation, one time-step takes 0.29 seconds for Oceananigans v0.63 but 0.39 seconds for Oceananigans v0.90. The script for Oceananigans v0.90 and the output logs are posted here:
- test_speed_v0.63.jl.txt
- test_speed_v0.90.jl.txt
- test_speed_output_v0.63.txt
-test_speed_output_v0.90.txt

In summary, I have two questions:
- Based on my test, can we say that Oceananigans v0.90 is 30% slower than Oceananigans v0.63 for the same simulation?
- Does using an immersed boundary condition make the simulation roughly 50% slower?

[anthony-meza]

@Yixiao-Zhang How did you get Oceananigans running on Satori? I'm running into a lot of bugs regarding CUDA

[simone-silvestri]

I just noticed this comment now:

@Yixiao-Zhang I strongly suspect that the reason Oceananigans is slower on version 0.90.0 that on version 0.63 is because we do advection in a very different way now. In 0.63.0 a high order scheme (like WENO5) was limited only once near the boundary to a centered scheme, now we progressively decrease the order as we approach the boundary (from 5 to 3 to finally first order in case of WENO5) by keeping the characteristic numerical error of the advection (in WENO case a diffusive instead of limiting to a dispersive scheme like centered second order).
This requires more computations since advection has to be performed typically more than twice but it's a hit that we probably need to take if we want to ensure that the advection works as advertised (i.e, you don't want to have to add some viscosity just because of advection near the boundary). You will probably notice that if you perform your benchmark again, this time on a fully Periodic domain that does not require limiting the advection order, the wall time per time step is probably the same.
However, with recent improvements in WENO performance you'll probably find out that if you run on main Oceananigans is now faster than 0.63.0 without having to compromise.
This said, it is very easy to modify the user interface to allow limiting only once to a centered scheme instead of a progressive order reduction, so if there is interest we can adapt the constructor to allow this possibility.

Regarding immersed grids, I do not have a figure in mind particularly on how slow an immersed grid is vs a non immersed grid, but Immersed boundaries do indeed put a strain on the performance because of the many if conditions that require loading the immersed boundary from memory.

@anthony-meza I routinely run on satori without any problem, what errors are you facing? Note that on Satori it is more convenient to build julia from source because of its architecture (PowerPC). Try following these steps:

• clone the julia repository https://github.com/JuliaLang/julia
• checkout to the v1.10.4 branch git checkout v1.10.4
• make
• make install
• run julia as ./julia from the install folder

Not that (I am not sure it is strictly necessary) before I launch a job I also source this setup file

# Upload modules
module purge all
module add spack
module add cuda/11.4
module load openmpi/3.1.6-cuda-pmi-ucx-slurm-jhklron

[glwagner]

This said, it is very easy to modify the user interface to allow limiting only once to a centered scheme instead of a progressive order reduction, so if there is interest we can adapt the constructor to allow this possibility.

If it's easy, this seems like a good idea.

@Yixiao-Zhang this hypothesis can be tested by doing the benchmark with advection = Centered() in which case you should find no performance difference.

[Yixiao-Zhang]

@simone-silvestri Thank you for your response! The perform of the latest version is fine for me. I was just curious about the reasons behind. I am closing this issue.

For the source setup file, I think it is only necessary to load cuda/11.4 if CUDA.set_runtime_version!(v"11.4", local_toolkit=true) has been used. The default behavior is to download an artifact. Besides, CUDA_Runtime_Discovery version 0.2 does not work on Satori in my experience.

@anthony-meza I think it is best to show your error messages first. Since I am closing this issue, you can open a new issue if you need to. I will be willing to help.