Optimize SplineEvaluator

[blegouix]

At

ddc/include/ddc/kernels/splines/spline_evaluator.hpp

Line 163 in 9f9292d

KOKKOS_CLASS_LAMBDA(typename batch_domain_type::discrete_element_type const j) {

, replacing:

ddc::parallel_for_each(
                exec_space(),
                batch_domain,
                KOKKOS_CLASS_LAMBDA(typename batch_domain_type::discrete_element_type const j) {
                    const auto spline_eval_1D = spline_eval[j];
                    const auto coords_eval_1D = coords_eval[j];
                    const auto spline_coef_1D = spline_coef[j];
                    for (auto const i : evaluation_domain) {
                        spline_eval_1D(i) = eval(coords_eval_1D(i), spline_coef_1D);
                    }
                });

With:

ddc::parallel_for_each(
                exec_space(),
                spline_eval.domain(),
                KOKKOS_CLASS_LAMBDA(typename batched_evaluation_domain_type::discrete_element_type const e) {
                   typename evaluation_domain_type::discrete_element_type const i(e);
                   typename batch_domain_type::discrete_element_type const j(e);

                    spline_eval(i, j) = eval(coords_eval(i, j), spline_coef[j]);
                });

Makes the performance reduce from1.59us to 0.56us (for a benchmark nx=1000, ny=100000).

The optimal solution may require hierarchical parallelism in ddc (#396) though and maybe transposition of spline_coef to make spline_coef[j] contiguous.

@tpadioleau should I address this ?

[tpadioleau]

Do you have a profiling of a spline interpolation on GPU ?

[blegouix]

I have this (for the current status):

(this is 1.36us in place of 1.59us but this is fluctuation, the case is the same)

I can provide .ncu-rep files if you want or the informations you need

[tpadioleau]

I would be interested to see a time trace like provided by nsys or from Kokkos simple kernel timer.

About the figures, I feel skeptical. I think they are of the order of magnitude of the latency of launching a GPU kernel, https://developer.nvidia.com/blog/understanding-the-visualization-of-overhead-and-latency-in-nsight-systems/#nsight_systems_overhead. Did you mean millisecond ?

[blegouix]

Yeaah, that's milliseconds sorry

[blegouix]

Anyway, in fact there is another problem with this table, it is for ny=10000, not 100000. And I realize the patch is propose is actually slower with ny=100000. So I close the issue.