Use 3D FFT transforms directly and drop the current 1D decomposition capable strategy #83
Conversation
src/cuda/kernels/complex.f90
Outdated
There was a problem hiding this comment.
Probably a good time to start documenting this function. At the very least I think it needs a description of arguments and a reference for the calculation being carried out. I presume this will need @slaizet's input?
There was a problem hiding this comment.
Also, could this file be renamed to something more descriptive? Suggestions:
spectral_transform.f90spectral_processing.f90poisson_solver.f90
There was a problem hiding this comment.
Added a description for a few arguments, and a reference too if someone wants more details. I need @slaizet's help with the remaining ones.
With the complex reorder functions gone, in this module we'll only carry out postprocessing in spectral space, so renaming is a good idea. I went with spectral_processing.f90.
| c_dev(1:SZ, 1:self%nx, 1:(self%ny*(self%nz/2 + 1))/SZ) => self%c_y_dev | ||
| ! tsize is different than SZ, because here we work on a 3D Cartesian | ||
| ! data structure, and free to specify any suitable thread/block size. | ||
| tsize = 16 |
There was a problem hiding this comment.
If this might be tuned later, can it be stored more publicly? E.g. as a parameter in common.f90. I'm worried this parameter will get forgotten later.
There was a problem hiding this comment.
At the moment the only use case for this variable is here in the postprocessing, and for this specific kernel. GPU kernels are generally written considering the block and thread sizes, so I think its a good idea to define it here and use right after. For example, if we need to process a 3D Cartesian array its likely that we'll require a different thread size. The variable here works well because the kernel we have requires 3 separate 1D arrays with global size to carry out the operation, probably we won't need a similar operation anywhere else. If we observe use of this elsewhere then maybe we can move somewhere else later on.
Co-authored-by: Jamie J Quinn <[email protected]>
We're moving away from doing our own FFTs using our specialist data structure to carry out 3D FFT transforms. The strategy CUDA backend implements prior to this PR was capable of being extended to 1D decomposition (already done earlier for the Thomas based approach), and as it was limited to 1D it was possible to take advantage of this for some optimisations. However, the distributed tridiagonal solver we have necessitates each slab/pencil to have certain amount of thickness, which limits a 1D decomposition strategy for going above 2k^3 ~ 4k^3 domain sizes.
This important shift in strategy was already mentioned in #32, and also #73.
The changes implemented up to so far includes switching from specialist data structure to cartesian data structure so that distributed FFT libraries can be used easily.
And currently, there is a bug in CUDA backend inFixed.poisson_fftthat prevents running with differentnx,ny,nzsizes. Possibly due to 3D FFT transform halving a dimension that is different than the current implementation assumes. Should be fixed soon as its not a big deal.Closes #73.