Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove assumption of CudaUVM in order to create Host team policies #227

Open
bartgol opened this issue Jun 24, 2022 · 0 comments
Open

Remove assumption of CudaUVM in order to create Host team policies #227

bartgol opened this issue Jun 24, 2022 · 0 comments
Labels
enhancement New feature or request

Comments

@bartgol
Copy link
Contributor

bartgol commented Jun 24, 2022

PR #218 added the ability to dispatch a kernel on host when CudaUVM is used, by creating a team policy on host. However, CudaUVM is still a device memory space, with behind-the-scenes sync between host and device. In particular, this means that a View with memory space HostSpace cannot be assigned to a view with memory space CudaUVM, and viceversa. This can have bad consequences for codes that use host mirrors, and are templated on the memory space. For instance, the following won't work:

template<typename MemSpace>
struct Types {
  using view = Kokkos::View<double*, MemSpace, Kokkos::LayoutRight>;
};

typename Types<Kokkos::CudaUVM>::view  my_view_d ("",10);
typename Types<HostSpace>::view my_view_h = Kokkos::create_host_mirror(my_view_d); // ERROR!

since the host mirror still has CudaUVM as mem space (b/c uvm is accessible from host), but CudaUVM is not assignable to HostSpace. On the other hand, CudaHostPinnedSpace is both assignable and accessible from device and host. The snippet above would work with CudaHostPinnedSpace in place of CudaUVM.

However, notice that nothing in the team policy requires to know what memory space the views that will be accessed will be on. In fact, the concept of "view" is orthogonal to that of a team policy. All that the team policy needs to know is the execution space where the kernel will be run on. We should therefore remove the requirement that CudaUVM is used, and allow the user to create a policy on host regardless. It is up to the user to ensure that the views used in the kernel are accessible on host.

The drawback would be that Kokkos does not offer a configuration option to make CudaHostPinnedSpace the default mem space for the Cuda execution space (while it does so for CudaUVM). Therefore, any usage of ExecSpace::memory_space would yield something different from the host pinned space.

@bartgol bartgol added the enhancement New feature or request label Jun 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant