-
Notifications
You must be signed in to change notification settings - Fork 122
GSoC 2024
Clad is an automatic differentiation (AD) clang plugin for C++. Given a C++ source code of a mathematical function, it can automatically generate C++ code for computing derivatives of the function. Clad has found uses in statistical analysis and uncertainty assessment applications. In scientific computing and machine learning, GPU multiprocessing can provide a significant boost in performance and scalability. This project focuses on enabling the automatic differentiation of CUDA GPU kernels using Clad. This will allow users to take advantage of the power of GPUs while benefiting from the accuracy and speed of automatic differentiation.
- Research about automatic differentiation of code involving CUDA GPU kernels. Prepare a report and an initial strategy to follow. This may involve brainstorming and the need for innovative solutions.
- Enable reverse-mode automatic differentiation of CUDA GPU kernels and calls to CUDA GPU kernels from the host code.
- Add proper tests and documentation.
- Automatic differentiation
- CUDA C++ programming
- C++ programming and Clang frontend
In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. Automatic differentiation is an alternative technique to Symbolic differentiation and Numerical differentiation (the method of finite differences). Clad is based on Clang which provides the necessary facilities for code transformation. The AD library can differentiate non-trivial functions, to find a partial derivative for trivial cases and has good unit test coverage.
The most heavily used entity in AD is a stack-like data structure called a tape. For example, the first-in last-out access pattern, which naturally occurs in the storage of intermediate values for reverse mode AD, lends itself towards asynchronous storage. Asynchronous prefetching of values during the reverse pass allows checkpoints deeper in the stack to be stored furthest away in the memory hierarchy. Checkpointing provides a mechanism to parallelize segments of a function that can be executed on independent cores. Inserting checkpoints in these segments using separate tapes enables keeping the memory local and not sharing memory between cores. We will research techniques for local parallelization of the gradient reverse pass, and extend it to achieve better scalability and/or lower constant overheads on CPUs and potentially accelerators. We will evaluate techniques for efficient memory use, such as multi-level checkpointing support. Combining already developed techniques will allow executing gradient segments across different cores or in heterogeneous computing systems. These techniques must be robust and user-friendly, and minimize required application code and build system changes.
This project aims to improve the efficiency of the clad tape and generalize it into a tool-agnostic facility that could be used outside of clad as well.
- Optimize the current tape by avoiding re-allocating on resize in favor of using connected slabs of array
- Enhance existing benchmarks demonstrating the efficiency of the new tape
- Add the tape thread safety
- Implement multilayer tape being stored in memory and on disk
- [Stretch goal] Support cpu-gpu transfer of the tape
- [Stretch goal] Add infrastructure to enable checkpointing offload to the new tape
- [Stretch goal] Performance benchmarks
- Automatic differentiation
- C++ programming
- Clang frontend
Clad is an automatic differentiation (AD) clang plugin for C++. Given a C++ source code of a mathematical function, it can automatically generate C++ code for computing derivatives of the function. Clad has found uses in statistical analysis and uncertainty assessment applications.
Object-oriented paradigms (OOP) provide a structured approach for complex use cases, allowing for modular components that can be reused & extended. OOP also allows for abstraction which makes code easier to reason about & maintain. Gaining full OOP support is an open research area for automatic differentiation codes.
This project focuses on improving support for differentiating object-oriented constructs in Clad. This will allow users to seamlessly compute derivatives to the algorithms in their projects which use an object-oriented model. C++ object-oriented constructs include but are not limited to: classes, inheritance, polymorphism, and related features such as operator overloading.
- Study the current object-oriented differentiable programming support in Clad. Prepare a report of missing constructs that should be added to support the automatic differentiation of object-oriented paradigms in both the forward mode AD and the reverse mode AD.
- Some of the missing constructs are: differentiation of constructors, limited support for differentiation of operator overloads, reference class members, and no way of specifying custom derivatives for constructors.
- Add support for the missing constructs.
- Add proper tests and documentation.
- Automatic differentiation
- C++ programming
- Clang frontend
In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to numerically evaluate the derivative of a function specified by a computer program. Automatic differentiation is an alternative technique to Symbolic differentiation and Numerical differentiation (the method of finite differences). Clad is based on Clang which provides the necessary facilities for code transformation. The AD library can differentiate non-trivial functions, to find a partial derivative for trivial cases and has good unit test coverage.
C++ provides the specifiers consteval and constexpr to allow compile time
evaluation of functions. constexpr declares a possibility, i.e the function
will be evaluated at compile time if possible, else at runtime; whereas
consteval makes it mandatory, i.e every call to the function must produce a
compile-time constant.
The aim of this project is to ensure that same semantics are followed by the generated derivative function, i.e if the primal function is evaluated at compile time (because of constexpr or consteval specifier), then the generated derivative code should also have the same specifier to be evaluatable at compile time.
This will enable clad to demonstrate the benefits of doing automatic differentiation directly on C++ frontend to utilize the benefits of clang's infrastructure.
After successful completion of the project the code snippet should work as expected:
#include <cstdio>
#include "clad/Differentiator/Differentiator.h"
constexpr double sq(double x) { return x*x; }
consteval double fn(double x, double y, double z) {
double res = sq(x) + sq(y) + sq(z);
return res;
}
int main() {
auto d_fn = clad::gradient(fn);
double dx = 0, dy = 0, dz = 0;
d_fn.execute(3, 4, 5, &dx, &dy, &dz);
printf("Gradient vector: [%.2f, %.2f, %.2f]", dx, dy, dz);
return 0;
}- Add support for differentiation with respect to consteval and constexpr functions in the forward mode.
- Add support for differentiation with respect to consteval and constexpr functions in the reverse mode.
- Extend the unit test coverage.
- Develop tutorials and documentation.
- Present the work at the relevant meetings and conferences.
- Automatic differentiation
- C++ programming
- Clang frontend
If you have interest in working on the project there is a list of things to do in order to maximize your chances to get selected:
- Contact the mentors and express interest in the project. Make sure you attach your CV;
- Download the source code of the project, build it and run the demos;
- Start familiarizing yourself with the codebase;
- If you have questions you can always contact a mentor.
The mentors are interested in working with all candidates but unfortunately the rules allow only one to be selected. There are a few tasks which give bonus points to candidate's application:
- Submit a valid bug -- demonstrates that the candidate has completed step 2 and 3 from the previous section.
- Fix a bug -- demonstrates the technical skills of the candidate and shows he/she can work independently on the project. The mentors can suggest looking into these good first issues. Fixing one issue may be enough to become a successful candidate.
Good luck!