WIP: design doc proposal

docs/developer/design-v2-2023-11-11.md

@@ -0,0 +1,198 @@
+# Design Document for Materialize Terraform Provider
+
+The proposed updates to the Materialize Terraform provider aim to streamline the management of resources within Materialize, offering a unified interface for global and regional resources. The provider will leverage a single authentication method, an app password, to derive the necessary configurations for seamless interaction with various services. It will also incorporate a dynamic client allocation mechanism to handle different resource types and their requirements.
+
+## Provider Overview
+
+### Objectives
+
+-   To provide a singular Terraform provider capable of interfacing with multiple backends and services.
+-   To facilitate the management of global resources and regional deployments through a consistent workflow.
+-   To enable a high degree of automation in resource provisioning and management, thereby reducing manual overhead and potential for errors.
+
+### Features
+
+-   **Single Authentication**: Utilizing an app password to access and manage resources without the need for multiple credentials.
+-   **Global and Regional Resource Management**: Allowing users to manage resources that are not bound by regional constraints, as well as those that are region-specific.
+-   **Dynamic Client Allocation**: Depending on the resource's requirement, the appropriate client (DB, Frontegg, Cloud API and etc.) will be allocated dynamically.
+
+## Architecture
+
+### Provider Configuration
+
+The provider will be initialized with parameters that determine its behavior and default settings:
+
+```hcl
+provider "materialize" {
+  app_password   = var.materialize_app_password
+  default_region = var.materialize_default_region
+  endpoint       = var.materialize_endpoint
+}
+```
+
+Users will specify the app password and the default region for operations. Optionally, the endpoint can be overridden for testing or staging purposes.
+
+### Multi-Client Structure
+
+The provider will maintain a map of clients for database operations across different regions and a client for interfacing with the Frontegg API.
+
+```hcl
+type ProviderMeta struct {
+    DB       map[Region]*clients.DBClient
+    Frontegg *clients.FronteggClient
+    // Future: CloudAPI *clients.CloudAPIClient
+}
+```
+
+This structure allows for scalability and extension of the provider to incorporate additional clients as necessary.
+
+## Resources and Operations
+
+### Region Resource
+
+Responsible for managing the availability of Materialize regions. It will interact with the Cloud API to enable or retrieve regions as requested by the user.
+
+```hcl
+resource "materialize_region" "aws_us_east_1" {
+  region = "aws/us-east-1"
+}
+```
+
+Operations:
+
+-   **Enable Region**: Uses the Cloud API to enable a new region.
+-   **Read Region**: Retrieves the status of a region from the Cloud API.
+-   **Disable Region**: Noop. Regions cannot be disabled once enabled.
+
+### User Resource
+
+Manages the lifecycle of a user within the Frontegg platform.
+
+```hcl
+resource "materialize_app_user" "example_user" {
+  email          = "[email protected]"
+  auth_provider  = "local"
+  role_ids       = ["role-id-1", "role-id-2"] # Logic to retrieve role IDs from role names?
+  # Additional attributes...
+}
+```
+
+Operations:
+
+-   **Create User**: Invokes the Frontegg API to create/invite a user.
+-   **Read User**: Retrieves user details from Frontegg.
+-   **Update User**: Attaches or detaches roles from the user.
+-   **Delete User**: Removes the user from Frontegg.
+
+### Organization Data Source?
+
+Generates a list of available organizations from Frontegg.
+
+```hcl
+data "materialize_app_organizations" "all" {}
+```
+
+Operations:
+
+-   **List Organizations**: Fetches all available organizations from Frontegg.
+
+### Role Data Source
+
+Generates a list of available roles from Frontegg.
+
+```hcl
+data "materialize_app_roles" "all" {}
+```
+
+Operations:
+
+-   **List Roles**: Fetches all available roles from Frontegg.
+
+## Testing Strategy
+
+Along the existing unit tests described in the [design document](../../DESIGN.md), the provider will include a comprehensive suite of tests for all resources and operations, including:
+
+-   **Unit Tests**: Will run against a test Materialize environment or a mock service to ensure that the provider functions correctly in isolation.
+-   **Integration Tests**: Will run against staging instance to confirm that the provider interacts correctly with the actual services.
+-   **Acceptance Tests**: То be determined.
+
+## State Management
+
+The provider will ensure that the Terraform state reflects the true state of resources in Materialize Cloud and Frontegg, using Terraform's state management capabilities.
+
+## User Experience and Documentation
+
+Users should be able to easily upgrade to the new provider version without having to make significant changes to their existing configurations.
+
+The release should include clear documentation and examples to help users understand the new features and how to use them.
+
+This should be clearly communicated to users through release notes and other channels.
+
+Clear and detailed documentation will accompany the provider, explaining usage, resource configuration, and troubleshooting.
+
+### Edge Case Management
+
+Further exploration of how the provider will handle edge cases, such as delays in region enablement or Frontegg API failures, will be included in the design document.
+
+## Database Resource
+
+### Integration with Frontegg and Global API
+
+The Materialize Terraform provider will integrate with the Frontegg API to dynamically fetch the necessary credentials and regional endpoint information.
+
+This approach streamlines the process of establishing SQL connections across various Materialize regions using a single app password.
+
+### Resource Definitions
+
+#### Cluster in an Existing Region
+
+To create a cluster in a region that has been previously enabled, users can define a `materialize_cluster` resource without creating a dependency on the region's resource within Terraform. This is useful when operating in regions activated outside the scope of the current Terraform configuration.
+
+```hcl
+resource "materialize_cluster" "eu_west_1_cluster" {
+  region = "aws/eu-west-1"
+  # Additional configuration options...
+}
+```
+
+#### Cluster in the Default Provider Region
+
+For convenience, users can create clusters without specifying a region, in which case the provider uses the default region specified in the provider's configuration block. This simplifies resource definitions and maintains consistency across Terraform configurations.
+
+```hcl
+resource "materialize_cluster" "default_region_cluster" {
+  # The provider will infer the region from the default settings.
+  # Additional configuration options...
+}
+```
+
+#### Cluster in a Terraform-Enabled Region
+
+When a region is enabled within Terraform, resource definitions can reference the `materialize_region` resource. This ensures that Terraform manages dependencies correctly, creating the cluster only after the region is fully enabled.
+
+```hcl
+resource "materialize_cluster" "us_east_1_cluster" {
+  region = materialize_region.aws_us_east_1.region
+  # Additional configuration options...
+}
+```
+
+### Enhanced Workflow
+
+The provider will handle the following workflow:
+
+1.  **Authentication**: Utilize the app password to retrieve the username associated with the provided token.
+2.  **Regional Endpoint Retrieval**: Query the global API to obtain the hostname and port for the specified region.
+3.  **SQL Connection Establishment**: Connect to the desired region's SQL interface using the retrieved credentials and endpoint information.
+4.  **Resource Management**: Facilitate the creation, updating, and deletion of resources within the connected region using the established SQL connection.
+
+### Error Handling and Region Validation
+
+The provider will have error handling to provide clear and actionable feedback when:
+
+-   A specified region is not enabled or in the process of being disabled.
+-   There are network or authentication issues while establishing the SQL connection.
+
+### Simplifying Configuration
+
+By abstracting away the complexities of managing credentials and endpoints, the provider allows users to focus on the high-level definitions of their Materialize infrastructure, promoting a simplified and more efficient configuration process.