Feat: Create fly.io template to launch beta9 in fly.io machines #722

jsun-m · 2024-11-19T19:13:51Z

Add templates and setup file to spin up an entire beta9 cluster in fly.io.

This includes templates for:

control plane redis
juicefs redis
blobcache redis
control plane postgres
control plane storage
image storage

Currently, there is no built in provider to automatically provision nodes. You would have to use the beta9 agent to manually add a node to the beta9 cluster.

jsun-m · 2024-11-22T03:16:21Z

sdk/src/beta9/cli/machine.py

@@ -177,7 +177,8 @@ def create_machine(service: ServiceClient, pool: str):
              --tailscale-url "{res.machine.tailscale_url}" \\
              --tailscale-auth "{res.machine.tailscale_auth}" \\
              --pool-name "{res.machine.pool_name}" \\
-              --provider-name "{res.machine.provider_name}"
+              --provider-name "{res.machine.provider_name}" \\
+              --gateway-url "https://{service._config.gateway_host}"


This would need to conditionally added since in beam the grpc host and the http host are on different subdomains

luke-lombardi · 2024-12-05T14:00:59Z

pkg/scheduler/pool_external.go

@@ -274,7 +274,8 @@ func (wpc *ExternalWorkerPoolController) createWorkerJob(workerId, machineId str
 	workerImage := fmt.Sprintf("%s/%s:%s",
 		wpc.config.Worker.ImageRegistry,
 		wpc.config.Worker.ImageName,
-		wpc.config.Worker.ImageTag,
+		// wpc.config.Worker.ImageTag,


revert this

luke-lombardi · 2024-12-05T14:01:30Z

pkg/scheduler/pool_external.go

@@ -370,6 +371,11 @@ func (wpc *ExternalWorkerPoolController) getWorkerEnvironment(workerId, machineI
 		podHostname = fmt.Sprintf("machine-%s.%s.%s", machineId, wpc.config.Tailscale.User, wpc.config.Tailscale.HostName)
 	}

+	grpcPort := 443
+	if wpc.config.GatewayService.GRPC.ExternalPort != 0 {


why not just set this in the config to 443 and remove the if clause?

luke-lombardi · 2024-12-05T14:02:46Z

pkg/types/config.go

+	Hostname string `key:"hostname" json:"hostname"`
+}
+
+type RootAgentConfig struct {


can we rename this?

luke-lombardi · 2024-12-05T14:03:19Z

pkg/types/config.go

+
+type RootAgentConfig struct {
+	ControlPlaneRedis AgentRedisConfig `key:"controlPlaneRedis" json:"control_plane_redis"`
+	JuicefsRedis      AgentRedisConfig `key:"juicefsRedis" json:"juicefs_redis"`


Not sure I like this config structure. We're basically only using it to override the URLs, feels like there's a more elegant approach here

i'm not a fan either. especially since we don't pass this to the agent.

we do this

remoteConfig.Database.Redis.Addrs[0] = fmt.Sprintf("%s:%d", redisHostname, 6379) remoteConfig.Storage.JuiceFS.RedisURI = fmt.Sprintf("%s://:%s@%s", scheme, juicefsRedisPassword, juiceFsRedisHostname) remoteConfig.BlobCache.BlobFs.Sources[idx].JuiceFS.RedisURI = remoteConfig.Storage.JuiceFS.RedisURI

I added a new variable config and removed the juicefs redis, controlplane redis and blobcache urls.
The new veriable is just called
DynamicServiceHosts

that encapsulate all of them

basically if its true, we use tailscale for the agent configs, if not it will use the host currently specified in config in the places that tailscale would have overrided

luke-lombardi · 2024-12-05T14:05:42Z

pkg/worker/lifecycle.go

@@ -364,6 +364,7 @@ func (s *Worker) getContainerEnvironment(request *types.ContainerRequest, option
 		fmt.Sprintf("CONTAINER_ID=%s", request.ContainerId),
 		fmt.Sprintf("BETA9_GATEWAY_HOST=%s", os.Getenv("BETA9_GATEWAY_HOST")),
 		fmt.Sprintf("BETA9_GATEWAY_PORT=%s", os.Getenv("BETA9_GATEWAY_PORT")),
+		fmt.Sprintf("BETA9_GATEWAY_TLS=%s", os.Getenv("BETA9_GATEWAY_TLS")),


I think we can just detect if a host has TLS enabled programatically, something like this:

def is_tls_enabled(url: str) -> bool: """Check if a given URL supports TLS. Args: url (str): The URL to check. Returns: bool: True if TLS is enabled, False otherwise. """ try: # Extract hostname and default to port 443 for HTTPS hostname = url.split("//")[-1].split("/")[0] port = 443 # Create SSL context and test the connection context = ssl.create_default_context() with socket.create_connection((hostname, port), timeout=5) as sock: with context.wrap_socket(sock, server_hostname=hostname): return True except (ssl.SSLError, socket.error): return False

Then we can remove the env var

this makes sense. the code here works

luke-lombardi · 2024-12-16T21:02:39Z

pkg/scheduler/pool_external.go

@@ -274,7 +274,8 @@ func (wpc *ExternalWorkerPoolController) createWorkerJob(workerId, machineId str
 	workerImage := fmt.Sprintf("%s/%s:%s",
 		wpc.config.Worker.ImageRegistry,
 		wpc.config.Worker.ImageName,
-		wpc.config.Worker.ImageTag,
+		// wpc.config.Worker.ImageTag,


luke-lombardi · 2024-12-16T21:03:21Z

sdk/src/beta9/config.py

@@ -176,6 +180,7 @@ def prompt_for_config_context(
    prompt_gateway_port = functools.partial(
        terminal.prompt, text="Gateway Port", default=gateway_port or settings.gateway_port
    )
+    prompt_tls = functools.partial(terminal.prompt, text="TLS", default=False)


why is this prompted? shouldn't it be using that function below?

is_tls_enabled

jsun-m changed the title ~~Create fly io template to launch beta9 in fly io machines~~ Create fly.io template to launch beta9 in fly.io machines Nov 19, 2024

Remove files from branch

14cb30d

jsun-m force-pushed the jm/fly-io-cluster branch from 9d2da82 to 14cb30d Compare November 19, 2024 19:20

jsun-m added 4 commits November 19, 2024 12:21

Remove config.json

6c4a9b3

Remove config.json

7dfcd8e

Complete deployment

3f422d7

handle complete deployment

3b41ef8

jsun-m changed the title ~~Create fly.io template to launch beta9 in fly.io machines~~ Feat:Create fly.io template to launch beta9 in fly.io machines Nov 19, 2024

jsun-m changed the title ~~Feat:Create fly.io template to launch beta9 in fly.io machines~~ Feat: Create fly.io template to launch beta9 in fly.io machines Nov 19, 2024

Testing worker releasE

636770e

jsun-m temporarily deployed to Release November 21, 2024 21:06 — with GitHub Actions Inactive

Update config from env

9eeec51

jsun-m temporarily deployed to Release November 21, 2024 21:54 — with GitHub Actions Inactive

Another worker build

3faa1ae

jsun-m temporarily deployed to Release November 22, 2024 01:57 — with GitHub Actions Inactive

jsun-m temporarily deployed to Release November 22, 2024 02:18 — with GitHub Actions Inactive

jsun-m added 2 commits November 21, 2024 19:32

First pass of review is ready

f43fe85

First pass of review is ready

82dae39

jsun-m requested review from nickpetrovic and luke-lombardi November 22, 2024 03:10

cleanup

5e7472e

jsun-m commented Nov 22, 2024

View reviewed changes

merge

2604299

luke-lombardi requested changes Dec 5, 2024

View reviewed changes

jsun-m added 2 commits December 5, 2024 10:32

merged

aca284b

Build and test worker image

87b9c88

jsun-m temporarily deployed to Release December 5, 2024 23:50 — with GitHub Actions Inactive

Rename config

259bc15

jsun-m requested a review from luke-lombardi December 6, 2024 01:03

Determine if the cmd text requires gateway url

a7380a2

luke-lombardi requested changes Dec 16, 2024

View reviewed changes

Remove envvar

9db8574

jsun-m requested a review from luke-lombardi December 20, 2024 16:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat: Create fly.io template to launch beta9 in fly.io machines #722

Feat: Create fly.io template to launch beta9 in fly.io machines #722

jsun-m commented Nov 19, 2024 •

edited

Loading

jsun-m Nov 22, 2024

luke-lombardi Dec 5, 2024

luke-lombardi Dec 16, 2024

luke-lombardi Dec 5, 2024

luke-lombardi Dec 5, 2024

luke-lombardi Dec 5, 2024

jsun-m Dec 5, 2024 •

edited

Loading

jsun-m Dec 5, 2024

jsun-m Dec 5, 2024

luke-lombardi Dec 5, 2024

luke-lombardi Dec 5, 2024

jsun-m Dec 5, 2024

luke-lombardi Dec 16, 2024

luke-lombardi Dec 16, 2024

jsun-m Dec 23, 2024

Feat: Create fly.io template to launch beta9 in fly.io machines #722

Are you sure you want to change the base?

Feat: Create fly.io template to launch beta9 in fly.io machines #722

Conversation

jsun-m commented Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsun-m Dec 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsun-m commented Nov 19, 2024 •

edited

Loading

jsun-m Dec 5, 2024 •

edited

Loading