Fix: De-centralize the network to support multiple clusters #82

codinja1188 · 2024-06-25T12:37:40Z

Description:

This pull request introduces changes to decentralize the network setup in the Terraform Equinix Metal Nutanix cluster module to support multiple clusters. Key modifications include:

Network Decentralization: Adjustments in network configurations to facilitate multiple cluster setups.
Variable Updates: Introduction of optional variables for netmask and other network parameters, allowing more flexible configurations.
Code Refinements: Various updates to templates and scripts to ensure proper functionality and connectivity between clusters.

Additional issues it fixes.

https://github.com/equinix-labs/terraform-equinix-metal-nutanix-cluster/issues/74

codinja1188 · 2024-06-25T12:53:14Z

@displague ,

Here are some snapshots

VRF

Metal Gateways

displague · 2024-06-27T15:43:45Z

main.tf

@@ -60,7 +61,8 @@ resource "equinix_metal_device" "bastion" {
  user_data = templatefile("${path.module}/templates/bastion-userdata.tmpl", {
    metal_vlan_id   = local.vxlan,
    address         = cidrhost(var.cluster_subnet, 2),
-    netmask         = cidrnetmask(var.cluster_subnet),
+    netmask         =  cidrnetmask(cidrsubnet(var.cluster_subnet, -1, -1)),


Netmask should be optional variable to the module.

When netmask is given, it will be used. Your example will pass the VRF /21 netmask in to this module.

When netmask is omitted (default, empty), a local will calculate it using the cidrnetmask function.

Since netmask is only used in one place in the userdata script, this seems sufficient.
https://github.com/equinix-labs/terraform-equinix-metal-nutanix-cluster/blob/main/templates/bastion-userdata.tmpl#L44

displague · 2024-06-27T15:43:54Z

main.tf

@@ -134,7 +136,8 @@ resource "equinix_metal_device" "nutanix" {
  wait_for_reservation_deprovision = length(var.nutanix_reservation_ids) > 0

  ip_address {
-    type = "private_ipv4"
+    type    = "private_ipv4"
+    cidr            = 21


The private_addr CIDR assigned on equinix_metal_device create can not be a /21.

ip_address represents the Layer3, per-server, IP assignments.

Modifying the VRF range to represent a /21 would be done in your example (#68), not in the main module.

The range passed to this module would be a /22, from which the gateway would be derived or its IP also passed in as a variable.

displague · 2024-06-27T15:45:28Z

templates/bastion-userdata.tmpl

@@ -51,12 +51,15 @@ write_files:
    dhcp-range=${host_dhcp_start},${host_dhcp_end},${lease_time}
    dhcp-mac=set:${set},${nutanix_mac}
    dhcp-range=tag:${set},${vm_dhcp_start},${vm_dhcp_end},${lease_time}
+    dhcp-option=option:netmask,${netmask}


I'm curious how this was working without the netmask previously defined. Perhaps the default behavior was to use the netmask and gateway from the host's interface where the DHCP range fits.

I think these explicit definitions do make sense.

Actually, it's blocking the connectivity to outside. Unable to download the image.

displague · 2024-06-27T15:46:22Z

templates/bastion-userdata.tmpl

 packages:
  - iptables-persistent
  - expect
  - sshpass
  - dnsmasq
 runcmd:
 - sysctl -p /etc/sysctl.d/10-ip-forwarding.conf
+- sysctl -w net.ipv4.ip_forward=1


This line is redundant to the line above it.

Updated in latest commits

codinja1188 · 2024-06-28T14:04:45Z

@displague,

How to verify/confirm metal gateways are reachable OR not?

codinja1188 · 2024-06-28T14:07:08Z

@displague,

Two clusters(like Bastion, Nutanix AHV, CVM controller) are to reachable to their gateways IP's

admin@NTNX-7WWG2N3-A-CVM:192.168.96.18:~$ ping 192.168.96.1
PING 192.168.96.1 (192.168.96.1) 56(84) bytes of data.
64 bytes from 192.168.96.1: icmp_seq=1 ttl=64 time=0.187 ms
64 bytes from 192.168.96.1: icmp_seq=2 ttl=64 time=0.204 ms
64 bytes from 192.168.96.1: icmp_seq=3 ttl=64 time=0.191 ms
^C
--- 192.168.96.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1999ms
rtt min/avg/max/mdev = 0.187/0.194/0.204/0.007 ms
admin@NTNX-7WWG2N3-A-CVM:192.168.96.18:~$ exit
logout
Connection to 192.168.96.18 closed.
[root@NTNX-7WWG2N3-A ~]# ping 192.168.96.1
PING 192.168.96.1 (192.168.96.1) 56(84) bytes of data.
64 bytes from 192.168.96.1: icmp_seq=1 ttl=64 time=0.363 ms
64 bytes from 192.168.96.1: icmp_seq=2 ttl=64 time=0.154 ms
64 bytes from 192.168.96.1: icmp_seq=3 ttl=64 time=0.148 ms
64 bytes from 192.168.96.1: icmp_seq=4 ttl=64 time=0.169 ms
^C
--- 192.168.96.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3057ms
rtt min/avg/max/mdev = 0.148/0.208/0.363/0.090 ms
[root@NTNX-7WWG2N3-A ~]# exit
logout
Connection to 192.168.96.4 closed.
root@bastion:~# ping 192.168.96.1 -c 5
PING 192.168.96.1 (192.168.96.1) 56(84) bytes of data.
64 bytes from 192.168.96.1: icmp_seq=1 ttl=64 time=0.189 ms
64 bytes from 192.168.96.1: icmp_seq=2 ttl=64 time=0.216 ms
64 bytes from 192.168.96.1: icmp_seq=3 ttl=64 time=0.274 ms
64 bytes from 192.168.96.1: icmp_seq=4 ttl=64 time=0.177 ms
64 bytes from 192.168.96.1: icmp_seq=5 ttl=64 time=0.240 ms

--- 192.168.96.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4090ms
rtt min/avg/max/mdev = 0.177/0.219/0.274/0.035 ms

Cluster -A is not reachable for Cluster -B Gateway, Do you think, do we have to add any firewall rules ?

codinja1188 · 2024-07-02T16:08:45Z

@displague ,

I can successfully ping VIPs between clusters.

ctreatma · 2024-07-02T16:48:35Z

variables.tf

+variable "cluster_gateway" {
+  description = "The cluster gateway IP address"
+  type        = string
+  default     = "192.168.96.1"


Why is this the default value?

@ctreatma ,

Updated in latest commits, Can you have a quick review

I removed the default value, made it as User Optional Input, In case user didn't provided it, module it's self generate from mandatory varialbe cluster_subnet.

README.md

codinja1188 · 2024-07-08T16:32:51Z

@displague ,

PR ready for review

displague · 2024-07-10T19:45:15Z

main.tf

@@ -20,6 +21,10 @@ resource "terraform_data" "input_validation" {
      condition     = (var.metal_project_name != "" && var.metal_project_id == "") || (var.metal_project_name == "" && var.metal_project_id != "")
      error_message = "One (and only one) of `metal_project_name` or `metal_project_id` is required"
    }
+    precondition {
+      condition     = (var.cluster_subnet != "" && var.cluster_subnet == "")


This condition is impossible. I don't believe we need a precondition for this at all.

Did you mean to check if var.cluster_subnet is not empty but local.cluster_gateway is empty, in which case cluster_gateway would be required?

variables.tf

main.tf

README.md

codinja1188 · 2024-07-11T17:17:08Z

@ctreatma / @displague ,

data.local_file.cvm_ip_address: Reading...
data.local_file.cvm_ip_address: Read complete after 0s [id=a77db81fae96a895fc3db05af25db50e6fc84a91]
╷
│ Error: error reserving IP address block: json: cannot unmarshal array into Go struct field ErrorResponse.errors of type string
│
│   with equinix_metal_reserved_ip_block.nutanix,
│   on main.tf line 109, in resource "equinix_metal_reserved_ip_block" "nutanix":
│  109: resource "equinix_metal_reserved_ip_block" "nutanix" {
│

Is there any known issues in Infra side. oberserved the issue in main branch too.

ctreatma · 2024-07-11T17:33:17Z

@codinja1188 I think someone else ran into a similar problem recently. Could you run with TF_LOG=debug in order to log HTTP details to stdout and share the request URL and response body that lead to this error?

One thing to note is that the parse error is happening on an error response from the API, which likely means there's something wrong with the attributes being passed in to terraform rather than a problem inside the terraform provider. When we've seen this parse error before, it seemed to be triggered by sending invalid IP addresses to the API, so you should double-check that you're not including unnecessary slashes or cidr notation in the network attribute or other attributes of the equinix_metal_reserved_ip_block resource.

codinja1188 · 2024-07-12T15:44:11Z

@displague / @ctreatma ,

I verified, it's working. plz check and approve it.

codinja1188 mentioned this pull request Jun 27, 2024

examples: Connect multiple Nutanix sites and migrate VMs between them #68

Closed

displague reviewed Jun 27, 2024

View reviewed changes

codinja1188 marked this pull request as ready for review July 1, 2024 15:42

ctreatma reviewed Jul 2, 2024

View reviewed changes

codinja1188 mentioned this pull request Jul 8, 2024

Terraform Destroy Fails to Delete VLAN's #74

Closed

displague reviewed Jul 8, 2024

View reviewed changes

README.md Show resolved Hide resolved

displague reviewed Jul 10, 2024

View reviewed changes

ctreatma reviewed Jul 10, 2024

View reviewed changes

README.md Show resolved Hide resolved

Fix: De-centralize the network support multiple clusters @build @publish

2dc54b0

displague mentioned this pull request Jul 13, 2024

improve delete of metal_port resources with reset_on_delete = true #86

Closed

displague approved these changes Jul 13, 2024

View reviewed changes

displague merged commit dff27b9 into equinix-labs:main Jul 13, 2024
1 check passed

displague mentioned this pull request Jul 13, 2024

feat: examples to connect multiple Nutanix sites and migrate VMs betwe… #71

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix: De-centralize the network to support multiple clusters #82

Fix: De-centralize the network to support multiple clusters #82

codinja1188 commented Jun 25, 2024 •

edited

Loading

codinja1188 commented Jun 25, 2024

displague Jun 27, 2024

displague Jun 27, 2024

displague Jun 27, 2024

codinja1188 Jul 2, 2024

displague Jun 27, 2024

codinja1188 Jul 8, 2024

codinja1188 commented Jun 28, 2024 •

edited

Loading

codinja1188 commented Jun 28, 2024 •

edited

Loading

codinja1188 commented Jul 2, 2024

ctreatma Jul 2, 2024

codinja1188 Jul 8, 2024

codinja1188 Jul 10, 2024

codinja1188 commented Jul 8, 2024

displague Jul 10, 2024

codinja1188 commented Jul 11, 2024

ctreatma commented Jul 11, 2024 •

edited

Loading

codinja1188 commented Jul 12, 2024

Fix: De-centralize the network to support multiple clusters #82

Fix: De-centralize the network to support multiple clusters #82

Conversation

codinja1188 commented Jun 25, 2024 • edited Loading

Description:

Additional issues it fixes.

codinja1188 commented Jun 25, 2024

VRF

Metal Gateways

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codinja1188 commented Jun 28, 2024 • edited Loading

codinja1188 commented Jun 28, 2024 • edited Loading

codinja1188 commented Jul 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codinja1188 commented Jul 8, 2024

Choose a reason for hiding this comment

codinja1188 commented Jul 11, 2024

ctreatma commented Jul 11, 2024 • edited Loading

codinja1188 commented Jul 12, 2024

codinja1188 commented Jun 25, 2024 •

edited

Loading

codinja1188 commented Jun 28, 2024 •

edited

Loading

codinja1188 commented Jun 28, 2024 •

edited

Loading

ctreatma commented Jul 11, 2024 •

edited

Loading