Add config option to assign VLANs to VLAN groups

[KPTheProf]

Discussed in #372

^{Originally posted by KPTheProf February 22, 2024}
Hi

I have a large number of VLAN's and multiple VLAN Groups configured in Netbox.
I have a VLAN Group for VMware VLAN's but when I import the data, it is imported into the same VLAN number but in a different group.
Is there a way to configure the settings.ini to specify which VLAN group I want the VLAN's added to or updated?

If possible, could I suggest that the currect VLAN options be extended to include a VLAN group or groups based on regex key / value pair similar to the tenant information.

vlan_group_relation_by_name = London/Vlan_.* = VLAN Group 1, Tokio/Vlan_.* = VLAN Group 2
vlan_group_relation_by_id = 1023-1042 = VLAN Group 1

[bb-Ricardo]

Mmhhh, this looks quite nice.

Thabk you. Will try to add it.

[thuguet]

We also would need that!
Also i could be extra nice to define VLAN based on the vSwitch name but if necessary we can do that using the portgroup name too

[KPTheProf]

Hi @bb-Ricardo, is there any progress with this update for multiple VLAN's and VLAN groups please?

[bb-Ricardo]

Hi @KPTheProf,

sorry for the delay, didn't had time the last few weeks to work on this project.

[Croes1]

Hi!

Are you working on this enhancement? It would be great to have this option to add VLANs to a group.

The script is really great and saved us a lot of time when importing data from vcenter!

[bb-Ricardo]

Just had a look at it and it seems a bit strange API wise. Maybe someone else can have a look as well.

I just tested it with NetBox 4.1.6. The UI specifies a VLAN ID's range but the API specification does not mention a VLAN ID. but when you request the object via API, it returns a vid_ranges

POST reuest schema:

GET response schema:

GET result on a test group:

What do you think?

[bb-Ricardo]

Ok, it's a known issue: netbox-community/netbox#17488

[Croes1]

Ah.. then there’s not much to do than wait until it’s fixed.

[bb-Ricardo]

I believe it still can be added.

Just wondering what's the expectation here.

• The VLAN groups exist in NetBox before and just get assigned to the VLAN according to the config
• Does the VLAN group need to be created as well?

Because there exists a discrepancy between the API and the UI.

When creating an VLAN group via UI, the VLAN ID range is mandatory. Creating a VLAN Group via API only requires a name and a slug.

What do you think?

[KPTheProf]

From my perspective, I would expect the VLAN group to already exist because I want to ensure that the VLAN's are only added to that group and not to another group that might have the same VLAN ID.

[bb-Ricardo]

Hi @Croes1, @KPTheProf, @thuguet,

I finally (sorry for this long wait) got some time to implement this feature. I pushed a new commit to the devlopment branch. Would you be able to test this?

[KPTheProf]

Hi @bb-Ricardo
I normally run netbox-sync using the docker image as I'm using the docker version of Netbox on Windows for testing.
I've tried to run the development branch using Python on Windows, but I'm getting strange errors.

File " ... \pyVmomi\SoapAdapter.py", line 1003, in <module>
    _SocketWrapper = ssl.wrap_socket
                     ^^^^^^^^^^^^^^^
AttributeError: module 'ssl' has no attribute 'wrap_socket'

Do have a docker development version that I can test against?

[KPTheProf]

Hi @bb-Ricardo
Don't worry, I've got it to work now.
I'm using Anaconda to setup my Python env and the latest version of pyvmomi was 7.0
I updated to version 8.0.3.0.1 using pip instead and I'm now able to run it.

[KPTheProf]

Hi @bb-Ricardo
Looks like I spoke too soon. It was running for quite some time, but then failed near the end.

Traceback (most recent call last):
  File "\netbox-sync\netbox-sync.py", line 146, in <module>
    main()
  File "\netbox-sync\netbox-sync.py", line 117, in main
    inventory.query_ptr_records_for_all_ips()
  File "\netbox-sync\module\netbox\inventory.py", line 426, in query_ptr_records_for_all_ips
    records = perform_ptr_lookups(data.get("ips"), data.get("servers"))

This is the only change I've made to my settings.ini for the test

vlan_group_relation_by_id = 1-4094 = Company CCS

[bb-Ricardo]

Hey,

thank you for the effort. I just created an beta image with the latest code changes here: docker pull bbricardo/netbox-sync:1.7.1-beta.1

[Croes1]

Thank you for working on this!

I agree, I expect the vlan group to already exist.

I’ll try to do some tests next week! Got some other stuff at work I have to complete first.

[KPTheProf]

Hi @bb-Ricardo

I've tried running the beta docker image, but it is failing with the below error.

docker run --rm -it -v .\settings.ini:/app/settings.ini bbricardo/netbox-sync:1.7.1-beta.1 --log_level DEBUG2 --dry_run 2>&1 | tee-object file2.txt
Traceback (most recent call last):
  File "/app/netbox-sync.py", line 23, in <module>
    from module.sources import instantiate_sources
  File "/app/module/sources/__init__.py", line 11, in <module>
    from module.sources.vmware.connection import VMWareHandler
  File "/app/module/sources/vmware/connection.py", line 29, in <module>
    from module.sources.common.source_base import SourceBase
  File "/app/module/sources/common/source_base.py", line 18, in <module>
    from module.sources.common.handle_vlan import FilterVLANByName, FilterVLANByID
  File "/app/module/sources/common/handle_vlan.py", line 26
    log.error(f"submitted VLAN {self.filter_type} string for VLAN was '{"'None'" if vlan is None else "empty" }'")
                                                                                                       ^^^^^
SyntaxError: f-string: expecting '}'

[bb-Ricardo]

hi, sorry for the hassle.

Yes, it was a bug which even my IDE warned me about but somehow just blindly ignored.

I just pushed another container, please try: docker pull bbricardo/netbox-sync:1.7.1-beta.2

Thank you

[KPTheProf]

Hi @bb-Ricardo
Ok so that ran this time, but I'm not getting the results that I would expect.
I'm seeing a few errors in the output. Any ideas?

2024-11-28 18:03:50,742 - INFO: Updating NetBox 'VLAN' object '330 (NDC Wolverhampton)' with data: {'group': 6}
2024-11-28 18:03:50,795 - ERROR: NetBox returned: PATCH /api/ipam/vlans/97/ Bad Request
2024-11-28 18:03:50,795 - ERROR: NetBox returned body: {'non_field_errors': ['The fields group, vid must make a unique set.']}
2024-11-28 18:03:50,796 - ERROR: Request Failed for VLAN. Used data: {'group': 6}
2024-11-28 18:03:50,796 - INFO: Updating NetBox 'VLAN' object '340 (NDC Wolverhampton)' with data: {'group': 6}
2024-11-28 18:03:50,863 - ERROR: NetBox returned: PATCH /api/ipam/vlans/98/ Bad Request
2024-11-28 18:03:50,863 - ERROR: NetBox returned body: {'__all__': ['VLAN is assigned to group Customer CCS (scope: Customer CCS); cannot also assign to site NDC Wolverhampton.']}
2024-11-28 18:03:50,863 - ERROR: Request Failed for VLAN. Used data: {'group': 6}
2024-11-28 18:03:50,864 - INFO: Updating NetBox 'VLAN' object '2000 (CH_POWER__________10.8.32.128/26)' with data: {'group': 6}
2024-11-28 18:03:50,916 - ERROR: NetBox returned: PATCH /api/ipam/vlans/19/ Bad Request
2024-11-28 18:03:50,916 - ERROR: NetBox returned body: {'non_field_errors': ['The fields group, vid must make a unique set.']}
2024-11-28 18:03:50,916 - ERROR: Request Failed for VLAN. Used data: {'group': 6}
2024-11-28 18:03:50,916 - INFO: Updating NetBox 'VLAN' object '2010 (CH_OOBM_____________10.8.32.0/26)' with data: {'group': 6}
2024-11-28 18:03:50,976 - ERROR: NetBox returned: PATCH /api/ipam/vlans/20/ Bad Request
2024-11-28 18:03:50,976 - ERROR: NetBox returned body: {'non_field_errors': ['The fields group, vid must make a unique set.']}
2024-11-28 18:03:50,976 - ERROR: Request Failed for VLAN. Used data: {'group': 6}
2024-11-28 18:03:50,976 - INFO: Updating NetBox 'VLAN' object '2020 (CH_MANAGMENT________10.8.36.0/24)' with data: {'group': 6}
2024-11-28 18:03:51,033 - ERROR: NetBox returned: PATCH /api/ipam/vlans/21/ Bad Request
2024-11-28 18:03:51,033 - ERROR: NetBox returned body: {'non_field_errors': ['The fields group, vid must make a unique set.']}
2024-11-28 18:03:51,033 - ERROR: Request Failed for VLAN. Used data: {'group': 6}
2024-11-28 18:03:51,033 - INFO: Updating NetBox 'VLAN' object '2040 (CH_VMOTION__________10.8.37.0/24)' with data: {'group': 6}
2024-11-28 18:03:51,084 - ERROR: NetBox returned: PATCH /api/ipam/vlans/23/ Bad Request
2024-11-28 18:03:51,084 - ERROR: NetBox returned body: {'non_field_errors': ['The fields group, vid must make a unique set.']}
2024-11-28 18:03:51,084 - ERROR: Request Failed for VLAN. Used data: {'group': 6}

[bb-Ricardo]

What are the scopes of your VLAN groups? Especially the ones in the error.

Guess the scopes I tested were all site scoped.

[KPTheProf]

The source vlan 330 has both site and group set, but the correct filtered vlan only has the group set.