Redis Queue is populated with thousands of jobs after the custom script has been successfully completed!

[Arshnika]

Hi,

My NetBox version is 4.1.6. I have written a custom script which updates more than 6000 NetBox devices.

When I run the script, the memory consumption of the NetBox server keeps increasing till no memory is availabe and it eventually makes the NetBox unresponsive!

Besides, when the script has successfully been completed, the Redis Queue is populated with thousands of jobs! And I have to wait for about 40 minutes for the queue to be cleared or I have to manually clear the queue!

Any idea what is causing the above issues?

Thanks.

[candlerb]

The problem is likely in your script, can you show it? A single custom script should create a single job.

[Arshnika]

Hi @candlerb,

Thanks for your reply.

Please find the attachment for the code.

#!/usr/bin/env python3
from typing import Protocol
from extras.scripts import *
from extras.models import Tag
from dcim.choices import DeviceStatusChoices, SiteStatusChoices
from ipam.choices import PrefixStatusChoices, VLANStatusChoices
from ipam.models import IPAddress, Prefix, VLAN
from circuits.choices import CircuitStatusChoices
from dcim.models import Device, DeviceRole, DeviceType, Manufacturer, Site, Region, Interface, SiteGroup
from dcim.utils import ContentType
from tenancy.models import Tenant, Contact, ContactAssignment
from timezone_field import TimeZoneField, TimeZoneFormField
from django.utils.text import slugify
from django.forms import PasswordInput
from utilities import querysets
from extras.models import Tag
from circuits.models import Provider, CircuitType, Circuit, CircuitTermination
from django.db import transaction
from netbox.authentication import Group
from django_rq import job

class Test_Scrip(Script):

    class Meta:
            name = 'Update NetBox Devices'
            description = 'Update NetBox Devices'
            commit_default = False
            scheduling_enabled = False

    #-----------------------------------------------------------
    #--- Get Values from HTML Form -----------------------------
    #-----------------------------------------------------------

    def run(self, data, commit):

        from scripts.Common_Functions import meraki_api
        import json
        
        #-------------------------------------------------------------
        #-------------------------------------------------------------
        #-------------------------------------------------------------

        mer_org_devices = meraki_api('getOrganizationDevices','','','','','','')
        mer_org_networks = meraki_api('getOrganizationNetworks','','','','','','')
        
        nb_sites = Site.objects.all()
        nb_devices = Device.objects.filter(
            device_type__manufacturer__name='Cisco Meraki'
        )
        nb_mer_dev_types = DeviceType.objects.filter(manufacturer__name='Cisco Meraki')

        #------------------------------------------------------------------------
        #--- Get All NetBox Sites -----------------------------------------------
        #------------------------------------------------------------------------

        nb_all_sites = []
        nb_all_mer_net_ids = []

        for site in nb_sites:

            try:
                nb_site_mer_netid = site.custom_field_data['meraki_netid']
            except:
                nb_site_mer_netid = 'None'

            nb_all_sites.append({
                'name' : site.name,
                'id' : site.id,
                'mer_net_id' : nb_site_mer_netid,
                'mer_site_name' : site.custom_field_data['meraki_sitename'],
                'tenant' : str(site.tenant).lower(),
                'tenant_id' : site.tenant_id,
                'group_id' : site.group.id
            })
            nb_all_mer_net_ids.append(nb_site_mer_netid)

        #------------------------------------------------------------------------
        #--- Get All NetBox Meraki Devices --------------------------------------
        #------------------------------------------------------------------------

        nb_all_devices =[]
        nb_all_devices_dict =[]

        for dev in nb_devices:

            dev_manufacturer = dev.device_type.manufacturer.name
            dev_name = dev.name

            try:
                dev_mer_netid = dev.site.custom_field_data['meraki_netid']
            except:
                dev_mer_netid = 'None'

            if dev_manufacturer == 'Cisco Meraki':
                nb_all_devices.append(dev_name)
                nb_all_devices_dict.append({
                    'name': dev_name.strip(),
                    'id' : dev.id,
                    'model' : dev.device_type.model,
                    'model_id' : dev.device_type.id,
                    'manufacturer' : dev_manufacturer,
                    'role' : dev.role.name,
                    'role_id' : dev.role.id,
                    'serial' : dev.serial,
                    'site' : dev.site.name,
                    'site_id' : dev.site.id,
                    'status' : dev.status,
                    'mer_net_id' : dev_mer_netid
                })

        #------------------------------------------------------------------------
        #--- Get All Meraki Devices ---------------------------------------------
        #------------------------------------------------------------------------

        meraki_all_devices = []
        meraki_all_devices_dict = []

        for mer_dev in mer_org_devices:

            dev_serial = ''
            dev_net_id = ''
            dev_type_name = ''
            dev_type_id = ''
            dev_model = ''
            dev_role_id = ''
            dev_ip = ''
            dev_notes = ''
            dev_url = ''
            dev_tenant = ''
            dev_tenant_id = ''
            dev_site = ''
            dev_site_id = ''
            dev_id = ''
            dev_mx_admin_url = ''
            dev_mx_landing_url = ''
            dev_ms_landing_url = ''
            mer_health_url = ''
            mx_wan1_ip = ''
            mx_wan1_gw = ''

            if mer_dev['name'] != '':
                dev_name = mer_dev['name'].strip()
            else:
                dev_name = mer_dev['mac']

            dev_serial = mer_dev['serial']
            dev_type_name = mer_dev['productType']
            dev_model = mer_dev['model']
            dev_url = mer_dev['url']
            dev_notes = mer_dev['notes']  
            
            for site in nb_all_sites:
                if site['mer_net_id'] == dev_net_id:
                    dev_site = site['name']
                    dev_site_id = site['id']
                    dev_tenant = site['tenant']
                    dev_tenant_id = site['tenant_id']
                    dev_group_id = site['group_id']
                    break

            for nb_device in nb_all_devices_dict:
                if nb_device['name'] == dev_name:
                    dev_id = nb_device['id']
                    break

            for dev_type in nb_mer_dev_types:
                if dev_type.model.lower() == dev_model.lower():
                    dev_type_id = dev_type.id

            if dev_type_name in ['switch','wireless','camera']:
                dev_ip = f"{mer_dev['lanIp']}/32"
            
            if dev_type_name == 'switch':
                dev_url_splits = dev_url.split('new_list')
                dev_ms_landing_url = dev_url_splits[0]
            elif dev_type_name == 'appliance':
                dev_mx_landing_url = dev_url

            if dev_type_name == 'switch' : dev_role_id = 8
            elif dev_type_name == 'appliance' : dev_role_id = 18
            elif dev_type_name == 'camera' : dev_role_id = 19
            elif dev_type_name == 'wireless' : dev_role_id = 17
            elif dev_type_name == 'sensor': dev_role_id = 20
            elif dev_type_name == 'cellularGateway': dev_role_id = 21

            meraki_all_devices.append(dev_name)
            meraki_all_devices_dict.append(
                {
                    'name' : dev_name,
                    'serial' : dev_serial,
                    'net_id' : dev_net_id,
                    'type' : dev_type_name,
                    'model' : dev_model,
                    'ip' : dev_ip,
                    'notes' : dev_notes,
                    'url' : dev_url,
                    'mx_admin_url' : dev_mx_admin_url,
                    'ms_landing_url' : dev_ms_landing_url,
                    'mer_health_url' : mer_health_url,
                    'mx_wan1_ip' : mx_wan1_ip,
                    'mx_wan1_gw' : mx_wan1_gw,
                    'nb_site_name' : dev_site,
                    'nb_site_id' : dev_site_id,
                    'nb_device_id' : dev_id,
                    'nb_tenant' : dev_tenant,
                    'nb_tenant_id' : dev_tenant_id,
                    'nb_role_id' : dev_role_id,
                    'nb_type_id' : dev_type_id,
                    'nb_group_id' : dev_group_id
                }
            )

        #------------------------------------------------------------------------
        #------------------------------------------------------------------------
        
        with transaction.atomic():

            #------------------------------------------------------------------------
            #--- Delete NetBox Devices ----------------------------------------------
            #------------------------------------------------------------------------

            for dev in nb_all_devices_dict:

                if dev['name'] not in meraki_all_devices and dev['status'] != 'planned':

                    try:

                        device = Device.objects.get(id=dev['id'])
                        device.delete()
                        
                    except Exception as e:

                        self.log_failure(f"Failed to delete the device {dev['name']}")
                        next

            #------------------------------------------------------------------------
            #--- Update/Create Devices & their IP addresses -------------------------
            #------------------------------------------------------------------------

            new_devices = []

            for dev in meraki_all_devices_dict:

                if dev['ip'] == 'None/32':
                    dev_status = 'offline'
                else:
                    dev_status = 'active'            

                if dev['name'] not in nb_all_devices:

                    #------------------------------------------------------------------------
                    #--- Create Devices -----------------------------------------------------
                    #------------------------------------------------------------------------

                    try:
                        device = Device(
                            name = dev['name'],
                            device_type_id = dev['nb_type_id'],
                            role_id = dev['nb_role_id'],
                            tenant_id = dev['nb_tenant_id'],
                            serial = dev['serial'],
                            site_id = dev['nb_site_id'],
                            status = dev_status
                        )
                        device.full_clean()
                        device.save()
                        
                        new_devices.append(dev['name'])
                        
                    except Exception as e:

                        if "'site': ['“” value must be an integer.']" in str(e):
                            self.log_failure(f"ADD >>> NetBox site missing --- {dev['name']} --- {dev['net_id']}")
                        else:
                            self.log_failure(f"ADD >>> {dev['name']} --- {dev['model']} --- {dev['net_id']} --- {str(e)}")
                        
                        continue
                    
                #------------------------------------------------------------------------
                #--- Update Devices -----------------------------------------------------
                #------------------------------------------------------------------------

                if dev['name'] in nb_all_devices and dev['name'] not in new_devices:

                    for nb_dev in nb_devices:

                        try:
                            if dev['name'] == str(nb_dev.name):
                        
                                try:

                                    nb_dev.name = dev['name']
                                    nb_dev.type_id = dev['nb_type_id']
                                    nb_dev.role_id = dev['nb_role_id']
                                    nb_dev.tenant_id = dev['nb_tenant_id']
                                    nb_dev.serial = dev['serial']
                                    nb_dev.site_id = dev['nb_site_id']
                                    nb_dev.status = dev_status
                                    nb_dev.full_clean()
                                    nb_dev.save()

                                    self.log_success(f"UPDATE >>> {dev['name']} --- {dev['model']} --- {dev['serial']} --- {str(e)}")
                                    
                                except Exception as e:
                                    
                                    self.log_failure(f"UPDATE >>> {dev['name']} --- {dev['model']} --- {dev['net_id']} --- {str(e)}")
                                    next
                        except Exception as e:
                            self.log_failure(f"UPDATE >>> {dev['name']} --- {dev['model']} --- {dev['net_id']} --- {str(e)}")
                            next

        return()

[candlerb]

Please can you edit your post and put three backticks (```) on a line of their own, before and after the code. This will make it format properly via Markdown.

[candlerb]

I noticed a few things.

You don't need transaction.atomic(): the script is already run inside a transaction that Netbox creates, and will be rolled back if there's an uncaught exception (or the user doesn't check the "keep changes" box). I don't know if a nested transaction will break things, but it would be safer to get rid of it.

You used "next" instead of "continue" in several places. That's wrong: AFAICS it will do nothing, as it's a function value which you're not calling.

% python3
Python 3.13.0 (main, Oct  7 2024, 05:02:14) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> next
<built-in function next>
>>>

OTOH, where you've put it, I don't think it matters and it could be removed completely without affecting the flow.

There are spurious ** at start at end; I guess they were not part of the code.

You iterate over the same QuerySet multiple times. I think this is allowed:

>>> d = Device.objects.filter(name__in=("gw1","gw2"))
>>> type(d)
<class 'extras.querysets.ConfigContextModelQuerySet'>
>>> for dev in d: print(dev.name)
...
gw1
gw2
>>> for dev in d: print(dev.name)
...
gw1
gw2

However, if you're inserting or deleting objects, the QuerySet may give different results when it is re-run (I don't know if it memoizes the first run). If you want to make a concrete list in memory instead of a QuerySet, then wrap with list(), e.g.

nb_devices = list(Device.objects.filter( ... ))

Otherwise, I'm afraid you'll have to debug this like any code. You could put plenty of log calls in to see where it gets to, and the contents of key variables at those points, and/or exit early.

At a glance I can't see how it would cause additional jobs to be spawned, but I do see nested loops so it could have an NxM running time. (Building a map of dev_id to device could avoid that)

Good luck!

[kkthxbye-code]

Most likely reason is that you have an event rule created for the device model, so each time you edit a device in the script, the event rule triggers a new script execution.

[mmfreitas]

Hello, do you handle script scheduling?

I have scripts scheduled to run every three hours, but for some reason—likely related to Redis—they started multiplying and overwhelming the workers with thousands of jobs. I haven’t been able to figure out the cause. The scripts execute normally regardless of the changes made to them, and they also run fine when scheduled. However, occasionally, they begin multiplying jobs by the second.