Get Environment Failure

[bogdancordos]

NetBox version

v3.0.2

Python version

3.8

Steps to Reproduce

The issue occurs for all my devices (Cisco ISR4321 and WS-C3850) using NApalm Diver "ios".
I can't reproduce this on the demo instance, as it needs live devices with a primary IP.

1.Go to a device detail page
2.Click on "Status"

Expected Behavior

Both "Device Facts" and "Environment" to be populated.

Observed Behavior

Only the "Device Facts" gets populated, and no information seems to be pulled for "Environment".
Getting error:

Need to mention that "LLDP Neighbors" and "Configuration" works fine.

[jeremystretch]

Can you provide the raw data being returned from the device? You should be able to extract & replicate the underlying API request in the browser by hitting F12 to open the developer console, selecting the "Network" tab, and reloading the page. You should see an API call to /api/devices/xxx/napalm/. If you can copy the data from the response to that request we can look at where exactly the bug is occurring. Thanks!

[bogdancordos]

The API returns as fallow:
{
"pageref": "page_1",
"startedDateTime": "2021-09-13T09:07:56.105+01:00",
"request": {
"bodySize": 0,
"method": "GET",
"url": "https://masked_server_name/api/dcim/devices/3/napalm/?method=get_facts&method=get_environment",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Host",
"value": "masked_server_name"
},
{
"name": "User-Agent",
"value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:92.0) Gecko/20100101 Firefox/92.0"
},
{
"name": "Accept",
"value": "/"
},
{
"name": "Accept-Language",
"value": "en-GB,en;q=0.5"
},
{
"name": "Accept-Encoding",
"value": "gzip, deflate, br"
},
{
"name": "Referer",
"value": "https://masked_server_name/dcim/devices/3/status/"
},
{
"name": "x-csrftoken",
"value": "masked_token"
},
{
"name": "DNT",
"value": "1"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Cookie",
"value": "csrftoken=masked_token; sessionid=8a5bwq1rdueu1w4bhw52oan7ifmj7mo4"
},
{
"name": "Sec-Fetch-Dest",
"value": "empty"
},
{
"name": "Sec-Fetch-Mode",
"value": "cors"
},
{
"name": "Sec-Fetch-Site",
"value": "same-origin"
}
],
"cookies": [
{
"name": "csrftoken",
"value": "masked_token"
},
{
"name": "sessionid",
"value": "8a5bwq1rdueu1w4bhw52oan7ifmj7mo4"
}
],
"queryString": [
{
"name": "method",
"value": "get_facts"
},
{
"name": "method",
"value": "get_environment"
}
],
"headersSize": 659
},
"response": {
"status": 200,
"statusText": "OK",
"httpVersion": "HTTP/1.1",
"headers": [
{
"name": "Server",
"value": "nginx/1.18.0 (Ubuntu)"
},
{
"name": "Date",
"value": "Mon, 13 Sep 2021 08:08:05 GMT"
},
{
"name": "Content-Type",
"value": "application/json"
},
{
"name": "Content-Length",
"value": "581"
},
{
"name": "Connection",
"value": "keep-alive"
},
{
"name": "Vary",
"value": "Accept, Cookie, Origin"
},
{
"name": "Allow",
"value": "GET, HEAD, OPTIONS"
},
{
"name": "X-Content-Type-Options",
"value": "nosniff"
},
{
"name": "Referrer-Policy",
"value": "same-origin"
},
{
"name": "X-Frame-Options",
"value": "SAMEORIGIN"
}
],
"cookies": [],
"content": {
"mimeType": "application/json",
"size": 581,
"text": "{"get_facts":{"uptime":1745460,"vendor":"Cisco","os_version":"ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S4b, RELEASE SOFTWARE (fc1)","serial_number":"masked_SN","model":"ISR4321/K9","hostname":"masked_hostanme","fqdn":"masked_hostanme.masked_domain","interface_list":["GigabitEthernet0/0/0","GigabitEthernet0/0/1","GigabitEthernet0","Virtual-Access1","Virtual-Access2","Vlan1"]},"get_environment":{"error":"Method get_environment failed: argument of type 'int' is not iterable"}}"
},
"redirectURL": "",
"headersSize": 313,
"bodySize": 894
},
"cache": {},
"timings": {
"blocked": 0,
"dns": 0,
"connect": 0,
"ssl": 0,
"send": 0,
"wait": 16301,
"receive": 0
},
"time": 16301,
"_securityState": "secure",
"serverIPAddress": "masked_server_ip",
"connection": "443"
}
]
}

I have saved the whole "Status" conversation on this file: https://github.com/bogdancordos/netbox_scripts/blob/2341e4d0c5508b441cdcc9cb1e87396cdac2c827/netbox_env_error

Thank you

[jeremystretch]

Looks like there's an error on the server side when processing the data being retrieved from the device. Can you enable debugging and get a complete stack trace when making the API call?

[bogdancordos]

Not sure if this is the info requested:
I have enabled debug in configuration.py and this is the stack trace output.

tu/<	        https://masked_server/static/status.js:1:121908
Li/<	        https://masked_server/static/status.js:1:1233
Li		https://masked_server/static/status.js:1:1053
tu		https://masked_server/static/status.js:1:121739
Do/<	https://masked_server/static/status.js:2:106
Li/<	        https://masked_server/static/status.js:1:1233
Li		https://masked_server/static/status.js:1:1053
Do		https://masked_server/static/status.js:2:68
Ro		https://masked_server/static/status.js:2:5971
(Async: EventListener.handleEvent) 
<anonymous>	https://masked_server/static/status.js:2:6365
<anonymous>	https://masked_server/static/status.js:2:6407

Thank you

[fskale]

I get the same.
I'm using the nxos_ssh driver. (netbox 3.0.1)
JSON answer: (without any errors)

{
  "get_environment" : {
    "cpu" : {
         "0" : {
            "%usage" : 78
         },
         "1" : {
            "%usage" : 26
         }
      },
      "fans" : {
         "invalid" : {
            "status" : true
         }
      },
      "memory" : {
         "available_ram" : 8253792,
         "used_ram" : 1903984
      },
      "power" : {
         "invalid" : {
            "capacity" : -1,
            "output" : -1,
            "status" : true
         }
      },
      "temperature" : {
         "Carmel-0-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 39
         },
         "Carmel-1-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 33
         },
         "Carmel-2-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 31
         },
         "Control-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 22
         },
         "Fan-side-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 22
         },
         "Intake-0-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 41
         },
         "Intake-1-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 49
         },
         "Outlet-1-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 52
         },
         "Port-side-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 38
         },
         "Sunnyvale-1" : {
            "is_alert" : false,
            "is_critical" : false,
            "temperature" : 38
         }
      }
   }
}

By querying the device, i get no errors, but…
The browser console clearly shows the error:

Uncaught (in promise) TypeError: right-hand side of 'in' should be an object, got undefined
    Nn https://netbox.example.domain/static/status.js:1
    so https://netbox.example.domain/static/status.js:2
    promise callback*so https://netbox.example.domain/static/status.js:2
    EventListener.handleEvent* https://netbox.example.domain/static/status.js:2
     https://netbox.example.domain/static/status.js:2

Reponse Header:

HTTP/2 200 OK
server: nginx
date: Wed, 15 Sep 2021 08:25:19 GMT
content-type: application/json
content-length: 951
vary: Accept, Cookie, Origin
allow: GET, HEAD, OPTIONS
x-content-type-options: nosniff
referrer-policy: same-origin
x-frame-options: SAMEORIGIN
server-timing: TimerPanel_utime;dur=1230.6889999999999;desc="User CPU time", TimerPanel_stime;dur=113.86499999999998;desc="System CPU time", TimerPanel_total;dur=1344.5539999999999;desc="Total CPU time", TimerPanel_total_time;dur=9629.544973373413;desc="Elapsed time", SQLPanel_sql_time;dur=21.278858184814453;desc="SQL 15 queries", CachePanel_total_time;dur=0;desc="Cache 0 Calls"
X-Firefox-Spdy: h2

An uncompressed version of status.js would be very helpful to debug !
Where can i obtain it ?

Best
Franz

[fskale]

Update:
I updated to version 3.0.2.
After restarting all Services (nginx, netbox, netbox-rq) the error went away.
@bogdancordos
Take a look at the nginx logfiles.
Especially the error.log.

Best
Franz

[bogdancordos]

I have both NetBox and nginx up to date. I have restarted nginx, netbox, netbox-rq, but no errors are being thrown by nginx.
The "Method get_environment failed: argument of type 'int' is not iterable" is still present.
This is not critical and I can live with.

[fskale]

@bogdancordos
The problem is not netbox.
It's the NAPALM driver itself.
I found a bug yesterday (nxos_ssh) and i'm sure you ran into the same problem.
As for the NXOS(SSH) driver, the parsing is very poor…
The output will not be trimmed which leads to an unexpected outcome.
I'll make a pull request today...

[jeremystretch]

Thanks @fskale. I'm going to close this out as it seems to be related to the underlying NAPALM driver. But if anyone can point to a specific change that needs to be made to NetBox, please feel free to ask that it be reopened.