Strange SNMP errors

[fizzkakz]

Hi all,

I recently turned on SNMP on my SPLAT gateways to be monitored by a Solarwinds Orion server. I have four gateways, all running identical configurations. All of a sudden my Solarwinds box has started complaining of the following errors:

14/11/2007 14:34 xxxx - FIREWALL - xxxx -Memory Buffers- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 xxxx - FIREWALL - xxxx -Swap Space- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 xxxx - FIREWALL - xxxx -Real Memory- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 xxxx - FIREWALL - xxxx -/var- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 xxxx - FIREWALL - xxxx -/sysimg- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 xxxx - FIREWALL - xxxx -/opt- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -Memory Buffers
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -Swap Space
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -Real Memory
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -/var
14/11/2007 14:34 xxxx - FIREWALL - xxxx -/boot- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -/sysimg
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -/opt
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx -/boot
14/11/2007 14:34 xxxx - FIREWALL - xxxx -/- Volume no longer exists. Data collection suspended.
14/11/2007 14:34 Added xxxx - FIREWALL - xxxx-/


Has anyone seen similar or know of a work around?

Cheers!

[fizzkakz]

I re-snmpwalked the device and it has added a number of the volumes that disappeared but with different IDs.

i.e. /boot was previously volume ID 48 and is now 64. On a Cisco device I would use "snmp-server ifindex persist" to make sure the index values were prersistant. Is there a simlar command for SPLAT? I have not been able to find one in the documentation.

[fizzkakz]

If anyone was interested, I managed to resolve this problem by switching from SNMPv3 to SNMPv2c. I am still unsure if this was a SPLAT or an Orion bug.

[lodown]

This is actually a bug in Orion. New versions should handle SNMP v3 better.

[fizzkakz]

Originally Posted by lodown

This is actually a bug in Orion. New versions should handle SNMP v3 better.

Is there a documented Orion bug for this? I am now experiencing some strange behaviour where I am seeing massive CPU spikes in my Orion data but looking at SmartView Monitor and running top on SPLAT they aren't actually happening. I suspect Orion.

[lodown]

I am not sure if it's a "documented" bug, but I worked with Solarwinds directly to identify the problem was theirs and not SPLAT. The problem with disappearing and reappearing resources was only experienced with SNMP v3. They sent us a patch that appeared to resolve the problem.

As for spikes, you will see a spike whenever you push firewall policy. It becomes a problem, as it skews the data graphs. With Cacti there is a perl script that can remove the policy push spikes, but I am not sure if there is a way to do this with Solarwinds.

lodown

[fizzkakz]

I don't think it is the result of pushing a policy. The spikes occur in the graphing data and slowly go back down over the course of a week or so. But if I jump onto the SPLAT box and use top or look at CPU util in SmartView Monitor they show the util being very, very low.

I have tried graphing CPU utilisation with PRTG to help isolate the problem. PRTG is just reporting 100% CPU utilisation constantly so I am now very confused. Three tools are giving me three different results.