5.6 on FGT-60E- SMNPD Crashes

[andrewbailey]

Hi folks,

I have a new Fortigate 60E running. It's using configuration from an FGT-80D translated via FortiConveter (not sure if this is or isn't relevant yet). Both devices are running 5.6 GA Build 1449.

I'm seeing system events like this:-

 [size="2"]Pid: 04450, application: snmpd, Firmware: FortiGate-60E v5.6.0,build1449b1449,170330 (GA) (Release), Signal 11 received, Backtrace: [0x00ce37ba] [0x56dcabf0][/size]

These were not being seen on the FGT-80D.The SNMPD (which this crash seems to refer to) appears to be disabled:-

routerXXXX # config system snmp sysinfo

 

routerXXXX (sysinfo) # get

status : disable

engine-id :

description :

contact-info :

location :

trap-high-cpu-threshold: 80

trap-low-memory-threshold: 80

trap-log-full-threshold: 90

Does anyone have any thoughts or ideas? Perhaps this one might be worth a ticket? Kind Regards, Andy.

[andreotta]

Hey Andy,

Could you paste a  show system interfaces ?

Regards

Andre

[andrewbailey]

Andre,

Yes, I'll include that below.

I must admit- that was one of my first checks too. I noticed that in the original config there was some repeated SNMP indexes (ie some interfaces using the same SNMP index).

I "unset snmp-index" on all interfaces and that seems to have cleared that issue, but the errors are still occuring. As they stand the snmp-indexes aren't very "logical" but I guess that shouldn't be too important as long as they are unique.

Here's the config for the interfaces anyway (with a few XXX's to replace any sensitive bits):-

router (interface) # show config system interface     edit "wan1"         set vdom "root"         set ip 0.0.0.0 255.255.255.255         set vlanforward enable         set ident-accept enable         set type physical         set scan-botnet-connections block         set description "Connection to XXX ISP"         set alias "WAN"         set estimated-upstream-bandwidth 19000         set estimated-downstream-bandwidth 74000         set role wan         set snmp-index 4     next     edit "wan2"         set vdom "root"         set mode dhcp         set allowaccess ping         set type physical         set role wan         set snmp-index 11     next     edit "dmz"         set vdom "root"         set ip 10.10.XXX.1 255.255.255.0         set type physical         set role dmz         set snmp-index 3     next     edit "internal1"         set vdom "root"         set ip 192.168.XX.1 255.255.255.0         set allowaccess ping https ssh         set vlanforward enable         set type physical         set scan-botnet-connections block         set alias "LAN1"         set device-identification enable         set lldp-transmission enable         set fortiheartbeat enable         set role lan         set snmp-index 1         config ipv6             set ip6-allowaccess ping https ssh             set ip6-address 2001:XXX:ba02:ed3f::100:1/120             set ip6-send-adv enable             set ip6-manage-flag enable             set ip6-other-flag enable             config ip6-prefix-list                 edit 2001:XXX:ba02:ed3f::100:0/120                     set autonomous-flag enable                     set onlink-flag enable                     set valid-life-time 7200                     set preferred-life-time 7200                 next             end         end     next     edit "internal2"         set vdom "root"         set ip 192.168.XX.1 255.255.255.0         set vlanforward enable         set type physical         set scan-botnet-connections block         set alias "LAN2"         set device-identification enable         set lldp-transmission enable         set fortiheartbeat enable         set role lan         set snmp-index 2         config ipv6             set ip6-address 2001:XXX:ba02:ed3f::200:1/120             set ip6-send-adv enable             set ip6-manage-flag enable             set ip6-other-flag enable             config ip6-prefix-list                 edit 2001:XXX:ba02:ed3f::200:1/128                     set autonomous-flag enable                     set onlink-flag enable                     set valid-life-time 3600                     set preferred-life-time 3600                 next             end         end     next     edit "internal3"         set vdom "root"         set type physical         set snmp-index 12     next     edit "internal4"         set vdom "root"         set type physical         set snmp-index 10     next     edit "modem"         set vdom "root"         set mode pppoe         set vlanforward enable         set type physical         set snmp-index 5         set lcp-echo-interval 1     next     edit "ssl.root"         set vdom "root"         set type tunnel         set alias "SSL VPN interface"         set snmp-index 6     next     edit "internal"         set vdom "root"         set ip 192.168.XX.99 255.255.255.0         set allowaccess ping https ssh         set type hard-switch         set stp enable         set role lan         set snmp-index 8     next     edit "WANIPv6"         set vdom "root"         set mode pppoe         set distance 10         set allowaccess ping         set type tunnel         set scan-botnet-connections block         set estimated-upstream-bandwidth 19000         set estimated-downstream-bandwidth 74000         set role wan         set snmp-index 7         config ipv6             set ip6-mode dhcp             set ip6-allowaccess ping             set dhcp6-prefix-delegation enable         end         set dns-server-override disable         set interface "wan1"     next     edit "iOS VPN"         set vdom "root"         set type tunnel         set snmp-index 9         set interface "WANIPv6"     next end

I also noticed the events are quite "bursty". After I had refreshed the SNMP indexes I had 8 within an hour (nearly two hours ago) and then nothing since. Each time the error occured the PID associated with the event increased which makes me wonder if I might ultimately hit some limit and see the Fortigate crash completely? Not sure how that side of things works.

Any other thoughts?

Kind Regards,

Andy.

[andreotta]

Hey Andy,

Yeah I really intended to check the snmp indexes, because I already had a problem with the same index in both interfaces. But you're already checked that.

I don't think about crash completely. The PID is only because with the signal 11 the process restart and each time with a new PID. I think this is a kind of bug, because nothing related to indexes or snmp config. More later I'll check here in my lab using 5.6.

Regards, Andre

[andrewbailey]

Andre,

Thanks for your support.

I've just rechecked the logs and I've had another few bursts of these events this morning. They seem to be occurring in bursts of 4 events in quick succession at the moment.

There was nothing after 5pm UK time last night until 1am this morning. Then three bursts this morning. No obvious consistency in what is going on when the bursts happen either.

Let me know if you need any more info.

Kind Regards,

Andy.

[andreotta]

Hey,

I checked here and no errors in my 5.6

Can you send this output ?

show full-configuration | grep -f snmp

I'm suspicious of anything wrong in FortiConverter, I used another time and gave me a lot of headache.

Regards, André

[andrewbailey]

Andre,

I had not seen this error reoccur since 13/07.

However, I went to collect the info you suggested and have had a batch of 4 of these errors occur just after I logged in.

I went to collect the data via FortiCloud and the remote access there. I'd not logged in via FortiCloud since the 13/07 (since I've been able to access the Fortigate locally over the last few days).

So I'm wondering if this is somehow caused by remote access from FortiCloud? Would that make any sense?

I'll still try and collect the data you have asked for later today. And I've noticed the SNMP Indexes are back to the "original" state with duplicated indexes. I think this occurred after a reboot which caused an older config to be loaded off the USB drive.

Kind Regards,

Andy.

[andrewbailey]

Andre,

Yes, I've logged into FortiCloud again today and opened a remote access session from there.

As soon as I've done that I've had a burst of the original application crash issues. It's well correlated and I think that pretty much confirms what I thought yesterday.

So, does remote access from FortiCloud trigger some kind of SNMP action?

I still seem to have duplicated SNMP indexes (follow by reboot) so I guess I could fix those again and see if the problem still exists- but I'm pretty certain it will given the history.

Any other thoughts anyone?

Kind Regards,

Andy

[andrewbailey]

Andre (and anyone who is interested),

I've updated to 5.6.1 (released last week) and this issue seems to have been resolved.

When I connect via FortiCloud I'm no longer seeing the snmpd application crash events. It's possible this could be a fix on the FortiCloud end too maybe?

Anyhow, for me the issue seems to be closed off.

Kind Regards,

Andy.