High CPU and Memory Usage

[Miata]

Hi guys

So my FG-60D running 5.2.3 has been at 100% CPU and about 90% memory recently so I thought I would run the diag sys top command as shown below. 

From this command I can see that the scanunitd and IPS engine it taking most of my CPU usage. I don't have vulnerability scanner but I have AV enabled on 17 different policies. I think the box is being overworked, but can I restart any processes or do you guys have any other advice?

Run Time: 42 days, 19 hours and 54 minutes 62U, 0N, 37S, 1I; 439T, 40F, 189KF scanunitd 7079 R < 68.4 3.7 ipsengine 602 S < 19.2 13.1 httpsd 7717 S 2.3 4.2 httpsd 7718 S 1.9 4.2 httpsd 7737 S 1.7 4.2

I also ran get sys performance - Output below

CPU states: 75% user 25% system 0% nice 0% idle CPU0 states: 75% user 25% system 0% nice 0% idle Memory states: 93% used Average network usage: 6282 kbps in 1 minute, 2754 kbps in 10 minutes, 2200 kbps in 30 minutes Average sessions: 1995 sessions in 1 minute, 2178 sessions in 10 minutes, 1824 sessions in 30 minutes

If you have any form of advice in terms of how to manage this more successfully or anything to restart/kill then please let me know, would be greatly appreciated.

Kind regards

Miata

[gschmitt]

Miata wrote:

scanunitd 7079 R < 68.4 3.7

diag sys kill 11 7079

It shouldn't get that high

[Miata]

Hi

Thanks for the comment.

My mistake, this is just an example of the diag sys top command, there are many others that show it fluctuates between the given value and 90%+.

Miata

[ecsupport]

I've noticed the same issue on 60D, 90D and others since upgrading from 5.0.10 to 5.2.3

CPU spikes from IPSengine primarily and scanunitd put average cpu about double what it was before upgrading.

I can kill/restart ipsengine but problem comes back. I disabled SSL cert inspection in case that was doing it but no go. Still occurs even on boxes that dont use ANY IPS policies (although app control is enabled on surfing).

Bug in 5.2.3??

[Miata]

Well I think that generally I'm over working the box, as it is only a 60D. These boxes can't really take a lot of tasks, especially if one of them is to scan every bit of traffic that comes in and out of the box, as well as a bunch of other tasks which I couldn't mention within the size of this text box!

I noticed the vulnerability task was enabled, and so assumed this would be scanning loads of traffic both incoming and outgoing, so I set this only to late at night to run security checks etc. I also killed/restarted the IPS engine which has also helped bring down the processing usage.

Miata

[vjoshi_FTNT]

Hi,

Yes, you can see high CPU/Memory if you have many task beyond the device capability, the box will exhaust.

However, best thing to do is to optimize the settings.

Like, reducing the session-ttl ( which is 3600 seconds may not be needed in most of the networks) and when can have increased session-ttl for specific protocols and ports if needed.

Also, tweaking the below values (these are not default, they are recommended values):

config system global set tcp-halfclose-timer 30 set tcp-halfopen-timer 30 set tcp-timewait-timer 0 set udp-idle-timer 60 end

Above techniques will help to optimize the performance of a device.

[vjoshi_FTNT]

Just to add, Even for the IPS profiles, instead of using the default sensor list, fine tune it by having specific signatures like, with Server based / OS based and so on.

Also, instead of killing a process, I would recommend restarting the application as shown below:

 # diagnose test application ipsmonitor IPS Engine Test Usage:    97: Start all IPS engines    98: Stop all IPS engines    99: Restart all IPS engines and monitor

[mscheiber]

We see the same symptons on our FGT60D since we  upgraded from 5.0.9 to 5.2.4 now CPU spike at 100% and of course response times are very very slow.

There was no change in the amount of sessions nor of the traffic which is going through the FGT60D and with 5.0.9 there was no problem cpu was idled most of the time. So 5.2.4 is doing something different causes high cpu usage.

Is it a bug? Or did someone find out what causes the the high cpu usage for the ipsengine/monitor since the upgrade to 5.2.4

[zeki893]

having same problem with 5.2.3 scanunitd is 100% and it won't kill the process when i try diag sys kill 11

Anybody know what to do if diag sys kill 11 doesn't work?

[bobm]

I'm running into the same thing too.  I have a 90D, and after upgrading from 5.0.13 to 5.2.7 the CPU is running much higher, and about once a day (usually after hours luckily) the box goes into conserve mode during an IPS scan.  I even disabled IPS in the "Features" tab in the GUI. Tried kill 11 also which didn't help.  Has anyone figured this out?

BTW - just to throw it out there, really not happy that my WAN load share/backup config got blown out thanks to the "New and improved" Virtual WAN IP.  Now I have to come in over the weekend and tear down every single policy in the box just to rebuild them after I configure the new failover.