Helpful ReplyHot!High CPU and Memory Usage

Page: 12 > Showing page 1 of 2
Author
Miata
Bronze Member
  • Total Posts : 56
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/04/08 04:28:49
  • Status: offline
2015/07/30 01:59:59 (permalink)
0

High CPU and Memory Usage

Hi guys
 
So my FG-60D running 5.2.3 has been at 100% CPU and about 90% memory recently so I thought I would run the diag sys top command as shown below. 
 
From this command I can see that the scanunitd and IPS engine it taking most of my CPU usage. I don't have vulnerability scanner but I have AV enabled on 17 different policies. I think the box is being overworked, but can I restart any processes or do you guys have any other advice?
 
Run Time: 42 days, 19 hours and 54 minutes

62U, 0N, 37S, 1I; 439T, 40F, 189KF

scanunitd 7079 R < 68.4 3.7

ipsengine 602 S < 19.2 13.1

httpsd 7717 S 2.3 4.2

httpsd 7718 S 1.9 4.2

httpsd 7737 S 1.7 4.2
 
I also ran get sys performance - Output below
 
CPU states: 75% user 25% system 0% nice 0% idle

CPU0 states: 75% user 25% system 0% nice 0% idle

Memory states: 93% used

Average network usage: 6282 kbps in 1 minute, 2754 kbps in 10 minutes, 2200 kbps in 30 minutes

Average sessions: 1995 sessions in 1 minute, 2178 sessions in 10 minutes, 1824 sessions in 30 minutes
 
If you have any form of advice in terms of how to manage this more successfully or anything to restart/kill then please let me know, would be greatly appreciated.
 
Kind regards
Miata
#1
gschmitt
Gold Member
  • Total Posts : 301
  • Scores: 14
  • Reward points: 0
  • Joined: 2015/04/21 04:25:35
  • Status: offline
Re: High CPU and Memory Usage 2015/07/30 02:40:35 (permalink)
0
Miata
 
scanunitd 7079 R < 68.4 3.7



diag sys kill 11 7079
 
It shouldn't get that high
post edited by gschmitt - 2015/07/30 03:01:00
#2
Miata
Bronze Member
  • Total Posts : 56
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/04/08 04:28:49
  • Status: offline
Re: High CPU and Memory Usage 2015/07/30 02:59:06 (permalink)
0
Hi

Thanks for the comment.
 
My mistake, this is just an example of the diag sys top command, there are many others that show it fluctuates between the given value and 90%+.
 
Miata
#3
ecsupport
New Member
  • Total Posts : 17
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/02/19 18:38:33
  • Status: offline
Re: High CPU and Memory Usage 2015/07/31 10:45:42 (permalink)
0
I've noticed the same issue on 60D, 90D and others since upgrading from 5.0.10 to 5.2.3
 
CPU spikes from IPSengine primarily and scanunitd put average cpu about double what it was before upgrading.
 
I can kill/restart ipsengine but problem comes back. I disabled SSL cert inspection in case that was doing it but no go. Still occurs even on boxes that dont use ANY IPS policies (although app control is enabled on surfing).
 
Bug in 5.2.3??
#4
Miata
Bronze Member
  • Total Posts : 56
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/04/08 04:28:49
  • Status: offline
Re: High CPU and Memory Usage 2015/08/10 01:47:04 (permalink)
0
Well I think that generally I'm over working the box, as it is only a 60D. These boxes can't really take a lot of tasks, especially if one of them is to scan every bit of traffic that comes in and out of the box, as well as a bunch of other tasks which I couldn't mention within the size of this text box!
 
I noticed the vulnerability task was enabled, and so assumed this would be scanning loads of traffic both incoming and outgoing, so I set this only to late at night to run security checks etc. I also killed/restarted the IPS engine which has also helped bring down the processing usage.

Miata
#5
vjoshi_FTNT
Gold Member
  • Total Posts : 135
  • Scores: 6
  • Reward points: 0
  • Joined: 2015/02/02 21:28:20
  • Status: offline
Re: High CPU and Memory Usage 2015/08/10 02:14:40 (permalink)
0
Hi,
 
Yes, you can see high CPU/Memory if you have many task beyond the device capability, the box will exhaust.
 
However, best thing to do is to optimize the settings.
 
Like, reducing the session-ttl ( which is 3600 seconds may not be needed in most of the networks) and when can have increased session-ttl for specific protocols and ports if needed.
 
Also, tweaking the below values (these are not default, they are recommended values):
 
config system global
set tcp-halfclose-timer 30
set tcp-halfopen-timer 30
set tcp-timewait-timer 0
set udp-idle-timer 60
end
 
Above techniques will help to optimize the performance of a device.
 
 
#6
vjoshi_FTNT
Gold Member
  • Total Posts : 135
  • Scores: 6
  • Reward points: 0
  • Joined: 2015/02/02 21:28:20
  • Status: offline
Re: High CPU and Memory Usage 2015/08/10 02:16:56 (permalink)
5 (1)
Just to add, Even for the IPS profiles, instead of using the default sensor list, fine tune it by having specific signatures like, with Server based / OS based and so on.
 
Also, instead of killing a process, I would recommend restarting the application as shown below:
 
 # diagnose test application ipsmonitor

IPS Engine Test Usage:

   97: Start all IPS engines
   98: Stop all IPS engines
   99: Restart all IPS engines and monitor
 
 
#7
mscheiber
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2014/01/13 07:01:09
  • Status: offline
Re: High CPU and Memory Usage 2015/11/17 00:09:39 (permalink)
0
We see the same symptons on our FGT60D since we  upgraded from 5.0.9 to 5.2.4 now CPU spike at 100% and of course response times are very very slow.
 
There was no change in the amount of sessions nor of the traffic which is going through the FGT60D and with 5.0.9 there was no problem cpu was idled most of the time. So 5.2.4 is doing something different causes high cpu usage.
 
Is it a bug? Or did someone find out what causes the the high cpu usage for the ipsengine/monitor since the upgrade to 5.2.4
 
#8
zeki893
Bronze Member
  • Total Posts : 36
  • Scores: 0
  • Reward points: 0
  • Joined: 2013/04/03 03:12:59
  • Status: offline
Re: High CPU and Memory Usage 2015/12/10 18:00:50 (permalink)
0
having same problem with 5.2.3 scanunitd is 100% and it won't kill the process when i try diag sys kill 11
 
Anybody know what to do if diag sys kill 11 doesn't work?
post edited by zeki893 - 2015/12/10 18:05:48
#9
bobm
Silver Member
  • Total Posts : 103
  • Scores: 0
  • Reward points: 0
  • Joined: 2013/07/15 12:32:22
  • Status: offline
Re: High CPU and Memory Usage 2016/04/25 07:32:57 (permalink)
0
I'm running into the same thing too.  I have a 90D, and after upgrading from 5.0.13 to 5.2.7 the CPU is running much higher, and about once a day (usually after hours luckily) the box goes into conserve mode during an IPS scan.  I even disabled IPS in the "Features" tab in the GUI. Tried kill 11 also which didn't help.  Has anyone figured this out?
 
BTW - just to throw it out there, really not happy that my WAN load share/backup config got blown out thanks to the "New and improved" Virtual WAN IP.  Now I have to come in over the weekend and tear down every single policy in the box just to rebuild them after I configure the new failover.
#10
frajico
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/05/31 02:04:48
  • Status: offline
Re: High CPU and Memory Usage 2016/05/31 09:51:50 (permalink)
0
bobm
I'm running into the same thing too.  I have a 90D, and after upgrading from 5.0.13 to 5.2.7 the CPU is running much higher, and about once a day (usually after hours luckily) the box goes into conserve mode during an IPS scan.  I even disabled IPS in the "Features" tab in the GUI. Tried kill 11 also which didn't help.  Has anyone figured this out?
 
BTW - just to throw it out there, really not happy that my WAN load share/backup config got blown out thanks to the "New and improved" Virtual WAN IP.  Now I have to come in over the weekend and tear down every single policy in the box just to rebuild them after I configure the new failover.




Same problem here with Fortigates 310B with 5.2.7 firmware .... high cpu with spikes without reason
#11
Rookie_tr
New Member
  • Total Posts : 3
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/05/09 02:06:16
  • Status: offline
Re: High CPU and Memory Usage 2016/08/09 03:42:40 (permalink)
0
ı have a some problem for fortigate 80c 5.2.8 firmware . always cpu shows %100 on the other hand memory shows %37 . ı dont know what ı have to do about this problem ?
 
#12
jakofall
New Member
  • Total Posts : 4
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/01/15 10:59:18
  • Status: offline
Re: High CPU and Memory Usage 2016/08/09 07:49:36 (permalink)
0
What services are you running currently with the CPU at 100%?
#13
sklenda
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/07/14 06:37:10
  • Status: offline
Re: High CPU and Memory Usage 2016/08/09 08:48:47 (permalink)
0
Hello everyone,
I have a Fortigate VM00 and I experience problem with high memory, a few minutes after restart the memory goes up to around 70% and it gets over 80% a few times a day, so I have to kill processes to lower it under 70% in order to to be able to do any configuration change.
From "diagnose sys top" I learnt that pyfcgid and httpsd processes consume together around 25% of memory. I found in some older forum postings that pyfcgid is helper process for the Fortigate GUI but no help how to avoid this problem.
I have the problem with FortiOS 5.2.8 and FortiOS 5.4.1 too.
Would anyone have a clue what to do with it?
Thank you
 
#14
Rookie_tr
New Member
  • Total Posts : 3
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/05/09 02:06:16
  • Status: offline
Re: High CPU and Memory Usage 2016/08/09 23:42:02 (permalink)
0
jakofall
What services are you running currently with the CPU at 100%?



ı am not using any service. Also ı closed ıps , web filter and antivirus in policy.  I try to use default settigins fortigate but still cpu shows %100. Also Environment is really hot now ı dont have air conditioner. İs that relayed about enviroment ?  
 
#15
frajico
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/05/31 02:04:48
  • Status: offline
Re: High CPU and Memory Usage 2016/08/12 01:26:30 (permalink) ☄ Helpfulby shahzad.butt 2016/10/02 23:11:07
0
Finally, we realized that some interfaces of Fortigate unit that were configured as trunk interfaces (multiple vlans), were receiving more traffic than they have to (have to receive only 1 vlan traffic, and was receiving 10 vlan traffic), so interface got oversubscribed and CPU of Fortigate raised almos al 100%. Allowing only the 1 vlan on the switch, solved the issue.

Check for overloaded / oversubscribed interfaces traffic.
post edited by frajico - 2016/08/18 04:50:43
#16
hussnainalijaved
New Member
  • Total Posts : 4
  • Scores: 0
  • Reward points: 0
  • Joined: 2015/05/19 23:57:52
  • Status: offline
Re: High CPU and Memory Usage 2016/10/07 00:59:33 (permalink)
0
Hello All,
 
I am having Fortigate 90D firewall, i recently updated it to Firmware Version v5.4.1,build1064. After update it started utilizing CPU constantly to 100%.
 
I see the issue is with reportd process which is utilizing 92% of CPU. I killed it multiple times but it comes back again and again.
 
is there any solution to permanently disable it ?
 
Regard
Ali Javed
 
#17
MikePruett
Platinum Member
  • Total Posts : 705
  • Scores: 17
  • Reward points: 0
  • Joined: 2014/01/08 19:39:40
  • Location: Montgomery, Al
  • Status: offline
Re: High CPU and Memory Usage 2016/10/07 14:09:43 (permalink)
0
hussnainalijaved
Hello All,
 
I am having Fortigate 90D firewall, i recently updated it to Firmware Version v5.4.1,build1064. After update it started utilizing CPU constantly to 100%.
 
I see the issue is with reportd process which is utilizing 92% of CPU. I killed it multiple times but it comes back again and again.
 
is there any solution to permanently disable it ?
 
Regard
Ali Javed
 


Did you follow the supported upgrade path (stepping the OS through the required versions to get to 5.4.1?)
#18
BarryM
New Member
  • Total Posts : 7
  • Scores: 2
  • Reward points: 0
  • Joined: 2016/11/14 13:28:31
  • Status: offline
Re: High CPU and Memory Usage 2016/11/15 08:06:21 (permalink)
0
I am having the same problem since the upgrade to 5.2.8 and 5.2.9 on my 300C.
My 300C does not have a big load on it and it should be able to handle up to 1500 devices with the services running.
My memory is only at 50% but my user cpu is 94%. Formerly the cpu load would be a typical 50%-60% during normal business hours.
I even stopped the IPS engines but that made no difference.
Here is my top output.
94U, 0N, 5S, 1I; 2016T, 1083F, 142KF
           smbcd       81      S       0.8     0.0
           fssod      107      S       0.4     0.4
           smbcd     7965      R       0.4     0.0
     proxyworker       86      S       0.2     1.3
           sqldb       79      S       0.0     2.9
       scanunitd    10391      S <     0.0     2.0
         src-vis       96      S       0.0     1.9
       scanunitd    26085      S <     0.0     1.9
       scanunitd       85      S <     0.0     1.9
          httpsd     1592      S       0.0     1.3
          httpsd      208      S       0.0     1.3
         pyfcgid     8946      S       0.0     1.1
         pyfcgid     8962      S       0.0     1.1
         pyfcgid     8966      S       0.0     1.1
         pyfcgid     8967      S       0.0     1.1
         cmdbsvr       43      S       0.0     1.0
         reportd       80      S       0.0     0.9
         miglogd       62      S       0.0     0.9
          httpsd       64      S       0.0     0.7
          httpsd      207      S       0.0     0.7
as you can see services running should not be running up the CPU.
I do have ports that are vlan trunks but that never caused issues before.
The configuration is not that complicated. I should be able to use the UTM services I choose in my environment. The Fortigate 300C was sized for my network infrastructure and included expected growth and increased internet speed.
 
edit.
after watching this for a while, I chose to restart the IPSmonitor engines. The system cpu went up to 18% and the user cpu went down to 81% while the IPS was reloading. It settled down after that. My system cpu is holding around 5% with the current settings. I have not noticed any performance degradation on the network nor have I had any complaints. I suspect this CPU problem is a bug in the 5.2.x firmware. It would be nice if they fixed it.
 
 
post edited by BarryM - 2016/11/15 10:43:49
#19
abgilson
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/11/17 01:45:19
  • Status: offline
Re: High CPU and Memory Usage 2016/11/17 17:20:59 (permalink)
0
I also have been experiencing very high CPU utilization on both 80CM & 140D,running 5.2.7 & 5.2.9. While you can change the settings above, which I also performed which did not seem to have any effect. I found that turning off inspect all ports under Policy and Objects SSL/SSH Inspection resolved my CPU issues for Both Models and both Level of Firmware.. Obviously this is not a fix, but at least my users can now get decent responses from the internet. Our High CPU made the internet very slow.. BTW you need to wait 3-4 minutes before you see CPU level coming down.
Hopefully Fortieth Engineering/Development can fix this issue to return all Fortinet back to there normally processing levels. 
 
 
#20
Page: 12 > Showing page 1 of 2
Jump to:
© 2021 APG vNext Commercial Version 5.5