Troubleshooting high memory usage

[Rick_H]

I' m running 4.3.10 on a FG100D. My CPU usage hovers around 5%, but the memory usage on my FG100D is usually upwards of 70%. I do have several VDOMs configured and do use UTM (webfilter and antivirus) within two of those VDOMs. I have 75ish users going through this one device. I' m attempting to troubleshoot to determine exactly where the high memory usage is coming from, but am having trouble. I figured top (get system performance top from the global VDOM) would be a good place to start, but I don' t really understand the output. Here is a sample:

 0U, 1S, 99I; 2019T, 393F, 411KF
  initXXXXXXXXXXX        1      S       0.0     0.5
          cmdbsvr       45      S       0.0     1.4
   zebos_launcher       49      S       0.0     0.6
              nsm       55      S       0.0     0.1
             ripd       56      S       0.0     0.1
           ripngd       57      S       0.0     0.1
            ospfd       58      S       0.0     0.1
           ospf6d       59      S       0.0     0.1
             bgpd       60      S       0.0     0.1
            isisd       61      S       0.0     0.1
          uploadd       62      S       0.0     0.5
          miglogd       63      S       0.0     0.8
          miglogd       64      S       0.0     0.5
           httpsd       65      S       0.0     0.9
           proxyd       66      S       0.0     0.5
        wad_diskd       67      S       0.0     0.6
             pimd       69      S       0.0     0.1
             pdmd       70      S       0.0     0.1
    proxyacceptor       71      S       0.0     0.0
      proxyworker       72      S       0.0     2.7
 

This is after hitting " M" to sort by memory. Can someone help me interpret this? Any other pointers on how to troubleshoot this particular issue? Cheers, Rick

[vanc]

Better upgrade to 4.3.12. There is a bug in 4.3.10 and 4.3.11 that " diag sys top" won' t give the right CPU usage, so the list is not properly sorted.

[Rick_H]

Did some research and ended up finding a writeup on top. Here it is for posterity:

About the diagnose sys top command You can use the diagnose sys top command from the FortiOS CLI to list the processes running on your FortiGate unit. The command also displays information about each process. Example output: CLI# diagnose sys top Run Time: 13 days, 13 hours and 58 minutes 0U, 0S, 98I; 123T, 25F, 32KF newcli 903 R 0.5 5.5 sshd 901 S 0.5 4.0 Where the codes displayed on the second output line mean the following: * U is % of user space applications using CPU. In the example, 0U means 0% of the user space applications are using CPU. * S is % of system processes (or kernel processes) using CPU. In the example, 0S means 0% of the system processes are using the CPU. * I is % of idle CPU. In the example, 98I means the CPU is 98% idle. * T is the total FortiOS system memory in Mb. In the example, 123T means there are 123 Mb of system memory. * F is free memory in Mb. In the example, 25F means there is 25 Mb of free memory. * KF is the total shared memory pages used. In the example, 32KF means the system is using 32 shared memory pages. Each additional line of the command output displays information for each of the processes running on the FortiGate unit. For example, the third line of the output is: newcli 903 R 0.5 5.5 Where: * newcli is the process name. Other process names can include ipsengine, sshd, cmdbsrv, httpsd, scanunitd, and miglogd. * 903 is the process ID. The process ID can be any number. * R is the state that the process is running in. The process state can be: o R running. o S sleep. o Z zombie. o D disk sleep. * 0.5 is the amount of CPU that the process is using. CPU usage can range from 0.0 for a process that is sleeping to higher values for a process that is taking a lot of CPU time. * 5.5 is the amount of memory that the process is using. Memory usage can range from 0.1 to 5.5 and higher. Interactive diagnose sys top commands You can enter the following single-key commands when diagnose sys top is running. * Press q to quit. * Press c to sort the processes by the amount of CPU that the processes are using. * Press m to sort the processes by the amount of memory that the processes are using.