Hot!Fortigate process " wad" consuming 62% of memory.

Page: 12 > Showing page 1 of 2
Author
plindgren
New Member
  • Total Posts : 9
  • Scores: 0
  • Reward points: 0
  • Joined: 2013/04/16 08:48:06
  • Status: offline
2013/07/31 05:58:56 (permalink)
0

Fortigate process " wad" consuming 62% of memory.

i get the " CFG_CMDBAPI_ERR" when i try to make changes on my fortigate. which is other than that operational.

I tired the command " diag test application ipsmonitor 99" but it did not work. So i used the command " diag sys top 1" to see what was hogging all that memory. And i found a process named " wad" that uses 62% of the memory.

However this machine is in production and i dont know what the process does and i cant seem to find it anywhere. Can i kill it? What does it do?

Is there a process reference for fortios out there somewhere?

" This solution needs to be idiot proof!"
" Why? Do you plan on hiring idiots?"
#1

22 Replies Related Threads

    pcraponi
    Gold Member
    • Total Posts : 451
    • Scores: 24
    • Reward points: 0
    • Joined: 2006/07/28 11:43:39
    • Location: Brazil
    • Status: offline
    RE: Fortigate process " wad" consuming 62% of memory. 2013/07/31 06:51:31 (permalink)
    0
    wad daemon is used for cache, explict-proxy and wan optimization.

    You use some of these resources? You are on 4.0 MR3 firmware? The next patch (patch15) will improve some wad resources... No ETA yet for release date.



    Regards,
    Paulo Raponi
    #2
    plindgren
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/04/16 08:48:06
    • Status: offline
    RE: Fortigate process " wad" consuming 62% of memory. 2013/07/31 23:50:46 (permalink)
    0
    We do use it, however it wont really matter if that function " goes down" since we have a good internetline which we never really had any problems with bandwidth(even prior to using wan opt).

    My question simply is, what will happen when i kill the process? Can it affect the whole system or is it safe to just kill it?

    Also how did you find out that wad was that? i cant seem to find any documentation of what all these processes do. Would be nice for future troubleshooting.

    edit:
    im on FortiOS 5.0.2

    Cant even turn off web cache and wan opt on my policy objects >.<
    < Message edited by plindgren -- 7/31/2013 11:55:18 PM >

    " This solution needs to be idiot proof!"
    " Why? Do you plan on hiring idiots?"
    #3
    plindgren
    New Member
    • Total Posts : 9
    • Scores: 0
    • Reward points: 0
    • Joined: 2013/04/16 08:48:06
    • Status: offline
    RE: Fortigate process " wad" consuming 62% of memory. 2013/08/02 01:36:47 (permalink)
    0
    I talked to Fortinet webchat and he told me it would be a problem to kill it

    i ran diag sys kill 11 <pid>
    where pid was process id.

    And it worked fine my fortigate is functioning normally

    " This solution needs to be idiot proof!"
    " Why? Do you plan on hiring idiots?"
    #4
    pcraponi
    Gold Member
    • Total Posts : 451
    • Scores: 24
    • Reward points: 0
    • Joined: 2006/07/28 11:43:39
    • Location: Brazil
    • Status: offline
    RE: Fortigate process " wad" consuming 62% of memory. 2013/08/05 04:54:44 (permalink)
    0
    Be carefully. When will kill the wad process, all users lost their HTTP/HTTPS access / downloads, etc... All sessions need be restarted.


    Regards,
    Paulo Raponi
    #5
    IyyappanD
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/09/26 12:43:18
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/09/26 12:53:28 (permalink)
    0
    @Paulo: Is there another than killing it ? Production was impacted already and Internet doesn't work. So seems better to having it killed and this seems to be a common issue till 5.4.10 which we are using. Not sure if the same is resolved in 5.6.X. Can someone comment on this
    #6
    sashag
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/09/27 06:24:03
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/09/27 06:31:48 (permalink)
    0
    Same with 5.6.6
    With upgrade from 5.6.3 and flow inspection mode to 5.6.6 and proxy mode, "wad" process ate 40% of memory in less than 10 hours. After reaching 90% of memory consumption fortigate entered "conserve mode" which killed all internet connections in office.
    Had to kill process and return to flow mode for further investigation.
    #7
    IyyappanD
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/09/26 12:43:18
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/09/27 07:09:51 (permalink)
    0
    We had an issue on 5.4.4 and were asked to upgrade 5.4.10 and did not solve the issue. Right now, we are doing failover and rebooting the affected box. This is an ongoing issue and i thought issue is resolved at 5.6.6.
    So is there a Solution to this now ?
    post edited by Mod_FTNT - 2018/09/27 07:32:46
    #8
    james.heyworth
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/29 06:15:50
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/10/29 05:34:03 (permalink)
    0
    Hi all
     
    We are on 5.6.6 1500D's and we experienced the same issue on both of our Firewalls (1500D's) and the WAD process was the culprit. I submitted a ticket to Fortinet to investigate, and provide us with a version that this issue is remediated in. I will keep the forum updated if possible.
    #9
    AtiT
    Platinum Member
    • Total Posts : 462
    • Scores: 38
    • Reward points: 0
    • Joined: 2012/04/18 12:13:27
    • Location: Prague / Czech Republic
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/10/29 10:01:36 (permalink)
    0
    We have the same problem on our 1500Ds FGT with 5.6.6 version of OS. It is configmed as a bug:
    Bug is fixed in FortiOS 6.0.3 GA and interim build 5.6.7
     
    We have to restart every 3-5 days the WAD process, sometimes the worker sometimes other process.
    When the 5.6.7 will be out the support could not tell as they did not have this info when I asked.
     
    We have a customer with 1000Ds in cluster and had the same issue on 5.6.4 and we updated it to 5.6.5 almost two months ago. They have did not experienced any issue so far. That is the reason we are not recommended to upgrade to 5.6.6 as it is confirmed having a bug.

    AtiT
    --------------------
    NSE 8, CCNP R+S
    #10
    james.heyworth
    New Member
    • Total Posts : 2
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/08/29 06:15:50
    • Status: offline
    Re: RE: Fortigate process " wad" consuming 62% of memory. 2018/10/29 10:49:40 (permalink)
    0
    Ok, we could more than likely revert back to 5.6.5 in order to remediate temporarily. However, I'd prefer steps forward rather than back.
     
    #11
    wolfgang.cernohorsky
    New Member
    • Total Posts : 1
    • Scores: 0
    • Reward points: 0
    • Joined: 2018/12/03 03:14:11
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2018/12/03 03:25:23 (permalink)
    0
    We had the same problem with Version 5.6.6 on our FortiGates 600D.  The good news are, since a vew days, release 5.6.7 is available and according to the release notes, "High memory usage on WAD" should be fixed.
     
    Hope this helps,
    Wolfgang
    #12
    Toshi Esumi
    Expert Member
    • Total Posts : 1566
    • Scores: 132
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2018/12/03 10:21:39 (permalink)
    0
    We're experiencing the same on one of our 1500Ds, which runs 5.6.6 for about a month. Before that it was running 5.4.8. We're going to kill wad tonight then schedule an upgrade to 5.6.7 soon.
    #13
    Toshi Esumi
    Expert Member
    • Total Posts : 1566
    • Scores: 132
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2018/12/03 21:56:50 (permalink)
    0
    Does anyone know what triggers this wad memory usage escalation, which was fixed with 5.6.7?
    When I checked it before killing them (fnsysctl killall wad) tonight, I found it came back down to normal level at about 50% two hours ago. Until then it had kept growing steadily for last two weeks based on our mem usage monitoring tool. I asked some coworkers if they did something to it but nobody seems to have done anything. It's very strange.
    #14
    sdlengua
    New Member
    • Total Posts : 5
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/03/02 12:01:25
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/01/28 09:25:15 (permalink)
    0
    Upgraded my 600C's to 5.6.7 last week. Same issue. Memory was up to 82% this morning and had to kill several WAD processes. Apparently not fixed in 5.6.7, just FYI. Calling in support ticket to see what's going on. Very frustrating. We are performing Full SSL and proxy based.
    #15
    Toshi Esumi
    Expert Member
    • Total Posts : 1566
    • Scores: 132
    • Reward points: 0
    • Joined: 2014/11/06 09:56:42
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/01/28 10:10:36 (permalink)
    0
    Thank you for the info. Please open a TT with TAC to claim it's still not fixed. We skipped 5.6.7 or even next 5.6.8 since this wad memory problem doesn't come back so far and another issue we've been waiting for a fix would not be in 5.6.8.
    #16
    AtiT
    Platinum Member
    • Total Posts : 462
    • Scores: 38
    • Reward points: 0
    • Joined: 2012/04/18 12:13:27
    • Location: Prague / Czech Republic
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/01/29 00:20:42 (permalink)
    0
    Hello,
    We upgraded our FGT-1500D A-P cluster 7 days ago to FortiOS 5.6.7, at this moment the memory is OK, but still the usage is increasing a little bit. We will see.
     
    I would recommend you to not kill the wad process but it is better to restart it.
    Always restart processes in case they have command for that.

    Check the overall CPU and memory status:
    # diagnose sys top-summary
    CPU [|||||||||||| ] 30.7%
    Mem [|||||||||||||||||||||||||||| ] 71.0% 11460M/16064M
    Processes: 20 (running=4 sleeping=239)

    PID RSS CPU% ^MEM% FDS TIME+ NAME
    * 262 6G 60.6 38.3 41675 48:06.38 wad [x14]
    247 1G 54.0 8.2 985 17:53.21 ipsmonitor [x12]
    287 1G 5.0 7.1 930 21:00.71 cw_acd
    256 645M 34.9 4.0 263 59:32.15 sslvpnd [x12]
    272 349M 2.5 2.2 20 51:14.84 urlfilter
    241 213M 4.8 1.3 61 19:52.33 miglogd [x7]
    244 147M 1.7 0.9 24 00:44.52 httpsd [x5]
    .....
     
    Use the "m" key on your keyboard to sort the process groups according to memory usage.
    Use "q" key on your keyboard to quit the diagnose command.
     
    Show the running processes:
    # diagnose sys top
    Use the "m" key on your keyboard to sort the processes according to memory usage.
    Use "q" key on your keyboard to quit the diagnose command.
     
    Locate your wad process and his process ID, let's say for now:
    wad 351 S 2.7 9.0
    The 351 is the process ID.
     
    Now reset and enable debuging:
    # diagnose debug reset
    # diagnose debug enable

    List all your wad processes and ocate your process ID (pid):
    # diagnose test application wad 1000
    Process [0]: WAD manager type=manager(0) pid=262 diagnosis=yes.
    Process [1]: type=dispatcher(1) index=0 pid=345 state=running
    diagnosis=no debug=enable valgrind=unsupported/disabled
    Process [2]: type=wanopt(2) index=0 pid=346 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [3]: type=worker(3) index=0 pid=347 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [4]: type=worker(3) index=1 pid=348 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [5]: type=worker(3) index=2 pid=349 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [6]: type=worker(3) index=3 pid=350 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [7]: type=worker(3) index=4 pid=351 state=running <=============================
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [8]: type=worker(3) index=5 pid=352 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [9]: type=worker(3) index=6 pid=353 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [10]: type=worker(3) index=7 pid=354 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [11]: type=worker(3) index=8 pid=355 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [12]: type=worker(3) index=9 pid=356 state=running
    diagnosis=no debug=enable valgrind=supported/disabled
    Process [13]: type=informer(4) index=0 pid=344 state=running
    diagnosis=no debug=enable valgrind=unsupported/disabled

    Now access your wad process - enter into the process menu:
    Remember:
    2xxx - the wad process always starts with 2.
    x3xx - the wad process type depending on whether it is a dispatcher, worker, informer etc. in brackets ().
    xx04 - the index number of the process (two digits). If the index is one digit, put 0 before the index.
     
    So it looks like this in our case for wad process with pid=351
    # diagnose test application wad 2304
    Set diagnosis process: type=worker index=4 pid=351
     
    Now you are working on your wad process and you can list the available options with issuing:
    # diagnose test application wad
    WAD process 351 test usage:
    1: display process status
    2: display total memory usage.
    99: restart all WAD processes
    1000: List all WAD processes.
    1001: dispaly debug level name and values
    1002: dispaly status of WANOpt storages
    ..... etc. There is a lot of them.
     
    What you need is to restart the process so use 99 like:
    # diagnose test application wad 99
     
    Reset and disable your debug:
    # diagnose debug disable
    # diagnose debug reset
     
    Now you can check the CPU and memory again with command:
    # diagnose sys top-summary
    Compare the results with the output at the beginning of this post.

    More work but better than just killing processes.

    AtiT
    --------------------
    NSE 8, CCNP R+S
    #17
    vdp
    New Member
    • Total Posts : 6
    • Scores: 0
    • Reward points: 0
    • Joined: 2015/10/22 06:07:16
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/05/01 23:42:48 (permalink)
    0
    Running 6.2 firmware, and experiencing the same wad high mem usage issue (conserve mode activated).
    I guess Fortinet didn't fix this issue.
    The only answer I got from Support is "buy a bigger device".
     
    #18
    JaapHoetmer
    Bronze Member
    • Total Posts : 56
    • Scores: 0
    • Reward points: 0
    • Joined: 2011/08/09 02:06:53
    • Location: Geneva, Switzerland
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/05/02 23:24:08 (permalink)
    0
    Hi there. I am experiencing the same issue on a 100E live-stby cluster, 89% memory loading, and the WAD process consuming 44% of total memory. Running v6.2.0 build0866 (GA).
     
    Two WAD processes were consuming 16.5% of memory each, and two additional WAD processes 7% and 4.7% of memory. I have restarted these processes using the instructions provided above, and that has fixed the issue for now. But it will probably return, as the same situation occurred yesterday morning.
     
     
    post edited by JaapHoetmer - 2019/05/02 23:47:08

    Kind regards,

    Jaap
    #19
    figge
    New Member
    • Total Posts : 17
    • Scores: 0
    • Reward points: 0
    • Joined: 2017/03/02 04:46:30
    • Status: offline
    Re: Fortigate process " wad" consuming 62% of memory. 2019/06/18 22:29:01 (permalink)
    0
    The release notes for 5.6.8 state that this WAD memory issue is resolved.
    Can anyone confirm this? Have you upgraded to 5.6.8 and gotten rid of the bug?
    #20
    Page: 12 > Showing page 1 of 2
    Jump to:
    © 2019 APG vNext Commercial Version 5.5