Helpful ReplyHot!Crashing of Application WAD

Author
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 4
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
2014/12/31 04:48:22 (permalink)
0

Crashing of Application WAD

 
Hello to you all,
 
I have a problem with crashing application within the Fortigate 1500D. So far I can see it is the WAD application (Cache & Wan optimization deamon). The strange thing that some traffic seems to have problems with it crashing. The next strange thing is that the feature “WAN Link Load Balancing” isn’t activated. What can be the cause of this application to be crashing all the time.
 
Firmware: V5.2.1
Type Fortigate: 1500D
 
Has somebody have a idea?
 
date=2014-12-31 time=13:24:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28447, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:20:50 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28442, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:19:52 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28441, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:19:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28435, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:18:08 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28431, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x01216efa] [0x012252d4] [0x0121378c] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
 
 
 
#1
Dave Hall
Expert Member
  • Total Posts : 1702
  • Scores: 174
  • Reward points: 0
  • Joined: 2012/05/11 07:55:58
  • Location: Canada
  • Status: offline
Re: Crashing of Application WAD 2014/12/31 05:26:34 (permalink)
0
Someone correct me on this, but I thought that daemon was also responsible for web caching.
 
If you are not actually using web caching on the Fortigate then I suggest disabling that feature. (e.g. zeroing the logdisk storage allocated to it, and also disabling any web caching features on your web firewall traffic policy.)
 
If this was a smaller unit (.e.g. <200B) I would say the logdisk is likely corrupted and needs to be reformatted.  Caution though if you choose to do this you will loose your logs, DLP files, backup configs, etc.  But I would wait to see what the other vets on the forums say about this.
 
 
post edited by Dave Hall - 2014/12/31 05:29:20

NSE4/FMG-VM64/FortiAnalyzer-VM/6.0 (FWF30E/FW92D/FGT200D/FGT101E/FGT81E)/ FAP220B/221C
#2
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 4
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
Re: Crashing of Application WAD 2015/01/05 08:07:26 (permalink)
0
I found a BUG ID 0259651 with a memory leak in the WAD Worker Process. It's been fixed in release 5.2.2. Fortinet support engineer could not verify that it's is the same bug as reported above. So I am going to find a maintenance window to update the Fortigate 1500D unit. To see if that issue has been fixed.
post edited by Jeroen - 2015/01/05 08:17:28
#3
ISOffice
Silver Member
  • Total Posts : 63
  • Scores: 2
  • Reward points: 0
  • Joined: 2013/11/25 03:23:57
  • Status: offline
Re: Crashing of Application WAD 2015/05/11 02:47:51 (permalink)
0
Hi all,
 
We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.
 
John P
#4
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 4
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
Re: Crashing of Application WAD 2015/05/12 02:04:21 (permalink) ☄ Helpfulby denkerzone 2016/11/08 06:30:44
0
We have updated the 1500D unit to version 5.2.2 and this solved the issue at our side. We have planned a update to version 5.2.3 as I see your message this is maybe not a good idea as the problems seems to return at version 5.2.3. I am curious at what Fortinet will return in the ticket.
 
ISOffice
Hi all,
 
We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.
 
John P




#5
Upendra
Bronze Member
  • Total Posts : 47
  • Scores: 0
  • Reward points: 0
  • Joined: 2012/10/10 22:59:24
  • Status: offline
Re: Crashing of Application WAD 2016/11/07 02:26:15 (permalink)
0
Hello
I having FortiGate-100D with FortiOS 5.4.0 and since from last 1 week. I am getting so many same error logs. Please help in this regards.
Troubleshooting done
1) exe formatlogdisk
2) Disable Local Reports.
3) Disable Threat weight.
------------------------------------------------------------------------------------------------------------
#3ActioncrashDate/Time15:50:26Level Log DescriptionApplication crashedLog ID32546MessagePid: 00548, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7ffa36a73c42]Sub TypesystemTimestamp11/7/2016, 3:50:26 PMVirtual Domainroot----------------------------
Regards
Upendra Makwana
#6
denkerzone
New Member
  • Total Posts : 1
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/11/08 06:23:08
  • Status: offline
Re: Crashing of Application WAD 2016/11/08 06:44:19 (permalink)
0
Hello
I am having a device FGT100D. The firmware is 5.4.0. In this firmware I faced some Application Crashed issues. The firmware is upgraded by me to 5.4.1, now. Still I am facing the same issue.
Few crash logs during 5.4.0:
date=2016-11-07 time=17:17:30 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00232, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 7 received, Backtrace: [0x7f04e7251be3]"

date=2016-11-07 time=17:11:13 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 04195, application: fort4195 �&#, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x0167491e] [0x01667988] [0x016659ea] [0x01625d6a] [0x01626734] [0x016677e1] [0x01665996] [0x0166b74e] [0x0166bbc2] [0x0076b1a6] [0x0076544b] [0x012139fc] [0x01213b62] [0x01213b89] [0x01213b89] [0x01213b89] [0x01213bf9] [0x00760c55] [0x0043d5e0] [0x004436e7] [0x004416b0] [0x00443318] [0x0043ab87] [0x7f1918cfd475] [0x0043ac61]"
 
Few crash logs during 5.4.1:
 
date=2016-11-08 time=18:31:28 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00220, application: ipsengine 03.279, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7f6b89bd48a1] [0x7f6b89bd634d] [0x7f6b899ac71c] [0x7f6b899ac759] [0x7f6b899b8983] [0x7f6b899b981a] [0x7f6b8995eefd] [0x7f6b8993c809] [0x00cfb8b4] [0x00cfbddc] [0x00cfc0e4] [0x00cfc177] [0x00cfe15a] [0x00427550] [0x00cfe94a] [0x00cff040] [0x00427550] [0x0042dd77] [0x0042b4d0] [0x0042d138] [0x0042d8d3] [0x7f6b8dab9475] [0x00424c49]"
 
date=2016-11-08 time=18:04:47 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00283, application: wad, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7fdd2da47e96] [0x0138a091] [0x014973e6] [0x014800e8] [0x0142e36a] [0x0142e596] [0x0142dacd] [0x0146c61e] [0x0042737c] [0x0042d51f] [0x7fdd2d91e475] [0x00424c49]"
Thank you.
Regards,
Harshil Gupta
 
 
post edited by denkerzone - 2016/11/08 06:46:16
#7
ronnyg
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/08/09 07:24:32
  • Status: offline
Re: Crashing of Application WAD 2017/08/09 08:04:39 (permalink)
0
With 5.6 and 5.6.1, same problem on a 101E :(
I was wondering some traffic was dropped,
disabling antivir in the rule, let it work. My first idea was the IDP is hanging, but it is the antivir rule.
Bad bug again ;(. It was there in FortiOS from 5.2 and comes back again and again ;-(
 
Event
Message Pid: 03367, application: wad, Firmware: FortiGate-101E v5.6.1,build1484b1484,170727 (GA) (Release), Signal 11 received, Backtrace: [0x0091bc9e] [0x0091d2a8] [0x0091eb50] [0x00928414] [0x00928a00] [0x009291f6] [0x0092a21e] [0x0092f226] [0x00930ba8] [0x00996146] [0x009963b0] [0x00975a14] [0x009a56a6] [0x00024044] [0x00027678] [0x36cbff86]
#8
a.ymeri
New Member
  • Total Posts : 4
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/03/07 08:20:22
  • Location: Tirana, Albania
  • Status: offline
Re: Crashing of Application WAD 2018/01/12 07:54:14 (permalink)
0
similar bug for sslvpn daemon application, FortiGate 51E, FortiOS 5.6.2
 application: sslvpnd,  Signal 11 received, Backtrace: [0x005502c8] [0x00551674] [0x005576c4] [0x007befbe] [0x007c6cf6] [0x007efc0c] [0x007efc2e] [0x007f1b34] [0x007f5a96] [0x007fab60] [0x0080f22e] [0x0080fed8] [0x008105de] [0x00810aa2] [0x000283ac] [0x0002c010] [0x0002a594] [0x0002b74a] [0x0002bde6] [0x56d46f86]
 
post edited by a.ymeri - 2018/01/12 07:55:36
#9
dieter
Bronze Member
  • Total Posts : 23
  • Scores: 2
  • Reward points: 0
  • Joined: 2018/01/04 06:04:13
  • Status: offline
Re: Crashing of Application WAD 2018/01/16 03:10:47 (permalink)
0
We're seeing similar wad crashes on a 500D cluster (happens only on the master).
 
Pid: 01795, application: wad, Firmware: FortiGate-500D v5.6.2,build1486b1486,170816 (GA) (Release), Signal 11 received, Backtrace: [0x7f4aedd789f9] [0x0159fe08] [0x01561065] [0x01bd39fb] [0x01bc7ee1] [0x015643d5] [0x01564fb6] [0x015791e0] [0x0153ba5a] [0x0158e3db] [0x0042a38c] [0x00430866] [0x7f4aedc69475] [0x004279e9]
 
Multiple crashes daily
 
#10
dieter
Bronze Member
  • Total Posts : 23
  • Scores: 2
  • Reward points: 0
  • Joined: 2018/01/04 06:04:13
  • Status: offline
Re: Crashing of Application WAD 2018/02/28 02:21:41 (permalink)
0
In our case the crashes seem to be related to specific ssl decrypted traffic.
TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...
#11
rickyrickuk
New Member
  • Total Posts : 5
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/11/07 07:40:11
  • Status: offline
Re: Crashing of Application WAD 2018/04/14 00:43:59 (permalink)
0
dieter
In our case the crashes seem to be related to specific ssl decrypted traffic.
TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...


We are getting these multiple times a minute (FG500E) during peak times and every couple of minutes during off peak, we are on 5.6.3 already so not sure that will fix.
#12
zack
Bronze Member
  • Total Posts : 43
  • Scores: 2
  • Reward points: 0
  • Joined: 2010/03/04 07:53:52
  • Status: offline
Re: Crashing of Application WAD 2020/03/14 09:05:08 (permalink)
0
Anyone find a solution to this? We have a Fortigate 200D cluster that is doing the same thing. The WAD daemon keeps crashing when we have UTM policies in place. It was happening on 5.6.8 and support recommended we upgrade. We're on 5.6.12 now and it's still happening. So far support is stumped.... If anyone came across a solution, sharing it here would be much appreciated. 
 
We're not using any caching..
post edited by zack - 2020/03/14 09:06:18
#13
muhkida
Bronze Member
  • Total Posts : 23
  • Scores: 5
  • Reward points: 0
  • Status: offline
Re: Crashing of Application WAD 2020/06/03 16:51:20 (permalink)
0
zack
Anyone find a solution to this? We have a Fortigate 200D cluster that is doing the same thing. The WAD daemon keeps crashing when we have UTM policies in place. It was happening on 5.6.8 and support recommended we upgrade. We're on 5.6.12 now and it's still happening. So far support is stumped.... If anyone came across a solution, sharing it here would be much appreciated. 
 
We're not using any caching..


Same issue on 5.6.11!! Any help?!
#14
zack
Bronze Member
  • Total Posts : 43
  • Scores: 2
  • Reward points: 0
  • Joined: 2010/03/04 07:53:52
  • Status: offline
Re: Crashing of Application WAD 2020/06/04 05:29:56 (permalink)
0
I opened a ticket with Fortinet and they acknowledged it was a bug. Furthermore, since our 200Ds were end of life and not receiving firmware updates anymore - there would be no "official" fix. Since I was under a support agreement and I escalated the issue (since they could offer no solution), Fortinet ended up creating a custom firmware image off of the 6.0.9 branch for us which contained the fix. 
 
The ticket number was:
3909060
 
We had to run for an extended time with our proxy based inspection down which meant no UTM policies. I was NOT happy about that. 
 
But there's good news. Fortinet sold us some new firewalls at a discount because of all the issues. We got 3 100Fs. Bad news - the WAD service still crashes on them too running 6.2.4. I have a ticket in on that one too. Ticket # 4088435. Still no fix!
 
My next firewall will NOT be a Fortinet. I buy these firewalls to run UTM policies. If I can't run them because it causes the firewall to crash, then WTF am I paying for? I can download a basic port filtering firewall/router for free! (Untangle of PFSense).
 
I know it sucks to hear, but my only recommendation can be to buy another brand at this point. 
post edited by zack - 2020/06/04 05:34:55
#15
zack
Bronze Member
  • Total Posts : 43
  • Scores: 2
  • Reward points: 0
  • Joined: 2010/03/04 07:53:52
  • Status: offline
Re: Crashing of Application WAD 2020/06/04 05:47:51 (permalink)
0
You have to disable proxy based inspection (SSL inspection). Set it to certificate only as opposed to full SSL inspection. By the way - this will disable the effectiveness of your UTM policies since the firewall can't look inside SSL sessions to inspect the traffic. But at least they'll stop crashing. 
 
You need to engage Fortinet support and ask for a custom firmware image for your firewalls that contain the fix. 
#16
Jump to:
© 2020 APG vNext Commercial Version 5.5