Helpful ReplyHot!Crashing of Application WAD

Author
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 0
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
2014/12/31 04:48:22 (permalink)
0

Crashing of Application WAD

 
Hello to you all,
 
I have a problem with crashing application within the Fortigate 1500D. So far I can see it is the WAD application (Cache & Wan optimization deamon). The strange thing that some traffic seems to have problems with it crashing. The next strange thing is that the feature “WAN Link Load Balancing” isn’t activated. What can be the cause of this application to be crashing all the time.
 
Firmware: V5.2.1
Type Fortigate: 1500D
 
Has somebody have a idea?
 
date=2014-12-31 time=13:24:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28447, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:20:50 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28442, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:19:52 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28441, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:19:02 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28435, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x012136fb] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
date=2014-12-31 time=13:18:08 logid=0100032546 type=event subtype=system level=warning vd="root" logdesc="Application crash" action=crash msg="Pid: 28431, application: wad, Firmware: FortiGate-1500D v5.2.1,build0618b618,140915 (GA) (Release), Signal 11 received, Backtrace: [0x011ad238] [0x0120cab8] [0x0120cfc0] [0x01216efa] [0x012252d4] [0x0121378c] [0x0126ee7a] [0x0126f0a6] [0x0126e5ed] [0x012a1a78] [0x0043d35c] [0x0043a3e3] [0x2a95c40475] [0x0043a889]"
 
 
 
#1
Dave Hall
Expert Member
  • Total Posts : 1230
  • Scores: 116
  • Reward points: 0
  • Joined: 2012/05/11 07:55:58
  • Location: Canada
  • Status: offline
Re: Crashing of Application WAD 2014/12/31 05:26:34 (permalink)
0
Someone correct me on this, but I thought that daemon was also responsible for web caching.
 
If you are not actually using web caching on the Fortigate then I suggest disabling that feature. (e.g. zeroing the logdisk storage allocated to it, and also disabling any web caching features on your web firewall traffic policy.)
 
If this was a smaller unit (.e.g. <200B) I would say the logdisk is likely corrupted and needs to be reformatted.  Caution though if you choose to do this you will loose your logs, DLP files, backup configs, etc.  But I would wait to see what the other vets on the forums say about this.
 
 
post edited by Dave Hall - 2014/12/31 05:29:20

FMG-VM64/FortiAnalyzer-VM/4.0/5.0/5.2/5.4 (FWF40C/FW92D/FGT200B/FGT200D) / FAP220B/221C
#2
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 0
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
Re: Crashing of Application WAD 2015/01/05 08:07:26 (permalink)
0
I found a BUG ID 0259651 with a memory leak in the WAD Worker Process. It's been fixed in release 5.2.2. Fortinet support engineer could not verify that it's is the same bug as reported above. So I am going to find a maintenance window to update the Fortigate 1500D unit. To see if that issue has been fixed.
post edited by Jeroen - 2015/01/05 08:17:28
#3
ISOffice
Bronze Member
  • Total Posts : 48
  • Scores: 2
  • Reward points: 0
  • Joined: 2013/11/25 03:23:57
  • Status: offline
Re: Crashing of Application WAD 2015/05/11 02:47:51 (permalink)
0
Hi all,
 
We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.
 
John P
#4
Jeroen
Bronze Member
  • Total Posts : 52
  • Scores: 0
  • Reward points: 0
  • Joined: 2014/11/27 08:36:44
  • Location: The Netherlands
  • Status: offline
Re: Crashing of Application WAD 2015/05/12 02:04:21 (permalink) ☄ Helpfulby denkerzone 2016/11/08 06:30:44
0
We have updated the 1500D unit to version 5.2.2 and this solved the issue at our side. We have planned a update to version 5.2.3 as I see your message this is maybe not a good idea as the problems seems to return at version 5.2.3. I am curious at what Fortinet will return in the ticket.
 
ISOffice
Hi all,
 
We have had an on-going issue with web caching with our 100D Cluster (v5.2.3, build 670) and I'm starting to see the same error messages as Jeroen. Due to caching not working on v5.2.2, we upgraded to v5.2.3. I have a call logged with Fortinet in relation to this issue and will post any further developments.
 
John P




#5
Upendra
Bronze Member
  • Total Posts : 47
  • Scores: 0
  • Reward points: 0
  • Joined: 2012/10/10 22:59:24
  • Status: offline
Re: Crashing of Application WAD 2016/11/07 02:26:15 (permalink)
0
Hello
I having FortiGate-100D with FortiOS 5.4.0 and since from last 1 week. I am getting so many same error logs. Please help in this regards.
Troubleshooting done
1) exe formatlogdisk
2) Disable Local Reports.
3) Disable Threat weight.
------------------------------------------------------------------------------------------------------------
#3ActioncrashDate/Time15:50:26Level Log DescriptionApplication crashedLog ID32546MessagePid: 00548, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x7ffa36a73c42]Sub TypesystemTimestamp11/7/2016, 3:50:26 PMVirtual Domainroot----------------------------
Regards
Upendra Makwana
#6
denkerzone
New Member
  • Total Posts : 1
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/11/08 06:23:08
  • Status: offline
Re: Crashing of Application WAD 2016/11/08 06:44:19 (permalink)
0
Hello
I am having a device FGT100D. The firmware is 5.4.0. In this firmware I faced some Application Crashed issues. The firmware is upgraded by me to 5.4.1, now. Still I am facing the same issue.
Few crash logs during 5.4.0:
date=2016-11-07 time=17:17:30 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00232, application: wad, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 7 received, Backtrace: [0x7f04e7251be3]"

date=2016-11-07 time=17:11:13 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 04195, application: fort4195 �&#, Firmware: FortiGate-100D v5.4.0,build1011b1011,151221 (GA) (Release), Signal 11 received, Backtrace: [0x0167491e] [0x01667988] [0x016659ea] [0x01625d6a] [0x01626734] [0x016677e1] [0x01665996] [0x0166b74e] [0x0166bbc2] [0x0076b1a6] [0x0076544b] [0x012139fc] [0x01213b62] [0x01213b89] [0x01213b89] [0x01213b89] [0x01213bf9] [0x00760c55] [0x0043d5e0] [0x004436e7] [0x004416b0] [0x00443318] [0x0043ab87] [0x7f1918cfd475] [0x0043ac61]"
 
Few crash logs during 5.4.1:
 
date=2016-11-08 time=18:31:28 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00220, application: ipsengine 03.279, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7f6b89bd48a1] [0x7f6b89bd634d] [0x7f6b899ac71c] [0x7f6b899ac759] [0x7f6b899b8983] [0x7f6b899b981a] [0x7f6b8995eefd] [0x7f6b8993c809] [0x00cfb8b4] [0x00cfbddc] [0x00cfc0e4] [0x00cfc177] [0x00cfe15a] [0x00427550] [0x00cfe94a] [0x00cff040] [0x00427550] [0x0042dd77] [0x0042b4d0] [0x0042d138] [0x0042d8d3] [0x7f6b8dab9475] [0x00424c49]"
 
date=2016-11-08 time=18:04:47 logid=0100032546 type=event subtype=system level=warning vd=root logdesc="Application crashed" action=crash msg="Pid: 00283, application: wad, Firmware: FortiGate-100D v5.4.1,build1064b1064,160608 (GA) (Release), Signal 11 received, Backtrace: [0x7fdd2da47e96] [0x0138a091] [0x014973e6] [0x014800e8] [0x0142e36a] [0x0142e596] [0x0142dacd] [0x0146c61e] [0x0042737c] [0x0042d51f] [0x7fdd2d91e475] [0x00424c49]"
Thank you.
Regards,
Harshil Gupta
 
 
post edited by denkerzone - 2016/11/08 06:46:16
#7
ronnyg
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/08/09 07:24:32
  • Status: offline
Re: Crashing of Application WAD 2017/08/09 08:04:39 (permalink)
0
With 5.6 and 5.6.1, same problem on a 101E :(
I was wondering some traffic was dropped,
disabling antivir in the rule, let it work. My first idea was the IDP is hanging, but it is the antivir rule.
Bad bug again ;(. It was there in FortiOS from 5.2 and comes back again and again ;-(
 
Event
Message Pid: 03367, application: wad, Firmware: FortiGate-101E v5.6.1,build1484b1484,170727 (GA) (Release), Signal 11 received, Backtrace: [0x0091bc9e] [0x0091d2a8] [0x0091eb50] [0x00928414] [0x00928a00] [0x009291f6] [0x0092a21e] [0x0092f226] [0x00930ba8] [0x00996146] [0x009963b0] [0x00975a14] [0x009a56a6] [0x00024044] [0x00027678] [0x36cbff86]
#8
a.ymeri
New Member
  • Total Posts : 3
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/03/07 08:20:22
  • Location: Tirana, Albania
  • Status: offline
Re: Crashing of Application WAD 2018/01/12 07:54:14 (permalink)
0
similar bug for sslvpn daemon application, FortiGate 51E, FortiOS 5.6.2
 application: sslvpnd,  Signal 11 received, Backtrace: [0x005502c8] [0x00551674] [0x005576c4] [0x007befbe] [0x007c6cf6] [0x007efc0c] [0x007efc2e] [0x007f1b34] [0x007f5a96] [0x007fab60] [0x0080f22e] [0x0080fed8] [0x008105de] [0x00810aa2] [0x000283ac] [0x0002c010] [0x0002a594] [0x0002b74a] [0x0002bde6] [0x56d46f86]
 
post edited by a.ymeri - 2018/01/12 07:55:36
#9
dieter
New Member
  • Total Posts : 15
  • Scores: 0
  • Reward points: 0
  • Joined: 2018/01/04 06:04:13
  • Status: offline
Re: Crashing of Application WAD 2018/01/16 03:10:47 (permalink)
0
We're seeing similar wad crashes on a 500D cluster (happens only on the master).
 
Pid: 01795, application: wad, Firmware: FortiGate-500D v5.6.2,build1486b1486,170816 (GA) (Release), Signal 11 received, Backtrace: [0x7f4aedd789f9] [0x0159fe08] [0x01561065] [0x01bd39fb] [0x01bc7ee1] [0x015643d5] [0x01564fb6] [0x015791e0] [0x0153ba5a] [0x0158e3db] [0x0042a38c] [0x00430866] [0x7f4aedc69475] [0x004279e9]
 
Multiple crashes daily
 
#10
dieter
New Member
  • Total Posts : 15
  • Scores: 0
  • Reward points: 0
  • Joined: 2018/01/04 06:04:13
  • Status: offline
Re: Crashing of Application WAD 2018/02/28 02:21:41 (permalink)
0
In our case the crashes seem to be related to specific ssl decrypted traffic.
TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...
#11
rickyrickuk
New Member
  • Total Posts : 3
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/11/07 07:40:11
  • Status: offline
Re: Crashing of Application WAD 2018/04/14 00:43:59 (permalink)
0
dieter
In our case the crashes seem to be related to specific ssl decrypted traffic.
TA suggested upgrading to 5.6.3, but without much explanation (bug report or other related issue)...


We are getting these multiple times a minute (FG500E) during peak times and every couple of minutes during off peak, we are on 5.6.3 already so not sure that will fix.
#12
Jump to:
© 2018 APG vNext Commercial Version 5.5