Helpful ReplyHot!Bad engine update???

Author
seadave
Gold Member
  • Total Posts : 256
  • Scores: 28
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
2017/09/28 17:22:06 (permalink)
5 (2)

Bad engine update???

At 4:23 PST today we started seeing 403 errors when trying to visit sites.  Only way to allow access is simple unfiltered NAT rule.  Searching form shows this happened in the past with a bad AV engine update.  I notice that support.fortinet.com is down with a 500 error so perhaps they self-inflicted the same.  TIME FOR AN ARCHITECTURE MODIFICATION!
 
Anyone else seeing this?
 
I'm seeing these in my debugs:
16330: 2017-09-28 17:03:53 <01449> firmware FortiGate-500D v5.4.5,build1138b1138,170531 (GA) (Release)
16331: 2017-09-28 17:03:53 <01449> application scanunit
16332: 2017-09-28 17:03:53 <01449> *** signal 11 (Segmentation fault) received ***
16333: 2017-09-28 17:03:53 <01449> AVDB 05004000AVDB00201-00052.00000-1709281424
16334: 2017-09-28 17:03:53 <01449> ETDB 05004000AVDB00701-00052.00000-1709281423
16335: 2017-09-28 17:03:53 <01449> EXDB 05004000AVDB00401-00001.00000-1210171547
16336: 2017-09-28 17:03:53 <01449> AVSO 04000000AVEN00701052471705041426
16337: 2017-09-28 17:03:53 <01449> Register dump:
16338: 2017-09-28 17:03:53 <01449> RAX: 0000000000000000 RBX: 00007fff067f82a0
16339: 2017-09-28 17:03:53 <01449> RCX: 00000000000000f8 RDX: 000000001258db56
16340: 2017-09-28 17:03:53 <01449> R8: 00000000000000ff R9: 0000000000000000
16341: 2017-09-28 17:03:53 <01449> R10: 0000000000000002 R11: 00007fb5625a0df0
16342: 2017-09-28 17:03:53 <01449> R12: 0000000000000046 R13: 00000000ffffffff
16343: 2017-09-28 17:03:53 <01449> R14: 00007fff067f82a0 R15: 00007fff067f81f0
16344: 2017-09-28 17:03:53 <01449> RSI: 0000000000000000 RDI: 0000000000000002
16345: 2017-09-28 17:03:53 <01449> RBP: 00000000ffffffff RSP: 00007fff067f80e0
16346: 2017-09-28 17:03:53 <01449> RIP: 00007fb5654af827 EFLAGS: 0000000000010212
16347: 2017-09-28 17:03:53 <01449> CS: 0033 FS: 0000 GS: 0000
16348: 2017-09-28 17:03:53 <01449> Trap: 000000000000000e Error: 0000000000000004
16349: 2017-09-28 17:03:53 <01449> OldMask: 0000000000000000
16350: 2017-09-28 17:03:53 <01449> CR2: 0000000000000014
16351: 2017-09-28 17:03:53 <01449> Backtrace:
16352: 2017-09-28 17:03:53 <01449> [0x7fb5654af827] => /data/lib/libav.so
16353: 2017-09-28 17:03:53 <01449> [0x7fb5654b7f45] => /data/lib/libav.so
16354: 2017-09-28 17:03:53 <01449> [0x7fb5654b86f3] => /data/lib/libav.so
16355: 2017-09-28 17:03:53 <01449> [0x7fb5654ab912] => /data/lib/libav.so
16356: 2017-09-28 17:03:53 <01449> [0x7fb5654b44c3] => /data/lib/libav.so
16357: 2017-09-28 17:03:53 <01449> [0x7fb5654baa39] => /data/lib/libav.so
16358: 2017-09-28 17:03:53 <01449> [0x7fb5654da895] => /data/lib/libav.so
16359: 2017-09-28 17:03:53 <01449> [0x7fb5654d87e7] => /data/lib/libav.so
16360: 2017-09-28 17:03:53 <01449> [0x7fb565494ad7] => /data/lib/libav.so (scanvirFile+0x00000187)
16361: 2017-09-28 17:03:53 <01449> [0x01a07ddf] => /bin/scanunitd
16362: 2017-09-28 17:03:53 <01449> [0x01a455ec] => /bin/scanunitd
16363: 2017-09-28 17:03:53 <01449> [0x01a466db] => /bin/scanunitd
16364: 2017-09-28 17:03:53 <01449> [0x010e54f0] => /bin/scanunitd
16365: 2017-09-28 17:03:53 <01449> [0x010e6599] => /bin/scanunitd
16366: 2017-09-28 17:03:53 <01449> [0x019b1c7c] => /bin/scanunitd
16367: 2017-09-28 17:03:53 <01449> [0x010e734d] => /bin/scanunitd
16368: 2017-09-28 17:03:53 <01449> [0x010e0616] => /bin/scanunitd
16369: 2017-09-28 17:03:53 <01449> [0x010e3fde] => /bin/scanunitd
16370: 2017-09-28 17:03:53 <01449> [0x00427c10] => /bin/scanunitd
16371: 2017-09-28 17:03:53 <01449> [0x0042e5c7] => /bin/scanunitd
16372: 2017-09-28 17:03:53 <01449> [0x0042bcf1] => /bin/scanunitd
16373: 2017-09-28 17:03:53 <01449> [0x0042d881] => /bin/scanunitd
16374: 2017-09-28 17:03:53 <01449> [0x0042deff] => /bin/scanunitd
16375: 2017-09-28 17:03:53 <01449> [0x7fb5690e4475] => /fortidev4-x86_64/lib/libc.so.6
16376: 2017-09-28 17:03:53 (__libc_start_main+0x000000f5) liboffset 00021475
16377: 2017-09-28 17:03:53 <01449> [0x00425065] => /bin/scanunitd
16378: 2017-09-28 17:03:53 [AV Engine <1449>] Last file info:
16379: 2017-09-28 17:03:53 [AV Engine <1449>] filename: bag, filesize: 7151
16380: 2017-09-28 17:03:53 [AV Engine <1449>] Native script imagebase: 0x12546000
16381: 2017-09-28 17:03:53 [AV Engine <1449>] cprl sigid: 489591, bintype: 00000400
16382: 2017-09-28 17:03:53 scanunit=worker pid=1449 exittype=signal code=11 total=7996 free=5679
16383: 2017-09-28 17:03:53 scanunit crash: signal=11, src-ip=172.21.11.126, dst-ip=104.80.89.9,
16384: 2017-09-28 17:03:53 request-uri=http://init-p01st.push.apple.com/bag
 
diag autoupdate ver output:
 
AV Engine
---------
Version: 5.00247
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu May 4 14:26:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
Virus Definitions
---------
Version: 52.00001
Contract Expiry Date: Mon Jul 16 2018
Last Updated using push update on Thu Sep 28 17:00:11 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: Updates Installed
Extended set
---------
Version: 52.00000
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Sep 28 14:23:00 2017
Last Update Attempt: n/a
Result: Updates Installed
Extreme set
---------
Version: 1.00000
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Wed Oct 17 15:47:00 2012
Last Update Attempt: n/a
Result: Updates Installed
Mobile Malware Definitions
---------
Version: 52.00000
Contract Expiry Date: Sat Jun 2 2018
Last Updated using push update on Thu Sep 28 17:00:11 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: Updates Installed
Attack Definitions
---------
Version: 6.00741
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Tue Dec 1 02:30:00 2015
Last Update Attempt: n/a
Result: Updates Installed
Attack Extended Definitions
---------
Version: 12.00234
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Sep 28 01:27:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
IPS Malicious URL Database
---------
Version: 1.00775
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Sep 28 07:29:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
Flow-based Virus Definitions
---------
Version: 52.00000
Contract Expiry Date: Mon Jul 16 2018
Last Updated using push update on Thu Sep 28 17:00:11 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: Updates Installed
Botnet Definitions
---------
Version: 4.00058
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Sep 28 10:00:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
IPS Attack Engine
---------
Version: 3.00430
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Tue Aug 22 20:13:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
Internet-service Database Apps
---------
Version: 2.00702
Contract Expiry Date: n/a
Last Updated using manual update on Wed Sep 27 11:15:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
Internet-service Database Maps
---------
Version: 2.00702
Contract Expiry Date: n/a
Last Updated using manual update on Wed Sep 27 11:15:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
Botnet Domain Database
---------
Version: 1.00505
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Aug 11 12:09:00 2016
Last Update Attempt: n/a
Result: Updates Installed
Modem List
---------
Version: 0.000
Device and OS Identification
---------
Version: 1.00061
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Fri Sep 8 17:49:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
IP Geography DB
---------
Version: 1.067
Contract Expiry Date: n/a
Last Update Date: Fri Aug 4 15:07:26 2017

Certificate Bundle
---------
Version: 1.00005
Last Update Date: Thu May 5 10:58:00 2016

FDS Address
---------
208.91.112.78-443

URL White list
---------
Version: 1.00810
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu Sep 28 08:05:00 2017
Last Update Attempt: Thu Sep 28 17:00:11 2017
Result: No Updates
#1
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/28 17:44:58 (permalink)
0
I see the same thing on a 300D w/ 5.4.5.
#2
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/28 17:57:17 (permalink)
0
I ran an "exec update-now".  They've already got a new set of virus definitions, 52.00002 instead of 52.00001.
 
Unfortunately, I'm still seeing the same sets of crashes, so it's not fixed yet.
 
#3
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/28 18:44:10 (permalink)
0
Looks like support.fortinet.com is back up.
 
Virus definitions have changed from 52.00001 to 52.00002 to 52.00003
Flow-based virus definitions have moved from 52.00001 to 52.00002.
 
Haven't seen any more crashes in the 10 minutes since I updated.  Fingers crossed.
#4
Yamada.Takahiro3@chuden.co.jp
New Member
  • Total Posts : 5
  • Scores: 2
  • Reward points: 0
  • Joined: 2015/08/18 21:10:13
  • Status: offline
Re: Bad engine update??? 2017/09/28 18:44:46 (permalink)
0
This is japan.
we ouccur same issue.
3600D 5.4.x
#5
seadave
Gold Member
  • Total Posts : 256
  • Scores: 28
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Bad engine update??? 2017/09/28 18:48:19 (permalink)
0
So I just go off the phone with the TAC.  Ticket 2382232 and it appears to be a bad AV Defs update.  The temp solution is to login to the console.
 
fw01 # config antivirus profile
fw01 (profile) # edit default
fw01 (default) # set inspection-mode flow-based
fw01 (default) # end
 
This is assuming you are using the "default" AV profile.  Change as needed.  This makes it not visible in the gui I think but it works.  Will update when I hear more.
post edited by seadave - 2017/09/29 07:33:11
#6
Cyrielr
New Member
  • Total Posts : 5
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/03/16 13:37:00
  • Status: offline
Re: Bad engine update??? 2017/09/28 18:53:31 (permalink)
0
I got the same issue and I confirm that an update of the AV database on 52.00003 solved the issue !
 
If you still have the issue you could disabled temporary the AV with : 
# diagnose antivirus bypass off
#7
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/28 19:05:17 (permalink) ☄ Helpfulby seadave 2017/09/29 07:34:01
5 (1)
@seadave,
 
Did TAC say anything about the newer virus definitions vs. the AV engine?  I thought we didn't get a new AV engine, just new virus definitions.
 
On our 300D and 100D (5.4.5) once virus definitions were updated from 52.00001 to 52.00003 and flow-based virus definitions were updated from 52.00001 to 52.00002 I stopped seeing the crashes.
#8
Yamada.Takahiro3@chuden.co.jp
New Member
  • Total Posts : 5
  • Scores: 2
  • Reward points: 0
  • Joined: 2015/08/18 21:10:13
  • Status: offline
Re: Bad engine update??? 2017/09/28 19:06:47 (permalink)
5 (1)
we seem to resoled. 52.0003
#9
ReseauSL
New Member
  • Total Posts : 4
  • Scores: 0
  • Reward points: 0
  • Joined: 2005/09/19 07:53:35
  • Status: offline
Re: Bad engine update??? 2017/09/28 19:22:45 (permalink)
0
I Had the same problem in Proxy-base Inspection Mode, ( 5.4.4 ).
I updated AV definition from 52.00000 to 52.00003 and it fixed the problem .
#10
Yamada.Takahiro3@chuden.co.jp
New Member
  • Total Posts : 5
  • Scores: 2
  • Reward points: 0
  • Joined: 2015/08/18 21:10:13
  • Status: offline
Re: Bad engine update??? 2017/09/28 21:01:15 (permalink)
0
Dose anyone have official announce or report from fortinet?
#11
seadave
Gold Member
  • Total Posts : 256
  • Scores: 28
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Bad engine update??? 2017/09/28 21:32:10 (permalink) ☄ Helpfulby tanr 2017/09/29 07:13:06
0
Not yet, I'll update when my ticket is updated.  If they do so at all.  It seems fairly obvious now that the cause was a bad AV Defs update.  I'm now on 52.00005 with no issues.  I started considering a large purchase of FortiSwitches today.  I guess this is my reward ;-)
 
You can check via the console with the "diag autoupdate ver" command:
 
AV Engine
---------
Version: 5.00247
Contract Expiry Date: Mon Jul 16 2018
Last Updated using manual update on Thu May 4 14:26:00 2017
Last Update Attempt: Thu Sep 28 20:36:17 2017
Result: No Updates
Virus Definitions
---------
Version: 52.00005
Contract Expiry Date: Mon Jul 16 2018
Last Updated using scheduled update on Thu Sep 28 20:36:17 2017
Last Update Attempt: Thu Sep 28 20:36:17 2017
Result: Updates Installed
 
If you have a FAZ 5.6, we realized today you can configure an event handler to alert you when the "app crash" event fires.
 
Should give you a small heads up when this is happening instead of the line of people knocking on your door.

Attached Image(s)

#12
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/29 07:13:00 (permalink)
0
Thanks for finding the FAZ application crash event handler! 
I see the same one with FAZ 5.4.5 and will add an alert for myself.
#13
seadave
Gold Member
  • Total Posts : 256
  • Scores: 28
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Bad engine update??? 2017/09/29 07:39:12 (permalink)
0
tanr
Thanks for finding the FAZ application crash event handler! 
I see the same one with FAZ 5.4.5 and will add an alert for myself.


We just went to a local Fortinet Tech refresh and they showed us a lot with the FAZ.  I think you should consider 5.6 it has a ton of nice new features.  You can download the VM and test with that.  I don't know how anyone with a network of more than 20 people can operate without the FAZ.  Critical tool for resolving things like this.  We first saw the App Crash via the Events/Event Handler view.  Ironically Fortinet defines it as a "Medium" event.  We changed to "Critical" and configured the alerts so we'll know right away next time.
 
Of course going in via the console and checking "diag debug crashlog read" gives you the same indicators in a slightly less refined way.  I wish the output of that was a widget on the Dashboard.
#14
tanr
Gold Member
  • Total Posts : 384
  • Scores: 12
  • Reward points: 0
  • Joined: 2016/05/09 17:09:43
  • Status: offline
Re: Bad engine update??? 2017/09/29 08:00:28 (permalink)
0
The FAZ 5.6 feature list looks nice.  I'm leery about changing our FAZ to a .0 release, though.  We had a number of problems with FAZ 5.4.0.  We didn't really consider it reliable till 5.4.3.
 
How has your experience with the FAZ 5.6.0 been?
#15
Cyrielr
New Member
  • Total Posts : 5
  • Scores: 0
  • Reward points: 0
  • Joined: 2017/03/16 13:37:00
  • Status: offline
Re: Bad engine update??? 2017/09/29 12:01:53 (permalink)
0
Thanks guys for this tips. It's now configured on our Faz ;)
#16
hmtay_FTNT
Gold Member
  • Total Posts : 209
  • Scores: 26
  • Reward points: 0
  • Joined: 2017/02/22 11:02:10
  • Status: online
Re: Bad engine update??? 2017/09/29 12:41:05 (permalink)
0
We are very sorry for the inconveniences caused with this crash. It should not have happened and we regretted it. I can confirm that there was a faulty signature that got through as a corner case. The latest signature database should have fixed the problem and preventive measures have been taken to prevent issues like this from happening again. Sorry once more.
#17
Jump to:
© 2017 APG vNext Commercial Version 5.5