Bad engine update???

[seadave]

At 4:23 PST today we started seeing 403 errors when trying to visit sites.  Only way to allow access is simple unfiltered NAT rule.  Searching form shows this happened in the past with a bad AV engine update.  I notice that support.fortinet.com is down with a 500 error so perhaps they self-inflicted the same.  TIME FOR AN ARCHITECTURE MODIFICATION!

Anyone else seeing this?

I'm seeing these in my debugs:

16330: 2017-09-28 17:03:53 <01449> firmware FortiGate-500D v5.4.5,build1138b1138,170531 (GA) (Release) 16331: 2017-09-28 17:03:53 <01449> application scanunit 16332: 2017-09-28 17:03:53 <01449> *** signal 11 (Segmentation fault) received *** 16333: 2017-09-28 17:03:53 <01449> AVDB 05004000AVDB00201-00052.00000-1709281424 16334: 2017-09-28 17:03:53 <01449> ETDB 05004000AVDB00701-00052.00000-1709281423 16335: 2017-09-28 17:03:53 <01449> EXDB 05004000AVDB00401-00001.00000-1210171547 16336: 2017-09-28 17:03:53 <01449> AVSO 04000000AVEN00701052471705041426 16337: 2017-09-28 17:03:53 <01449> Register dump: 16338: 2017-09-28 17:03:53 <01449> RAX: 0000000000000000 RBX: 00007fff067f82a0 16339: 2017-09-28 17:03:53 <01449> RCX: 00000000000000f8 RDX: 000000001258db56 16340: 2017-09-28 17:03:53 <01449> R8: 00000000000000ff R9: 0000000000000000 16341: 2017-09-28 17:03:53 <01449> R10: 0000000000000002 R11: 00007fb5625a0df0 16342: 2017-09-28 17:03:53 <01449> R12: 0000000000000046 R13: 00000000ffffffff 16343: 2017-09-28 17:03:53 <01449> R14: 00007fff067f82a0 R15: 00007fff067f81f0 16344: 2017-09-28 17:03:53 <01449> RSI: 0000000000000000 RDI: 0000000000000002 16345: 2017-09-28 17:03:53 <01449> RBP: 00000000ffffffff RSP: 00007fff067f80e0 16346: 2017-09-28 17:03:53 <01449> RIP: 00007fb5654af827 EFLAGS: 0000000000010212 16347: 2017-09-28 17:03:53 <01449> CS: 0033 FS: 0000 GS: 0000 16348: 2017-09-28 17:03:53 <01449> Trap: 000000000000000e Error: 0000000000000004 16349: 2017-09-28 17:03:53 <01449> OldMask: 0000000000000000 16350: 2017-09-28 17:03:53 <01449> CR2: 0000000000000014 16351: 2017-09-28 17:03:53 <01449> Backtrace: 16352: 2017-09-28 17:03:53 <01449> [0x7fb5654af827] => /data/lib/libav.so 16353: 2017-09-28 17:03:53 <01449> [0x7fb5654b7f45] => /data/lib/libav.so 16354: 2017-09-28 17:03:53 <01449> [0x7fb5654b86f3] => /data/lib/libav.so 16355: 2017-09-28 17:03:53 <01449> [0x7fb5654ab912] => /data/lib/libav.so 16356: 2017-09-28 17:03:53 <01449> [0x7fb5654b44c3] => /data/lib/libav.so 16357: 2017-09-28 17:03:53 <01449> [0x7fb5654baa39] => /data/lib/libav.so 16358: 2017-09-28 17:03:53 <01449> [0x7fb5654da895] => /data/lib/libav.so 16359: 2017-09-28 17:03:53 <01449> [0x7fb5654d87e7] => /data/lib/libav.so 16360: 2017-09-28 17:03:53 <01449> [0x7fb565494ad7] => /data/lib/libav.so (scanvirFile+0x00000187) 16361: 2017-09-28 17:03:53 <01449> [0x01a07ddf] => /bin/scanunitd 16362: 2017-09-28 17:03:53 <01449> [0x01a455ec] => /bin/scanunitd 16363: 2017-09-28 17:03:53 <01449> [0x01a466db] => /bin/scanunitd 16364: 2017-09-28 17:03:53 <01449> [0x010e54f0] => /bin/scanunitd 16365: 2017-09-28 17:03:53 <01449> [0x010e6599] => /bin/scanunitd 16366: 2017-09-28 17:03:53 <01449> [0x019b1c7c] => /bin/scanunitd 16367: 2017-09-28 17:03:53 <01449> [0x010e734d] => /bin/scanunitd 16368: 2017-09-28 17:03:53 <01449> [0x010e0616] => /bin/scanunitd 16369: 2017-09-28 17:03:53 <01449> [0x010e3fde] => /bin/scanunitd 16370: 2017-09-28 17:03:53 <01449> [0x00427c10] => /bin/scanunitd 16371: 2017-09-28 17:03:53 <01449> [0x0042e5c7] => /bin/scanunitd 16372: 2017-09-28 17:03:53 <01449> [0x0042bcf1] => /bin/scanunitd 16373: 2017-09-28 17:03:53 <01449> [0x0042d881] => /bin/scanunitd 16374: 2017-09-28 17:03:53 <01449> [0x0042deff] => /bin/scanunitd 16375: 2017-09-28 17:03:53 <01449> [0x7fb5690e4475] => /fortidev4-x86_64/lib/libc.so.6 16376: 2017-09-28 17:03:53 (__libc_start_main+0x000000f5) liboffset 00021475 16377: 2017-09-28 17:03:53 <01449> [0x00425065] => /bin/scanunitd 16378: 2017-09-28 17:03:53 [AV Engine <1449>] Last file info: 16379: 2017-09-28 17:03:53 [AV Engine <1449>] filename: bag, filesize: 7151 16380: 2017-09-28 17:03:53 [AV Engine <1449>] Native script imagebase: 0x12546000 16381: 2017-09-28 17:03:53 [AV Engine <1449>] cprl sigid: 489591, bintype: 00000400 16382: 2017-09-28 17:03:53 scanunit=worker pid=1449 exittype=signal code=11 total=7996 free=5679 16383: 2017-09-28 17:03:53 scanunit crash: signal=11, src-ip=172.21.11.126, dst-ip=104.80.89.9, 16384: 2017-09-28 17:03:53 request-uri=http://init-p01st.push.apple.com/bag

diag autoupdate ver output:

AV Engine --------- Version: 5.00247 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu May 4 14:26:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Virus Definitions --------- Version: 52.00001 Contract Expiry Date: Mon Jul 16 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Extended set --------- Version: 52.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 14:23:00 2017 Last Update Attempt: n/a Result: Updates Installed

Extreme set --------- Version: 1.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Wed Oct 17 15:47:00 2012 Last Update Attempt: n/a Result: Updates Installed

Mobile Malware Definitions --------- Version: 52.00000 Contract Expiry Date: Sat Jun 2 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Attack Definitions --------- Version: 6.00741 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Tue Dec 1 02:30:00 2015 Last Update Attempt: n/a Result: Updates Installed

Attack Extended Definitions --------- Version: 12.00234 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 01:27:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IPS Malicious URL Database --------- Version: 1.00775 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 07:29:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Flow-based Virus Definitions --------- Version: 52.00000 Contract Expiry Date: Mon Jul 16 2018 Last Updated using push update on Thu Sep 28 17:00:11 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: Updates Installed

Botnet Definitions --------- Version: 4.00058 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 10:00:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IPS Attack Engine --------- Version: 3.00430 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Tue Aug 22 20:13:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Internet-service Database Apps --------- Version: 2.00702 Contract Expiry Date: n/a Last Updated using manual update on Wed Sep 27 11:15:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Internet-service Database Maps --------- Version: 2.00702 Contract Expiry Date: n/a Last Updated using manual update on Wed Sep 27 11:15:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

Botnet Domain Database --------- Version: 1.00505 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Aug 11 12:09:00 2016 Last Update Attempt: n/a Result: Updates Installed

Modem List --------- Version: 0.000

Device and OS Identification --------- Version: 1.00061 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Fri Sep 8 17:49:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

IP Geography DB --------- Version: 1.067 Contract Expiry Date: n/a Last Update Date: Fri Aug 4 15:07:26 2017

Certificate Bundle --------- Version: 1.00005 Last Update Date: Thu May 5 10:58:00 2016

FDS Address --------- 208.91.112.78-443

URL White list --------- Version: 1.00810 Contract Expiry Date: Mon Jul 16 2018 Last Updated using manual update on Thu Sep 28 08:05:00 2017 Last Update Attempt: Thu Sep 28 17:00:11 2017 Result: No Updates

[tanr · Accepted solution]

@seadave,

Did TAC say anything about the newer virus definitions vs. the AV engine?  I thought we didn't get a new AV engine, just new virus definitions.

On our 300D and 100D (5.4.5) once virus definitions were updated from 52.00001 to 52.00003 and flow-based virus definitions were updated from 52.00001 to 52.00002 I stopped seeing the crashes.

[tanr]

I see the same thing on a 300D w/ 5.4.5.

[tanr]

I ran an "exec update-now".  They've already got a new set of virus definitions, 52.00002 instead of 52.00001.

Unfortunately, I'm still seeing the same sets of crashes, so it's not fixed yet.

[tanr]

Looks like support.fortinet.com is back up.

Virus definitions have changed from 52.00001 to 52.00002 to 52.00003

Flow-based virus definitions have moved from 52.00001 to 52.00002.

Haven't seen any more crashes in the 10 minutes since I updated.  Fingers crossed.

[Yamada_Takahiro3]

This is japan.

we ouccur same issue.

3600D 5.4.x

[seadave]

So I just go off the phone with the TAC.  Ticket 2382232 and it appears to be a bad AV Defs update.  The temp solution is to login to the console.

fw01 # config antivirus profile

fw01 (profile) # edit default

fw01 (default) # set inspection-mode flow-based

fw01 (default) # end

This is assuming you are using the "default" AV profile.  Change as needed.  This makes it not visible in the gui I think but it works.  Will update when I hear more.

[Cyrielr]

I got the same issue and I confirm that an update of the AV database on 52.00003 solved the issue !

If you still have the issue you could disabled temporary the AV with : 

# diagnose antivirus bypass off

[tanr · Accepted solution]

@seadave,

Did TAC say anything about the newer virus definitions vs. the AV engine?  I thought we didn't get a new AV engine, just new virus definitions.

On our 300D and 100D (5.4.5) once virus definitions were updated from 52.00001 to 52.00003 and flow-based virus definitions were updated from 52.00001 to 52.00002 I stopped seeing the crashes.

[Yamada_Takahiro3]

we seem to resoled. 52.0003

[ReseauSL]

I Had the same problem in Proxy-base Inspection Mode, ( 5.4.4 ). I updated AV definition from 52.00000 to 52.00003 and it fixed the problem .