FortiOS 6.2.3 - high Memory, low CPU, DNS Filter unreachable

[Jannik]

Hi everyone,

FG61E active-active HA

Since v.6.2.3 we have very high memory usage. The Fortigates go into  conserve mode all the time and i can't get the memory any lower than 77%. I disabled unrequiered features, switched some policies from proxy-based to flow-based, reduced the session timers, logging etc. Nothing helps. The memory overflows while the CPU runs on <5% most of the time. I have a customer with about 20x FG 30E. Some of these do nothing but IP-Sec VPN and nothing else yet but the memory there is also on 68% while the CPU is <5%.  What can I do to reduce my memory usage? Or is it a firmware issue? There were problems with the memory management in v6.2.0 and 6.2.2 in the past.

Also I noticed that the FortiGuard DNS Filter Server is unreachable in v6.2.3. I configured the DNS Filter IP from v.6.2.2 (on which it works) and it doesn't work on v6.2.3 either. I already have a case open with fortinet about the DNS Filter issue.

-Jannik

[andrewbailey]

Hi Jannik,

I don't have any immediate answers for you- but I am seeing similar problems on a 60E and a 30E. I believe they are firmware related.

Both have high memory use on initial reboot and the memory use grows over time until conserve mode is reached. The 30E was particularly bad- and I have ended up downgrading to 6.2.2 (which is slightly better and just takes longer to reach converse mode).

The release notes for 6.2.3 have been revised this week:-

https://docs.fortinet.com/document/fortigate/6.2.3/fortios-release-notes/760203/introduction-and-sup...

and they show quite a long list of known issues, including:-

[ul]

563250 Shared memory does not empty out properly under /tmp.(Anti-virus)

582374 License shows expiry date of 0000-00-00. (DNS Filter)

594598 Enabling proxy policies (+400) increases memory by 30% and up to 80% total. (Explicit Proxies).

575224 WAD high memory usage from worker process causing conserve mode and traffic issues (Proxy).

etc etc....[/ul]

Any or all of these could be causing your problems.

I have two tickets open (one for a 60E and one for a 30E). So far neither ticket has identified anything new or a resolution for my issues. Slightly frustrating really.......

I hope that helps you a little at least!

Kind Regards,

Andy.

[Jannik]

Hi Andy,

yes it is really frustrating because the memory issues are going on for quite some time/versions now. I would roll back to 6.0.X if it weren't for the better Wifi GUI and features in 6.2.X. How is it with the DNS Filter Server on your units? Also unreachable?

[andrewbailey]

Hi again Jannik,

I'm pretty sure that I have the DNS Filter is working. But, you are making me doubt slightly so I'll double check again later for you.

However, if I remember correctly one of the 6.2.X releases has introduced HTTPS support for Fortiguard updates. I've seen some intermittent/ long delay issues with Fortiguard servers (Web Filtering, DNS Filter servers ) over HTTPS using the default port 443 (but also with some UDP options too). I've ended up using HTTPS over port 53 which for me seems to be more reliable and stable here in Europe.

Of course using HTTPS rather than UDP for the FortiGaurd updates makes sense from a security point of view but it almost seems to be the feature has been enabled in the FortiOS before the back end infrastructure has been fully deployed.

Also worth noting that I'm using enforcing DNS over TLS and using non-standard (ie not Fortinet) DNS servers. In my case I'm using CloudFlare and Quad9 (both of which seem fast and support DNS over TLS).

The DNS settings in the GUI give quite a good almost real time view of the server reachability for DNS, Web filtering etc. So I keep an eye on that to see what is happening.

So, perhaps some configuration settings your could try changing? Might help improve what you are seeing currently?

There is also some troubleshooting tips in the Cookbook on the docs website which might be useful. See here:-

https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/126629/dns-troubleshooting

Best of luck, and I'll update you later on anything else I see.

Kind Regards,

Andy.

[andrewbailey]

Hi Jannik,

I've done some checking tonight and I definitely have DNS Filter Servers showing as "reachable" in 6.2.3 on the 60E.

I can see in the DNS logs that blocked sites are being reported as being in a "blocked" category, but the redirect to the port IP (in my case the default fortinet portal at the moment) doesn't seem to happen. Instead the webfilter is catching the result and webfilter replacement page is being hit.

The DNS filter policy has the "Redirect botnet C&C requests to Block Portal" option enabled and if I select one of the Botnet addresses and try browsing to that it again triggers the webfilter rather than DNS filter (with a "Spam URL" category).

So different results to you, but someone what suspicious.

If you manage to get the DNS Server Filters as "reachable" let me know what you see- but I'm thinking I may be raising another case over the next 24 hours or so!

Kind Regards,

Andy.

[JerryPWhite1]

I haven't seen the memory related issues however I'm still having DNS related issues. And I can't get Fortiguard to work without using UDP=53. HTTPS=anything won't work. They did write a script when I was on 6.2.2 that clears the memory that might help you. Under -Security Fabric-Automation click on new and then cli. Call it what you want, they called it restart_wad. Under frequency they put daily @ 3:00AM. And then the code below under 1st Action script

diagnose debug enable diagnose test application wad 97 diagnose debug disable

I do notice whenever this happens my session rate climbs. I'm assuming maybe because it can't reach dns and/or fortiguard it gets held in the queue? Don't know but I have 500-600 users screaming when it happens.

[Jannik]

I opened a case with Fortinet regarding the DNS Filter issue. There is now a bug id 582374 . 

"The issue lies with a failure in the FortiGates DNS message to the SDNS servers. The DNS message RRs are not properly pulling license information as well as the list of Secure DNS severs which interrupts the expected behavior of the DNS Filtering Service.A fix for this has been implemented server side already, however we are awaiting for a confirmation of the fix on the FortiOS side in 6.2. The developers noted they were actively working on it and should have the fix confirmed shortly.This would then be expected to release with the next 6.2 upgrade for FortiOS, 6.2.4.I will followup once the fix for 6.2 has been confirmed, "

As for now I have deactivated DNS Filter on all my ipv4 policies.

[andrewbailey]

Hi Jannik and JerryPWhite_FTNT,

I meant to ask you if you had seen the "anycast" section in the FortiGate 6.2.3 cookbook (https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/734277/fortiguard-third-party-ssl-valida...)?

That may help you?

I have enabled anycast and it seems to be working well and reliably for HTTPS over 443 (at least for server reachability).

I'm still not certain if the behavior looks right- maybe Jannik's bug will resolve some of the issues perhaps.

Kind Regards,

Andy.

[Jannik]

Andy Bailey wrote:

Hi Jannik and JerryPWhite_FTNT,

I meant to ask you if you had seen the "anycast" section in the FortiGate 6.2.3 cookbook (https://docs.fortinet.com/document/fortigate/6.2.3/cookbook/734277/fortiguard-third-party-ssl-valida...)?

That may help you?

I have enabled anycast and it seems to be working well and reliably for HTTPS over 443 (at least for server reachability).

I'm still not certain if the behavior looks right- maybe Jannik's bug will resolve some of the issues perhaps.

Kind Regards,

Andy.

Hi Andy,

I tried the anycast enable and as source both fortinet and aws but it remains unreachable. 

-Jannik

[Yurisk · Accepted solution]

Just stumbled on Fortigate 80F with this 6.2.3 having a weekly time off entering Conserve Mode on high memory usage, with 3 security rules and average 250-300 browsing sessions, no DNS Filter is used. Advised client to upgrade FortiOS ASAP.