Helpful ReplyHot!Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel

Author
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
2020/02/13 15:45:41 (permalink)
0

Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel

Hello-
 
Yesterday we updated our HA 501E pair from 6.0.5 to 6.0.7.  All seemed to be fine, but today, users connected to our SSL VPN tunnel are having Outlook 2016 Connectivity issues.  We have Outlook on-prem and only clients coming in over the SSL VPN full tunnel are having the issue.
 
Outlook sync log keeps generating errors similar to:
14:58:32 Synchronizer Version 16.0.11328
14:58:32 Synchronizing Mailbox 'Joe User'
14:58:32 Synchronizing Hierarchy
14:58:33 Synchronizing server changes in folder 'Inbox'
14:58:33 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com'
14:58:34 Downloading from server 'd10db7e0-492a-4521-8c0a-fe49225ad443@company.com'
14:58:34 Error synchronizing folder
14:58:34 [80040115-514-80040115-0]
14:58:34 Network problems are preventing connection to Microsoft Exchange.
14:58:34 Microsoft Exchange Information Store
14:58:34 For more information on this failure, click the URL below:
14:58:34 https://www.microsoft.com...0040115-514-80040115-0
14:58:34 Done
14:58:34 Microsoft Exchange offline address book
14:58:34 Download successful
 
Of course the support link goes to a generic MS help page and is unrelated.
 
If we put any of our clients on a mobile hotspot and VPN in, this happens.  If we take them off and reconnect to wifi, no more errors.  So it seems like something is happening to the traffic as it passes through the tunnel.
 
We are using a full tunnel, without split DNS.  Here are the settings:
 
config vpn ssl settings
set reqclientcert disable
set tlsv1-0 disable
set tlsv1-1 disable
set tlsv1-2 enable
unset banned-cipher
set ssl-insert-empty-fragment enable
set https-redirect enable
set x-content-type-options enable
set ssl-client-renegotiation enable
set force-two-factor-auth disable
set servercert "XXXXXXXX"  (Obscured for post)
set algorithm high
set idle-timeout 7200
set auth-timeout 28800
set login-attempt-limit 2
set login-block-time 60
set login-timeout 30
set dtls-hello-timeout 10
set tunnel-ip-pools "SSLVPN_TUNNEL_ADDR1"
set tunnel-ipv6-pools "SSLVPN_TUNNEL_IPv6_ADDR1"
set dns-suffix "company.local"
set dns-server1 XXXXXXXXXXX (Obscured for post)
set dns-server2 XXXXXXXXXXX (Obscured for post)
set wins-server1 0.0.0.0
set wins-server2 0.0.0.0
set ipv6-dns-server1 ::
set ipv6-dns-server2 ::
set ipv6-wins-server1 ::
set ipv6-wins-server2 ::
set url-obscuration enable
set http-compression disable
set http-only-cookie enable
set port 443
set port-precedence enable
set auto-tunnel-static-route enable
set header-x-forwarded-for add
set source-interface "WAN"
set source-address "all"
set source-address-negate disable
set source-address6 "all"
set source-address6-negate disable
set default-portal "SSL_VPN_TS"
config authentication-rule
edit 8
set groups "SSL_VPN_FULL"
set portal "SSL_VPN_FULL"
set realm ''
set client-cert disable
set cipher high
set auth any
next
set dtls-tunnel enable
set check-referer enable
set http-request-header-timeout 20
set http-request-body-timeout 30
set unsafe-legacy-renegotiation disable
end
 
We attempted to sniff the traffic and the PCAPS do show RSTs occurring.
 
We have used the Default Web Proxy for ever with these connections and tried to use a custom one with RPC over HTTP enabled but it made no change to the behavior.
 
config vpn ssl web portal
edit "SSL_VPN_FULL"
set tunnel-mode enable
set ipv6-tunnel-mode disable <----- This was enabled and we set to disabled, but no luck
set web-mode enable
set host-check none
set limit-user-logins enable
set mac-addr-check disable
set os-check disable
set forticlient-download disable
set ip-mode range
set auto-connect disable
set keep-alive disable
set save-password disable
set ip-pools "SSLVPN_TUNNEL_ADDR1"
set exclusive-routing disable
set service-restriction disable
set split-tunneling disable
set dns-server1 0.0.0.0
set dns-server2 0.0.0.0
set dns-suffix ''
set wins-server1 0.0.0.0
set wins-server2 0.0.0.0
set display-bookmark enable
set user-bookmark enable
set allow-user-access web ftp smb telnet ssh vnc rdp ping citrix portforward
set user-group-bookmark enable
set display-connection-tools enable
set display-history enable
set display-status enable
set heading "Omeros SSL VPN"
set redir-url ''
set theme blue
set custom-lang ''
set smb-ntlmv1-auth disable
set smbv1 disable
set hide-sso-credential enable
next
end
 
Here is a RST frame.
 
Frame 2247: 40 bytes on wire (320 bits), 40 bytes captured (320 bits)
Encapsulation type: Raw IP (7)
Arrival Time: Feb 13, 2020 14:23:33.631440000 Pacific Standard Time
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1581632613.631440000 seconds
[Time delta from previous captured frame: 1.469173000 seconds]
[Time delta from previous displayed frame: 9.668200000 seconds]
[Time since reference or first frame: 66.766862000 seconds]
Frame Number: 2247
Frame Length: 40 bytes (320 bits)
Capture Length: 40 bytes (320 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: raw:ip:tcp]
[Coloring Rule Name: TCP RST]
[Coloring Rule String: tcp.flags.reset eq 1]
Raw packet data
Internet Protocol Version 4, Src: vpnclient.company.local (10.253.1.17), Dst: exchange.company.com (172.16.10.44)
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 40
Identification: 0xe397 (58263)
Flags: 0x4000, Don't fragment
0... .... .... .... = Reserved bit: Not set
.1.. .... .... .... = Don't fragment: Set
..0. .... .... .... = More fragments: Not set
...0 0000 0000 0000 = Fragment offset: 0
Time to live: 128
Protocol: TCP (6)
Header checksum: 0x54ee [validation disabled]
[Header checksum status: Unverified]
Source: vpnclient.company.local (10.253.1.17)
Destination: exchange.company.com (172.16.10.44)
Transmission Control Protocol, Src Port: 63189, Dst Port: 443, Seq: 50, Ack: 1, Len: 0
Source Port: 63189
Destination Port: 443
[Stream index: 28]
[TCP Segment Len: 0]
Sequence number: 50 (relative sequence number)
[Next sequence number: 50 (relative sequence number)]
Acknowledgment number: 1 (relative ack number)
0101 .... = Header Length: 20 bytes (5)
Flags: 0x014 (RST, ACK)
000. .... .... = Reserved: Not set
...0 .... .... = Nonce: Not set
.... 0... .... = Congestion Window Reduced (CWR): Not set
.... .0.. .... = ECN-Echo: Not set
.... ..0. .... = Urgent: Not set
.... ...1 .... = Acknowledgment: Set
.... .... 0... = Push: Not set
.... .... .1.. = Reset: Set
[Expert Info (Warning/Sequence): Connection reset (RST)]
[Connection reset (RST)]
[Severity level: Warning]
[Group: Sequence]
.... .... ..0. = Syn: Not set
.... .... ...0 = Fin: Not set
[TCP Flags: ·······A·R··]
Window size value: 0
[Calculated window size: 0]
[Window size scaling factor: -1 (unknown)]
Checksum: 0xbf9e [unverified]
[Checksum Status: Unverified]
Urgent pointer: 0
[Timestamps]
[Time since first frame in this TCP stream: 19.268674000 seconds]
[Time since previous frame in this TCP stream: 9.668200000 seconds]
 
We are looking at DNS possibly as an issue as records show up for multiple VPN client hostnames with the same IP, but other than that can't figure out why this is happening.  Any ideas?  Is there a good diag debug command for VPN sessions we could use?
 
#1
Toshi Esumi
Expert Member
  • Total Posts : 1928
  • Scores: 168
  • Reward points: 0
  • Joined: 2014/11/06 09:56:42
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/14 00:13:44 (permalink) ☄ Helpfulby seadave 2020/02/16 16:00:18
3 (1)
If swapping back to 6.0.5 (just change the boot partition) fixes the problem, it's likely caused by one of those SSL VPN problems with 6.0.7-6.0.9 reported in this forum. Then open a case and get it verified by TAC. Depending on which one of the problems, 6.0.9 might solve the issue or might need to wait until 6.0.10 like our case.
#2
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/14 10:03:50 (permalink)
0
That is what I feared, but thanks.  What problems are you seeing?
#3
Toshi Esumi
Expert Member
  • Total Posts : 1928
  • Scores: 168
  • Reward points: 0
  • Joined: 2014/11/06 09:56:42
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/14 21:31:45 (permalink)
0
RDP sessions through SSL VPNs drop unexpectedly with 6.0.8. But not all cases. Only when some unknown conditions are met. We were told the fix will be in 6.0.10 by TAC.
#4
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/16 09:24:38 (permalink)
0
Thanks.  I've updated the code to 6.0.9 but the problem persists.  I'm begining to think this might be a DNS issue.  When a remote system with FortiClient connect its local IP AND the VPN issue IP are registered in DNS.  The other problem we have is that as clients connect/disconnect their record will remain in DNS and a new client with the re-issued IP will be added.  So we'll see things like:
 
LTP01    10.253.1.10
LTP02    10.253.1.10
LTP03    10.253.1.10
 
I've spent an insane amount of time tweaking scavenging and dynamic update settings in our 2016 AD DNS servers.  We only have this problem with Fortigate related records, I'm assuming because some sort of ack isn't being sent to indicate the record is stale.  Still digging.
#5
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/16 14:08:51 (permalink)
0
Running out of options.  Have attempted to play with DTLS settings on Gate and Client (enabled on both) to no avail.  Using FC 6.0.90277.
 
Confirmed DNS caching is working as expected.
Disabled ASIC and NPU offload for SSLVPN policy with no impact.
 
Wireshark continues to show duplicate ACKs, RSTs, Out of Order, Spurious Retransmissions, when SSLVPN is up.
 
Will update our ticket.
#6
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/16 15:51:06 (permalink) ☄ Helpfulby tanr 2020/02/16 16:27:07
5 (1)
Okay, I got a fix from TAC.  Issue is 607433 related to timeouts and sessions not found.  One way you can test is to do a diag.
 
diag debug en (enable debugging)
diag debug flow filter add X.X.X.X (the IP address of the client experiencing the issue)
diag debug flow trace start 999  (999 is number of flows to capture, you may need more)
 
search for "Session mismatch" or "Session missing"
 
When done reset debug:
 
diag debug reset
diag debug disable
 
The fix is a special build FGT_XXXX-v6-build8661-FORTINET.out (your model number XXXX)  TAC can provide if you request.
 
I was on 6.0.9 so not sure if you need to be on that first.  We installed it and while I still saw occasional resets and retransmissions via WireShark on the client side, Outlook now works perfectly.  No Sychronization errors, etc.
 
Tech said that 6.0.10 and 6.2.4 should be released with the fix.
 
Thanks Fortinet support!!!
 
 
#7
ede_pfau
Expert Member
  • Total Posts : 6184
  • Scores: 510
  • Reward points: 0
  • Joined: 2004/03/09 01:20:18
  • Location: Heidelberg, Germany
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/16 23:00:40 (permalink)
0
Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.

Ede

" Kernel panic: Aiee, killing interrupt handler!"
#8
grax88
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2020/02/17 04:55:51
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/17 04:59:24 (permalink)
0
Provided by support firmware 8660 (for FG 100F), RDP retransmission fixed but still problem with HTTPS. There is lot 
 
dstintf="unknown-0" msg="no session matched"
 
messages. I was asked to downgrade do 6.0.6.
#9
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/17 09:50:34 (permalink)
0
grax88
Provided by support firmware 8660 (for FG 100F), RDP retransmission fixed but still problem with HTTPS. There is lot 
 
dstintf="unknown-0" msg="no session matched"
 
messages. I was asked to downgrade do 6.0.6.


 
Thanks for posting the proper log message display.  I didn't catch it when working with TAC.  "no session matched" NOT missing.  Sorry.

Note the build I was issued was 8661 which is newer than what they provided you with.  Also Fs are the newest chip-set.  I find it interesting that the problem spans E and F revisions.  Suggests it is more of a coding than hardware issue. 
 
I was watching a laptop connected via SSLVPN over a mobile hotspot with Wireshark real-time as I updated.  As soon as I applied this build and 501Es rebooted (so nice that HA doesn't drop VPN sessions!!!), the number of RST and re-transmission ACKs dropped by 80% (best guess).  It did NOT eliminate them, but Outlook no longer produced Synchronization errors every few minutes and I was able to send/receive messages without delay.
 
So I agree some low level TCP packet issues may remain, but 8661 was WAY better than 6.0.9GA.  In WireShark, the packets marked TLSv1.2 have no issues (I assume that is cert negotiations) whereas the TCP protocols are the ones that show re-transmissions.  Need to look into that more.
 
We went from 6.0.5 to 6.0.7 initially, maybe we should have tried .6 first.
post edited by seadave - 2020/02/17 09:54:11
#10
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/17 09:52:15 (permalink)
0
ede_pfau
Excellent work and support for all of us! Thanks a lot for your efforts, this might save a lot of headaches.


Thanks, your knowledge has helped me many times over the years. Happy to contribute.  This is one of the better vendor forums out there in my opinion due to the time and effort put in by so many end users.
#11
grax88
New Member
  • Total Posts : 2
  • Scores: 0
  • Reward points: 0
  • Joined: 2020/02/17 04:55:51
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/17 10:59:17 (permalink)
0
Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tommorow i should have remote assistance with support.
 
This is my first bigger solution (2x 100F in HA) and I am really disapointed with overall service and fortigate support.
#12
seadave
Expert Member
  • Total Posts : 341
  • Scores: 50
  • Reward points: 0
  • Joined: 2004/11/03 18:02:09
  • Location: Seattle, WA
  • Status: offline
Re: Outlook Clients Connection Errors 6.0.7 SSL-VPN Full Tunnel 2020/02/17 15:38:19 (permalink)
0
grax88
Firmware downgrade to 6.0.6 doesn't resolve the issue, i have several messages (50) in minute with dropped packet and unknown-0 destination. For tomorrow I should have remote assistance with support.
 
This is my first bigger solution (2x 100F in HA) and I am really disappointed with overall service and fortigate support.


Having used Fortigates since 2004, I would suggest you not give up based on this one situation.  There have been periods where there appears to be a wave of bad releases.  5.0 comes to mind.  Contact the TAC and see if you can obtain the 8661 build to see if that helps.  That was my mistake, I didn't call first and it caused me to waste several hours troubleshooting the issue. 
 
Always wait a few weeks when a new build is released and monitor these forums for others reporting bugs before you upgrade.  The usual advice regarding making good backups and keeping a copy of the prior firmware on hand just in case also applies.
 
The F series sound like they are supper powerful but perhaps the code for them isn't up to snuff yet.  I would agree that the one downside of Fortinet is they tend to favor getting some new products on the market before they are as stable as one would expect them to be.
 
I think in the long run you'll find they provide good value and features for the money.
#13
Jump to:
© 2020 APG vNext Commercial Version 5.5