Helpful ReplyHot!Problem with 2nd network reaching internet

Author
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
2019/08/15 15:01:58 (permalink)
0

Problem with 2nd network reaching internet

 
I'm definitely a beginner with Fortigate, but have some experience with stateful firewalls (from iptables). I'm setting up a system (Fortigate 30E) which should look like:
 
SD-WAN (wan + lan4) -> two different internet connections, select ports forwarded to Internal (below)
Local (lan3): Single computer that is a local server and responsible for most of our load (including stuff it forwards), unlimited outgoing connections via SD-WAN.
Lan (lan1 & lan2): connect to some WiFi routers; unlimited outgoing connections via SD-WAN, limited to select ports to Local.
 
My Configuration so far is outlined below. Note that I know I do not have rules to permit LAN to access anything on Local right now.
 
What works: Internet connections to Local for selected ports only; Local accessing internet; Load balancing with SD-WAN.
What Doesn't work: Lan accessing internet. Using the packet sniffer, I determined that for DNS requests going out from Lan, the returning packets from the nameserver (I tested with 8.8.8.8) are rejected by the Fortigate, with an ICMP response sent to the DNS server that the destination port is not accessible.
 
The same traffic from Local seems to work just fine.
 
Any thoughts on what I've done wrong, or how to identify what I've done wrong would be greatly appreciated.
 
Thanks,
Doug
 
(Names and addresses have been sanitized, and the number of ports reduced).
Network:
lan (lan1 & lan2) Role LAN
Addressing Manual nnn.nnn..1.99/24
DHCP server starting nnn.nnn.1.100 ending nnn.nnn.1.200
STP on

local (lan3) Role LAN
Addressing Manual nnn.nnn..2.1/29
DHCP server starting & ending at nnn.nnn.2.2

SD-WAN combines two interfaces
wan2 (lan4) Role WAN
Addressing Manual XXX.XXX.XXX.XXX to ISP1 (gateway configure in SD-WAN page)
wan Role WAN
Addressing Manual YYY.YYY.YYY.YYY to ISP2 (gateway configure in SD-WAN page)

Static routes:
0.0.0.0/0            SD-WAN
nnn.nnn.0.0/16    0.0.0.0        local (lan3)
nnn.nnn.1.0/24    0.0.0.0        lan

Virtual IPs:
ip1: XXX.XXX.XXX.XXX tcp port 20 to nnn.nnn.2.2, interface any
ip2: YYY.YYY.YYY.YYY tcp port 20 to nnn.nnn.2.2, interface any
ip3: XXX.XXX.XXX.XXX tcp port 22 to nnn.nnn.2.2, interface any
ip4: YYY.YYY.YYY.YYY tcp port 22 to nnn.nnn.2.2, interface any

Virtual IP Groups
ipg1: Interface lan4
      Members ip1 and 1p3
ipg2: Interface wan
      Members ip2 and 1p4

IP Pool (not used):
nnn.nnn.2.100-nnn.nnn.2.200


Policies:
0: Implicit Deny (built in)
1: To Local  sd-wan -> lan3
   Source all
   Dest ipg1, ipg2
   Schedule Always
   Service All
   Accept
   NAT Disabled
2: To Internet lan3 -> sd-wan
   Source all
   Dest all
   Schedule always
   Service all
   NAT Enabled
   Use Outgoing Interface Address
3: To Internet2 lan -> sd-wan
   Source all
   Dest all
   Schedule always
   Service all
   NAT Enabled
   Use Outgoing Interface Address

Observed lan -> sd-wan fails because (at least) DNS responses are rejected by Fortigate (which sends ICMP: Port Not Accessible to DNS server).
#1
orani
Silver Member
  • Total Posts : 88
  • Scores: 1
  • Reward points: 0
  • Joined: 2019/07/11 12:54:18
  • Location: Athens
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/15 23:15:43 (permalink)
0
The implicit deny rule must be the last rule you have.
#2
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 10:41:07 (permalink)
0
Orani,
Thanks for your reply. Since the deny rule is implicit, it's position is determined by the Fortigate software. I imagine if you check your rules, you will find the implicit rule (if you enable seeing it) shows up as rule 0 also. Also, if it really were first, nothing would work at all. As it is most things work, just not packets returning from the internet to the second internal interface.
Regards,
Doug
#3
orani
Silver Member
  • Total Posts : 88
  • Scores: 1
  • Reward points: 0
  • Joined: 2019/07/11 12:54:18
  • Location: Athens
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 11:42:32 (permalink) ☄ Helpfulby dcs 2019/08/19 12:16:18
3 (1)
Try to place the rule with id 2 before (higher) the rule with id 1
#4
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 12:05:28 (permalink)
0
orani,
Thanks for the quick reply. I haven't been ordering the rules myself. The numbering is generated by the FortiGate GUI, and the actual order they occur in that interface is based on the interfaces involved. How do I
1) Determine the actual order of priority of the rules, and
2) Control that order.
Regards,
Doug
#5
orani
Silver Member
  • Total Posts : 88
  • Scores: 1
  • Reward points: 0
  • Joined: 2019/07/11 12:54:18
  • Location: Athens
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 12:44:04 (permalink) ☄ Helpfulby dcs 2019/08/19 12:56:19
4 (1)
On the upper right corner check the sequence view. Then check you rules priority. The rule wich is higher is the first to execute... then you can drag and drop the rule to move upper
#6
Dave Hall
Expert Member
  • Total Posts : 1475
  • Scores: 162
  • Reward points: 0
  • Joined: 2012/05/11 07:55:58
  • Location: Canada
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 12:48:18 (permalink) ☄ Helpfulby dcs 2019/08/19 14:47:33
3 (1)
Try removing the static routes for local and lan - the networks tied to these interfaces should already be in the routing table (but don't do this unless you are on site or have alternate access to get into the fgt.)
 
If it's just DNS-related only issue, check the WAN interfaces to see if DNS override is enabled or not. 
 
If would help if you provide the packet sniffer output and/or show us the routing monitor output.
 
Edit: Have you confirmed or isolated the issue on the client side behind the lan network and/or confirmed the information handed out by the DHCP server to devices on that lan is correct?
dcs
Static routes:
0.0.0.0/0            SD-WAN
nnn.nnn.0.0/16    0.0.0.0        local (lan3)
nnn.nnn.1.0/24    0.0.0.0        lan



post edited by Dave Hall - 2019/08/19 12:52:19

NSE4/FMG-VM64/FortiAnalyzer-VM/5.4/6.0 (FWF40C/FW92D/FGT200D/FGT101E)/ FAP220B/221C
#7
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 12:56:46 (permalink)
0
orani,
Thank you. I tried to drag before, but it was in interface pair view and would not let me. Now I see the priority (from highest to lowest) is 1, 2, 3, 0. My problem is with traffic from rule 3; rule 2 traffic works correctly. Should I move rule 3 above 1? Also, it sounds like the VIP in rule 1 takes effect first even if rule 1 is last -- but that VIP is restricted to ports which are not related to the problem, so I'm hoping it doesn't matter.
Doug
 
#8
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 13:07:46 (permalink)
0
Dave,
I've already tried without the static rules, and (because of our internal network topology) they are required.
I don't see DNS override on either interface.
I'll check DHCP and edit the packet logs to sanitize them so I can post them after lunch.
Thanks for the tips,
Doug
 
#9
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 14:24:33 (permalink)
0
Here are the packet logs. I didn't filter for the Fortinet DNS servers, so you don't see the ICMP replies to them, or the DNS responses coming from them. However, in another trace I did see the same thing for those as I see for the google name server here. Note that only DNS is captured here, because everything else is waiting on name resolution.
 
<Serial Number> # diag sniffer packet any "host nnn.nnn.1.100 or host 8.8.8.8" 4 20 a
interfaces=[any]
filters=[host nnn.nnn.1.100 or host 8.8.8.8]
2019-08-16 19:00:42.452130 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 33
2019-08-16 19:00:42.452193 lan in nnn.nnn.1.100.48860 -> 208.91.112.52.53: udp 33
2019-08-16 19:00:42.452221 lan in nnn.nnn.1.100.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.452243 lan4 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.452247 eth0 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.452253 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 33
2019-08-16 19:00:42.452511 lan in nnn.nnn.1.100.48860 -> 208.91.112.52.53: udp 33
2019-08-16 19:00:42.452526 lan in nnn.nnn.1.100.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.452535 lan4 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.452539 eth0 out YYY.YYY.YYY.YYY.48860 -> 8.8.8.8.53: udp 33
2019-08-16 19:00:42.459044 lan4 in 8.8.8.8.53 -> YYY.YYY.YYY.YYY.48860: udp 49
2019-08-16 19:00:42.459081 lan4 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable
2019-08-16 19:00:42.459086 eth0 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable
2019-08-16 19:00:42.469131 lan4 in 8.8.8.8.53 -> YYY.YYY.YYY.YYY.48860: udp 93
2019-08-16 19:00:42.469151 lan4 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable
2019-08-16 19:00:42.469155 eth0 out YYY.YYY.YYY.YYY -> 8.8.8.8: icmp: YYY.YYY.YYY.YYY udp port 48860 unreachable
2019-08-16 19:00:43.013578 lan in nnn.nnn.1.100.34520 -> 8.8.8.8.53: udp 39
2019-08-16 19:00:43.013619 lan4 out YYY.YYY.YYY.YYY.34520 -> 8.8.8.8.53: udp 39
2019-08-16 19:00:43.013623 eth0 out YYY.YYY.YYY.YYY.34520 -> 8.8.8.8.53: udp 39
2019-08-16 19:00:43.014352 lan in nnn.nnn.1.100.48860 -> 208.91.112.53.53: udp 39


 
#10
orani
Silver Member
  • Total Posts : 88
  • Scores: 1
  • Reward points: 0
  • Joined: 2019/07/11 12:54:18
  • Location: Athens
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 15:32:05 (permalink)
0
Try to set the rules with the following sequence 2,3,1,0. If that not work, then try to enable nat at policy 1
#11
dcs
New Member
  • Total Posts : 7
  • Scores: 0
  • Reward points: 0
  • Joined: 2019/08/15 14:24:10
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 16:42:04 (permalink)
0
We have urgent network activity right now -- when I get a chance I'll try  your suggestions.
Thanks,
Doug
 
#12
orani
Silver Member
  • Total Posts : 88
  • Scores: 1
  • Reward points: 0
  • Joined: 2019/07/11 12:54:18
  • Location: Athens
  • Status: offline
Re: Problem with 2nd network reaching internet 2019/08/19 17:06:49 (permalink)
0
Ok, but don't worry. You will not have any downtime doing those changes. Let us know when you do the tests.
#13
Jump to:
© 2019 APG vNext Commercial Version 5.5