Helpful ReplyFortigate Internal traffic Problem

Author
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
2015/01/09 03:46:46 (permalink)
0

Fortigate Internal traffic Problem

Hi All
 
I have a really strange issue at my remote office. 
 
Basically I have a FortiWifi-40c V5.2.2 with a pretty basic configuration:
WAN1 -> Router
Internal1 -> LAN Switch
Internal 5 -> Workshop switch
 
We have an IPSEC tunnel up to our Head Office.
 
The issues we are having is that when I plug a device into the LAN (My laptop for example), I fail to get any internet breakout. After investigating, I found that the machine kept getting and IP from our local windows DHCP server, but then it would loose the IP and try and renew again (This will carry on in a constant loop). If i get on to the windows server, the DHCP adress leases get full of "BAD_Request" objects. Looking through the event logs on the windows server came back with no errors being logged.
 
Now the interesting this is... When I unplug the Fortigate, bam...I get a DHCP address. I then plug the Fortigate back in to the LAN switch and I can browse and access various resources.
 
Another interesting fact is that when connecting to the wireless on the Fortigate, I get an IP and can browse, access network resources etc, except for one thing... The branch printer (A small simple HP malfunction). I can ping the printer, but cannot print, scan or access it's web interface.
 
It's almost as if the Fortigate is killing internal traffic somehow. We have this same device and a very similar setup at some of our clients and have no issues. 
 
Yesterday I factoried the Fortigate and re-built the config from scratch, but still the issues persists. I'm pretty sure the issue started after the 5.2 upgrade, but I am unfortunately not 100% sure as most devices are wired and have never been disconnected and connected back to the network.
 
Today I had a look through the switch config and could not find any issues there either (Also very basic) - None the less, I firmwared the switch to the latest version in case.
 
Any assistance/guidance would be greatly appreciated. I would prefer avoiding a downgrade of the FortiOS if possible.
 
 
Regards 
#1
Christopher McMullan_FTNT
Platinum Member
  • Total Posts : 415
  • Scores: 36
  • Reward points: 0
  • Joined: 2014/09/08 08:00:33
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/09 05:13:39 (permalink)
0
Just an initial thought...does the FortiGate have a DHCP server set up on the internal interface facing the LAN switch?
 
I'm trying to think of where the BAD_request messages could be coming from. Is there a way you could show them via a screenshot? If the Windows logs don't yield any errors, it's not compromising the service itself. Would the messages be received by broadcast, maybe?
 
There are mechanisms for detecting address conflicts...I'm wondering if the FGT could be broadcasting BAD_request messages that get picked up by the Windows server.
 
Just some thoughts off-the-cuff as I nurse a coffee...
#2
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/09 05:42:15 (permalink)
0
Hi Christopher
 
No DHCP on the LAN interface of the Fortigate. There is however one running on our "Workshop" interface - Even though this should make no difference, I tried disabling, even unplugging this to ensure it was not causing any issues...No difference there.
 
I'm also thinking a broadcast of sorts. Very odd...
 
Screenshot attached.

Attached Image(s)

#3
Christopher McMullan_FTNT
Platinum Member
  • Total Posts : 415
  • Scores: 36
  • Reward points: 0
  • Joined: 2014/09/08 08:00:33
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/09 05:46:08 (permalink)
0
It looks like the 'name' could be Windows-generated. There still seem to be address conflicts, which was my hunch. Now, as to the source...
 
We could run packet captures to determine who is generating the conflicts. Two options: 'diag sniffer packet any "port 67 or port 68" 6 0 a' would show all traffic matching the filter, but the converted output in Wireshark will obfuscate the MAC addresses. It may be better to run simultaneous sniffs on individual interfaces to preserve the Layer-2 information:
 
diag sniff pack Internal1 "port 67 or port 68" 6 0 a
diag sniff pack Internal5 "port 67 or port 68" 6 0 a

Regards,
Chris McMullan
Fortinet Ottawa
#4
Dave Hall
Expert Member
  • Total Posts : 1636
  • Scores: 174
  • Reward points: 0
  • Joined: 2012/05/11 07:55:58
  • Location: Canada
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/09 07:15:08 (permalink)
0
xkalib3r
Screenshot attached.

Maybe I'm reading this wrong, but what's unusual is the MAC Address (Unique ID) listed for the bad addresses are not full MAC addresses -- they are missing some digits and the lease exp dates are all the same 2015-01-09 (RAS? IPSec?).  If I didn't know better, it looks like the DHCP lease table is corrupted on the Windows server.
 
Edit: see this forum thread about a similar issue.
post edited by Dave Hall - 2015/01/09 07:30:21

NSE4/FMG-VM64/FortiAnalyzer-VM/6.0 (FWF30E/FW92D/FGT200D/FGT101E/FGT81E)/ FAP220B/221C
#5
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/13 05:55:03 (permalink)
0
Hi Guys
 
Thanks for all the feedback - Apologies for the late response, I have been out of the office.
 
Well... I came in this morning and everything is working again :-/ the even more strange part... No changes have been by myself or my team. 
 
I was also reading up about DCHP databases being corrupt - I was going to look in to that this morning but them bam all is good again :-/. The thing with the DHCP DB being corrupt is odd though cause if I unplugged the FGT from the LAN, everything worked (DHCP wise of course)
#6
Christopher McMullan_FTNT
Platinum Member
  • Total Posts : 415
  • Scores: 36
  • Reward points: 0
  • Joined: 2014/09/08 08:00:33
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/13 06:04:26 (permalink)
0
That could have something to do with DHCP auto-configuration, based on my reading for another internal, unrelated case.
 
DHCP option 116 allows the server to respond with an address of 0x00000000 if the pool is exhausted. The FortiGate's default setting is:
config system dhcp server
edit <int>
...
set auto-configuration {enable | disable} default: enable
end
 
Having auto-configuration enabled means the FortiGate will *not* send a null address if the client broadcast includes the option. The {enable | disable} logic seems backwards, but I have that from the bug that created the feature in 2010, so unless it's wrong...
 
Assuming the Windows DHCP server would respond to client DHCP option 116 broadcasts, effectively triggering the client's APIPA addressing, where the FortiGate would not (by default), that might explain why removing the FortiGate seemed to resolve the issue in your case, if there were address conflicts, exhausted pools, database corruption, etc.
 
Just a thought...

Regards,
Chris McMullan
Fortinet Ottawa
#7
Dave Hall
Expert Member
  • Total Posts : 1636
  • Scores: 174
  • Reward points: 0
  • Joined: 2012/05/11 07:55:58
  • Location: Canada
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/13 07:13:07 (permalink)
0
One of the comments made in the thread I posted, was a laptop with IPV6 installed and the LAN/Wireless NICs were set in bridge mode.  If the 40C is the only source of wifi then I'm wondering if there isn't a laptop on your network causing something similar.
 

NSE4/FMG-VM64/FortiAnalyzer-VM/6.0 (FWF30E/FW92D/FGT200D/FGT101E/FGT81E)/ FAP220B/221C
#8
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/15 01:21:07 (permalink)
0
Hi All
 
Looks like the issue is still there.
 
Christopher - I ran the diag commands but it did not help much. I just saw My laptop in there and or any other device that I tried to connect.
 
In order to rule out any machines/devices causing the issue I did the following:
 
1. Unplugged everything from our switch (Including the firewall)
2. Logged on to the server and cleared all DHCP addresses (Note this was also after removing and re-adding the DHCP role on the box)
3. Plugged the server in to the switch
4. Plugged the firewall in to the switch.
5. Plugged a device in to the switch (I tried a phone, printer, laptop and desktop)
 
None of the devices picked up an IP. I then unplugged the firewall, plugged in a single device and bam, IP was successfully obtained. 
 
I also completely removed the config for the workshop network, leaving only our internal interface as active. 
 
Something to note here as well is that we have the internal (port1) and wireless interface bridged so that laptops can obtain an IP on the same subnet. Another interesting point is that when I connect a laptop to the wireless, I am connected and working 100%.
 
I may try and separate the wireless and LAN interfaces later today to see if that makes any difference. If that is the issue, it seems it's a v5.2.2 bug.
 
Doing an arp -a on the server showed me all the MAC's associated with the bad addresses in DHCP - These MAC's were just of any device that was plugged in while the firewall was plugged in. Even when testing with only one device on the lan, the bad address arp lookup showed the single device.
 
#9
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/15 02:17:46 (permalink)
0
Hi All
 
I found the issue! Well kinda...
 
I decided to remove the corporate wireless SSID from the soft switch and all is now working. Devices happily get IP's again!
 
So now the question becomes why... This has always worked and is a very standard config across all of the smaller devices that we manage :-/
#10
xkalib3r
New Member
  • Total Posts : 19
  • Scores: 4
  • Reward points: 0
  • Joined: 2015/01/09 00:28:16
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/01/15 02:33:51 (permalink) ☄ Helpfulby gowest 2015/02/19 08:07:25
5 (2)
Hi again
 
OK so I've narrowed it down even further...
 
After reading through this post http://community.spiceworks.com/topic/366137-dhcp-bad_address-yes-i-ve-searched-other-topics right at the end there is a mention of device tracking. This prompted me to check if detect and identify devices was enabled on any of the interfaces. I had only checked the softswitch interface...I had a look at the SSID interface and found that this was enabled. I disabled this and added the SSID back to the softswitch and all is still working!
 
While I do not require the detect device feature on the internal network, i am still curious to know if there is any way to have this enabled. I tried enabling this on the softswitch interface only and DHCP broke again...
 
Anyhow, at least it's working!
 
I hope this helps others with the same issue.
 
Thanks again for everyone's assistance on this! Much appreciated!
#11
Fortiwalle
New Member
  • Total Posts : 1
  • Scores: 0
  • Reward points: 0
  • Joined: 2014/08/06 15:10:53
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/04/15 08:46:55 (permalink)
0
Unchecking detect and identify devices solved the DHCP/BAD_ADDRESS in my lab environment as well on firmware 5.2.3.
 
Thanks for the post!
#12
ChrisS
New Member
  • Total Posts : 7
  • Scores: 2
  • Reward points: 0
  • Joined: 2012/05/24 01:08:13
  • Location: Switzerland
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/07/23 03:22:58 (permalink)
0
Hi all,
 
I've also this DHCP issue "BAD_ADDRESS". DHCP fills with "BAD_ADDRESS" entries. My FortiWiFi has FortiOS 5.2.3 installed. LAN and WLAN are connected through a softswitch. I checked the setting for device identification but this setting is disabled on all my interfaces. When i delete the softswitch and configure WLAN and LAN to be in separate subnets, everything is working fine. Switching back to softswitch configuration will cause DHCP to be filled with BAD_ADDRESS entries. Does anybody have an idea what happen here?
 
Regards Christoph
#13
Camshaft007
Bronze Member
  • Total Posts : 40
  • Scores: 6
  • Reward points: 0
  • Joined: 2014/05/06 19:25:48
  • Status: offline
Re: Fortigate Internal traffic Problem 2015/08/04 21:20:59 (permalink)
0
I experienced this problem and replaced the offending machine's NIC and viola!  Problem solved, with that said, I only had two Win7Pro clients, one SMB Server (All roles) and a FWF30D (flat network 192.168.0.0/24).  But I could replicate the BAD Addressing on both the Windows SMB DHCP server AND the FWF30D DHCP Server.  BTW, this problem will demolish your network performance every time it comes back.
 
What prompted me to replace the NIC was the fact that the offending NIC kept connecting, then dropping when I went to Control Panel->Network Adapters. 
 
BTW, I'm 4+ weeks strong after correcting this problem and users are really happy and the issue has not come back.
 
Hope this helps!

" The Linux philosophy is ' Laugh in the face of danger' . Oops. Wrong One. ' Do it yourself' . Yes, that' s it." - Linus Torvalds
#14
Chris Carson
New Member
  • Total Posts : 15
  • Scores: 0
  • Reward points: 0
  • Joined: 2012/10/26 13:53:24
  • Location: Southeast USA
  • Status: offline
Re: Fortigate Internal traffic Problem 2016/01/06 12:37:55 (permalink)
0
I had some weird ARP issues with my softswitch configuration.  We had a laptop that would refuse to work, and found that ARP broadcasts was not working properly on our softswitch with internal1, internal3, internal4, but internal 2 would work. <nuts>
I fixed it by upgraded to 5.2.5.
 
The 5.2.5 release notes have a few mentions about Virtual Switch fixes...
http://docs.fortinet.com/uploaded/files/2762/fortios-v5.2.5-release-notes.pdf
 
Thanks for the help guys!
Chris
#15
torlok2002
New Member
  • Total Posts : 1
  • Scores: 0
  • Reward points: 0
  • Joined: 2016/04/06 15:03:34
  • Status: offline
Re: Fortigate Internal traffic Problem 2016/04/06 15:17:34 (permalink)
0
Christoph Schneider
Hi all,
 
I've also this DHCP issue "BAD_ADDRESS". DHCP fills with "BAD_ADDRESS" entries. My FortiWiFi has FortiOS 5.2.3 installed. LAN and WLAN are connected through a softswitch. I checked the setting for device identification but this setting is disabled on all my interfaces. When i delete the softswitch and configure WLAN and LAN to be in separate subnets, everything is working fine. Switching back to softswitch configuration will cause DHCP to be filled with BAD_ADDRESS entries. Does anybody have an idea what happen here?
 
Regards Christoph


I also found this to be an issue on a 90D, completely making their network useless for any new devices which needed an IP address. I'm thinking it would start when a user was connected with WiFi, then plugged into wired connection at his desk. "Detect devices" on the interfaces was not enabled on the soft switch or the WiFi SSID.
 
Removing the WiFi SSID from the soft switch and problem would go away, but not an ideal solution.
 
I upgraded to the newest 5.4 build 1011, as a coworker said this resolved the same issue for him at another location. I'll update with the status.
#16
Jump to:
© 2020 APG vNext Commercial Version 5.5