Another weird issue.....this time its WiFi!

S

Smoblikat

Limp Gawd
Joined
May 28, 2020
Messages
444
Hello all, ive got another strange scenario that I was hoping to get some help with, be warned though.......its an apple issue. I have a guest network, very basic setup, it has its own VLAN (we can call it 5) and I am using a virtual interface on my firewall as the gateway for the internet (same setup as my printer subnet from my other routing thread). I am assigning DHCP from my server subnet, with an IP helper on the firewall interface, and there is just one policy on the FW that points the guest interface to the WAN interface, nothing special whatsoever, no communication with my other subnets. DHCP is handing out 8.8.8.8 as the only DNS server, and all clients can receive the appropriate IP address on the subnet. All of my non apple clients can get to the internet beautifully, windows devices, android devices, chrome/pixel devices, and even my personal macbook that runs OSX 10.6 (the last usable OS apple ever made), the issue I am having is that none of the iPhones/iPads and modern crapple computers can actually browse the internet. I can verify that they are at least able to communicate out to the WAN and onto the internet by browsing directly to an IP rather than DNS name, but most websites simply return some answer about not allowing traffic directly to an IP address, but the fact that I can get to the website to see that answer pretty much proves that the connection is being made. Obviously this seems like some sort of DNS issue, but its ONLY on modern apple devices, so im wondering if there are any other apple admins out there that know of some trick to make these devices play nicely? Im going to play around with some stuff now, but I wanted to get the question out there first to see if there is any low hanging fruit im missing. Im borrowing someones iphone to test with now, ive done all the things as far as forgetting the network, resetting network settings etc..

The devices all do seem to work on my production network, but as far as configuration goes the only difference is that I am only handing out 8.8.8.8 as DNS rather than my main DCs address. I am specifying a DNS suffix, but I have already tried removing that option from the DHCP settings.

Things I will try next:
Make a new SSID on the same VLAN
Add a policy to allow the subnet to communicate with my main DC that allows only DNS querries (I did try this before, but I might have made a mistake so ill try it again)
Reboot all the things in general
More as I think of it
 
This is going to be really tough to troubleshoot without seeing your firewall policies. Have you verified the client devices are getting the proper DNS settings from DHCP?

You are correct that this does sound like a DNS problem. If you can ping 8.8.8.8 but can't browse to any website via hostname then your internet is working but you are just having issues with name resolution.
 
Cmustang87 said:
This is going to be really tough to troubleshoot without seeing your firewall policies. Have you verified the client devices are getting the proper DNS settings from DHCP?

You are correct that this does sound like a DNS problem. If you can ping 8.8.8.8 but can't browse to any website via hostname then your internet is working but you are just having issues with name resolution.
Click to expand...

I attached the ONLY policy I have for this connection, its literally just a straight shot from the interface to WAN, the VLAN is just layer 2 up until this firewall interface, and there is an IP helper that provides DHCP. I have confirmed that clients are recieving the appropriate DNS servers, ive tried all of cloudflares, googles, and I was able to confirm that DNS resolution was working to my internal DC when I had the policy to allow that (which has since been shut off since it didnt help). What would make name resolution different on modern apple devices compared to ANY other device in the world?
 

Attachments

  • FWPol.PNG
    FWPol.PNG
    6.6 KB · Views: 0
Are you using interface pair view or sequence view in your firewall policy section?
 
are you SURE they are connected to the same network? I had an issue where i thought they were, my 2.4 and 5ghz were the same SSID, some clients were connecting to the 2.4 and working just fine, others were connecting via 5ghz and weren't working, I ended up having to split the SSID's to do further testing, eventually found out that it wasn't vlan tagging properly on the 5ghz, but it did when I split it
 
That's unfortunate. Drag that Guest policy all the way to the top and test again. Have you reset your counters and checked which policy is being hit?
 
Smoblikat said:
What would make name resolution different on modern apple devices compared to ANY other device in the world?
Click to expand...
Have you used the cli on the clients the be certain that you are using the DNS resolvers you think you are? If things look correct there do you see the look ups in in the firewall logs?
 
scrappymouse said:
are you SURE they are connected to the same network? I had an issue where i thought they were, my 2.4 and 5ghz were the same SSID, some clients were connecting to the 2.4 and working just fine, others were connecting via 5ghz and weren't working, I ended up having to split the SSID's to do further testing, eventually found out that it wasn't vlan tagging properly on the 5ghz, but it did when I split it
Click to expand...

Good point, I went to another site today with basically the same configuration, my suspicion was some sort of split between 2.4/5ghz, as the apple devices do somehow pick up a random IPV6 address, I was thinking maybe theyre trying to resolve names against that? The working site has some basic access rules set on the SSID itself, I mirrored that to my other site, but I havent been back to test yet. I will go there later today for some other issues, and I will be sure to look into this first. It might be an issue with the AP/controller itself too, the problem site is on a drastically older version than my working one, and I brought 2 clients with me that werent working, and they both fire up just fine over here. Both apple devices too, with the same firewall policy and the same DNS config pointing it to 8.8.8.8.


Cmustang87 said:
That's unfortunate. Drag that Guest policy all the way to the top and test again. Have you reset your counters and checked which policy is being hit?
Click to expand...

I did move the policy as close to the top as I felt comfortable (I have some EXTREMELY important security policies configured for some unique security systems that I dare not impede) and that didnt help, though I will reset the counter on the policy and try to connect some problem devices to it, see if/when it gets hit.

Nicklebon said:
Have you used the cli on the clients the be certain that you are using the DNS resolvers you think you are? If things look correct there do you see the look ups in in the firewall logs?
Click to expand...

I have CLI access to the laptops, but on iphones/ipads I am somewhat screwed as far as what I can make them tell me. So far everything that has gotten a DHCP address has pulled the right DNS servers, I do have one random macbook that isnt pulling DHCP, but im pretty sure that system is broken in general, which is a whole other issue....I will look at the logs and see if I get any info about lookups, I might re-instate my policy that makes my on-site DC the primary DNS just so I can view the hit-count on DNS lookups from there too.

Nicklebon said:
When troubleshooting interface pair will make your life easier, especially so when dealing with a large number of policies.
Click to expand...

Maybe my use case doesnt have enough policies (less than 100) but I cant see any reason how interface pair would make it easier? It seems to show me the same exact information, but now I need to click the + sign in order to see it, rather than it being presented to me without any additional work. Ill try using it, im always looking for ways to improve my workflow, but I am having a hard time visualizing how its better, how many polcies do you usually deal with in a FW? Attached is a screenshot of the polciy at my working site using interface pair view, looks the same to me except ive got that navy blue bar and a + sign to deal with now.


Thanks for the replies everyone, I thought I was going to get away with working less than 12 hours today, but we have a massive meeting later plus a bunch of other stuff going on, maybe ill be able to sleep this weekend assuming the second VOIP cutover goes smooth on friday night..........
 

Attachments

  • fwworking.PNG
    fwworking.PNG
    10.5 KB · Views: 0
Nicklebon said:
When troubleshooting interface pair will make your life easier, especially so when dealing with a large number of policies.
Click to expand...

I 100% agree with this. I could never go back to sequence view. Interface pair view is much more powerful and easy to manage when you get beyond a handful of policies.
 
Smoblikat said:
Maybe my use case doesnt have enough policies (less than 100) but I cant see any reason how interface pair would make it easier?
Click to expand...

Interface pair view allows you to view policies as it relates to traffic flow and less chance you screw something up with "any any" policies or policies that contradict each other in out of order sequences. Interface pair view lets you have separate "sequence views" but only as it relates to the specific packet flow you are intending. For example, each policy in interface pair view when you have Guest to WAN1 traffic will follow sequence logic, but you don't have to worry about any policies above that would contradict or "win". A perfect example of this would be blocking DNS outbound from a specific subnet but having an allow DNS rule below it. The deny would catch it first. If you did it with Interface Pair view this problem goes away.
 
So just to keep you guys updated, I tried interface pair view, I still dont like it. Maybe in the future I will put more effort into understanding it, but for now im doing fine in sequence view. Between all the staff coming back I have had very little time to dig into this network issue, but I have it isolated to JUST wireless clients on this one SSID. I used an ethernet adapter on one of the macbooks that wouldnt work on wifi, picked up an address and got to the internet just fine. This seems to be some sort of issue with the AP's only, our airwave server was down, AND our master AP is also down for some reason (that I just noticed). Im coming in tomorrow to get wifi cleaned up, im thinking once I get the wireless network back up to where it should be, this issue will hopefully be solved? Still doesnt explain why it only affects apple devices and not any of the other ones, but at least its a startign point. Moving the apple stuff to the production network does allow them to connect to the internet, and the guest network was added after airwave went down and all of the subnets changed, while production has never changed, im thinking some of the AP's arent picking up the new information/reporting to the master AP and its causing confusion (again, just for apple stuff, no clue....)

Ill keep you all posted, thanks for the replies.
 
OK, im done for now. I cant waste more time on this random issue, for now its solved and I will investigate it more in the future. I limited the radio to 2.4ghz only for the SSID (which didnt help) and then I moved the DHCP server to the IAP controller (I tried it on the gateway port on the FW and my actual server that hands out DHCP, only non apple clients got an IP) but having it on the AP itself allows all clients to get an IP and reach the internet, still just using 8.8.8.8 as DNS, no config changes were made to the network. I dont think limiting the radio to 2.4ghz helped, but the fact that its working at all means im done touching it for now, I think my other sites are going to have some issues down the line but I will deal with that when it happens.

Appreciate the help, if anyone wants to ponder on this a bit please feel free to do so, I still cant explain why this only affected apple devices.
 
Smoblikat said:
I am specifying a DNS suffix, but I have already tried removing that option from the DHCP settings.
Click to expand...

I doubt this is your issue, but I keep thinking about it so I'm going to throw it out there. What DNS suffix is being handed out? Apple devices do not play well when they end in ".local". Pfsense actually has a warning about this where you enter the domain name.

There is one other case I read somewhere about someone having issues (I think it was Googles mesh wi-fi system and ".lan" but don't quote me) with a certain dns suffix.

I know you said you tried removing the suffix so it's likely this isn't the issue, but it's the only thing I can think of where only certain devices aren't working properly. I'd double check what suffix the devices have and make sure that suffix isn't being used by some other protocol somewhere.
 
Last edited:
Cypher- said:
I doubt this is your issue, but I keep thinking about it so I'm going to throw it out there. What DNS suffix is being handed out? Apple devices do not play well when they end in ".local". Pfsense actually has a warning about this where you enter the domain name.

There is one other case I read somewhere about someone having issues (I think it was Googles mesh wi-fi system and ".lan" but don't quote me) with a certain dns suffix.

I know you said you tried removing the suffix so it's likely this isn't the issue, but it's the only thing I can think of where only certain devices aren't working properly. I'd double check what suffix the devices have and make sure that suffix isn't being used by some other protocol somewhere.
Click to expand...

Thanks for the reply, the DNS suffix I was supplying was a .net, and even after specifying no suffix/playing around with different ones it didnt seem to help. I wound up posting this problem on an apple specific forum and the only person to reply confirmed a suspicion that I was having, but didnt have time to investigate. Firmware. My other sites were on firmware 8.5.0.10 or 8.6.somthing-else, while the problem site was on 8.3.whatever (all aruba instant access points, with airwave for monitoring only) I finally got a chance to update the firmware on my problem site tonight, which did NOT go smoothly, I learned that the mac address of the virtual controller is dynamic and not tied at all to the master AP, even though all my DHCP reservations were set to the mac address of the master controller and its working fine everywhere else. After about an hour of nail-biting fearing that the firmware update failed and I now have a building full of bricks 2 days before school starts, I dug into the arp cache to see if there was some misconfiguration of what the IP of the virtual controller should be, the arp cache read a completely different mac address than what I expected it to be for the port the master was plugged into, as soon as I updated my DHCP reservations with the new mac address (seriously, how is a mac address not a static object????) and rebooted the AP, I was greeted with the aruba instant login page, and airwave started showing all the other access points as coming up. Initial tests were promising, my boss was able to make phone calls on wifi (which was apparently a problem before, I never knew since I have a flip phone, 15 years old and still the most reliable piece of tech I own) but I was so not up for the task of reconfiguring the guest network to test out if the firmware did it or not. I have another late night tomorrow as our VOIP guys are coming in to put the finishing touches on the paging system, so I will move guest over then while they are working on that. Im almost glad I had this problem, ive learned that sometimes things literally dont work for absolutely no reason at all, and that a MAC address is merely a suggestion. I will keep you posted on my test results tomorrow, comcast finalized our fiber switches in 2 of the sites today and the configuration they gave me is nothing at all like I envisioned it to be, so I imagine I will have a third "WTF is this weird issue??" thread soon enough :D

This is the apple thread I posted, so far the dude seems spot on with his recommendation of a firmware update. I would have done it sooner but when I say I barely have time to breathe right now, I mean it :p
https://forums.macrumors.com/threads/strange-issue-only-affecting-apple-devices.2251883/

Thank you,
Smoblikat
 
So the firmware update helped in the sense that now all devices are able to at least get an IP address, but the apple devices still arent able to get to any websites. I might experiment a bit more later in the year when I get a block of time where I can take guest down, but for now ive just decided to keep the DHCP on the IAP itself, and forget about it. Technically this has the added benefit of devices not needing to touch my production network at all (My initial goal was for DHCP to come from the actual DHCP server), so im going to pretend this was all part of the plan and be thankful that I got it working at all. I think I would prefer the firewalls gateway to hand out the DHCP as opposed to the AP's themselves, but whatever!

this-is-fine.jpg
 
You must log in or register to reply here.
Back
Top