My first design

I’m wrapping up my first predictive design – 3 floors of a building that is being heavily renovated for a client. The renos are in progress so much of the construction does not yet match the drawings; and the client wanted to know where to have the electricians run cabling for APs.

Channeling Keith Parsons – the most important part of this design is the validation stage; which hasn’t happened yet; but WILL be happening once renovations are near completion. When that happens, I’ll find out how far off the predictive design was, and make adjustments to make sure I’ve met the requirements.

I’d like to share the process I went through for two reasons:

  1. So my experienced colleagues on twitter can offer advice that I can use to improve
  2. So my novice colleagues on twitter can see the process and the expert advice

I realize that I will make mistakes, but I will learn from them, and hopefully give some of the other up and coming WLAN experts something to glean. I’m definitely uncomfortable about posting my first design for the community to see, but I’m confident that a lot of good can come from it. With that in mind, please be gentle! Continue reading “My first design”

Asymmetric routing is not for your LAN

UPDATED: Figured it out! Go to the bottom for the fix.


 

I know I’ll continue to see it everywhere. It just happens by accident. They didn’t realize what they were doing.

But they expect me to clean up the mess. Without downtime. Where’s my hammer?

Yet another unintentional asymmetric routing scenario! This one is bugging me a bit. I am in the process of fixing this client’s LAN topology and will remove the asymmetric routing in the process, however, I need to transition services onto a single firewall with as little downtime as possible, without being on site. Not uncommon so far.

Here’s the gist:

curse-you-asymmetric-routing
Asymmetric routing is ok… on the WAN, on the internet, and in some cases, if you INTEND to do it. Most of the time, it’s just EVIL.

Today, the remote access client tunnel terminates on the branch default gateway (10.1.1.1). I’m reconfiguring the newer firewall (10.1.1.2) to take over that role. The newer firewall was installed to terminate a VPN across the WAN (not internet) circuit to the remote branch. It is now destined to be the sole/main/one ring to rule them all firewall for the single subnet LAN.

Both firewalls are Cisco ASAs. I know that, in order to make this work until I can change the default gateway for the LAN to 10.1.1.2, I will need TCP state bypass on the default gateway (10.1.1.1), right? Right.

Lo’ and behold – TCP state bypass is already configured, for a LOT of traffic – because the remote branch (10.2.2.0/24) is hosting some critical services, and it is also accessible over a different gateway, which is on the same subnet as the default gateway.

Me: “Were you guys aware that you were using TCP state bypass for all traffic to the branch office?”

Client: “Hm, that sounds familiar… where have I seen that before…?”

Right. So the client googled something and figured out how to make it work, but really doesn’t understand what they’re doing (turning off core firewall functionality). But it worked!

I’ve been here before, I’ll be here again.

SO I configure the remote access VPN on the newer firewall (10.1.1.2), without state bypass, and can’t RDP to the DNS server. I expected that. In the diagram, steps 1-3 happen as expected, but step 4 doesn’t happen. Enable the state bypass, problem solved, TCP traffic works. Steps 1-5 work as expected.

Here’s the part that is bugging me though. I’ve made it work with a kludge, knowing full well that I’m going to fix the core problem soon. But I haven’t had time to really figure out what the problem is…

VPN clients couldn’t do name resolution. UDP lookups to port 53 were not getting a response. Since I am working remotely from the site, embedded PCAP is the best tool I have.

PCAP on new ASA (10.1.1.2) inside interface indicates that the DNS query from the VPN client is transmitted on the inside interface to 10.1.1.80. No response is received from the DNS server.

PCAP on default gateway ASA (10.1.1.1) inside interface indicates that the DNS server is, in fact, sending a response to the client at 10.10.10.X, and it arrives INBOUND on the inside interface. An outbound flow is not captured, matching the PCAP on the new ASA.

So looking back at the diagram, steps 1-3 are fine. Step 4 never proceeds.

So… UDP shouldn’t have the same asymmetric routing issue as TCP right (it is TCP state bypass after all)? There is no “state” to check.

Hairpinning is indeed enabled on the default gateway ASA. Informational level logging didn’t indicate anything interesting, but it would still seem that this firewall is dropping the traffic.

Fix? Static routing entry on the DNS server (10.1.1.80) for 10.10.10.0/24 via 10.1.1.2. (FOR SHAME BRENNAN!). 

I know, I would NEVER leave this in production. But I needed to confirm quickly where the issue lies.

At the moment, the reason for the drop is not coming to me. Let me know in the comments, or on twitter (@bmroute) if you have an idea.


 

FIX: Credit to my mentor, CCIE Security @rwatt13. Default DNS inspection treats the UDP query/response similarly to a TCP state in that the ASA doesn’t like to see a DNS response without a query first. Disabling DNS guard on the gateway ASA fixed the problem. See some explanation of DNS guard here.

YES, I REMOVED THE STATIC ROUTE FROM THE WINDOWS SERVER. I’m not an animal.