Meraki Auto-VPN over MPLS

Here’s a quick review of a recent Meraki MX deployment I wrapped up this week. The customer has 4 sites across Canada; one HQ with an Internet and MPLS connection, and 3 branches with only MPLS connections. We replaced their aging Juniper SSGs with MX65s.

Here’s the basic topology:

Initial State

The branch MXs reach the cloud controller through the HQ internet connection – internet connections are a requirement for running the Meraki gear; but it doesn’t have to be local. The branch MX uplink ports are connected to the MPLS cloud. At HQ, the MPLS cloud connects to a LAN port – keep this in mind.

Of course, site-to-site connectivity was required. Each site has two internal subnets/VLANs: one data, one voice. Simple enough right? Auto-VPN to the rescue!

VPN Setting

Turn on VPN at each site – we want full mesh, so every site is a hub. Then advertise the two local subnets into the VPN cloud at each site:

VPN networks

Done. Simple. Result:

Spoke to Spoke Only

Fancy green lines mean successfully established VPN connectivity! Auto-VPN is smart enough to realize that each branch/spoke appears on the internet at the same IP, but that the locally set uplink addresses are all reachable from spoke to spoke, so those VPNs form without a problem. But this is not what we wanted. Spoke-to-spoke traffic is good for voice calls, but all of our servers (including the call manager) are at HQ… and there are no fancy green lines to HQ.

What is going on? Well I can’t say with 100% certainty what exactly the limitation is, but I know one thing about the Meraki MX and VPNs – they won’t establish VPN SAs over non-uplink ports.

There are also some limitations with hairpinning – in this case, in order to establish an SA with the HQ uplink (Internet) port, the branches would need to exit the HQ uplink port, and do a 180° turn to form an ISAMKP SA. I have some intel that says Meraki has tested this feature, but I can’t say when it will be deployed to customers.

So in the meantime, we need another solution – and Meraki has just the feature.

Remember VPN concentrators?

Add Concentrator

We can put in another MX at the HQ in VPN concentrator mode. It’s a normal MX (in this case an MX64 because we only need the one port). We connect the uplink to another LAN port on the HQ MX, in this case, we gave it a separate subnet.

We can only use the uplink port on the MX when it is in this mode, and then it will serve as a dedicated hairpinning interface, effectively putting that uplink interface inside the HQ LAN.

passthrough setting

When we do this, the VPN settings page changes. On the concentrator, we manually specify the subnets at HQ that should be reachable over the VPN:

concentrator VPN settings

No routes required – the concentrator will send everything to it’s own default gateway, which in this case is the HQ MX, where both these subnets are homed.

We then turn OFF the VPN at the HQ MX, and create static routes on the HQ MX pointing to the concentrator.

The new result:

End Result

Now we have fancy blue lines showing the VPNs between the branches and HQ, resulting in a full mesh.

Rumor has it that we’ll see two improvements in these deployments sometime in the near future – the ability to use the second uplink port without an internet connection to terminate VPNs, as long as the primary uplink can reach the cloud controller; and the ability to hairpin out the uplink as I mentioned earlier. Either of these should have replaced the need for a dedicated concentrator.

In the meantime, here’s the Meraki deployment guide for the VPN concentrator role:

https://documentation.meraki.com/MX-Z/Deployment_Guides/Configuring_VPN_Concentrator_for_the_Data_Center

 

 

My first design

I’m wrapping up my first predictive design – 3 floors of a building that is being heavily renovated for a client. The renos are in progress so much of the construction does not yet match the drawings; and the client wanted to know where to have the electricians run cabling for APs.

Channeling Keith Parsons – the most important part of this design is the validation stage; which hasn’t happened yet; but WILL be happening once renovations are near completion. When that happens, I’ll find out how far off the predictive design was, and make adjustments to make sure I’ve met the requirements.

I’d like to share the process I went through for two reasons:

  1. So my experienced colleagues on twitter can offer advice that I can use to improve
  2. So my novice colleagues on twitter can see the process and the expert advice

I realize that I will make mistakes, but I will learn from them, and hopefully give some of the other up and coming WLAN experts something to glean. I’m definitely uncomfortable about posting my first design for the community to see, but I’m confident that a lot of good can come from it. With that in mind, please be gentle! Continue reading “My first design”

Mac – Set your static IP address fast!

A quick update on the last post – but for OS X!

The OS X terminal is more comfortable than the Windows CLI for route/switch guys like me. Here’s how I make quick IP changes in OS X.

You can create command aliases just like in Cisco IOS by adding them to the profile that terminal loads when you open the app. Just open terminal, and type “open .profile”. Here I’ve done just that, and you can see my terminal profile:

profile Sublime Text, Today at 10.21.49 AM

If you’ve never messed with the terminal profile, it will likely be empty. But the basic syntax is straightforward. Save the file and close/reopen terminal for the aliases to become available.

Here are a couple of my examples:

  • “setair 1.2.3.4 255.255.255.0” will statically set my Wi-Fi adapter address
  • “setethdhcp” will revert my ethernet NIC to DHCP
  • “shair” is like ipconfig in Windows, or a simpler version of ifconfig for OS X:

shair— -bash — 139×39 Terminal, Today at 10.26.43 AM.png

Note that any commands running SUDO will prompt you for your elevated password.

Set your static IP address fast!

Hello! Sorry for the radio silence – I have been a little sidetracked from welcoming my second daughter into the world on March 21st! Lack of sleep and going back to work this week are my excuses for a little blogging break – I’m not sure my ramblings would have made much sense anyhow…

But I was inspired by this tweet today:

John here wrote a really cool post about some of the ways he uses the netsh command in the Windows CLI. I knew about the “netsh WLAN show …” commands already, but John has demonstrated a neat way to monitor roaming with a simple loop. It’s elegant because it’s simple and works.

This got me thinking about a little trick I’ve been using for years now with netsh to avoid a certain pain in the arse I would otherwise be dealing with in Windows. Maybe I can help someone with my trick like John helped me with his.

I’m a CLI guy – if you’re a Cisco Route/Switch person, you know why. Heck, I try to avoid taking my hands off of the keyboard as much as possible.

If there is one thing I do a lot in Windows, it’s set static IP addresses on my NICs. And then set them back to DHCP. Sometimes several times a day.

Worse yet, when I need to bootstrap a router/switch/controller/network voodoo box, 9 times out of 10, I’m using wifi to remain internet connected while using my wired NIC to connect to the new unit – a lot of which provide DHCP out of the box. The problem with this is that my Windows wired NIC is preferred and I’m getting a default gateway from the DHCP server on the box I’m bootstrapping, even though that’s a dead end. There goes my internet until I set a static IP without a gateway on my wired NIC.

The default work flow to change the IP settings on your NIC is terrible (Windows 8.1 example):

Start > Control Panel > Network and Sharing Centre > Change Adapter Settings > Right-click adapter > Properties > TCP/IPv4 > Properties: Set your IP/Mask/Gateway if required > OK.

And don’t forget the three windows you now have to close. Then do it all over again to set back to DHCP. Sure, there are some shortcuts you can use to this process, but I still have nightmares about this. Enough complaining – here’s what I do.

First, I renamed my NICs to “wired” and “wireless” in the “network connections” window (after “Change Adapter Settings” in the work flow above). Just right click and rename, it won’t hurt anything and makes it so much easier to read CLI output.

Then I write myself a little script in notepad:

@ECHO off
cls
:start
ECHO.
set /p ipadd=Enter address and mask, press enter:
netsh interf ip set add wireless static %ipadd%
timeout /t 5
ipconfig
pause
goto end
:end
exit

Then I save it as a batch file called “Wireless Static.bat”. But wait, there’s more…

I right-click that sucker and choose “create shortcut”, which creates a “Wireless Static.bat – shortcut” file in the same folder. I rename it to “Wireless Static”, then right-click again and go to the file properties – here’s the magic:

CMD Batch file

The “Target” field will have the full path to the file. I prepend the path to the cmd.exe executable like so:

C:\Windows\System32\cmd.exe /c “full path to script”, then hit “Ok”. Now I have two files, the original batch file, and a shortcut with the CMD icon:

IP bat and shortcut

That CMD shortcut right there can be pinned to the Start page or Taskbar – the batch file can’t.

Still with me? Here’s the result! I pin mine to the Start page, so I hit the Windows key, and click the shortcut I want.

Start Page.png

Here’s the prompt, after I slap in my new static IP. The format is IP address, Mask, Gateway (if required), with a space between each.

Enter Address

I press enter, the script runs the netsh command, and counts down for 5 seconds before running an ipconfig to verify the result (I was remote desktop-ed to my Windows notebook when I took the screenshots, so I erased the wireless NIC addressing output to avoid confusion).

Static Wait.png

Finally, the “pause” parameter in the batch file leaves the output there for me to look at until I’m ready to dismiss it with a tap of the “Enter” key.

Three buttons, one click, and typing the addressing. YES.

You can do another script for returning to DHCP. I’ve got separate shortcuts for static and DHCP for each NIC, but you can modify the batch file to make the user type the NIC name and put it into the netsh command as well so you can use one shortcut for any NIC, if you prefer (this is where it helps to rename the NICs).

Here attached are my basic batch files for anyone who wants to try them out. PDF because wordpress won’t allow me to attach text files.

IP Static

IP DHCP

Wireless Static

Wireless DHCP

 

CWDP!

I sat the CWDP exam today. I didn’t really expect to pass, but I felt like I should give it a shot to keep the momentum going, and I was pleasantly surprised to walk away successful!

CWNP Safari, Today at 7.52.46 PM

I was due to renew my CWNA in June, so this is a better refresh than doing the CWNA again. I had to make a conscious effort to step away from Wi-Fi for a while to complete my CCIE (R/S), but I am glad to be back on wireless.

If you’re working on completing the CWDP certification, I highly recommend the CWDP study guide from the PW0-250 version of the exam, from 2011 (Amazon: here) This is the second newest version and includes a much higher level of detail than the current book. Frankly, I didn’t feel like the current study guide covered nearly enough info to feel like I would have earned the cert.

So… onto CWAP! I think CWAP will help fill in a lot of the gaps I feel like I have in my understanding; and I think it will align well with the way I think and troubleshoot. I’m looking forward to spending more time with some PCAPs!

Mining Borer Wireless Connectivity

See this?

When I started working in mining, all of the mines I worked with were using Cisco APs mounted on the borer to facilitate a network connection to the less mobile equipment. The root of the link is usually the end of an extensible conveyor belt, and the PLCs controlling the belt communicate with the PLCs on the borer. There are things like safety interlocks which shut down the belt if the communication is interrupted. The extensible belt root AP is also uplinked to the rest of the mine network.

The Cisco bridges worked reasonably well at reasonable distances. On the bright side, EIRP was not a concern since we were far enough underground to prevent any sort of radiation into the open atmosphere, so very high gain antennas were standard operating procedure. I tried to find a pic, but just think parabolic dish.
But the borer moves away from the extensible as it does its work. We’re talking thousands of feet, and the cuts that the borer makes don’t stay level over the entire cut – so LoS is not maintained. On one hand, hard rock does an alright job bouncing signal along. On the other, throughput and data rate still faded quickly.

Can’t pull fiber and power in behind a borer – it will need to come back out the way it went in, and moving heavy machinery makes that idea asking for trouble. So ….

Yes, that IS two 1242 APs in a NEMA enclosure with back to back patch antennas, forming a repeater; AND I’m happy to see you!

It worked. No channel reuse (sorry Keith) because it was more important for unskilled miners to be able to move and install them with zero config. The entire set of root, repeaters and end bridge are hard coded to the same frequency.

Oh and if you could see to the ground in that picture, you’d find a car battery and some alligator clips. Fancy.

And yes that’s repeaters plural. The borer just got that far away. Sometimes it goes around a corner and then you need a repeater too.

So it’s not surprising to know that throughput was… not great. When VoIP was first brought out to the borers, we had a problem with the PLC interlock between the borer and the extensible belt dropping anytime someone picked up the phone. Neither the PLC interlock nor the VoIP call required much bandwidth, but there was enough latency or loss to starve the interlock, and VoIP tends to get prioritized by default. We had to do a little QoS reservation for the PLCs to keep them running at the expense of the calls.

So a year or two ago the instrumentation guys on site started playing with these:

This is a Ubiquiti PowerBridge M3. 3.5GHz licensed frequency? No problem – we’re thousands of meters underground! That’s a 20dBi integrated panel antenna, and a 2 SS radio.They use a root AP with a 26dBi antenna, and 12dBi repeaters, if necessary (sorry, I neglected to record the antenna specs on the Cisco gear when we installed them years back).

The results were rather stunning. At distances without repeaters, they can maintain near 100Mbps. With repeaters, they stay stable at distances of six to eight thousand feet without LoS due to elevation change. In the particular set I checked recently, the bridge was operating at the 32.5Mbps data rate (not throughput) over a 5Mhz wide channel, with 25dBm output power on the borer AP (100Mbps data rate was at 20MHz channel widths). The root has a 26dBi antenna, and the repeaters 12dBi. I didn’t have a chance to grab the models and power settings on those though.

Just for fun, here’s an idea of the perspective of the borer AP:

IMG_4655

So these are significant (though unquantified) improvements from the 1242 and 1310 Cisco bridges. Had I thought to keep specs from then, perhaps I would have been able to provide more than just anecdotal commentary. But there are a few points that make it clear we’re not making an apples to apples comparison:

  • 3Ghz is not Wi-Fi At a rough level, it also goes farther with less loss.
  • 5Mhz wide channels are also not Wi-Fi. While there are some benefits to narrower channels being a little more robust against interference, interference generally isn’t a concern in this environment
  • Ubiquiti isn’t exactly known for playing by the rules in general. There are some proprietary enhancements at play – the data sheet for the M3 lists it as a MIMO TDMA protocol device

I’m no Ubiquiti fan. I know their gear is a lower quality hardware, and that they’ve caused some trouble in the unlicensed space. But that’s not the point.

This is a unique environment. Disposable hardware is favoured – service contracts are less useful here, so spares are kept on the shelf for quick deployment, which means low cost is a bonus. Performance is priority, and we’re not constrained by the rules of the surface world.

The engineers and technicians on site like a lot of things about the Ubiquiti bridges, including the simple (yet effective) GUI, and the physical signal strength indicators on the bridge, which simplify aiming.

In the end, I guess it comes back to designing for requirements. Each solution may shine or fail horribly in different scenarios.

I’m not the most experienced Wi-Fi engineer, but I’m learning, and I found this an interesting educational experience. I am curious to hear what the more experienced member of the Wi-Fi community think about Ubiquiti and their place in these less traditional deployments.

 

Asymmetric routing is not for your LAN

UPDATED: Figured it out! Go to the bottom for the fix.


 

I know I’ll continue to see it everywhere. It just happens by accident. They didn’t realize what they were doing.

But they expect me to clean up the mess. Without downtime. Where’s my hammer?

Yet another unintentional asymmetric routing scenario! This one is bugging me a bit. I am in the process of fixing this client’s LAN topology and will remove the asymmetric routing in the process, however, I need to transition services onto a single firewall with as little downtime as possible, without being on site. Not uncommon so far.

Here’s the gist:

curse-you-asymmetric-routing
Asymmetric routing is ok… on the WAN, on the internet, and in some cases, if you INTEND to do it. Most of the time, it’s just EVIL.

Today, the remote access client tunnel terminates on the branch default gateway (10.1.1.1). I’m reconfiguring the newer firewall (10.1.1.2) to take over that role. The newer firewall was installed to terminate a VPN across the WAN (not internet) circuit to the remote branch. It is now destined to be the sole/main/one ring to rule them all firewall for the single subnet LAN.

Both firewalls are Cisco ASAs. I know that, in order to make this work until I can change the default gateway for the LAN to 10.1.1.2, I will need TCP state bypass on the default gateway (10.1.1.1), right? Right.

Lo’ and behold – TCP state bypass is already configured, for a LOT of traffic – because the remote branch (10.2.2.0/24) is hosting some critical services, and it is also accessible over a different gateway, which is on the same subnet as the default gateway.

Me: “Were you guys aware that you were using TCP state bypass for all traffic to the branch office?”

Client: “Hm, that sounds familiar… where have I seen that before…?”

Right. So the client googled something and figured out how to make it work, but really doesn’t understand what they’re doing (turning off core firewall functionality). But it worked!

I’ve been here before, I’ll be here again.

SO I configure the remote access VPN on the newer firewall (10.1.1.2), without state bypass, and can’t RDP to the DNS server. I expected that. In the diagram, steps 1-3 happen as expected, but step 4 doesn’t happen. Enable the state bypass, problem solved, TCP traffic works. Steps 1-5 work as expected.

Here’s the part that is bugging me though. I’ve made it work with a kludge, knowing full well that I’m going to fix the core problem soon. But I haven’t had time to really figure out what the problem is…

VPN clients couldn’t do name resolution. UDP lookups to port 53 were not getting a response. Since I am working remotely from the site, embedded PCAP is the best tool I have.

PCAP on new ASA (10.1.1.2) inside interface indicates that the DNS query from the VPN client is transmitted on the inside interface to 10.1.1.80. No response is received from the DNS server.

PCAP on default gateway ASA (10.1.1.1) inside interface indicates that the DNS server is, in fact, sending a response to the client at 10.10.10.X, and it arrives INBOUND on the inside interface. An outbound flow is not captured, matching the PCAP on the new ASA.

So looking back at the diagram, steps 1-3 are fine. Step 4 never proceeds.

So… UDP shouldn’t have the same asymmetric routing issue as TCP right (it is TCP state bypass after all)? There is no “state” to check.

Hairpinning is indeed enabled on the default gateway ASA. Informational level logging didn’t indicate anything interesting, but it would still seem that this firewall is dropping the traffic.

Fix? Static routing entry on the DNS server (10.1.1.80) for 10.10.10.0/24 via 10.1.1.2. (FOR SHAME BRENNAN!). 

I know, I would NEVER leave this in production. But I needed to confirm quickly where the issue lies.

At the moment, the reason for the drop is not coming to me. Let me know in the comments, or on twitter (@bmroute) if you have an idea.


 

FIX: Credit to my mentor, CCIE Security @rwatt13. Default DNS inspection treats the UDP query/response similarly to a TCP state in that the ASA doesn’t like to see a DNS response without a query first. Disabling DNS guard on the gateway ASA fixed the problem. See some explanation of DNS guard here.

YES, I REMOVED THE STATIC ROUTE FROM THE WINDOWS SERVER. I’m not an animal.