I have a client that recently opened their second location in the Washington DC area. Both sites have fiber internet (5Gbps) provided by the same provider, Allied Telecom. During the contract discussions for the second site’s fiber connection I asked if there was a way for us to connect the two sites together (other than Site Magic or dedicated VPNs) and Allied confirmed they could provision a Point to Point PTP fiber link for us. The fee for this added service was $0/month as basically what they are doing is running a VLAN on top of the normal internet fiber connection at each site and that VLAN provides connection between the sites via VLANs the configure in their data center. As an aside, the overall speed at each site becomes a Dynamic circuit where by the 5Gbps bandwidth is SHARED between Internet access and the site to site PTP connection. This is likely why they can do this for $0/month.
Allied installed fiber and connected our Site2 location in mid-March. Their inbound fiber went to a box that “split” the fiber into two ports/fiber handoffs - one fiber was internet access which connected to the UXG-Pro WAN port and the other fiber was a layer2 connection to the other site. I tested the second fiber and found that my laptop would indeed get a LAN IP address at the other site, so the link was working!
How to permanently connect the sites - Layer 3 Switching Approach
I have been working on the design for how to connect the two sites permanently using our Ubiquiti switches and routers. Our networks at Site1 and Site2 are completely separate - separate routers, separate cloud key controllers, different subnets. My initial plan was to use Layer3 switching over the fiber link between sites as it is essentially just a “long ethernet cable”. I did a lot of reading on the subject and testing in a home office test lab but I could not get the layer 3 switching to work.
The L3 switching design I tried to implement was to create a new “Transit” VLAN at Site1 and Site2 with the same network and same VLAN ID - 192.168.2.0/24 VLAN2 - The gateway at one site would be 192.168.2.1 (the L3 switch SVI on that side) and the gateway at the other site would be 192.168.2.2 for that side’s L3 switch SVI. When creating the new network on both sides in the Unifi interface I could choose the option to have the routing for the new VLAN2 network be done by either the router (UXG Pro) or the L3 switch (USW-Enterprise-XG-24). For my preferred L3 routing design I chose the L3 switch so that I could use Layer3 Switching and keep this traffic off of my Internet router.
So after creating the transit VLAN, the last step was to tell each side how to reach the other network via the L3 switch (acting as a router) so I created a static route on each network site to point to the other side’s switch as the gateway for routing to the other network, in this case:
static route 1: 10.1.10.0/24 via 192.168.2.2
static route 2: 10.1.11.0/24 via 192.168.2.1
I created these static routes in the UNIFI controller web UI and it happily accepted the static route config to be applied to the L3 switch on each side of the PTP fiber link. But this setup never worked for me as I was never able to ping the other network.
I brought in Willie Howe and he worked on it with me in my test environment and we discovered that L3 switching in this scenario with Ubiquiti gear may not be possible at this time. One of the key issues currently with Ubiquiti and L3 switching is that there does not seem to be a way to configure Static Routes properly through the Unifi controller Web UI. You can create the route, but if you SSH into the L3 switch and do a netstat -r, you will see that the route is not added! In fact I later found errors in the logs during the provisioning process of the switch that may be the culprit.
Willie was very helpful to me in showing me how to look at the routing tables in the devices and also how to see what interfaces/VLANs are configured in each network device. While he could not just “fix” my issue, he taught me a lot and confirmed a few things I was not 100% sure about. Thank you, Willie!
How to permanently connect the sites - UXG Pro Router Approach
So back to the drawing board and I decided to try and configure the site to site link using our UXG Pro Routers on either side. I set-up the same VLAN2 Transit network 192.168.2.0/24 on both sides with 192.168.2.1 and 192.168.2.2 as gateways on Site1 and Site2, respectively. But this time the VLANs were created such that the UXG Pro routers would do the routing instead of the L3 Switches. Then I created the static routes and viola! Everything is happy.
FYI - I also configured the ports on the switches where the Point to Point fiber connected at each site to be VLAN2 NATIVE and to BLOCK ALL other VLANs. I believe this is necessary to prevent broadcasts like DHCP requests (and others) from being broadcast across the PTP fiber link. I did not want a computer at Site1 to accidentally get an IP address at Site2 and visa versa.
FYI2 - I think my transit VLAN could be changed to a /31 network which would have exactly two IP addresses for the two gateways on either side. Likely a better approach for security?
So in the end, I have two sites, each with their own Internet WAN connection, and also with a PTP link to the other site so that employees can access server resources at the other site as needed.
Willie and I thought this might be helpful to folks using Ubiquiti gear. I was disappointed that the L3 Routing feature in the Unifi interface is still very limited and especially displeased that static routes can be configured but they don’t get provisioned to the equipment properly which cost me many hours of time troubleshooting! But my hope is that Ubiquiti will add a more mature feature set in the near future to enable my scenario to work using L3 switching instead of the UXG Pro routers. This would allow me to switch from a “router on a stick” approach to a more streamlined, lower latency, faster implementation using L3 Switching.