Internet Edge Architecture

xtecharc Route and Switch, Route and Switch, Security Leave a Comment

I have written before about some Internet designs, specifically SIP architectures when it comes to the Internet Edge. I recommended that these services stay on the perimeter, next to the firewall, and that you front end these with a router for QOS. To broaden this discussion, I want to talk about multi homed internet as well as firewall and other security services and how this all fits together.


Let’s start with the security services. These devices, specifically firewalls, URL filters, IPS, and VPN can be seen in many manufacturers as all-in-one solutions. This is not always the best idea, and in large deployments, it is typically a pretty bad one. All firewall manufacturers will boast about their speeds and feeds AND all of the services the firewall offers; they will tell you that this all-in-one solute solves rack space issues, power issues and management issues. The facts are that by putting all of these into a single platform, you have introduced a huge issue…yes the O in TOMES – the operational issue. Typically the SLA for remote VPN users is different than my business partner VPNs, which is still different than my Internet access. Because all of these have unique SLAs, I may not be able to make changes to the devices that control these at the same time, causing a change order nightmare when I need to make a simple change to my Internet firewall to add a NAT and access rule, or another VPN.


Solving this requires that you break apart your functions, and the larger the company the more you break this apart. The good news is that by breaking these apart you typically end up with smaller firewalls, each serving a separate function, also allowing for best of breed selections. I have many customers who have broken this into user VPN termination, vendor VPN termination and Internet firewall functions. Three distinct pairs of firewalls with three SLAs, allowing for intervention far more often.


So, how do you secure each of these with an IPS, which is required more than ever in today’s environments? The best way is to not impregnate your firewalls with this function. Just like anything else, the more load you add, the slower it goes. Move this to its own appliance behind all of your perimeter devices that take traffic. To reiterate a past article, your voice gateway for Internet SIP should avoid an IPS if at all possible. I would recommend wiring around the IPS for this device, in fact putting it on its own network.


You can put the IPS behind the three firewalls above in two methods:

  1. Put them all into the same VLAN, and as discussed in the article about VDCs, you can run a pair of VLANs to force the routing through the IPS.
  2. Put each of the firewall pairs in a separate network, ensuring traffic going from one firewall to another (lets say a user VPN to a vendor) passes through the IPS. To do this you would either need to use an 802.1q trunk to the IPS from the core switch, which contains all three SVIs and again using the VLAN pairs from the VDC article, or you can use 3 separate links with 3 pairs of interfaces on the IPS. Again, you many need to have a switch between the IPS and the firewalls with 3 separate VLANs just to allow for HA firewalls.


In the second option above, you will want to rely on the HA of the firewalls to reduce failover times. You can also tune your routing protocols should you want to failover to a secondary data center.


For URL filtering, you many want to include this functionality in the IPS, buy a large enough appliance that can service both of these loads. If this is not an option, there are many ways to do URL filtering, from cloud based, to firewall redirects or another inline appliance with the Internet firewalls. All of these work with the individual firewall pairs which require these functions, so the other firewalls do not get slowed down with needless processing.


DMZs could even have their own firewall or utilize your Internet firewall. Most utilize the Internet firewall for this, and if desired, pull this through their IPS using another pair of interfaces on the IPS.


Outside the fence:

If you look at the firewalls and other externally facing devices (SIP gateways) as a fence you have to now decide what outside the fence looks like. Many people want to put their IPS there; others are more concerned with ensuring constant Internet availability, both inbound and outbound.


For the IPS, I have one word – DON’T. Don’t put your IPS on the outside of your firewall, it will be looking at traffic that is derived from script kiddies and would have been blocked by your firewall to begin with. If you are curious as to what is occurring out in that wild jungle, use a SPAN port on a switch outside the fence and setup another device as an IDS (intrusion detection system), or you can even do this on your correctly sized IPS that you are using inside, again by spanning to the port and setting that port as IDS only.


Now for my ISP recommendations:

I am going to only talk about the more difficult situation, two different ISPs connecting to you where you want both inbound and outbound redundancy. Most of you have already read about how to do AS Path Prepending and how to ensure you don’t become a transit zone – I can cover that if you want (please leave comments below), but Cisco has some incredible papers, which are located here. What I want to talk about is the idea of what BGP routes you should take and why. If you have two providers, first never just take default routes, unless one of the links is just too slow and should only be used as a last resort. You should always take partial and sometimes full routes. I typically recommend partial.


For each ISP you should have a dedicated router in my opinion. There is a reason you split providers, for redundancy, you should mimic this with your hardware. Most providers will give you a /30 on the link between you and them, meaning you only have room for one device anyway. You are also able to buy smaller routers typically. As you will see below, using iBGP between the external routers is how you share the routes. iBGP unlike other IGP routing protocols, does not add costs between other iBGP peers, so all peers in the AS see the same way out as the best route. When working with two providers, I recomend connecting the routers directly over a dedicated link. When working with more than two, I recommend still using a dedicated interface, but consider a shared broadcast domain by connecting that dedicated interface to a VLAN on an external switch. Ensure you tune your routing hello timers as necessary, but I talk a bit more about your options on this below.


When should you take Partial routes?

First what are partial routes? Typically partial routes are those which are routes owned or managed directly from that provider. If that provider has a direct link to that route, they should advertise it to you. This allows you to go to those autonomous systems which are only 1 or 2 AS hops away. Anything further would come to you as a default route. By taking partial routes, you can typically lower the requirements on your routers memory.


Take partial routes when you have two or more well-known carriers and your traffic is mostly domestic. If each well-known carrier provides partial routes to you, you will get a small subset of the full Internet routing table, but you will get a huge percentage of the routes you truly care about. Your external routers (yes routers, don’t put this sort of BGP into your firewalls unless you like to write resumes) will choose the default route to insert into the table based upon the costs of the routes provided by the carriers. You can skew this by changing the weights or local preferences.


Full Internet Routes:

Only take the entire Internet routing table from two providers when one of the following is true.


  1. You do a lot of international business and want the best routes back to your international customers.
  2. You are using smaller carriers, and partial routes simply are not going to provide you enough routes to matter.


If you fall into number two, you will ask your ISPs for full routes and you will want to do the following to still get the benefits of partial routes.

  1. Ask for a default route from each of the providers anyway (you will see why in a minute).
  2. Filter the incoming routes based on AS Path length. Typically your smaller carriers are peered with large carriers, and those are the routes you really want. Create a filter, which looks for AS Paths that are no greater than 3 in length. These three would be the AS of your carrier, the AS of their upstream and the AS the upstream is attached to (their customers). You may need to adjust this to 2 if 3 is too large, as the upstream provider may peer to another provider which has more routes than you really care about. This is very carrier dependent. Also allow through your filter the default route from part 1 above. This gives you the gateway of last resort for everything else and becomes a defacto partial routing table.


How do you take care of getting this information to your firewalls?


First do not send all of these routes to your firewalls – they don’t care and they don’t’ need to process all of this (again, unless resume writing is fun). Only send the default routes to them. Again, two options.

  1. Setup OSPF or EIGRP with your firewalls and send the default route from each router down to them, ensuring that one costs more than the other to reduce equal cost multipath routing over two ISPs which would cause buffer issues on the receiving end as that router tries to reassemble packets and possible drops.
  2. Use HSRP on the routers and setup a static route on the firewalls to the HSRP address.


Second, connect your external routers together using iBGP, sharing routes they have learned. This way they each know that the other may have a better path to the destination. When a packet comes up from the ASAs to the preferred router (either HSRP or lower cost advertised default route), the router will examine its routing table and send the packet either out its ISP or across the wire to the other router to go out the other ISP as it may be closer to the destination. If you do not setup this cross connect, you have just lost all of the reason you put in the partial routes to begin with as one router will get all of the traffic and not know that the other router exists and won’t send traffic to it. I prefer to set this connection up as a dedicated interface between the routers, but you can use the shared network, which lies between the routers and the firewalls if you wish, but that puts more traffic on those interfaces facing downstream.


I always enjoy a good Internet edge design – they are sometimes the most challenging to get right, as there are so many competing objectives. I hope this helps.

Share this Post

Fatal error: Call to undefined function x_google_authorship_meta() in /home/allowu6/public_html/ on line 20