Anyway, the fact that you have to use IPs brings up some interesting problems.  Let’s say you have a server in a DMZ that needs to query Google for some content.  Since you’re a hard-ass network guy like I am, you tell the admin that they have provide the data flow they want to use — source IP, destination IP, protocol, port.  They come back and tell you that they need their server to connect via HTTP to 74.125.45.100.  You put in the rules as given, but the IP has suddenly changed on you.

Google (and lots of other big sites) uses some tricks to keep the load down on their servers and to help with availability, and one such trick is to use round robin DNS, which rotates the A record so everyone doesn’t slam the same boxes.  You can query google.com once and get an address, but, when you query it again, you may get a different address.  That means that when your new rules don’t work, you have to check the logs, see what got denied, open that up, rinse, and repeat.  That sucks.

An easier way might be to create an object-group that includes IPs as you discover them.  You put in rules based on an object-group, then, when it fails, you just add to the object-group so you don’t have to put in any more rules.  The problem is that you’ll spend a lot of time building up a good baseline.  If only there were a way to get a list of IP addresses that Google uses.  Hmmm.  *segue*

Have you ever heard of SPF netblock records?  SPF is an email security mechanism that allows an email server to verify that an email message is coming from an authorized email source.  In other words, when a mail server receives mail, it can check to see if the sending server is actually allowed to send mail on behalf of the source domain.  It supposed to cut down on spam and whatnot, but I don’t follow it closely enough to know if it’s working.  The moral of the story is that is involves a list of IP addresses that an organization maintains; Google happens to be a participant in SPF.

If you query for the TXT record _netblocks.google.com, you get back a text record that looks like this.

[jac@holland ~]$ dig +short txt _netblocks.google.com
“v=spf1 ip4:216.239.32.0/19 ip4:64.233.160.0/19 ip4:66.249.80.0/20 ip4:72.14.192.0/18 ip4:209.85.128.0/17 ip4:66.102.0.0/20 ip4:74.125.0.0/16 ip4:64.18.0.0/20 ip4:207.126.144.0/20 ip4:173.194.0.0/16 ?all”

This record includes all IP addresses that Google says is authorized to send email from google.com.  That’s a lot of IP addresses, isn’t it?  It might make sense that this list might also be the definitive list of Google production IPs.

My company has used this TXT record in the past to open access to Google.  We had an app that needed to query Google maps, and one of our engineers was tired of nickel and diming it to death, so he found the SPF block and put them all in.  Works like a champ.

There are always dangers when you rely on information from somebody else, though, right?  Google’s usually pretty good about stuff like this, but what if you did the same for another company who only half-heartedly kept their records up-to-date?  You may only have half of their IPs in your object-gropu.  You might even wind up opening access to or from a cable modem system or from another company who bought the IP addresses.

I’ll also  note that there aren’t that many domains using this technique, so finding SPF netblock records may be a challenge.  It’s worth the time to do a simply query, though; it might save you some time.

Send any carved pumpkins questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

Welcome to the modern world.  A world of wonder.  A world of quickly-advancing technology.  A world where clusters of machines sit behind load balancers for scalability and availability.  A world where those clusters need access to other clusters.  A world where your firewall rulebase gets so big that it’s unreadable without some help.

Enough with the drama already.  I would say I hate the cheesy stuff, but I think my whole blog is nothing but cheesy stuff, right?  To the point.  Enterprise firewall configurations can get quite large with ACLs applied in different directions to different interfaces.  Our ACL entries number in the 6000 range, but the firewall we’re running says we’re only at 5% utilization in the ACLE memory space. That means that our not-top-of-the-line firewall is designed to handle 120k lines of ACLs.  That can be quite a handful to configure by hand.  There may be an easier-to-maintain solution, though.

Let’s say you have a cluster of servers behind your CSM that all need to access a database.   Since there’s a nice ASA, FWSM, or PIX between the servers and database (as there should be), you have to open up access for this connection.  Let’s say that you have four servers with the IPs of 192.168.100.101-104 that need access to 10.10.10.1 on the mySQL port (TCP/3306).

access-list LIST1 permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306
access-list LIST1 permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306
access-list LIST1 permit tcp host 192.168.100.103 host 10.10.10.1 eq 3306
access-list LIST1 permit tcp host 192.168.100.104 host 10.10.10.1 eq 3306

Where are your remarks?  Why don’t you document something for once in your life?

Anyway, that’s easy, right.  Four configuation lines isn’t so bad, but some of the server admins come to you one day and tell you that the company actually marketed the new web app and that tey are adding 37 more servers to the cluster.  Now the 37 new servers need the same rules, right?  The server dudes also tell you that, since the app has grown so much, the DBAs have set up a split-read-write scenario where the current database handles the reads and a new database handles the writes.  That’s 78 new rules (37 to the old and 41 for the new).  That’s a lot of rules.

Object-groups to the rescue.  An object-group is a logical group of objects (duh!) that you can use to create ACLEs.  You can create a group of hosts, a group of network, or a group of ports.  For our example, let’s create an object-group that includes all the hosts in the new huge cluster.

object-group network CLUSTER1
   description The Huge Cluster (that's what she said)
   network-object host 192.168.100.101
   network-object host 192.168.100.102
...
   network-object host 192.168.100.141

What do we do with it, though?  You treat it (almost) just like it was a host in an ACL.  Remember we wanted to open access to the old database on TCP/3306, right?

access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306

If you do a show access-list LIST1 now, you’ll see that a new rules has been added for each object in the object-group.  It should look something like this.

access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 eq 3306
   access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0)
   access-list LIST extended permit tcp host 192.168.100.102 host 10.10.10.1 eq 3306 (hitcnt=0)
...
   access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0)

Notice that the firewall created 41 rules for you out of your one configured line, but now the rules are indented. The indention means that the rules is generated automagically instead of by hand. Since you can only take out rules that you put in by hand, so you can’t take out the line allowing 192.168.100.123 access; it’s an all-or-nothing scenario.  Be aware of that.

You can use object-group for ports, too.  Let’s add to our example and say that the cluster will need to access the memcached instance on the database server as well.  Those processes run on TCP ports 15000 – 15100.

First we build an object-group for the ports we need.

object-group service DBPORTS
   description mySQL and memcached ports
   service-object tcp eq 3306
   service-object tcp range 15000 15100

Now let’s apply it to the ACL.

access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORT

What does the ACL look like now?  Well, it’s a Duesenberg.

access-list LIST1 permit tcp object-group CLUSTER1 host 10.10.10.1 object-group DBPORTS
   access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 3306 (hitcnt=0)
   access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15000 (hitcnt=0)
...
   access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15099 (hitcnt=0)
   access-list LIST extended permit tcp host 192.168.100.101 host 10.10.10.1 eq 15100 (hitcnt=0)
...
   access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 3306 (hitcnt=0)
   access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15000 (hitcnt=0)
...
   access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15099 (hitcnt=0)
   access-list LIST extended permit tcp host 192.168.100.141 host 10.10.10.1 eq 15100 (hitcnt=0)

That’s a lot of ACL entries for one configuration line, isn’t it? Let’s see. 102 ports times 41 servers is 4182 lines in the ACL. You can see how might be to your advantage to use object-groups at times.

Send any candy corn questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

I’ve got a real-world example for you today.  We have an ASA 5540 installed at a business unit with interfaces in multiple networks, including one containing the production servers and another containing the accounting servers.  The production network sits on a 7600 that’s not ours, so, to avoid IP conflicts, we are statically NATting connections into that network.  The 7600 has with many, many VLANs, and, since the firewall production servers are on different VLANs, there’s an interface VLAN between us.  Sounds pretty straightforward, but it just wasn’t working when we try to connect between the interfaces.

When we tried to connect from the accounting servers to the production gear, the firewall saw the SYN, built the outbound connection, sent the packet on, and waited.  Nothing back.  SYN timeout.  The vendor on the production side checked the routing.  Fine.  Checked the ACLs.  None installed.  When the (other) vendor ran TCPDump on the production servers, they saw the SYN landing and the SYN-ACK leaving, but it never got to the ASA.  We even looked at the inline IDS and still didn’t see the SYS-ACK hitting the firewall.  It was simply not getting passed on.

I got tired of walking people through stuff over the phone, so I drove up there to see what I could find.  When I checked the ARP table on the 7600, I noticed that the statically NATted IP we were serving was conveniently incomplete.  For those who don’t know, that means that the 7600 was ARPing for the address, but nothing was answering for it.  Obviously, our ASA should be answering, right?  To make the situation a little more dire, I did a debug arp (or something close) on the firewall and generated an ARP request; the firewall saw the request but just ignored it.  Ugh!

If you couldn’t tell by the title, it turns out that the solution was to enable proxy ARP.  It’s off by default for good reason, but here’s how to enable it.

no sysopt noproxyarp PRODUCTIONINTERFACE

Enabling proxy ARP, however, could be a security issue.  Any time you use the word “proxy”, there is a potential to spoof addresses, and, in this case, an attacker could (potentially) use the firewall to discover hosts that are on the other side of it.  That wouldn’t be good.

A more-secure solution is to use static ARP entries.  In our case, we added a static ARP entry on the 7600 that points our NATted IP to the MAC address of the firewall.  Now, when you ping the IP, the 7600 doesn’t ARP; it already has the MAC in the ARP table, so it just sends the packet on.  Since we only have one static translation in this case, it’s no big deal, but, if we had a whole class-C of addresses to NAT, there would be a management problem.

A part of me wants to do the simple thing and enable proxy ARP, but the vast majority of article, blogs, forums, lists, etc., that I’ve ready say to turn it off for security and efficiency purposes.  The more I think about it, though, the more Iwonder why proxy ARP needs to be enabled to make staic NATs work.  I looked back at an old PIX running 6.x,  and proxy ARP is on by default.  The same holds true for an FWSM running 2.x.  I’m going to have to ask Cisco what’s up with that.

Send any misconfigured subnet masks questions my way.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

To forward the port, I would use the static directive, but there are two ways to do that. I can either set up a one-to-one NAT or a port redirection. In the one-to-one NAT, you have a outside address that’s mapped directly to an inside address, and any traffic to that IP is passed to the inside host (if it passes ACLS, of course). One of the limitation, though, of using this setup is that you can’t use that IP as your PAT address, and, since I only have one IP, no other inside hosts would have a outside address to which to be NATted. The other method — port redirection — is a much better solution. In this setup, I actually forward a protocol/port on a outside address to a protocol/port on an inside address. Since there are other ports available on that outside address, the address is still available for other hosts to use as a NAT address.

In an enterprise, I would probably use an address out of my pool for the port forwarding, but, since I only have one address at home, I’ve got another decision to make. I can configure the static statement with an IP address or I can use the reserved word interface to indicate the IP that is on an interface. This is a great feature, actually, since my outside IP could potentially change without notice. I’m going to use that feature, too.

static (inside,outside) tcp interface 80 10.10.10.10 80

This is pretty simple, but I’ll explain.  The ASA will take any request that comes in on TCP/80 (HTTP) on its outside interface’s IP and forward it to TCP/80 of 10.10.10.10.  If my webserver ran on TCP/81 on my box, I could just change the last 80 to 81 to make it work.

The port is redirecting, but I still need to open the ACL.  When that’s done, everything should work as expected.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts

Anyway, we won’t be going over any configs this time, but we can get some in the future. I just wanted to go over some stuff that you may need to know to get an FWSM working.

The FWSM does not have any physical ports on it. It uses the backplane of the switch to get a foot in the networks that you need it to. What does that give you? Well, it’s ultrafast on the 256Gbps backplane very fast with a 5Gbps uplink to the backplane, which is potentially a lot better than the 1Gbps on a physical interface. It also lets you have more interfaces than you could ever have on a firewall appliance (I think you can have 128 interfaces…someone correct me). I don’t know any firewall appliance that has more than 20 or so interfaces. It also is good for your cabling guy since there are no cables. I love it since I’m the cabling guy, too.

The FWSM can be set up with several security contexts. That means that you can chop up the resources of the module into “virtual firewalls” that operate separately from each other. This is pretty cool, actually, since you can, say, make a firewall for each of your hosting customers so that configurations on one don’t affect the others. You can also set the thing up in single context mode so it acts just like a single firewall. You’ll have to do some research to figure out what you want to do, and there are a lot of possibilities.

You can also set each context up in either routed or transparent mode. Routed mode is the “traditional” way to set up a firewall, where each interface is a different IP network with packets being routed through the FWSM. Transparent mode is where the FWSM has two interfaces that bridge the same IP network. I’ve always used routed mode everywhere, but you may have some use for transparent mode.

You also have to do some stuff on the 6500 to get the FWSM working. You make a vlan-group (a list of VLANs) to present to the FWSM and then assign that to the module. If you have more than one FWSM, you can have multiple vlan-groups; different FWSMs can control access in and out of different networks. Remember the context thing? If you have an FWSM in multiple contexts, you present the module with all the VLANs on all the contexts and split them up when you get to the context config part.

No configs. This is weird, but you’ll need to get all this information straightened out before you can get an FWSM configured properly.

Aaron Conaway

I like to lean my head to the left, hit it with the palm of my right hand, and document what knowledge falls out.

Website - More Posts