Category Archives: Cisco

Upgrading a Cisco ASA firmware in CLI

I decided to write up the steps so I could always refer back to this if I get hit in the head really hard and forget, which is very likely to happen. Remember to always read the release notes of the firmware you are installing.

In this case I have Cisco ASA 5505 running newer 9.X firmware, and just want both the ASA and ASDM images to be the latest suggested. In this scenario I am using TFTP64 to copy the files over but if I had a Flash drive handy I would have went that route.

First I downloaded the newest images from Cisco, both for the ASDM and the ASA firmware.

Then I moved those over to my TFTP server directory. On the ASA I will run these commands to copy the files to flash, then set options to boot to those images.

VIP-ASA# copy tftp flash
Address or name of remote host: 192.168.19.10
Source filename: asa917-12-k8.bin
Destination filename:asa917-12-k8.bin

Accessing tftp://192.168.19.10/asa917-12-k8.bin…!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!                                                                           !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
26709020 bytes copied in 33.860 secs (809364 bytes/sec)

Now to copy over the ASDM image

VIP-ASA# copy tftp: flash
Address or name of remote host: 192.168.19.10
Source filename: asdm-771.bin
Destination filename: asdm-771.bin
Accessing tftp://192.168.19.10/asdm-771.bin…!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
26709020 bytes copied in 33.860 secs (809364 bytes/sec)

Great, our images are on the flash, now we need to set our boot variables.

VIP-ASA(config)# boot system flash:/asa917-12-k8.bin
INFO: Converting flash:/asa917-12-k8.bin to disk0:/asa917-12-k8.bin

Remove the old boot variable

VIP-ASA# show boot

BOOT variable = disk0:/asa917-12-k8.bin
Current BOOT variable = disk0:/asa917-12-k8.bin;disk0:/asa821-k8.bin
CONFIG_FILE variable =
Current CONFIG_FILE variable =
VIP-ASA# config t
VIP-ASA(config)# no boot system disk0:/asa821-k8.bin
VIP-ASA(config)# show boot

BOOT variable = disk0:/asa917-12-k8.bin
Current BOOT variable = disk0:/asa917-12-k8.bin
CONFIG_FILE variable =
Current CONFIG_FILE variable =
VIP-ASA(config)#

Set your ASDM image to the new one, and then check your ASDM boot image

config t
VIP-ASA(config)# asdm image disk0:/asdm-771.bin

VIP-ASA# show asdm image
Device Manager image file, disk0:/asdm-771.bin

Now save config, and reboot – That’s it!  Reading the release notes is super important to know what has changed, and if there is a certain firmware you need to be at before upgrading. In this example I upgraded to 9.1.7 and ASDM 7.7.1. If you were upgrading VIA USB it is basically the same config, except replace tftp with usb.

 

Cisco WLC AP cert issue: %DTLS-3-HANDSHAKE_FAILURE

Recently we were troubleshooting some network issues with a Cisco 1242  AP that suddenly stopped communicating with our WLC.

Controller firmware is 8.0.133

We consoled into the AP and found logs that looked like below. Then we logged into the WLC and saw similar logs.

*spamApTask0:  %LWAPP-3-PAYLOAD_MISSING: spam_lrad.c:6433  Join request does not contain BOARD_DATA payload
*spamApTask5:  %CAPWAP-3-DECODE_ERR: capwap_ac_sm.c:4702 Error decoding Join request from AP 00:26:0b:10:34:70
*spamApTask6:  %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:844 Failed to complete DTLS handshake with peer 10.192.112.241
*spamApTask2:  %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:844 Failed to complete DTLS handshake with peer 10.10.91.20
*spamApTask6:  %CAPWAP-3-DECODE_ERR: capwap_ac_sm.c:4702 Error decoding Join request from AP 00:26:0b:10:34:70
*spamApTask5:  %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:844 Failed to complete DTLS handshake with peer 10.10.75.23
*spamApTask0:  %LWAPP-3-DECODE_ERR: spam_lrad.c:2364 Error decoding join request from AP 00:19:aa:35:10:88
*spamApTask0:  %LWAPP-3-KEY_ERR3: spam_crypto.c:1630 The system is unable to free public key for AP 00:19:aa:35:10:88
*spamApTask0:  %LWAPP-3-PAYLOAD_ERR: spam_lrad.c:7931 Join request does not contain valid certificate in certificate payload – AP 00:19:aa:35:10:88

Seems the AP cert had expired. To get around this we had to enable a command in the WLC that ignored the AP cert. The happened because the Manufacturer Installed Certificate (MIC) has now become older than ten years and has expired. This will not be accepted now.

I SSH’d into the controller and ran the below command:

config ap cert-expiry-ignore mic enable

This allowed the AP to come back online immediately.

If you happen to be on 7.0.252.X the command and logs are different:

log example:

Failed to complete DTLS handshake with peer 10.32.41.96 for AP 00:1d:45:36:97:30
*spamReceiveTask: Sep 19 21:42:59.855: %DTLS-3-HANDSHAKE_FAILURE: openssl_dtls.c:631 Failed to complete DTLS handshake with peer 10.32.41.101 for AP 00:1d:45:56:b6:1c

Each log will of course be different due to IP/MAC. The commands to resolve this:

config ap lifetime-check {mic|ssc} enable

So:

config ap lifetime-check mic enable

config ap lifetime-check ssc enable

 

That should do it!

Cisco information:

https://supportforums.cisco.com/t5/wireless-mobility-documents/lightweight-ap-fail-to-create-capwap-lwapp-connection-due-to/ta-p/3155111

 

Cisco ASA VPN Spoke to Spoke communication in 8.3 and later

This configuration was in ASA 8.4

Spoke to spoke communication has always been super easy in ASA Site to Site VPNs. As long as your CRYPTO ACL has the remote subnets in it, and NO-NAT Statements are there  everything pretty much works.

The other day I had an issue getting it to work. After some research I was still struggling. All of my remote sites were in my Crypto ACL, my VPN was up and working to the hub, and any subnet behind the hub would work, but access to other IPSEC tunnels connected behind were not working. See rough sketch of the network below.

diagram

I checked Nat statements, looked great, but my traffic was not flowing. I decided to debug via ASDM this is the error I received.

asdm-error

Routing failed to locate next hop for ICMP then my outside (Louisville), and inside (Italy) address.

Other examples are:

Routing failed to locate next hop for TCP then my outside (Louisville), and inside (Italy) address.

Routing failed to locate next hop for UDP then my outside (Louisville), and inside (Italy) address.

Well, 192.168.17.0/24 does not live inside my firewall – it should be connected to the outside (US-Signal) VIA the VPN. Boom, that’s when it clicked. My nat statement is wrong, well not wrong, just missing. Since these connections are connecting to my outside network, and then going to my outside network – I need to create the nat statement with the source interface and destination interface being US-Signal.

A few things to note about the below statement – I put it at the top of my manual nat entries, and notice the interface – both are US-Signal my outside interface.

object network Louisville-Subnet
 subnet 10.26.0.0 255.255.0.0

object network Italy-Subnet
 subnet 192.168.17.0 255.255.255.0

nat (US-Signal,US-Signal) source static Louisville-Subnet Louisville-Subnet destination static Italy-Subnet Italy-Subnet no-proxy-arp route-lookup

As soon as I added this statement everything worked great. All of my spoke to spoke communication flowed through the hub perfectly.

 

 

 

 

Cisco ASA 8.4+ manual nat – the only way to nat!

Before learning the more about Manual or “Twice Nat” I would use individual object NAT (Auto NAT) for my incoming services, and use Manual NAT for my No-NAT or if I had to NAT VPN traffic before encryption (Policy NAT).

Recently though I started using it for everything. Once you get the hang of it, it is much more applicable to everyday NAT needs.

Something to note about Manual NAT:

  • Processed before Auto NAT (Nat under the object command)
  • Considers source, or source and destination together (Policy)
    • For example – I need to  NAT traffic to this IP, only when it goes to this network
  • Configured directly from global config
  • Uses objects only, cannot specify direct IPs
  • Can specify to come after auto NAT.

Lets get started with a few examples. A list of all examples is below:

Static NAT – Public address/server to Private address/service

Group/Range of services forwarded into private server

Port redirection

Dynamic NAT

Policy Based Nat

Something to always note – 8.3 and above firmware’s require you to put in the private or real ip address of destination , not the public or Natted address.

Static NAT – Public address/server to Private address/service

Lets say my internal web servers  is at 10.20.20.10. The external IP I am using is 23.4.3.10. I want to NAT in only HTTP (80) traffic to my server. No problem.

Lets create the our address Objects

object service OBJ-TCP-80
 service tcp source eq 80

object network OBJ-10.20.20.10
 host 10.20.20.10
object network OBJ-23.4.3.10
 host 23.4.3.10

Great! Now lets create our Manual nat rule to allow that traffic in.

nat (inside,outside) source static OBJ-10.20.20.10 23.4.3.10 service OBJ-TCP-80 OBJ-TCP-80

Thats it, we would create our ACL to allow traffic to the Private host (Remember that, big time change in 8.3) and that’s it. Our traffic would be natted from Public, to Private port 80.

The next example will be a 1 to 1 NAT from our private object created above, to our public object also created above.

NAT (inside,outside) source static OBJ-10.20.20.10 OBJ-23.4.3.10

Modify the ACL to allow traffic to 10.20.20.10 and traffic should make it to the host.

 

Forward in a group or range of services

In this example we will forward in a group of service objects. Lets say HTTP, HTTPS, and SSH . Still using our local/public hosts. You have two options, 1 create a service group full of existing objects, or create a group of Service-objects. Int the below example I will create a group of predefined objects. This is due to having so many different configurations of groups.

object service OBJ-TCP-80
 service tcp source eq 80

object service OBJ-TCP-443
 service tcp source eq 443

object service OBJ-TCP-22
 service tcp source eq 22

object-group service Web-services
group-object OBJ-TCP-80
group-object OBJ-TCP-443
group-object OBJ-TCP-22

object network OBJ-10.20.20.10
 host 10.20.20.10
object network OBJ-23.4.3.10
 host 23.4.3.10

So now our NAT rule:

nat (inside,outside) source static OBJ-10.20.20.10 23.4.3.10 service Web-services Web-services

 

Port redirection

Sometimes we have the need to make a port such as 8080 on the outside go to our websever on the inside at port 80. The below example shows how to do that. In this example we will forward in port 8080 on our public IP to port 80 on our private webserver. First we need to create the objects for the service and networking addresses, and then apply the nat rule – an don’t forget our ACL. To help visualize whats happening here look at the format of the rule:

nat (source interface,destination interface) source static object ((private) IP) object ( Natted (Public IP))  service Private-Service Public-Service

object service OBJ-TCP-80
 service tcp source eq 80

object service OBJ-TCP-8080
 service tcp source eq 8080

object network OBJ-10.20.20.10
 host 10.20.20.10
object network OBJ-23.4.3.10
 host 23.4.3.10

So now our nat rule:
nat (inside,outside) source static OBJ-10.20.20.10 23.4.3.10 service OBJ-TCP-80 OBJ-TCP-8080

 

Dynamic NAT

I like to usually do this through Auto nat, but you can most definitely do this through Manual.

object network OBJ-10.0.0.0/8
host 10.0.0.0/8

nat (inside,outside) source dynamic OBJ-10.0.0.0/8 interface

You could also specify “any” instead of the internal address object, or specify the public IP you want to be natted to instead of “interface”.

 

Policy Based manual NAT

Manual NAT is the only way I believe that Policy based natting is done. You would use this if you had to NAT traffic into some other IP when going to a certain destination address. In this example lets say we need to NAT traffic from 10.0.0.0/8 int 1.1.1.1 when going to destination 3.3.3.3. This comes up a lot in healthcare when both sides need to nat into a Public address so there are no address conflicts.

Lets first create our objects, then our Nat rule.

object network OBJ-10.0.0.0/8
 host 10.0.0.0/8
object network OBJ-3.3.3.3
 host 3.3.3
object network OBJ-1.1.1.1
 host 1.1.1.1

So now our nat rule:

nat (inside,outside) source static OBJ-10.0.0.0/8 OBJ-1.1.1.1 destination static OBJ-3.3.3.3 OBJ-3.3.3.3

This reads that whenever 10.0.0.0/8 is going to 3.3.3.3, nat 10.0.0.0/8 into 1.1.1.1. This might help:

nat (inside,outside) source static Private-IP Natted-IP destination static Real-destination Natted-Destination

So, if this was used for a VPN you would just create an Crypto-ACL and the source would be your Natted-IP, and destination would be your 3.3.3.3 or whatever address lives across the tunnel that you set as your NAT destination.

 

 

 

 

Cisco Errdisable and recovery options

Errdisable is an extremely cool feature on Cisco switches that can place a port into a disabled state due to some reason/errors on the port. There are many reasons a port can be disabled:
Duplex mismatch
Port channel misconfiguration
BPDU guard violation
UniDirectional Link Detection (UDLD) condition
Link-flap detection
Security violation
Port Aggregation Protocol (PAgP) flap
DHCP snooping rate-limit
Incorrect GBIC / Small Form-Factor Pluggable (SFP) module or cable

And many more. Here is Cisco’s Documentation :http://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/69980-errdisable-recovery.html

The beauty of this feature is that if I screw something up, or if for example a I configure Port security and there’s and error it will shut down the port so that horrible things like loops or security violations are not allowed. By default Err-disable will shut down the port and it will take a manual shut/no shut of the port.

Finding out what ports and why they were put into ERRDisable

It is very frustrating to see ports come online, and then get shut off for some unknown reason. We can find out why they were shut off with a few simple commands

to find out what ports might be having errdisable problems we can do a :

show interfaces status errdisable

This command will show us all ports that are currently shutdown due to errdisable and the reason why. You can also get more specific with the :

err-reason

show interfaces gig 1/0/12 status errdisable

to get more information just from that port.

You can of course also see what is happening through the logs or syslog showing something like this

%SPANTREE-SP-2-BLOCK_BPDUGUARD: 
   Received BPDU on port GigabitEthernet4/1 with BPDU Guard enabled. Disabling port.

Auto Recovery options

So how can we make this a temporary setting – what if I was putting a switch in a school, and I want to make sure that if someone plugs up another switch, and I see a BPDU, I shutdown the port and then want that port to come back online in x amount of time. There are two parts to that problem. 1, you have to set BPDU Guard on the port or whole switch. Once that is setup, it will automatically be put into Err-disable state. Now, to bring it out of that state automatically, we have to modify the err-disable recovery option, and the cause option (unless we want all causes to automatically come back up – which might not be good). There are a few commands to help us figure out what has been set already:

Show errdisable recovery

This command will report back to you any recovery options that have been set, and the default recovery value of 300 seconds.

recovery

Show errdisable detect

This command will show you if we are detecting this error. By default all should be detecting.

err-detect

So, lets say I only want BPDUguard to recovery iteself every 60 seconds. This is what I would do:

Config t

errdisable recovery cause bpduguard

errdisable recovery interval 60

This will effectively enable recovery only for BPDUguard, and will change ALL recovery times to 60 seconds.

The following is the show recovery after the change:

err-after-change

Errdisable is a great feature that Cisco implements in almost all of their switches. It can really save some pain if you incorrectly configure a etherchannel, or have a bad cable that is really sending a ton of CRCs.

Cisco Router IOS Policy-based NAT for VPN traffic

I thought I would blog on this. It could be useful for someone who might have an IOS router instead of an ASA and need to create a IPSEC Site-to-Site VPN to a remote peer, then NAT VPN traffic to a different address or subnet if needed, or the local subnets conflict with each other.

Here is a nice little Visio to kind of show what I am going for with the traffic:

vis

Because of duplicate subnets on both sides, I need to nat traffic going to 172.90.0.20 from 192.168.10.10, otherwise traffic should flow normally. How can I achieve conditional nat? By using a route-map and then natting only the traffic in the Route-map. So, lets get our VPN setup first. Remember, we add the NAT network or host IP to our interesting traffic ACL that will be used to define our Phase2

These are my commands:

ip access-list extended VPN-to-Remote
 permit ip host 10.255.232.10 host 172.20.0.192

crypto isakmp policy 50
 encr 3des
 authentication pre-share
 group 2
 lifetime 28800

crypto isakmp key … address 1.1.1.1 no-xauth

crypto ipsec transform-set Transform esp-3des esp-sha-hmac

crypto map Crypto 6 ipsec-isakmp
 set peer 1.1.1.1
 set transform-set Transform
 match address VPN-to-Remote

That pretty much gets the VPN up and going. Now for the interesting part – we need to create a new ACL, match my private 192.168.10.10 address and the destination address of the remote server, then match that ACL in my Route-map.

ip access-list extended Nat-for-VPN
 permit ip host 192.168.10.10 host 172.20.0.192

route-map VPN-to-REMOTE permit 10
 match ip address Nat-for-VPN
!

Great! So, we now have the route-map created.. so now what? We need to create a NAT statement that references my Route-Map. Then of course with any VPN we need to modify the “NO-NAT” ACL to include the traffic for both the 192.168.10.10, and the 10.255.232.10 to my remote destination.

ip nat inside source static 192.168.10.10 10.255.232.10 route-map VPN-to-HCN extendable

ip access-list extended NO-NAT
 deny   ip host 10.255.232.10 host 172.20.0.192
 deny   ip host 192.168.10.10 host 172.20.0.192

Now, if we try to access the remote side, does it work? Yes it does, but lets check to see if our nat is really working. It is! As you can see, 192.168.10.10 going to 172.20.0.192 is being natted into 10.255.232.10, but all other traffic gets natted out of the WAN interface.

nat1

Lets just check for translations of 10.255.232.10

2

Bingo, everything works great. Lets make sure that we are getting hits on our Route-Map.

route-map

Cisco Duplicate IP address 0.0.0.0 ERROR – IP Device Tracking/NMSP

Recently I was seeing this error pop up on many Windows desktop clients:

The system detected an address conflict for IP address 0.0.0.0 with the system having network hardware address Ed-Ef-A9-B8-CC-2E. Network operations on this system may be disrupted as a result. Mac will vary.

After some research I found http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/8021x/116529-problemsolution-product-00.html

To give some highlights : “Cisco IOS® uses the Address Resolution Protocol (ARP) Probe sourced from an address of 0.0.0.0 in order to maintain the IP device-tracking cache when IP device tracking and a feature that uses it is enabled (such as 802.1x) on a Cisco IOS switch.

If the switch sends out an ARP Probe for the client while the Microsoft Windows PC is in its duplicate-address detection phase, Microsoft Windows detects the probe as a duplicate IP address and presents the user with a message that a duplicate IP address was found on the network for 0.0.0.0

So we now know the issue is with IP Device tracking, but what the heck does this do? IP Device tracking keeps an active list of devices that are connected VIA ARP. The function has as Cisco put it “Always been around”, is extremely beneficial when using MAC ACLs or using 802.1x. Recently it has really been used with Network Mobility Services Protocol (NMSP), this feature manages communication between the mobility service engine and the wireless controller in newer switches.

So how it works – When a  link is detected, it sends unicast Address Resolution Protocol (ARP) probe with a default interval of 30 seconds; these probes are sent to the MAC address of the host connected on the other side of the link, and use Layer 2 (L2) as the default source the MAC address of the physical interface out of which the ARP goes and a sender IP address of 0.0.0.0, — Bingo there’s are default IP that pops up.

So how do we remove device tracking? Easy huh.. just “no ip device-tracking” – this currently gives an error in certain firmwares. Firmware 03.02.02.SE and below give, the error is:

% IP device tracking is disabled at the interface level by removing the relevant configs

So, you could upgrade to 3.3 and then use the no ip device-tracking command, or if you cannot upgrade still disable all the features of IP device tracking. To do this:

Under each interface use commands:

nmsp attach suppress

no ip device-tracking max

I would recommend using a range command to get all the ports at once. This has fixed the issue for me.

Cisco Router with AT&T DSL internet connection

I would say no one will find this useful, but I have DSL at home .. I know. The following config is how to add a Cisco router use AT&T DSL. I have done this many times but not in the last two years or so, I had trouble finding good documentation on google for exactly what I needed to do, so I thought I would share.

The main parts of the config needed are creating a dialer interface, setting the authentication and user/pass. Then associating that dialer interface with your Physical Ethernet interface, and lastly configuring the default route.

First lets configure the Dialer Interface

config t

interface Dialer1
ip address negotiated
ip mtu 1492
ip nat outside
encapsulation ppp
ip tcp adjust-mss 1460
dialer pool 1
dialer-group 1
ppp authentication chap pap callin optional
ppp chap hostname username@att.net
ppp chap password password
ppp pap sent-username username@att.net password password

Now lets associate that with a real interface:

interface FastEthernet0/1
no ip address
duplex auto
speed auto
pppoe enable group global
pppoe-client dial-pool-number 1
!

Next lets set our default route and point it out of the Dialer interface

ip route 0.0.0.0 0.0.0.0 Dialer1

Given we have all our user/pass correct in a minute you should be able to do a “Show IP int br” and see the ip address passed to us by our ATT DSL Modem in bridge mode on the dialer interface.

Cisco ASA 9.1+ Static Nat example

Below shows how to configure Static nat for a web server or some kind of application running on a internal host. Basically we are port forwarding port 80 from our public IP of 1.1.1.2 to port 80 of our internal IP at 10.1.1.2. In this example Auto Nat will be used. You could also use Manual nat, I have written another blog entry on this.  This is way different than 8.2 and below. Here we create an object and then modify the object with the Static port forward we want. I

In this example my ASA outside IP is 1.1.1.1, and I want the web server to answer on 1.1.1.2.

object network 1.1.1.2
host 1.1.1.2
exit

object network 10.1.1.2
host 10.1.1.2
nat (inside,outside) static 1.1.1.2 service tcp 80 80

Somethings to note- We could name the objects anything, I just chose to use the actual IP address. For example, you could do the command ” object network Webserver-Outside” and use that name to reference the outside IP address.

Next, if I want to allow access to 10.1.1.2 from the outside world, I will need an ACL.

access-list Outside-In permit tcp any 10.1.1.2 eq 80

access-group Outside-In in interface outside

Notice the internal IP specified in the ACL – that is there on purpose. Instead of referencing the External IP  you now reference the internal.

What if the outside address answering for my web server is the outside IP of the ASA?

No problem, just have to modify that one NAT entry. Instead of the public NAT object we use the “interface” keyword.

object network 1.1.1.2
host 1.1.1.2
exit

object network 10.1.1.2
host 10.1.1.2
nat (inside,outside) static interface service tcp 80 80

Cisco – Combing T1 interfaces to increase speed

Recently I was working on a project that had a very remote office that could not get high speed connections to its location. So, they wanted to combine 4 T1s that were already in the building to boost throughput. I had done this exact thing with many different ISPs but never Verizon/MCI . They do things just a bit different, so this entry is about what is needed to combine these links on Verizon’s network.

The key here is to use a Multilink Frame relay interface. This is a Virtual interface that will combine the individual interfaces. Its very similar to a BVI in concept. As Cisco says it: The Multilink Frame Relay feature enables you to create a virtual interface called a bundle or bundle interface. The bundle interface serves as the Frame Relay data link and performs the same functions as a physical interface. 

At the bottom of this entry all the parts of the config are listed

First we needed to create the MFR interface itself. The number is up to you, I created 34 – feel free to change that.

interface MFR34
mtu 4470 — MTU Verizon said to set
no ip address
no ip redirects
encapsulation frame-relay IETF – Encapsulation used.
frame-relay multilink bid u11111-11
frame-relay lmi-type ansi
!

Next we need to create the sub interface , which will be used as our DLCI and our L3 interface.
interface MFR34.500 point-to-point
description multilink:MLFR:3xT1
bandwidth 6000
ip address x.x.x.x x.x.x.x
ip nat outside
snmp trap link-status
no cdp enable
no arp frame-relay
frame-relay interface-dlci 500 IETF
!

Then we will bond the interface to our MFR interface.

interface Serial0/0/0:0
mtu 4470
bandwidth 1536
no ip address
 encapsulation frame-relay MFR34
load-interval 30
cdp enable
no arp frame-relay
!

The T1 and MFR config are below:

controller T1 0/0/0
cablelength long 0db
channel-group 0 timeslots 1-24
!
controller T1 0/0/1
cablelength long 0db
channel-group 0 timeslots 1-24
!
controller T1 0/0/2
cablelength long 0db
channel-group 0 timeslots 1-24
!
controller T1 0/0/3
cablelength long 0db
channel-group 0 timeslots 1-24
!

interface MFR34
mtu 4470
no ip address
no ip redirects
encapsulation frame-relay IETF
frame-relay multilink bid u11111-11
frame-relay lmi-type ansi
!
interface MFR34.500 point-to-point
description multilink:MLFR:3xT1
bandwidth 6000
ip address x.x.x.x x.x.x.x
ip nat outside
snmp trap link-status
no cdp enable
no arp frame-relay
frame-relay interface-dlci 500 IETF
!

interface Serial0/0/0:0
mtu 4470
bandwidth 1536
no ip address
no ip redirects
no ip proxy-arp
encapsulation frame-relay MFR34
load-interval 30
cdp enable
no arp frame-relay
!
interface Serial0/0/1:0
mtu 4470
bandwidth 1536
no ip address
no ip redirects
no ip proxy-arp
encapsulation frame-relay MFR34
load-interval 30
no arp frame-relay
!
interface Serial0/0/2:0
mtu 4470
bandwidth 1536
no ip address
no ip redirects
no ip proxy-arp
encapsulation frame-relay MFR34
load-interval 30
no arp frame-relay
!
interface Serial0/0/3:0
mtu 4470
bandwidth 1536
no ip address
no ip redirects
no ip proxy-arp
encapsulation frame-relay MFR34
load-interval 30
no arp frame-relay
!