Pete's Packet

Limitless

Archive for the ‘Troubleshooting’ Category

Troubleshooting BGP

Posted by Peter Kurdziel on February 29, 2012

Main Troubleshooting Flowchart

bgp_trouble_main.jpg

Troubleshooting BGP Neighbor Establishment

bgp_trouble_neighbor.jpg

Note: *Sample log messages to be checked when neighbor is not coming up:

BGP_SESSION-5-ADJCHANGE: neighbor[ip address] IPv4 Unicast topology base removed
  from session Peer closed the session
BGP_SESSION-5-ADJCHANGE: neighbor[ip address] IPv4 Unicast topology base removed
  from session Unknown path error

Note: **Example of ping with packet size and enable does not fragment bit in IP header:

Router#ping 10.10.10.2 size 1400 df-bit

Type escape sequence to abort.
Sending 5, 1400-byte ICMP Echos to 10.10.10.2, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/37/84 ms

Troubleshooting Routes Missing from the Routing Table

bgp_trouble_route_missing.jpg

Note: In the debug ip bgp x.x.x.x updates command, x.x.x.x is the neighbor to which the route should be advertised.

Troubleshooting Multihoming Inbound

bgp_trouble_multi_in.jpg

Troubleshooting BGP Route Advertisement

bgp_trouble_route_adv.jpg

Troubleshooting Multihoming Outbound

bgp_trouble_multi_out.jpg

Source:

http://www.cisco.com/en/US/tech/tk365/technologies_tech_note09186a008009478a.shtml#bgp_trouble_neighbor

 

Posted in BGP, Troubleshooting | Leave a Comment »

The Cisco Catalyst 6000/6500 Series switch receives the VLAN(s) not available in Port Manager error message when it attempts to create a VLAN

Posted by Peter Kurdziel on February 14, 2011

The Cisco Catalyst 6000/6500 Series switch receives the VLAN(s) not available in Port Manager error message when it attempts to create a VLAN

VERSION 2 Click to view document history

Core issue

Layer 3 LAN ports, WAN interfaces and subinterfaces, and some software features use internal VLANs in the extended range. You cannot use an extended range VLAN that has been allocated for internal use.

The same VLAN cannot be recreated if it is used internally by the switch. Issue the show vlan internal usage command in order to check the internal status of a VLAN.

 

Resolution

In order to resolve this issue, configure the internal VLAN allocation policy either in ascending or descending order, as shown:

Router(config)#vlan internal allocation policy ? 
              ascending Allocate internal VLAN in ascending order 
              descending Allocate internal VLAN in descending order 

During the configuration of the internal VLAN allocation policy, select one of these procedures:

 

  • Enter the ascending keyword in order to allocate internal VLANs from 1006 and higher.
  • Enter the descending keyword in order to allocate internal VLAN from 4094 and lower.                 

Note: The internal VLAN allocation policy is applied only after a reload.

Alternatively, if you cannot reload the switch to make the vlan allocation change, you can translate the higher dot1q vlan-id to a lower dot1q vlan-id not being used by internal processes.

Refer to the Configuring the Internal VLAN Allocation Policy section of Configuring VLANs for more information.

 

Device connected to switch

Another Switch

Error message

VLAN not available in Port Manager

Posted in Troubleshooting | Leave a Comment »

Configuration Register Values and their Meaning

Posted by Peter Kurdziel on September 17, 2010

Configuration Register Values and their Meaning

Configuration Register Setting

Router Behavior

0×102

·         Ignores break

·         9600 console baud

0×1202

·         1200 baud rate

0×2101

·         Boots into bootstrap

·         Ignores break

·         Boots into ROM if initial boot fails

·         9600 console baud rate

0×2102

·         Ignores break

·         Boots into ROM if initial boot fails

·         9600 console baud rate default value for most platforms

0×2120

·         Boots into ROMmon

·         19200 console speed

0×2122

·         Ignores break

·         Boots into ROM if initial boot fails

·         19200 console baud rate

0×2124

·         NetBoot

·         Ignores break

·         Boots into ROM if initial boot fails

·         19200 console speed

0×2142

·         Ignores break

·         Boots into ROM if initial boot fails

·         9600 console baud rate

·         Ignores the contents of Non-Volatile RAM (NVRAM) (ignores configuration)

0×2902

·         Ignores break

·         Boots into ROM if initial boot fails

·         4800 console baud rate

0×2922

·         Ignores break

·         Boots into ROM if initial boot fails

·         38400 console baud rate

0×3122

·         Ignores break

·         Boots into ROM if initial boot fails

·         57600 console baud rate

0×3902

·         Ignores break

·         Boots into ROM if initial boot fails

·         2400 console baud rate

0×3922

·         Ignores break

·         Boots into ROM if initial boot fails

·         115200 console baud rate

 

Posted in Troubleshooting | Leave a Comment »

Displaying BGP routes

Posted by Peter Kurdziel on September 15, 2010

show ip bgp neighbors <x.x.x.x> advertised-routes
This shows you the routes that you advertised to this peer.

show ip bgp neighbors <x.x.x.x> routes
This shows you the routes you received from this peer that made it through the inbound filters.

show ip bgp neighbors <x.x.x.x> received-routes
This shows you the routes you received from this peer, even those that were denied(filtered).

In order to use this last command you must configure:
 bgp <as>
 neighbor <x.x.x.x> soft-reconfiguration inbound

Posted in BGP, Troubleshooting | 1 Comment »

Cisco Notification Alert -7600-08/14/2010 16:52 GMT

Posted by Peter Kurdziel on August 15, 2010

Cisco Notification Service Alert:
________________________________________________________________________________

Cisco Notification Alert -7600-08/14/2010 16:52 GMT

Security Advisories-Cisco 7600 Series Routers-08/13/2010 03:34 GMT-08/14/2010 07:34 GMT

Title: Cisco Security Advisory: Cisco IOS Software TCP Denial of Service Vulnerability
Url: http://www.cisco.com/en/US/products/products_security_advisory09186a0080b4095e.shtml
Description: Cisco IOS Software Release, 15.1(2)T is affected by a denial of service (DoS) vulnerability during the TCP establishment phase.
Date: 2010-08-13 14:00:00.0

For more information; you can visit Cisco Security Advisories & Responses Index: http://www.cisco.com/en/US/products/products_security_advisories_listing.html

Posted in Troubleshooting | Leave a Comment »

In order for VoIP traffic to not be in violation of the RTP standards and best practices, even/odd pairing of ports for RTP and RTCP traffic for SIP ALG, Skinny and H.323 has been made available.

Posted by Peter Kurdziel on July 15, 2010

User Defined Source Port Ranges for PAT Overview

In order for VoIP traffic to not be in violation of the RTP standards and best practices, even/odd pairing of ports for RTP and RTCP traffic for SIP ALG, Skinny and H.323 has been made available.

Following is a scenario of what happens to VoIP traffic translated using PAT without user defined ports.

The first VoIP traffic getting translated using PAT, would request for port 16384 and would get to use port 16384 for its RTP traffic.

The second VoIP traffic stream getting translated using PAT would also request 16384 for its RTP. Since this port number is already in use by the first call, PAT would translate the 16384 source port for the second phone to 1024 (assuming the port was free) and this would be in violation of the RTP standards/best practices.

A third call would end up using port 1025 and others would increment from there.

Each call after the first call would end up having its inside source port translated to an external port assignment that is out of specifications for RTP, and this would continue until PAT binding fir the first call expires.

Problems associated with RTP traffic being assigned to a non-standard port by PAT:

Inability for compressed RTP (cRTP) to be invoked in the return direction, as it only operates on RTP flows with compliant port numbers.

Difficulty in properly classifying voice traffic for corresponding QoS treatment.

Violation of standard firewall policies that specifically account for RTP/TRCP traffic by specified standard port range.

Here is an example from a debug ip nat sip I did:
02:02:14: NAT: SIP: [1] translate embedded port 1029->5060

Here is an example from show ip nat trans:
Pro Inside global      Inside local       Outside local      Outside global
udp 192.1.271:1024 10.1.1.252:5060   192.168.8.13:5060 192.168.8.13:5060

Even Port Parity

Cisco IOS NAT SIP gateways normally select the next available port+1 for SIP fixup in the NAT translations. The NAT gateway does not check for even/odd pair for RTP/TRCP port numbers, and as a result issues may arise with SIP user agents that are strictly following the encouraged even/odd parity for RTP/RTCP port numbers.

Even port parity for SIP, H.323, and skinny is supported by default and it can be turned off forcing the odd RTP ports allocation.

User Defined Source Port Ranges for PAT: Example

The following examples shows how to assign a set of ports and associate a map to them.

ip nat portmap NAT-I

 cisco-rtp-h323-low
 appl sip-rtp startport 32128 size 128
 appl sip-rtp startport 32000 size 64
ip nat inside source list 1 pool A overload portmap NAT-I

Table 1 Macro Names and Ports

Macro Name

Ports

Application

cisco-rtp-h323-low

16384-32767

H.323

cisco-rtp-h323-high

49152-65535

H.323

cisco-rtp-skinny-low

16384-32767

Skinny

cisco-rtp-skinny-high

49152-65535

Skinny

cisco-rtp-sip-low

16384-32767

SIP

cisco-rtp-sip-high

49152-65535

SIP

Configuration Examples for Even Port Parity

Even Port Parity: Example

The following example enables even port parity for H.323.

ip nat service allow-h323-even-rtp-ports

The following example enables even port parity for SIP.

ip nat service allow-sip-even-rtp-ports

The following example enables even port parity for the skinny protocol.

ip nat service allow-skinny-even-rtp-ports

source: http://www.cisco.com/en/US/docs/ios/ipaddr/configuration/guide/iad_pat_pt_rng.html

http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6640/prod_white_paper0900aecd80597bc7.html

Posted in BGP, Troubleshooting | Leave a Comment »

BGP neighbor failover taking too long.

Posted by Peter Kurdziel on July 5, 2010

BGP neighbor failover taking too long.

Try lowering the timer values. BGP will use the lower value.
Try using neighbor ip-address fall-over [bfd | route-map map-name]
Try using OER/PfR

ICMP Echo Example
The following example, starting in global configuration mode, configures an active probe using an ICMP echo (ping) message. The 10.5.5.55 address is the target. No explicit configuration is required on the target device.

Router(config)# oer master
Router(config-oer-mc)# active-probe echo 10.5.5.55

TCP Connection Example
The following example, starting in global configuration mode, configures an active probe using a TCP connection message. The 10.5.55.56 address is the target. The target port number must be specified when configuring this type of probe.

Router(config)# oer master
Router(config-oer-mc)# active-probe tcp-conn 10.5.5.56 target-port 23

Posted in BGP, Troubleshooting | Leave a Comment »

troubleshooting high CPU utilization

Posted by Peter Kurdziel on February 22, 2010

For troubleshooting high CPU utilization in specific processes, refer
to the appropriate link:

  • ARP Input—ARP Input section of the document Troubleshooting High CPU
    Utilization Due to Processes
  • BGP Router—High CPU due to BGP Router Process section of the document
    Troubleshooting High CPU Caused by the BGP Scanner or BGP Router
    Process
  • BGP Scanner—High CPU due to BGP Scanner section of the document
    Troubleshooting High CPU Caused by the BGP Scanner or BGP Router
    Process
  • EXEC—High CPU
    Utilization in the EXEC and Virtual EXEC Processes
  • HyBridge Input—Troubleshooting High CPU Utilization Caused by the HyBridge Input
    Process on Routers With ATM Interfaces
  • IP Input—Troubleshooting High CPU Utilization Due to the IP Input
    Process
  • IP Simple Network Management Protocol (SNMP)—IP Simple Network Management Protocol
    (SNMP) Causes High CPU Utilization
  • LC ADJ Updater—What Causes High CPU Utilization in the LC Adjacency Updater
    Process on a Cisco 12000 Series Internet Router?
  • TCP Timer—TCP Timer section of the document Troubleshooting High CPU
    Utilization due to Processes
  • TTY Background—TTY Background section of the document Troubleshooting High
    CPU Utilization due to Processes
  • Virtual EXEC—High CPU Utilization in Exec and Virtual Exec
    Processes
  • Vtemplate Backgr—Virtual Template Background section of the document
    Troubleshooting High CPU Utilization due to Processes
  • Other processes—Other Processes section of the document Troubleshooting High
    CPU Utilization due to Processes

Source: http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a00800a70f2.shtml

Posted in Troubleshooting | Leave a Comment »

Hardware Troubleshooting for the Cisco 12000 Series Internet Router

Posted by Peter Kurdziel on February 22, 2010

Hardware Troubleshooting for the Cisco 12000 Series Internet Router

http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a00800949bb.shtml

Misleading Symptoms

There are a few issues that can be misinterpreted as hardware
problems, when, in fact, they are not. Some of the more common issues
are when the router stops responding or “hangs”. Another one is a
failure following a new hardware installation. It is very uncommon for
any of these symptoms to be caused by a chassis component. The table
below lists symptoms, explanations, and troubleshooting steps for these
commonly misinterpreted issues:

Symptom

Explanation/Troubleshooting

The Cisco 12000 hangs during normal operation

This is usually caused by software problems, but can also be caused by hardware. See Troubleshooting Router Hangs for this issue.

A new line card is not recognized

Use the Software Advisor (registered customers only)
tool to determine if the new card is supported in your current Cisco
IOS software version. If the LC is supported, then configure service upgrade all, save the configuration with the copy run start
command and power-cycle the router. Sometimes a reload is not
sufficient, but a power-cycle fixes the problem. If the new card is not
supported in your current Cisco IOS software version, verify that you
have enough route memory installed on the line card before upgrading
the Cisco IOS software version. For release 12.0(21)S, 256 MB of route
memory is required, especially if Border Gateway Protocol (BGP) is
configured with many peers and many routes.

The CPU utilization is running very high

While there are hardware problems that can cause this, it is much
more likely that the router is either mis-configured or something on
the network is causing the problem. See Troubleshooting High CPU Utilization on a Cisco Router to troubleshoot this issue.

Memory allocation errors are seen on the Gigabit Route Processor (GRP)

Memory allocation errors are almost never caused by hardware
problems. Troubleshooting tips for memory allocations errors are
located on the Troubleshooting Memory Problems page.

An increasing number of input drops is seen in the output of the show interfaces command

This is never due to a hardware issue with the router. See Troubleshooting Input Drops on the Cisco 12000 Series Internet Router to troubleshoot this problem.

An increasing number of ignored messages is seen in the output of the show interfaces command

One of the line cards is most likely overloaded. Follow the steps detailed in Troubleshooting Ignored Errors and No Memory Drops on the Cisco 12000 Series Internet Router.

Forwarding Information Base (FIB) error messages are seen on the GRP

Use the Cisco Error Message Decoder (registered customers only)
Tool to find information about the meaning of this error message. Some
of them point to a hardware issue on either the line card or a switch
fabric card (SFC or CSC); others indicate a Cisco IOS software bug or a
hardware issue on another part of the router. Some FIB and CEF-related
messages are explained in Troubleshooting CEF-Related Error Messages.

Inter Process-Communication (IPC)-related messages are seen on the GRP.

You can use the Cisco Error Message Decoder (registered customers only)
Tool to find information about the meaning of this error message. Some
of them point to a hardware issue on either the line card or a switch
fabric card (SFC or CSC); others indicate a Cisco IOS software bug or a
hardware issue on another part of the router. Some IPC-related messages
are explained in Cisco 12000, 10000, 7600, and 7500 Series Routers: Troubleshooting IPC-3-NOBUFF Messages.

The following error messages are seen on the GRP:

%GRP-3-FABRIC_UNI: Unicast send timed out (1)
%GRP-3-COREDUMP: Core dump incident on slot 1, 
error: Fabric ping failure

Fabric ping failures occur when either a line card or the secondary
GRP fails to respond to a fabric ping request from the primary GRP over
the switch fabric. Such failures are a problem symptom that should be
investigated. You can find more information about this issue at Troubleshooting Fabric Ping Timeouts and Failures on the Cisco 12000 Series Internet Router.

The following error message is seen on the GRP:

%GRP-3-UCODEFAIL: Download failed to slot 5<br>

The image that was downloaded to the line card has been rejected by
the line card. You can try to reload the microcode using the microcode reload
configuration command. If the error message recurs, try to upgrade the
MBUS Agent ROM, MBUS Agent RAM, Fabric-downloader using the upgrade all slot command as explained in Upgrading Line Card Firmware on a Cisco 12000 Series Internet Router. You can also refer to the symptom “A new line card is not recognized” in this table.

Posted in Troubleshooting | Leave a Comment »

Hardware Troubleshooting for Cisco 12000 Series Internet Router Line Card Failures

Posted by Peter Kurdziel on February 22, 2010

Hardware Troubleshooting for Cisco 12000 Series Internet Router Line Card

Highlights:

  • show context summary = slot crashes
  • show logging
  • show logging summary
  • show diag <slot>
  • show context slot <slot>
  • show led
  • diag 7 verbose <span style="font-weight: normal; font-style: italic;">tftp tftp://x.x.x.x ( this test will run 5 - 20 min and the card will not be usable durung the test)

  • diag <slot> previous

http://www.cisco.com/en/US/products/hw/routers/ps167/products_tech_note09186a00800cdd58.shtml

Posted in Troubleshooting | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.