Pete's Packet

Limitless

Archive for the ‘Best practices’ Category

Troubleshooting the Catalyst 6500

Posted by Peter Kurdziel on January 12, 2010

Troubleshooting Hardware and Common Issues on Catalyst 6500/6000 Series Switches Running Cisco IOS System Software


Troubleshoot Error Messages in the Syslog or Console



The show diagnostic sanity Command



Supervisor Engine or Module Problems


     
Supervisor Engine LED in Red/Amber or Status Indicates faulty

     
Switch Is in Continuous Booting Loop, in ROMmon mode, or Missing the System Image

     
Standby Supervisor Engine Module Is Not On Line or Status Indicates unknown

     
Show Module Output Gives “not applicable” for SPA Module

     
Standby Supervisor Engine Reloads Unexpectedly

     
Even After You Remove the Modules, the show run Command Still Shows Information About the Removed Module Interfaces

     
Switch Has Reset/Rebooted on Its Own

     
DFC-Equipped Module Has Reset on Its Own

     
Troubleshoot a Module That Does Not Come On Line or Indicates faulty or other Status

     
Inband Communication Failure

     
Error “System returned to ROM by power-on (SP by abort)”

     
Error: NVRAM: nv->magic != NVMAGIC, invalid nvram

     
Error: Switching Bus FIFO counter stuck

     
SYSTEM INIT: INSUFFICIENT MEMORY TO BOOT THE IMAGE!


Troubleshoot CatOS to Cisco IOS Software or Cisco IOS Software to CatOS Conversion


     
Problem when User Attempts to Access the NVRAM After Cisco IOS to CatOS Conversion

     
Unable to Boot with Cisco IOS Software when User Converts from CatOS to Cisco IOS


Interface/Module Connectivity Problems


     
Connectivity Problem or Packet Loss with WS-X6548-GE-TX and WS-X6148-GE-TX Modules used in a Server Farm

     
Workstation Is Unable to Log In to Network During Startup/Unable to Obtain DHCP Address

     
Troubleshoot NIC Compatibility Issues

     
Interface Is in errdisable Status

     
Troubleshoot Interface Errors

     
You Receive %PM_SCP-SP-3-GBIC_BAD: GBIC integrity check on port x failed: bad key Error Messages

     
You Get COIL Error Messages on WS-X6x48 Module Interfaces

     
Troubleshoot WS-X6x48 Module Connectivity Problems

     
Troubleshoot STP Issues

     
Unable to Use Telnet Command to Connect to Switch

     
Giant Packet Counters on VSL Interfaces


Power Supply and Fan Problems


     
Power Supply INPUT OK LED Does Not Light Up

     
Troubleshoot
C6KPWR-4-POWRDENIED: insufficient power, module in slot [dec] power
denied or %C6KPWR-SP-4-POWRDENIED: insufficient power, module in slot
[dec] power denied Error Messages


     
FAN LED Is Red or Shows failed in the show environment status Command Output

     
“Diagnostic level complete” causes a crash on 6500


Cisco Support Community – Featured Conversations



Related Information

Catalyst 6500/6000 Switches ARP or CAM Table Issues Troubleshooting


Troubleshoot ARP or CAM Related Issues


     
Loss of Dynamic MAC Addresses with Distributed Switching

     
CEF Drops Packets at Regular Intervals

     
Switch Filter All-Zero MAC Addresses from the CAM Table

     
Unicast Flooding in the Network Every 5 Minutes

     
ARP Issues in Hybrid CatOS

     
Error EARL-2-EARL4LOOKUPRAMERROR During the CAM Table Lookup

     
Static CAM Entries Lost After Supervisor Switchover

     
%ACL-5-TCAMFULL: acl engine TCAM table is full

     
Ping Issues Occur when the MSFC Does Not Respond to the ARP Request in Catalyst 6500 Series Switches

     
Multiple Entries in MAC Address Table

     
Virtual IP Address Used by Microsoft Load Balancing is Not Reachable


Troubleshooting Input Queue Drops and Output Queue Drops


Processing and Switching



Input Queue Drops


     
Troubleshoot Input Queue Drops


Output Queue Drops


     
Troubleshoot Output Queue Drops


Commands to Obtain More Information


     
show interfaces switching

     
show interfaces stats

     
ip accounting mac-address

     
show interfaces mac-accounting

More info here: http://www.cisco.com/en/US/products/hw/switches/ps708/prod_tech_notes_list.html

Posted in CATALYST, Real World, Troubleshooting | Leave a Comment »

look up vendors bases on OUIs and other MAC address prefixes

Posted by Peter Kurdziel on December 19, 2009

http://www.wireshark.org/tools/oui-lookup.html

Posted in Real World | Leave a Comment »

Advantage Now! Accelerating Partner Success in Small Business

Posted by Peter Kurdziel on November 20, 2009

My company being a Cisco Partner was invited to this event.  It looks very interesting and I am looking forward to going.

Details

Small and medium-sized businesses will be leading the recovery again as
optimism rises. Find out how you can capitalize on the opportunity that
is about to unfold. Exclusively for Cisco® partners, Advantage Now! Accelerating Partner Success in Small Business is a live event designed to help send your sales figures through the roof.

By registering now, you’ll give yourself the chance to learn:

  • How a complete Cisco Small Business Pro Series portfolio creates new opportunities for you
  • How two new Pro Series solutions can lead to even more revenue
  • How the Cisco Small Business Advantage can help you become more profitable
  • Watch Broadcasts from Top Cisco Executives and Small Business Experts

  • John Chambers, Chairman and CEO of Cisco
  • Andrew Sage, Vice President, Worldwide Small Business Sales, and Rick Moran, Vice President, Small Business Marketing, discuss Cisco’s small business strategy and vision
  • Gene Marks, small business consultant, business
    owner, author, and journalist, talks about 10 things smart business
    owners learned from the recession and why they’ll make more money in
    2010
  • And many more
  • We have the products NOW.

    We have the tools for you to sell them NOW.

    All you need to do is register NOW.

    Agenda

    Eastern Standard Time

    12:00 pm John Chambers Introduction

    12:05 pm “S” Vision LIVE by Rick Moran and Andrew Sage

    12:30 pm What’s New in Products and Services LIVE with local hosts

    1:15 pm What’s New in Channels and Marketing Update LIVE with local hosts

    2:00 pm Keynote – 10 Things Smart Business Owners Learned from the Recession…and How They’ll Profit in 2010 by Gene Marks

    2:45 pm Wrap

    Posted in Real World | Leave a Comment »

    Configuring and Troubleshooting Ethernet 10/100/1000Mb Half/Full Duplex Auto-Negotiation

    Posted by Peter Kurdziel on November 14, 2009

    This is a subject that still confuses many people.  It is best that you hard code your ports.

    More info here:
    http://www.cisco.com/en/US/partner/tech/tk389/tk214/technologies_tech_note09186a0080094781.shtml

    Posted in Best practices, Real World, Troubleshooting | 3 Comments »

    Netflow tools

    Posted by Peter Kurdziel on October 27, 2009

    Stager is a system for aggregating and presenting network statistics.
    Stager is generic and can be customized to present and process any kind
    of network statistics. The backend collects data and stores reports in
    a database, automatically handling the aggregation of hourly statistics
    into days, weeks, and months. The Web frontend presents data in tables,
    matrices, or plots. The reports are fully customizable, and their
    definitions are stored in an XML file.

    http://software.uninett.no/stager/

    The nfdump tools collect and process netflow data on the command line
    http://nfdump.sourceforge.net/

    NfSen is a graphical web based front end for the nfdump
    netflow tools.
    http://nfsen.sourceforge.net/

    Posted in Other, Real World, Security | Leave a Comment »

    Cisco Config Best Practices

    Posted by Peter Kurdziel on October 22, 2009

    Router configuration Best Practices

    DOCUMENT- DOCUMENT- DOCUMENT

    I can not stress that enough. Always get a copy of the current running config and keep it in a safe place. I had one router where a vendor configured it, I just happen to get a copy for my records. 5 months later they took a power hit and guess what, the vendor had never saved the config to NVRAM. Without my notes, this would have been very bad.

    To protect against a “smurf” attack use the following comand:
    !
    no ip directed-broadcast
    !

    Other easy security measures are:
    !
    no service tcp-small-server
    no service udp-small-server
    !

    You should have names that make sense. Just an IP or nothing at all makes troubleshooting much more difficult then it needs to be.

    To configure the host name:

    router(config) hostname california_wan ; note the lower case. Not all software can handle uppercase correctly so lower case is “safer”

    Remember that SNMP can get this name from the sysName variable so again, use a name that makes sense.

    Interfaces:

    Always.. always use a description for each interface. A very good idea with WAN links is to use the circuit number as part of the description. When you are on the phone trying to troubleshoot a down link, this small detail can be a lifesaver for you.
    Including the contact and phone number adds to your work load as this information tends to be rather transitory nowdays

    !
    interface serial 0.1 point to point
    description San Fransico to New York PVC, circuit 001BHAC56789-001
    ip address 123.456.789.1
    !

    You can see how easy this is to read and when you are trying to troubleshoot problems, this is what you want, nice clear descriptions

    Get into the habit of specifing the bandwidth even if it’s not needed. Some protocols like OSPF use the bandwidth to help figure out the metrics using this information.

    If your link is slower the 256K, you *may* want to use the following command to make more buffer available depending on the link load
    !
    no ip route-cache
    !

    Always configure a loopback address. This provides several positive things.
    OSPF will by default use the loopback as the router ID or use the highest IP number as the router ID. If you plan this right, you can make your OSPF IDs make sense
    10.10.10.1
    10.10.10.2
    10.10.10.3 etc

    Also with OSPF, each time a link “flaps” all the routers must recalculate the route changes. Since the loopback doesnt “flap”, the network will be more stable
    You can telnet to it without regard to whether the interface is “up” or not. SNMP polling is the same thing.
    A stable interface is very important to protocols like SNA which is very sensitive to time delays and outage. This also applies to DLSW, STUN and RSBR
    IP or PPP from a laptop if you find yourself in a bind.

    SNMP

    SNMP is one of those double edged swords. It can be very useful but dangerous to your peace of mind if not handled well. SNMP has two types of communities. Read Only and Read/Write. The read/write is the dangerous one. With this string, you are god on the router and there is not any password checking ( normally)
    Read/write SNMP is a way to get out of the nasty box of configuring the enable password and then promptly forgetting it or mistyping it. Not that this EVER happens ( dont ask how I know this one)

    It very easy to configure

    router(config)snmp-server community string RO; read only snmp string
    router(config)snmp-server community string RW; read/write snmp string

    Do NOT use common names, your name, words like sex and the like. There are dictionary based SNMP crackers out there so be careful with your choices. Better yet, get a cracker and look at the dictionary to get an idea of what is in them.

    I alway use the snmp-server chassis-id serial-number to ID the router so I can get the SN remotely.

    You can specify access lists to restrict the number of workstations with access to the SNMP info.

    !
    access-list 60 permit 123.456.789.1 0.0.0.0 ; limits access to a single IP
    !

    If you want to be very paranoid then consider the following comand

    !
    snmp-server trap-authenication
    !

    This sends traps to your management station whenever a invalid community string is tried. Fun huh?

    A very good friend is called SYSLOG. This is a great way to get a nice log file about things that happen to interfaces, events and debugging. I happen to use a Wintel syslog deamon from Kiwi software. There are many to choose from from both Wintel and Unix/linux
    To work with syslog, use the following commands.

    !
    logging on
    logging buffered
    !
    loggin 123.456.789.1 ; the IP address is the syslog management workstation
    !

    CDP

    Very useful to both you and hackers. So the rule is if you plan to use it( good idea), make sure you turn it OFF on any outside interfaces.

    !
    interface ethernet 0
    ip address 123.456.789.2 255.255.255.0
    !
    no cdp enabled
    !

    Telnet access is something many people ignore. It’s prefectly acceptable to lock down your telnet ports to some degree. Just dont make it too restrictive ( like having to hit a certain router then bounce back)
    use a strong password, again, not common names etc. Dont leave printouts of the running-config laying around. There are several password crackers that can be used to compromise your passwords if you give someone a chance

    !
    access-list 1 permit 1.2.3.0 0.0.0.255
    !
    line vty 0 4
    access-class 1 in
    login
    password xxxxx
    exec-timeout 5 0
    !
    source:http://www.tek-tips.com/faqs.cfm?fid=404

    Posted in Best practices, Real World | Leave a Comment »

    Operational Best Practices for the Cisco Catalyst 6500 Series

    Posted by Peter Kurdziel on October 22, 2009

    Operational Best Practices for the Cisco Catalyst 6500 Series

    INTRODUCTION

    Companies today place high demands on network infrastructure. The ability to maintain performance, availability, security, and manageability of those network devices is of paramount importance to the successful operation of their data center. With the wide-scale deployment of the Cisco® Catalyst® 6500 Series in networks worldwide, Cisco Systems® has been able to better understand the demands placed on the switching infrastructure. More importantly, Cisco Systems has been able to direct the development of the Catalyst 6500 family of switches to meet the challenges corporate networks place on these devices.

    This document will attempt to present a best practices guide for operations management for features that customers can implement on the Catalyst 6500 to meet these challenges. It will focus on the areas of performance, scalability, security, availability, and manageability. Examples of the command-line interface (CLI) will be used in strategic places to show how commands can be used to deploy the proper best practice guidelines. This document will predominantly use command examples from the Cisco IOS® Software CLI, although references to the Catalyst OS commands will be made where appropriate.

    More here: http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd805457cc.html

    Posted in Best practices, CATALYST, Real World | Leave a Comment »

    High Availability Performance Management: Best Practices White Paper

    Posted by Peter Kurdziel on October 22, 2009

    High Availability

    Performance Management: Best Practices White Paper


    Introduction

    Performance management involves optimization of network service response time and management of the consistency and quality of individual and overall network services. The most important service is the need to measure the user/application response time. For most users, response time is the critical performance success factor. This variable shapes the perception of network success by both your users and application administrators.

    More here: http://www.cisco.com/en/US/tech/tk869/tk769/technologies_white_paper09186a008014fbf3.shtml

    Posted in Best practices | Leave a Comment »

    Control Plane Policing Implementation Best Practices

    Posted by Peter Kurdziel on October 22, 2009


    Control Plane Policing Implementation Best Practices

    Introduction: Network Device Operations

    IP networks provide users with connectivity to networked resources such as corporate servers, extranet partners, multimedia content, the Internet, and any other application envisioned within IP networks. While these networks function to carry data plane (user-generated) packets, they are also created and operated by control plane and management plane packets. Unlike legacy network technologies such as ISDN, Frame Relay, and ATM that defined separate data and control channels, IP carries all packets within a single pipe. Thus, IP network devices such as routers and switches must be able to distinguish between data plane, control plane, and management plane packets to treat each packet appropriately.

    Read the rest here: http://www.cisco.com/web/about/security/intelligence/coppwp_gs.html

    Posted in Best practices, Multicast | Leave a Comment »

    IP Multicast Best Practices for Enterprise Customers

    Posted by Peter Kurdziel on October 22, 2009

    IP Multicast Best Practices for Enterprise Customers

    customer networks. Although many of the practices in this document were developed for Financial customers to deliver Market Data the general principles apply to any Enterprise Multicast Deployment. It describes ways to optimize multicast delivery according to basic design principals including:

    • Resiliency

    – Path diversity

    – Redundancy

    – Load sharing or splitting

    • Latency

    • Security

    These recommendations are consistent with the existing Solution Reference Network Designs (SRND) listed below. They should be consulted for further information.

    High Availability Campus Network Design-Routed Access Layer using EIGRP or OSPF: http://www.cisco.com/application/pdf/en/us/guest/netsol/ns432/c649/ccmigration_09186a0080811468.pdf

    General information about IP Multicast: http://www.cisco.com/go/ipmulticast

    Read the rest here: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6552/ps6592/whitepaper_c11-474791.html

    Posted in Best practices, Multicast | Leave a Comment »

     
    Follow

    Get every new post delivered to your Inbox.