Cisco edge switch timeouts

Pete
Pete
in SAN


The issue we are running into is in the errors logs snippet below. looks like load.sh times out which is preventing us from getting continues data from our Cisco edge switches.  Data pulled from the core switches seem to find and both are on the same IP subnet. Is there a setting we can tweak to address these gaps? 



 Errors:

Adding to menu : W : CISCO9513E: Health status
Adding SAN port:
end time       : Wed Nov 21 11:21:46 CST 2018
An error occured, check /opt/stor2rrd/logs/error.log and output of load.sh
 
$ tail -701 /opt/stor2rrd/logs/error.log
 
………….
Wed Nov 21 11:05:01 2018: Could not find ports for Fabric VSAN00014xxxxxxx /opt/stor2rrd/bin/san.pl:1883
Wed Nov 21 11:05:01 2018: Could not find ports for Fabric VSAN00015xxxxxxx /opt/stor2rrd/bin/san.pl:1883
^* matches null string many times in regex; marked by <-- HERE in m/^* <-- HERE ENCAPSPOOL*$/ at /opt/stor2rrd/bin/storage.pl line 16823.
^* matches null string many times in regex; marked by <-- HERE in m/^* <-- HERE ENCAPSPOOL*$/ at /opt/stor2rrd/bin/storage.pl line 16823.
………..
Wed Nov 21 11:18:38 2018: /opt/stor2rrd/bin/custom_ext.pl timed out after : 600 seconds
Wed Nov 21 11:21:46 CST 2018


Comments

  • Karel
    Karel
    Hi Pete,

    Send us logs pls. 
    Note a short problem description in the text field of the upload form.

    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir
    tar cvhf logs.tar logs etc tmp/*txt
    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.stor2rrd.com
  • Karel
    Karel
    Hi Pete,

    data collection takes more than 10 minutes.

    Mon Nov 19 09:50:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 10:00:38 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 10:30:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 11:30:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 12:00:06 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 12:10:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 12:20:04 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : No such file or directory
    Mon Nov 19 12:30:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 13:10:02 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : No such file or directory
    Mon Nov 19 13:20:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 13:40:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 14:20:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Mon Nov 19 15:40:03 2018: Data collection via snmp timed out after 600 seconds : /opt/stor2rrd/bin/sanperf.pl:91 : Interrupted system call
    Can you increase this timeout (max is 14 minutes)? 
    Use the following procedure:

    su - stor2rrd
    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir
    echo "export SAN_DATA_COLLECTION_TIMEOUT=840" >>  etc/.magic

    Change also crontab entry.
    From:

    # SAN agent
    #4,14,24,34,44,54 * * * * /opt/stor2rrd/load_sanperf.sh >/opt/stor2rrd/logs/load_sanperf.out 2>&1
    */10 * * * * /opt/stor2rrd/load_sanperf.sh >/opt/stor2rrd/logs/load_sanperf.out 2>&1

    To:

    # SAN agent
    #4,14,24,34,44,54 * * * * /opt/stor2rrd/load_sanperf.sh >/opt/stor2rrd/logs/load_sanperf.out 2>&1
    0,12,24,36,48 * * * * /opt/stor2rrd/load_sanperf.sh >/opt/stor2rrd/logs/load_sanperf.out 2>&1

    Let us know if that helped.
  • Pete
    Pete
    I made the changes and will give it a few hours to run through its paces. Thank you for quick assistance.
  • Pete
    Pete
    No changes to captured data. We still have major gaps in pulled data. 
    From first glance at the logs, I do not see timeouts as in the past, so not sure why these gaps only show up on the edges. 

  • Pavel
    Pavel
    Hi,

    I affraid we cannot do too much here, switches are not able to response to snmp queries within given time period.
    We are planing to do some optimalisation in the code at start of new year, it might help a bit.
    It is unfortunatelly all we can do for now :(
Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Categories

In this Discussion