Http timeout with DS6200

Hello,
I stopped receiving IO data from my Lenovo DS6200. Error log contains 

Thu Jan  9 12:54:52 2020: Thu Jan  9 12:54:52 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:06:32 2020: Thu Jan  9 13:06:32 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:18:12 2020: Thu Jan  9 13:18:12 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call

But sometimes data comes in. I changed polling period from 5 to 10 minutes, it didn't help.

Comments

  • Hi,

    send us logs.
    Any change in yoru environment on 16th of Dec? I see that before it worked. Any storage firmware upgrade etc?

    Note a short problem description in the text field of the upload form.

    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.stor2rrd.com

  • Hi Pavel,
    I sent you logs. I have no information about any changes on the storage on 15th or 16th Dec.
  • NoOne
    edited June 2020
    Hi there.
    We have the same problem with Lenovo DS6200.
    A lot of performance gaps, error messages in email:

    STOR2RRD Data Health Status event for device Lenovo-DS6200-1 Storage does not provide performance statistics longer than 6 hours.

    As I think it started to happen in December too. It might be connected with the firmware update.

    But after that we had some more firmware updates, but the problem is still there.
    And we have Lenovo DS2200 with the same firmware level, and no problem.

    STOR2RRD is 2.80.

    I have uploaded logs.

    Thank you.
  • Hi,

    try this:
    1. wait for time period when there is no data coming
    2. run this cmd (change IP/username/password)

    Does it provide any output?
    How long it takes? (you get time stats once cmd finishes)


  • NoOne
    edited June 2020
    Unfortunately, I have this result:
    $ time /usr/bin/wbemcli -nl -t -noverify ei 'https://monitoring:***@10.***:5989/root/smis:CIM_BlockStorageStatisticalData'
    *
    * /usr/bin/wbemcli: Http Exception: Server returned nothing (no headers, no data)
    *

    real 1m10.434s
    user 0m0.069s
    sys 0m0.115s



    And in the same time DS2200 returns good result in two seconds. Does it mean, that there are some issues with the storage array?

  • Hi,

    We already seen that, some storage timing our or return this.
    Try to upgrade firmware, it helped in some cases.
    Raise a call with the storage vendor, send output from above cmd from working and not working storage (which are ideally in same subnet to avoid firewall issues)


  • Hi.

    We opened an issue ticket with the vendor. As I understand, they confirmed the problem and escalated it to developers.

    Unfortunately, the deadlines for solving the problem have not been set.

    So, will be waiting for their response.

  • FYI, Lenovo had released new firmware for DS6200 (1.09 GN280R008), and it seems, that finally fixed the issue.
    We have updated the storage array firmware, and there are no any problems for more than one day.
  • thanks for letting us know.
  • Hi,
    We updated firmware, and the problem came back a week later.
  • Hi,

    just to make it clear, you have above firmware and the issue re-appeared?

    How long takes wbemcli command in example above?
  • We have 2 node. The first node does not provide any output, and the second node returns data in

    real    0m1.319s
    user    0m0.013s
    sys     0m0.004s

  • it looks good, send us logs if you can

    Note a short problem description in the text field of the upload form.

    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.stor2rrd.com

  • opps, did not read it all :)

    open the support case with the vendor to check why the first node does not provide data
  • NoOne
    edited December 2020
    Hi there.
    I have to agree with @kuznetsovmyu
    Unfortunately, the problem is back after one week.
    So we'd resumed previous case with the vendor.

  • @Pavel Vendor is asking to double the period between information capture for Lenovo DS6200.

    I changed crontab, but it seems that STOR2RRD still does all requests every 5 minutes.

    Is there any simple way to change the period to 10 minutes? (at least for testing purpose)
  • Hi,


    -rwxrwxr-x 1 stor2rrd stor2rrd 109101 May 18 14:56 bin/dothillperf.pl
    Gunzip it and copy to /home/stor2rrd/stor2rrd/bin/dothillperf.pl (755, stor2rrd owner)
    If your web browser gunzips it automatically then just rename it: mv dothillperf.pl.gz dothillperf.pl

    Assure that file size is the same as on above example


    1.
    umask 0022
    cd /home/stor2rrd/stor2rrd
    echo "export DOTHILL_SAMPLE_RATE=10" >> etc/.magic


    2. kill all runnning dothillperf processe
    ps -ef| grep dothillperf


    Now it will get data every 10 minutes


  • @Pavel thanks a lot. I will test and come back a bit later.
  • FYI, Lenovo had resolved the issue described in this discussion. But right now the hotfix is available only via a service request. It will be published in the next firmware service release.
  • Hi,

    thanks for such info, do you know which firmware will contain that fix?
    Can you find it out?

    Thanks!
  • Unfortunately, I don't have this information, but as I understand, the patch will be included in the next firmware release (when it will be released)
Sign In or Register to comment.