› ›

Http timeout with DS6200

kuznetsovmyukuznetsovmyu
in Lenovo
Hello,
I stopped receiving IO data from my Lenovo DS6200. Error log contains 

Thu Jan  9 12:54:52 2020: Thu Jan  9 12:54:52 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:06:32 2020: Thu Jan  9 13:06:32 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:18:12 2020: Thu Jan  9 13:18:12 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call

But sometimes data comes in. I changed polling period from 5 to 10 minutes, it didn't help.

Comments

  • PavelPavel
    Hi,

    send us logs.
    Any change in yoru environment on 16th of Dec? I see that before it worked. Any storage firmware upgrade etc?

    Note a short problem description in the text field of the upload form.

    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.stor2rrd.com

  • kuznetsovmyukuznetsovmyu
    Hi Pavel,
    I sent you logs. I have no information about any changes on the storage on 15th or 16th Dec.
  • NoOneNoOne
    edited June 2020
    Hi there.
    We have the same problem with Lenovo DS6200.
    A lot of performance gaps, error messages in email:

    STOR2RRD Data Health Status event for device Lenovo-DS6200-1 Storage does not provide performance statistics longer than 6 hours.

    As I think it started to happen in December too. It might be connected with the firmware update.

    But after that we had some more firmware updates, but the problem is still there.
    And we have Lenovo DS2200 with the same firmware level, and no problem.

    STOR2RRD is 2.80.

    I have uploaded logs.

    Thank you.
  • PavelPavel
    Hi,

    try this:
    1. wait for time period when there is no data coming
    2. run this cmd (change IP/username/password)

    Does it provide any output?
    How long it takes? (you get time stats once cmd finishes)


  • NoOneNoOne
    edited June 2020
    Unfortunately, I have this result: 
    $ time /usr/bin/wbemcli -nl -t -noverify ei 'https://monitoring:***@10.***:5989/root/smis:CIM_BlockStorageStatisticalData'
    *
    * /usr/bin/wbemcli: Http Exception: Server returned nothing (no headers, no data)
    *

    real    1m10.434s
    user    0m0.069s
    sys     0m0.115s



    And in the same time DS2200 returns good result in two seconds. Does it mean, that there are some issues with the storage array?

  • PavelPavel
    Hi,

    We already seen that, some storage timing our or return this.
    Try to upgrade firmware, it helped in some cases.
    Raise a call with the storage vendor, send output from above cmd from working and not working storage (which are ideally in same subnet to avoid firewall issues)


  • NoOneNoOne
    Hi.

    We opened an issue ticket with the vendor. As I understand, they confirmed the problem and escalated it to developers.

    Unfortunately, the deadlines for solving the problem have not been set.

    So, will be waiting for their response.

  • NoOneNoOne
    FYI, Lenovo had released new firmware for DS6200 (1.09 GN280R008), and it seems, that finally fixed the issue.
    We have updated the storage array firmware, and there are no any problems for more than one day.
  • PavelPavel
    thanks for letting us know.
  • kuznetsovmyukuznetsovmyu
    Hi,
    We updated firmware, and the problem came back a week later.
  • PavelPavel
    Hi,

    just to make it clear, you have above firmware and the issue re-appeared?

    How long takes wbemcli command in example above?
  • kuznetsovmyukuznetsovmyu
    We have 2 node. The first node does not provide any output, and the second node returns data in

    real    0m1.319s
    user    0m0.013s
    sys     0m0.004s

  • PavelPavel
    it looks good, send us logs if you can

    Note a short problem description in the text field of the upload form.

    cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

    tar cvhf logs.tar logs tmp/*txt

    gzip -9 logs.tar

    Send us logs.tar.gz via https://upload.stor2rrd.com

  • PavelPavel
    opps, did not read it all :)

    open the support case with the vendor to check why the first node does not provide data
  • NoOneNoOne
    edited December 2020
    Hi there.
    I have to agree with @kuznetsovmyu
    Unfortunately, the problem is back after one week.
    So we'd resumed previous case with the vendor.

Sign In or Register to comment.