Http timeout with DS6200

kuznetsovmyu

Hello,
I stopped receiving IO data from my Lenovo DS6200. Error log contains 

Thu Jan  9 12:54:52 2020: Thu Jan  9 12:54:52 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:06:32 2020: Thu Jan  9 13:06:32 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call
Thu Jan  9 13:18:12 2020: Thu Jan  9 13:18:12 2020 command: /usr/bin/wbemcli -nl -t -noverify ei 'http://stor2rrd:Test_1234@10.200.8.59:5988/root/smis:CIM_BlockStorageStatisticalData' timed out after: 700 seconds /var/lib/stor2rrd/bin/dothillperf.pl:2446 : Interrupted system call

But sometimes data comes in. I changed polling period from 5 to 10 minutes, it didn't help.

Pavel

Hi,

send us logs.

Any change in yoru environment on 16th of Dec? I see that before it worked. Any storage firmware upgrade etc?

Note a short problem description in the text field of the upload form.

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

tar cvhf logs.tar logs tmp/*txt

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.stor2rrd.com

kuznetsovmyu

Hi Pavel,
I sent you logs. I have no information about any changes on the storage on 15th or 16th Dec.

NoOne

Hi there.

We have the same problem with Lenovo DS6200.

A lot of performance gaps, error messages in email:

STOR2RRD Data Health Status event for device Lenovo-DS6200-1 Storage does not provide performance statistics longer than 6 hours.

As I think it started to happen in December too. It might be connected with the firmware update.

But after that we had some more firmware updates, but the problem is still there.

And we have Lenovo DS2200 with the same firmware level, and no problem.

STOR2RRD is 2.80.

I have uploaded logs.

Thank you.

Pavel

Hi,

try this:

1. wait for time period when there is no data coming

2. run this cmd (change IP/username/password)

time /usr/bin/wbemcli -nl -t -noverify ei 'https://stor2rrdro:XXXXX@10.11.12.156:5989/root/dhs:CIM_BlockStorageStatisticalData'

Does it provide any output?

How long it takes? (you get time stats once cmd finishes)

NoOne

Unfortunately, I have this result:

$ time /usr/bin/wbemcli -nl -t -noverify ei 'https://monitoring:***@10.***:5989/root/smis:CIM_BlockStorageStatisticalData'
*
* /usr/bin/wbemcli: Http Exception: Server returned nothing (no headers, no data)
*

real    1m10.434s
user    0m0.069s
sys     0m0.115s

And in the same time DS2200 returns good result in two seconds. Does it mean, that there are some issues with the storage array?

Pavel

Hi,

We already seen that, some storage timing our or return this.
Try to upgrade firmware, it helped in some cases.
Raise a call with the storage vendor, send output from above cmd from working and not working storage (which are ideally in same subnet to avoid firewall issues)

NoOne

Hi.

We opened an issue ticket with the vendor. As I understand, they confirmed the problem and escalated it to developers.

Unfortunately, the deadlines for solving the problem have not been set.

So, will be waiting for their response.

NoOne

FYI, Lenovo had released new firmware for DS6200 (1.09 GN280R008), and it seems, that finally fixed the issue.

We have updated the storage array firmware, and there are no any problems for more than one day.

Pavel

thanks for letting us know.

kuznetsovmyu

Hi,
We updated firmware, and the problem came back a week later.

Pavel

Hi,

just to make it clear, you have above firmware and the issue re-appeared?

How long takes wbemcli command in example above?

kuznetsovmyu

We have 2 node. The first node does not provide any output, and the second node returns data in

real    0m1.319s

user    0m0.013s

sys     0m0.004s

Pavel

it looks good, send us logs if you can

Note a short problem description in the text field of the upload form.

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

tar cvhf logs.tar logs tmp/*txt

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.stor2rrd.com

Pavel

opps, did not read it all

open the support case with the vendor to check why the first node does not provide data

NoOne

Hi there.

I have to agree with @kuznetsovmyu

Unfortunately, the problem is back after one week.

So we'd resumed previous case with the vendor.

NoOne

@Pavel Vendor is asking to double the period between information capture for Lenovo DS6200.

I changed crontab, but it seems that STOR2RRD still does all requests every 5 minutes.

Is there any simple way to change the period to 10 minutes? (at least for testing purpose)

Pavel

Hi,

place there this file:
https://www.stor2rrd.com/download-static/dothillperf.pl.gz

-rwxrwxr-x 1 stor2rrd stor2rrd 109101 May 18 14:56 bin/dothillperf.pl

Gunzip it and copy to /home/stor2rrd/stor2rrd/bin/dothillperf.pl (755, stor2rrd owner)

If your web browser gunzips it automatically then just rename it: mv dothillperf.pl.gz dothillperf.pl

Assure that file size is the same as on above example

1.
umask 0022

cd /home/stor2rrd/stor2rrd

echo "export DOTHILL_SAMPLE_RATE=10" >> etc/.magic

2. kill all runnning dothillperf processe

ps -ef| grep dothillperf

Now it will get data every 10 minutes

NoOne

@Pavel thanks a lot. I will test and come back a bit later.

NoOne

FYI, Lenovo had resolved the issue described in this discussion. But right now the hotfix is available only via a service request. It will be published in the next firmware service release.

Pavel

Hi,

thanks for such info, do you know which firmware will contain that fix?

Can you find it out?

Thanks!

NoOne

@Pavel

Unfortunately, I don't have this information, but as I understand, the patch will be included in the next firmware release (when it will be released)