Capacity of a VSP-G Pool

mbayard

Hi,

I'm not sure to understand what is displayed in stor2rrd regarding pool capacity.

From Hitachi Command Suite : Yesterday

From Hitachi Command Suite : Today (8TB more in Used Capacity)

From stor2rrd 2.50 :

The Capacity in HCS seems to be = Used + Free in stor2rrd.

However both values in stor2rrd seem wrong : Used max is 114.03TB while the actual Used in HCS is 128.75TB

Moreover, the "blank period" of the stor2rrd graph seems to be taken into account for the average calculation.

The 4 weeks graph is also strange, it doesn't show any increase while during the 24h the used capacity increased by 8TB at least (seen in HCS).

Am I missing something ?

Lukas

Hello,

it is possible that configuration has not been done for a long time. Please this output


cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

ls -l data/<STORAGE_ALIAS>/IOSTATS/configuration/

where <STORAGE_ALIAS> is name of storage in etc/storage-list.cfg


Thank you

mbayard

Indeed, it seems that files are not very fresh :

[stor2rrd@xxx stor2rrd_BEMUTU]$ ls -l data/xxx/IOSTATS/configuration/

total 332
-rw-r--r-x 1 stor2rrd stor2rrd  10608 Dec 23 23:52 config-HOST1.txt
-rw-r--r-- 1 stor2rrd stor2rrd   1728 Feb 27 11:48 config-HOST1.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd  60608 Dec 23 23:55 config-HOST2.txt
-rw-r--r-- 1 stor2rrd stor2rrd   3540 Feb 27 12:03 config-HOST2.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd   3936 Dec 23 23:36 config-HOST3.txt
-rw-r--r-- 1 stor2rrd stor2rrd   3540 Feb 27 12:17 config-HOST3.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd 188968 Dec 24 00:00 config-LDEV.txt
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:34 config-LDEV.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd    199 Dec 24 00:00 config-POOL1.txt
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:35 config-POOL1.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd    171 Dec 24 00:00 config-POOL2.txt
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:35 config-POOL2.txt-tmp
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:35 config-POOL3.txt-tmp
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:35 config-POOL4.txt-tmp
-rw-r--r-x 1 stor2rrd stor2rrd   5337 Dec 24 00:00 config-PORT.txt
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:36 config-PORT.txt-tmp
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 12:18 config-SNAPSHOT1.txt-tmp
-rw-r--r-- 1 stor2rrd stor2rrd    158 Feb 27 11:34 config-STORAGE_FIRMWARE.txt
-rw-r--r-x 1 stor2rrd stor2rrd    404 Dec 24 00:00 config-STORAGE.txt
-rw-r--r-- 1 stor2rrd stor2rrd     27 Feb 27 11:34 config-STORAGE.txt-tmp

Lukas

Hello,

this problem has already been reported by several users. Try these commands: do they work?

raidcom -login <login> <password> <-Ihorcm_id>

raidcom raidcom get pool <-Ihorcm_id>

raidcom -logout <-Ihorcm_id>

where <login> is user name, <password> is password, <-Ihorcm_id> is number horcm.conf file

example

for line in etc/storage-list.cfg

VSPG-400:VSPG:vspg_SVP_host.example.com:stor2rrd:JT4mXVI9N0BgCg==:/etc/horcm1.conf

raidcom -login stor2rrd xorux -I1

raidcom raidcom get pool -I1

raidcom -logout -I1

Thank you

mbayard

Hi Lukas,

Indeed there is a problem with these commands, I think it's a problem with the user I use ... 

When I tried to login with my "stor2rrd" user (created as specifier in www.stor2rrd.com/userfiles/file/Hitachi-arrays-setup.pdf), I had a timeout, note that it worked fine in the past ...

I created another user in the same groups and the login worked perfectly :

[stor2rrd@xxx tmp]$ raidcom -login -I19
User for Serial#[xx] : martin
Password :
[stor2rrd@xxx tmp]$

I deleted the user "stor2rrd", and when I tried to login with : timeout again :

[stor2rrd@xxx tmp]$ raidcom -login -I19 
User for Serial#[xx] : stor2rrd
Password :
raidcom: [EX_EWSTOT] Timeout waiting for specified status

While if I try to login with a user that doesn't exist (and never existed), I have a clear error message :

[stor2rrd@xxx tmp]$ raidcom -login -I19
User for Serial#[xx] : azerty
Password :
raidcom : User authentication has failed(stat = 2) on Serial#(xxx).
raidcom: [EX_ENAUTH] Authentication failed with User

I then recreated "stor2rrd" and "stor3rrd" :

And when I try to login :

[stor2rrd@xxx tmp]$ raidcom -login -I19
User for Serial#[xx] : stor2rrd
Password :
raidcom: [EX_EWSTOT] Timeout waiting for specified status

[stor2rrd@xxx tmp]$ raidcom -login -I19
User for Serial#[xx] : stor3rrd
Password :

Only "stor3rrd" worked, it's like the "stor2rrd" user was not usable anymore at all ... I could simply create another user and update my stor2rrd configuration, but what will happen if the new one gets blocked aswell ? Any advise ?

Note that the tests were made on the stor2rrd servers where all the tasks are running, so maybe the checks from the crontab interfered ...

Thanks very much for your help !

Lukas

Hello,

 
This problem has already been reported by several users. The raidcom commands do not work. One of them wrote this

Looks like i found an issue in SVOS. Im not sure whats ging on. raidcom via "stor2rrd" dosent work also if i delete and add the user brand new.
Login via GUI ist working. I have to do some tests. On the second HDS all is fine with a new user "rrd_user".

Just create a new user and update stor2rrd. If it stops working again, it will not affect performance as the current situation (configuration has been done at least once).

try the storage connection test (after creating a new user)

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

./bin/config_check.sh <STORAGE_NAME>

question

Did you upgrade your CCI (24 December)?

can you send us logs before?

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir

tar cvhf logs.tar logs tmp/*txt /HORCM/log19/*

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.stor2rrd.com
You might even attach screenshots when it helps in understanding of the issue.

Thank you

mbayard

Hi,

I recreated another user on all my HDS arrays, upgraded CCI to latest version (and also ExportTools as some arrays have been upgraded in the meanwhile). Also enabled snmp monitoring (new in 2.50), and so far, it works again ...

Let's see with time if the user gets *blocked* again.

Thanks for your suggestions.

mbayard

Hi,

It's broken again, see the file list :

[stor2rrd@ypstor001s stor2rrd_BEMUTU]$ ll data/SPDVSP02B/IOSTATS/configuration/
total 1144
drwxr-xr-x  2 stor2rrd stor2rrd   4096 Mar 13 08:36 .
drwxr-xr-x 30 stor2rrd stor2rrd   4096 Mar  4 18:44 ..
-rw-r--r--  1 stor2rrd stor2rrd  45332 Mar  3 23:45 config-HOST1.txt
-rw-r--r--  1 stor2rrd stor2rrd   1728 Mar 13 06:59 config-HOST1.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd 286523 Mar  3 23:52 config-HOST2.txt
-rw-r--r--  1 stor2rrd stor2rrd  25143 Mar 13 08:36 config-HOST2.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd 117862 Mar  3 23:58 config-HOST3.txt
-rw-r--r--  1 stor2rrd stor2rrd  19853 Mar 13 09:51 config-HOST3.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd 531253 Mar  4 00:00 config-LDEV.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:45 config-LDEV.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    293 Mar  3 23:43 config-POOL1.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:46 config-POOL1.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    256 Mar  3 23:43 config-POOL2.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:46 config-POOL2.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    291 Mar  3 23:43 config-POOL3.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:46 config-POOL3.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    384 Mar  3 23:43 config-POOL4.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:46 config-POOL4.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd   5337 Mar  3 23:43 config-PORT.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:47 config-PORT.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd  26001 Mar  3 23:44 config-RG2.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:48 config-RG2.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd   4233 Mar  3 23:43 config-RG.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:48 config-RG.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:42 config-SNAPSHOT1.txt-tmp
-rw-r--r-x  1 stor2rrd stor2rrd      0 Sep 17 13:56 config-SPDVSP02B.touch
-rw-r--r--  1 stor2rrd stor2rrd    158 Mar 13 06:45 config-STORAGE_FIRMWARE.txt
-rw-r--r--  1 stor2rrd stor2rrd    680 Mar  4 00:00 config-STORAGE.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 06:45 config-STORAGE.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd      0 Mar 13 09:30 health_status_check.touch

[stor2rrd@ypstor001s stor2rrd_BEMUTU]$ ll data/SPDG1K03B/IOSTATS/configuration/
total 420
drwxr-xr-x  2 stor2rrd stor2rrd   4096 Mar 13 09:52 .
drwxr-xr-x 28 stor2rrd stor2rrd   4096 Mar  4 18:44 ..
-rw-r--r--  1 stor2rrd stor2rrd  13700 Mar  3 23:59 config-HOST1.txt
-rw-r--r--  1 stor2rrd stor2rrd   1728 Mar 13 08:58 config-HOST1.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd  82190 Mar  3 23:51 config-HOST2.txt
-rw-r--r--  1 stor2rrd stor2rrd   5554 Mar 13 09:21 config-HOST2.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd  11790 Mar  3 23:47 config-HOST3.txt
-rw-r--r--  1 stor2rrd stor2rrd   5554 Mar 13 09:43 config-HOST3.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd 221698 Mar  3 23:57 config-LDEV.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:50 config-LDEV.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    199 Mar  3 23:57 config-POOL1.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:51 config-POOL1.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    171 Mar  3 23:57 config-POOL2.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:51 config-POOL2.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    194 Mar  3 23:57 config-POOL3.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:51 config-POOL3.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    256 Mar  3 23:57 config-POOL4.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:51 config-POOL4.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd   5337 Mar  3 23:57 config-PORT.txt
-rw-r--r--  1 stor2rrd stor2rrd      0 Mar 13 09:52 config-PORT.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:44 config-SNAPSHOT1.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd    158 Mar 13 09:50 config-STORAGE_FIRMWARE.txt
-rw-r--r--  1 stor2rrd stor2rrd    386 Mar  3 23:57 config-STORAGE.txt
-rw-r--r--  1 stor2rrd stor2rrd     27 Mar 13 09:50 config-STORAGE.txt-tmp
-rw-r--r--  1 stor2rrd stor2rrd      0 Mar 13 09:13 health_status_check.touch

I noticed that all the last "valid" files are from 03/03/2019 just before midnight, and that we scheduled an automatic reboot of the SVPs every sunday at 23:59 ... I guess that when the SVP reboots, the raidcom sessions get broken, and now the user is "locked" again ...

Lukas

Hello,

with this problem we will do nothing. We will recommended to raise a call with Hitachi support. Let us know what solution Hitachi support suggests.

Thank you.

mbayard

I understand, but maybe the detection of this kind of behavior could be improved.

I only detected it because I noticed that the capacity of my pools were not up to date, but the date under the graphic was saying that it was refreshed recenlty.

Of course I can find in logs that there is an authentication problem, but from GUI it's not visible...

Alex_Rogov

We have many different HPE/Hitachi boxes, this problem is typical for VSP G1000/1500/F1500 and HPE XP7. Mid-range Hitachis do not exhibit such behaviour. In our case, killing "stuck" processes that perform data/configuration load helped.

ps -ef|grep load

ps -ef|grep perf

Look for any processes that relate to your storage and were started more than a day ago. Issue kill -9 <process-id>

If your instance of RAIDCOM works via IP, try using a command device instead. It is usually more reliable.

As I have suggested to the Support before, stor2rrd should at least provide some indication in the GUI that it has problems loading configuration and/or metrics data. AFAIK, they have it on their to-do list.

Alex_Rogov

Oh, and don't forget to restart the HORCM service after the processes have been killed:

/HORCM/usr/bin/horcmshutdown.sh <instance_id>

/HORCM/usr/bin/horcmstart.sh <instance_id>