XIV volumes data not displayed after linux host reboot
Hi
We are running stor2rrd v2.40 on a RHEL 7.5. After applying patches , we do not see all the LUNS in the web GUI.
looks at stor2rrd CLI , we see the following error in logs/error.log-XIVNAMEHERE
WANING Tue Jul 2 15:10:10 2019 xivperf.pl: Same sample: IBM.2812-7860226.volume.9e814d0006a = 20190629071904.000000+000, skipping
There are hundreds if not thousands of lines above this. I have looked at the code on github , this happens when previous volume data has no time stamp. Is there any fix like deleting the stale files to kick start this back to life.
Atif
Comments
-
This is the code that executes ref https://github.com/dago/stor2rrd/blob/master/dist_storage/bin/xivperf.plLine 335 - 339 ,
&warning("Same sample: ".$cacheKey." = ".$stat->{'StatisticTime'}.", skipping."); $retry_cnt = 0;
last;if($debug) { print("=\n"); } if ( $storage_epoch == $cached_raw_counters->{'storage_timestamp'} ) {
-
Hi,try to remove these filestor2rrd/data/XIVNAMEHERE/tmp/cache.file
stor2rrd/data/XIVNAMEHERE/tmp/config.fileLet us know if that helps
-
Thanks for your reply.This location had no files stor2rrd/data/XIVNAMEHERE/tmp/*Instead I found the files herestor2rrd/data/XIVNAMEHERE/cache.filestor2rrd/data/XIVNAMEHERE/config.fileI have backed up the original file and deleted them. I re-ran load.sh and I will give it a couple of hours to see if that helps, will keep you posted.
-
Only one host is displaying the correct IO rate and data rate on the Stor2rrd interface. Rest of the hosts show "-nan" values in the graphs.
-
WE have dig through the time line, the day data stopped collecting was a reboot , interestingly this XIV was a target XIV in mirrored relationship. We switched to this XIV as primary and removed the mirrored relationships.After doing that, Stor2rrd got in the state where it displays -nan in the graphs
-
The linux host reboot and mirror relationship change happened at the same day, stor2rrd never worked properly after the bounce
-
Hello,
Send us logs pls.
Note a short problem description in the text field of the upload form.cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dirtar cvhf logs.tar logs tmp/*txtgzip -9 logs.tar
Send us logs.tar.gz via https://upload.stor2rrd.com
You might even attach screenshots when it helps in understanding of the issue.
thank you
-
Hi,pls screenshots1. host which is working,2. some other host3. volume aggregated graph4. ls -l data/<storage>HOST5. cat data/<storage>/HOST/host*txt
-
Historical data on this XIV is not important as it was a mirrored target until we changed it to source recently.Any way we can remove this XIV completely from stor2rrd and rediscover it again.
-
you can remove data/<storage name> directory, then everything wil be rediscovered
-
I removed the old data using rm -Rf data/xivnamehere*Rediscovery worked and now we have some hosts with values in the charts and some state the following error in the GUI.Error happenedCheck- $$LPAR2RRD_HOME$$/logs/error-cgi.log- Web server error logWhen I checked the log/error.log-xivname , I see it flooded with entries like this oneWarning Tue Jul 9 11:50:13 2019 xivperf.pl: Same sample: IBM.2812-7860226.volume.xxxxxxxxxxxx = 20190629071904.000000+000, skippingwhere xxxxxxxxxxx changes every line. Interesting fact , June 29 is when we stopped the remote mirror relationships and unlocked the volumes in this XIV and made them independent volumes. It looks like it has nothing to do with the reboot but it is somehow effected by stopping the relationship and unlocking the volumes on XIV.XIV itself is working fine but stor2rrd is not working properly after the replication change.
Howdy, Stranger!
Categories
- 1.6K All Categories
- 43 XORMON NG
- 25 XORMON
- 152 LPAR2RRD
- 13 VMware
- 16 IBM i
- 2 oVirt / RHV
- 4 MS Windows and Hyper-V
- Solaris / OracleVM
- XenServer / Citrix
- Nutanix
- 7 Database
- 2 Cloud
- 10 Kubernetes / OpenShift / Docker
- 122 STOR2RRD
- 19 SAN
- 7 LAN
- 17 IBM
- 3 EMC
- 12 Hitachi
- 5 NetApp
- 15 HPE
- Lenovo
- 1 Huawei
- 1 Dell
- Fujitsu
- 2 DataCore
- INFINIDAT
- 3 Pure Storage
- Oracle