graphs not updating after disk full

mpacyna

We're running stor2rrd 7.10-1 as a docker container, it's been running fine for a while for few weeks but few days ago the guest os running the container ran out of space.  After adding more space and restarting the container few times the graphs aren't updating anymore, they're just showing values from few days ago and then blank (it's been few hours since the disk space issue was fixed and container restarted).  I see the following errors in ~stor2rrd/stor2rrd/logs/error.log (we're using ibm v5030)

Thu Jul  8 17:03:59 2021: v5k2 : get_pool_csv_report error, no source: resp_t:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379

Thu Jul  8 17:04:06 2021: v5k2 : get_pool_csv_report error, no source: data_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:04:12 2021: v5k2 : get_pool_csv_report error, no source: io_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:04:21 2021: v5k2 : get_pool_csv_report error, no source: used:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: resp_t:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: data_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:34 2021: v5k1 : get_pool_csv_report error, no source: io_rate:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

Thu Jul  8 17:05:38 2021: v5k1 : get_pool_csv_report error, no source: used:h /home/stor2rrd/stor2rrd/bin/storage.pl:23379 

any pointers on how to fix this would be appreciated

Pavel

Hi,

run this from the cmd line to see if there is not any corrupted files

cd /home/stor2rrd/stor2rrd
./bin/check_rrdtool.sh

It just prints files, remove them manually if there is any,

let us know if that does not resolve it.

mpacyna

Thanks Pavel,

unfortunately it doesn't seem to have found anything (script name is slightly different though)

4633c117e624:~/stor2rrd$ ./bin/check_rrdtool_stor2rrd.sh 
...Checked files: 360
No corrupted files have been found

Pavel

send us logs

Note a short problem description in the text field of the upload form.

cd /home/lpar2rrd/lpar2rrd  # or where is your LPAR2RRD working dir

tar cvhf logs.tar logs tmp/*txt tmp/*json

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.lpar2rrd.com