Graphs for SAN switches are not being generated after perf files corruption

Brano

Hello guys,

I will need your help. I'm running the latest 2.61 sto2rrd. Our graphs for SAN switches are not being generated after 4th of December. In error.log I see some mentions of "data/*/*.out.tmp" files being probably corrupted.
There is no hard line in time since the graphs are not being generated. Looks like there was some network issues and not all data was getting through (in pic. below)



I have checked https://forum.xorux.com/discussion/comment/1726/#Comment_1726 where is the similar problem. But the thread is quite old and solution is promised in newer version, which I believe I have.

Using:

ls -l data/*/*_sanperf_* | wc -l

it shows I have 116205 files.

The CPU and Memory graphs are being generated fine.

Any ideas how to solve this? Where should I upload the log files?

Brano

I checked and the oldest *.out.tmp files are from Dec  5 17:30. then till Dec  8 05:55 the file sizes are irregular and in many occasions zero. After Dec  8 05:55 the sizes looks fairly similar.

Karel

Hello Brano,

remove those corrupted files (from Dec  5 17:30 to  Dec  8 05:55). 

After then:

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir
rm tmp/san_last_up*

Wait about 1-2 hours. There should not be any sanperf file after then.

In the meantime, you can check if count of stucked sanperf files is reduced.

ls -l data/*/*_sanperf_* | wc -l

Let us know, thanks.

Brano

Hi Karel,

It works! Thank you very much for support.