stor2rrd not generating RRD files after AIX upgrade from 7.1 to 7.2

EchoMax

Hello,

I am experiencing an issue with stor2rrd after upgrading the operating system from AIX 7.1 to AIX 7.2. Since the OS upgrade, stor2rrd no longer generates RRD files.

Starting from the upgrade date, inside the directory /home/stor2rrd/stor2rrd/data/SAN01 I see a large number of files being generated with names like "_sanperf_20251210_1125.out.tmp" and "_sanconf_20251210_1125.out.tmp". These .out.tmp files remain in the directory and are generated again every time stor2rrd starts, but they are never processed into RRD files.

I checked the logs under /home/stor2rrd/stor2rrd/logs and I do not see any explicit errors. I also upgraded stor2rrd to version 8.06, but the problem remains unchanged.

Below is the output of load.sh

##########################

start time san : Thu Jan 15 10:55:02 GMT 2026

STOR2RRD free version 8.06

Alarm          : 3600

RRDp    version: 1.4008

RRDtool version: 1.7.2

Switches found : 8

creating graph : Totals:Data:san_data_sum_tot_in:d

creating graph : Totals:Data:san_data_sum_tot_in:w

creating graph : Totals:Data:san_data_sum_tot_in:m

creating graph : Totals:Data:san_data_sum_tot_in:y

creating graph : Totals:Data:san_data_sum_tot_in_trend:y

creating graph : Totals:Data:san_data_sum_tot_out:d

.......

creating graph : Totals:Frames:san_io_sum_tot_pcs_errors:m

creating graph : Totals:Frames:san_io_sum_tot_pcs_errors:y

Fabrics found  : 4 (vSAN = 2)

creating graph : Totals:Fabric:san_fabric_data_in:d

creating graph : Totals:Fabric:san_fabric_data_in:w

.........

creating graph : Totals:Fabric:san_fabric_pcs_errors:w

creating graph : Totals:Fabric:san_fabric_pcs_errors:m

creating graph : Totals:Fabric:san_fabric_pcs_errors:y

ISL found      : 20

creating graph : Totals:ISL:san_isl_data_in:d

creating graph : Totals:ISL:san_isl_data_in:w

creating graph : Totals:ISL:san_isl_data_in:m

creating graph : Totals:ISL:san_isl_data_in:y

creating graph : Totals:ISL:san_isl_data_out:d

creating graph : Totals:ISL:san_isl_data_out:w

.......

creating graph : Totals:ISL:san_isl_pcs_errors:w

creating graph : Totals:ISL:san_isl_pcs_errors:m

creating graph : Totals:ISL:san_isl_pcs_errors:y

start time lan : Thu Jan 15 10:55:24 GMT 2026

STOR2RRD free version 8.06

Alarm          : 3600

RRDp    version: 1.4008

RRDtool version: 1.7.2

alert last_rec : limit is 6 hours

alert last_rec : SAN200 : nok : do not send alert again, last_rec=1765360818

alert last_rec : SAN201 : nok : do not send alert again, last_rec=1765360817

alert last_rec : SAN300 : nok : do not send alert again, last_rec=1765360819

alert last_rec : SAN301 : nok : do not send alert again, last_rec=1765360817

alert last_rec : SAN200 : nok : do not send alert again, last_rec=1765360817

alert last_rec : SAN201 : nok : do not send alert again, last_rec=1765360816

alert last_rec : SAN300 : nok : do not send alert again, last_rec=1765360818

alert last_rec : SAN301 : nok : do not send alert again, last_rec=1765360818

Custom start    : free : 1

Finished       : Thu Jan 15 10:55:24 2026, run time: 0 secs

heatmap-stor   : start Thu Jan 15 10:55:25 2026

heatmap-stor   : set SAN Data IN utilization for 48 ports

heatmap-stor   : set SAN Data OUT utilization for 48 ports

heatmap-stor   : set resp utilization for 0 pools

heatmap-stor   : set resp utilization for 0 volumes

heatmap-stor   : set capacity utilization for 0 pools/tiers

heatmap-stor   : end Thu Jan 15 10:55:26 2026

trim logs      : not this time 15 == 15

reporter START : Thu Jan 15 10:55:27 2026

load cfg       : /home/stor2rrd/stor2rrd/etc/web_config/reporter.json

product        : STOR2RRD

reporter       : not this time 15 == 15

reporter END   : Thu Jan 15 10:55:27 2026

Auto Conn test : Thu Jan 15 10:55:27 GMT 2026 start on the background

start time html: Thu Jan 15 10:55:27 GMT 2026

installing WWW : install-html.sh

Host identif   : AIX server01 2 7

Apparently nothing new, install_html.sh exiting

end time       : Thu Jan 15 10:55:27 GMT 2026

Inactive volumes start  : Thu Jan 15 10:55:27 GMT 2026

Inactive volumes end    : Thu Jan 15 10:55:27 GMT 2026

Thu Jan 15 10:55:27 GMT 2026

##########################

And below are the logs from load_sanperf.sh

##########################

10.1.2.26: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.26

10.1.2.154: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.154

10.1.2.24: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.24

10.1.2.155: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.155

10.1.2.153: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.153

10.1.2.23: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.23

10.1.2.156: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.156

10.1.2.25: Thu Jan 15 10:58:17 GMT 2026 : starting

Continue now   : 10.1.2.25

Waiting for 12976618

Waiting for 23527814

Waiting for 26476976

.......

Waiting for 18743572

Waiting for 17563910

##########################

Has anyone seen a similar behavior on AIX 7.2? I am wondering if this could be related to changes in system utilities, shell behavior, or command output parsing introduced with AIX 7.2 that prevent stor2rrd from processing the temporary output files.

Any suggestion or hint would be appreciated. Thanks.

Pavel

every AIX TL upgrade can screw Perl dependency, upgrade major version definitely does that.

AIX feature ....

Send us logs. 

Note a short problem description in the text field of the upload form.

under stor2rrd user (use lpar2rrd user on our virtual appliance)

cd `cat ~/.stor2rrd_home`

tar cvhf logs.tar logs tmp/*txt

gzip -9 logs.tar

Send us logs.tar.gz via https://upload.stor2rrd.com

EchoMax

Thanks Pavel,

the problem has been fixed by following the steps here:

https://forum.xorux.com/discussion/893/graphs-for-san-switches-are-not-being-generated-after-perf-files-corruption

Removed corrupted *.out.tmp files generated on the day of the upgrade, and removed these files:

cd /home/stor2rrd/stor2rrd # or where is your STOR2RRD working dir
rm tmp/san_last_up*

Thanks