Mismatch between CPU tab and CPU OS tab.

kbarth

Over the weekend, we had a situation where we had some Nagios processes fall into a runaway state and spike the CPU on a number of our AIX LPARs. LPAR2RRD captured this spike accurately on both the CPU and CPU OS tab. This morning we cleaned up those processes and the CPU utilization returned to normal (via vmstat, nmon, and topas outputs). When we looked at the LPAR2RRD graphs, the CPU OS (run from the agent) reflected the drop in utilization, but the CPU graph continues to show a high CPU utilization. 

I checked the LPAR2RRD error.log and found nothing. I reran the load.sh and it completed without any errors or warnings in the data collection output, yet the CPU graph still reflects incorrect information. 

Any ideas on how to correct this issue?

kbarth

kbarth

kbarth

A further update. It appears that the OS tab graph returns to reporting correct information 5 hours after the issue was corrected.

kbarth

Our sample rate is set to 60.

We did upgrade the HMCs to V9R1 M920 recently to prepare for eventual migration to IBM Power9 hardware.

Pavel

wrong time on the HMC.

Check last update time, it is quite different in both examples.

su - lpar2rrd

cd /home/lpar2rrd/lpar2rrd

./bin/sample_rate.sh

HMC time and lslprutil time must be more less same for that server.

Pavel

HMC  upgrade resets timezone often.

Note: changing TZ require HMC reboot!

kbarth

Pavel, you were correct. This version upgrade of the HMC did reset the TZ to UTC, the prior version did not.

kbarth

It is interesting to note that the graph kept the local time settings. The update time showed the difference.

Pavel

I know

This is not definitelly only upgrade which resets TZ, we seen it many times in the past already.

kbarth

Time Zone changed and we will be checking to make sure that the graphs are now consistent. I found it interesting that the bottom line of the graph had the local time, but the graph line followed UTC.

Pavel

both times should be same, both are from HMC.

Send a screenshot example.