Apache error after upgrading to AIX Toolbox httpd.ppc 2.4.34-1

kbarth

A recent security vulnerability has forced us to upgrade our http package to AIX Toolbox version 2.4.34-1. The upgrade did not create any configuration errors and LPAR2RRD loads without issue, but when we bring up the web interface, the pane on the left side of the screen loads normally, but none of the graphs load. I checked all the logs and the only errors I find are in the /var/log/httpd/error_log. They are as follows:

[Wed Oct 31 10:31:56.565111 2018] [core:error] [pid 16384262] (70007)The timeout specified has expired: [client 10.2.220.168:50865] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: http://10.10.8.121/lpar2rrd/?menu=4dc8084&tab=0
[Wed Oct 31 10:31:56.565053 2018] [cgi:warn] [pid 16384262] [client 10.2.220.168:50865] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail.sh, referer: http://10.10.8.121/lpar2rrd/?menu=4dc8084&tab=0
[Wed Oct 31 10:31:37.274074 2018] [core:error] [pid 11272548] (70007)The timeout specified has expired: [client 10.2.220.168:50455] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:31:37.273942 2018] [cgi:warn] [pid 11272548] [client 10.2.220.168:50455] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:30:38.804267 2018] [cgi:warn] [pid 19202434] [client 10.2.220.168:50475] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail-graph.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:30:38.471892 2018] [cgi:warn] [pid 13435332] [client 10.2.220.168:50474] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail-graph.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:30:37.961232 2018] [cgi:warn] [pid 8258016] [client 10.2.220.168:50472] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail-graph.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:30:37.946534 2018] [cgi:warn] [pid 11010502] [client 10.2.220.168:50473] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail-graph.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0
[Wed Oct 31 10:30:37.416176 2018] [cgi:warn] [pid 18088432] [client 10.2.220.168:50469] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail-graph.sh, referer: http://10.10.8.121/lpar2rrd/?menu=a8de6fa&tab=0

Are there adjustments I need to make in lpar2rrd configs so that this loads properly?

Thanks,

Keith Barth

Pavel

Hi Keith,

t can timeout after 60 seconds.
Is that happen for every graph?
Was that so slow even before?

Increase timeout in Apache httpd.conf to 10 minutes to be safe:
                KeepAlive On
                KeepAliveTimeout 5
                TimeOut 1200

Under root:
apachectl restart

kbarth

It is happening on every graph and it was not this slow before. I am going to adjust the timeout settings in the http.conf file and restart the httpd daemons. I will let you know if that corrects the issue.

kbarth

It may be my unfamiliarity with this version of httpd on AIX, but I am not seeing any parameters in the httpd.conf file where KeepAlive or TimeOut can be set. Is this a set of new lines that need to be added?

Pavel

just add them there

kbarth

I added the lines to the httpd.conf file and restarted the httpd daemons. Unfortunately, it did not correct the issue. Now the graph load hangs for the duration of the time out setting and then makes the same entry in the httpd error_log.

Additionally, I now am noting a CPU spike on the server with the offending process being:

apache 20251104 10224020  65 13:07:51      -  1:13 /usr/bin/perl /home/lpar2rrd/lpar2rrd/bin/detail-cgi.pl

kbarth

Took the KeepAlive and Timeout parameters out of httpd.conf. Noticed that the new httpd was owned by the apache user instead of the nobody user (from the older IBMHS version). Went back and set the limits correctly for the apache user. While the issue is still present, I am getting a different error entry in the httpd error_log:

[Wed Oct 31 15:36:06.005296 2018] [core:error] [pid 15401388] (70007)The timeout specified has expired: [client 10.2.220.168:58589] AH00574: ap_content_length_filter: apr_bucket_read() failed, referer: http://10.10.8.121/lpar2rrd/?menu=b2d37ae&tab=4
[Wed Oct 31 15:36:06.005207 2018] [cgi:warn] [pid 15401388] [client 10.2.220.168:58589] AH01220: Timeout waiting for output from CGI script /home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/detail.sh, referer: http://10.10.8.121/lpar2rrd/?menu=b2d37ae&tab=4

Any thoughts?

Pavel

Hi,

under root:

rm /var/tmp/lpar2rrd-realt-error.log

assure that files owned nobody are owned by apache here (if there is any)

/home/lpoar2rrd/lpar2rrd/etc/web_config

refresh the GUI to get any graph

tail -f /var/tmp/lpar2rrd-realt-error.log

kbarth

I cleared the log, made sure that the httpd was started by and owned by the apache user. When I refreshed the GUI, I got the following error:

root@osknim01: /var/tmp >cat lpar2rrd-realt-error.log

Could not load program /opt/freeware/bin/rrdtool:

Could not load module /usr/lib/libcairo.a(libcairo.so.2).

        Dependent module /usr/lib/libXrender.a(libXrender.so.1) could not be loaded.

        Member libXrender.so.1 is not found in archive

I found this odd because the apache and lpar2rrd user profiles all have their LIBPATH set  to:

LIBPATH=/opt/freeware/lib:/opt/freeware/lib64:/usr/lib

Pavel

Check this page: http://www.lpar2rrd.com/aix_install.htm --> Install tab --> Problems and their resolutions