Most graphs missing & no error-cgi log

andyfsimon

Hello,
We had been testing the appliance with success for a few weeks. Now we moved to a linux VM and installed the tar version of Stor2rrd but we don't seem to be able to replicate the results.

All 7 storage systems (Hitachi VSPs) & 80 Brocade switches connect fine, but most graphs (not all) fail to be shown and end up with the error message: "Error happened Check  - $$LPAR2RRD_HOME$$/logs/error-cgi.log - Web server error log".

The error-cgi.log file is completely empty & no errors are present in Apache error log. Furthermore, some graphs are correctly shown so it's not a generic graphing or Apache2 issue.

Where could we be looking for troubleshooting?

Thx.

Pavel

Hi,

1.send us a screenshoot with that error

2. tail -50 /var/tmp/stor2rrd-realt-error.log

https://upload.stor2rrd.com

andyfsimon

Hi Pavel,
Thanks for coming back to me so quickly. I have uploaded the screenshot.
As for the file, it is empty.
In fact, it didn't even exist at the beginning (and the error-cgi link was red) so I created it with "touch" (from stor2rrd user).

Pavel

Hi,

do not create it with touch under stor2rrd user.

It must be owned by apache user!

Just remove it.

Try re-load the page?

is the file appears with any error?

if not then apache error_log (somewhere in/var : find /var -name error_log)

tail error_log

andyfsimon

Hi,
I deleted it and restarted Apache, reloaded page multiple times. Still no file.
Also, no errors on Apache error log.

I really don't know what to think - the same config on the appliance works, so log collection is happening.

andyfsimon

I noticed something: it is only "aggregated" stats that seem to fail.
If I select something under "<Storage>-->Volume-->Items--><itemXX>, for instance, the graph is shown correctly.
Anything above (like <Storage>-->Volume-->IO for example) gives the error.

Pavel

what is your rrdtool version?

rpm -qa| grep -i rrdtool

andyfsimon

rrdtool-perl-1.4.8-9.el7.x86_64

rrdtool-1.4.8-9.el7.x86_64

Pavel

hi,

there must be somwthing in /var/tmp/stor2rrd-realt-error.log

If you use private tmp feature on the Linux then the log could be here:

ls -l /var/tmp/systemd-private*httpd*/tmp/*real*
ls -l /var/tmp/systemd-private*apache*/tmp/*real*

tail ...

andyfsimon

Hi, I still haven't seen the logs there (I lack permissions for now) but I saw some "permission denied" errors in the VSP-related error logs. It appears that stor2rrd cannot access files starting with a dash (-). We have some of them for example in the POOL directory (-1.rrd, -1.first...). 
Could you confirm this would be a problem?
Here's an example of the error:

XXX-VSPXXX:POOL:sum_io_trend:sum:0: XXX-VSPXXX:POOL:sum_io_trend failed ERROR: opening '/home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/-1.rrd': Permission denied at /opt/stor2rrd/bin/detail-graph-cgi.pl line 4600. /opt/stor2rrd/bin/detail-graph-cgi.pl:646

The file permissions are stor2rrd:stor2rrd  rw/r/r and the user can open it just fine.

Pavel

ls -l /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/

andyfsimon

Permissions seem fine everywhere. This is the directory you requested:

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 11.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 11.rrd

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 12.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 12.rrd

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 13.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 13.rrd

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 -1.first

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 1.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 -1.rrd

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 1.rrd

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 2.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 2.rrd

-rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 3.first

-rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 3.rrd

-rw-r--r--. 1 stor2rrd stor2rrd     301 Jan 29 16:16 pools.col

andyfsimon

Hello,
Any suggestion? 
The "error-cgi" file still doesn't exist, but if I click on the relative link on the web page I do get a log and it's full of "Permission denied" for rrd files.
Spoiler: the permissions on the filesystem are ok (see above) and SELinux is in Permissive mode.

Pavel

ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL

ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX

ls -ld /home/stor2rrd/stor2rrd/data/

ls -ld /home/stor2rrd/stor2rrd/

ls -ld /home/stor2rrd/

ls -ld /home/

All directories must be read + execute for everyone

andyfsimon

Hi Pavel,
They're all ok: 

drwxrwxrwx. 2 stor2rrd stor2rrd 4096 Jan 29 16:16 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/

drwxrwxrwx. 13 stor2rrd stor2rrd 4096 Feb  4 08:49 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX

drwxr-xr-x. 89 stor2rrd stor2rrd 4096 Jan 29 16:50 /home/stor2rrd/stor2rrd/data

lrwxrwxrwx. 1 stor2rrd stor2rrd 14 Jan 22 10:09 /home/stor2rrd/stor2rrd -> /opt/stor2rrd/

drwxr-xr-x. 17 stor2rrd stor2rrd 4096 Feb  4 08:50 /opt/stor2rrd

drwx------. 7 stor2rrd stor2rrd 4096 Feb  1 15:17 /home/stor2rrd

drwxr-xr-x. 11 root root 4096 Jan 25 09:38 /home

The only one not having it right is /home/stor2rrd, which has standard user-only configuration. But Apache does access it through /opt/stor2rrd so I fail to see how this could be relevant. 
Plus, all the stuff that works fine is using the same data... I cannot understand what could be making a difference.

This is a problem because we're trying to switch to stor2rrd from HDS support, but if we cannot make it work I won't be able to convince mgmt to get a license.

Pavel

no no, apache must read /home/stor2rrd/stor2rrd sym link, it is not able

You can workaround it by configuring direct /opt/stor2rrd in httpd.conf (instead of /home/stor2rrd/stor2rrd)

andyfsimon

But apache is already pointing to /opt/stor2rrd. What I don't understand is why it is even looking into that path (/home/stor2rrd/stor2rrd/data) in the first place.

Pavel

is even cgi-bin apache setup pointing to /opt/stor2rrd ?

andyfsimon

Yes:

Alias /stor2rrd  "/opt/stor2rrd/www/"

<Directory "/opt/stor2rrd/www/">

    AllowOverride AuthConfig FileInfo

    Options Indexes FollowSymLinks Includes MultiViews

    AuthType basic

    AuthName "STOR2RRD authorization"

    AuthUserFile /etc/httpd/conf/htpasswd

    Require valid-user

</Directory>

# CGI-BIN

ScriptAlias /stor2rrd-cgi/ "/opt/stor2rrd/stor2rrd-cgi/"

<Directory "/opt/stor2rrd/stor2rrd-cgi">

AllowOverride AuthConfig FileInfo

    #AllowOverride AuthConfig

    AddHandler cgi-script .pl

    AddHandler cgi-script .sh

    SetHandler cgi-script

    Options ExecCGI Includes FollowSymLinks

    AuthType basic

    AuthName "STOR2RRD authorization"

    AuthUserFile /etc/httpd/conf/htpasswd

    Require valid-user

</Directory>

Pavel

ok, can you at least try to change /home/stor2rrd right to see if it fixes the issu?

andyfsimon

I've already asked the admins to do so. Will see if that works around the issue.
I will let you know once done.
Thanks!!

andyfsimon

I believe I fixed it!  - The issue was not on the /home folder permission, but instead that I was calling the CRONTAB using the /home/stor2rrd path instead of /opt/stor2rrd !
I copied the crontab script from the website install instructions but didn't think of it!

I figured it out when I saw this in the load.sh script:

# Load STOR2RRD configuration

. `dirname $0`/etc/stor2rrd.cfg

Then I thought, well it must only affect it when I load it manually... but then I checked the crontab and bingo! 

I have since modified to use /opt/stor2rrd to start the scripts and now the graphs are showing!!

Thanks for you help in this!