Most graphs missing & no error-cgi log

We had been testing the appliance with success for a few weeks. Now we moved to a linux VM and installed the tar version of Stor2rrd but we don't seem to be able to replicate the results.

All 7 storage systems (Hitachi VSPs) & 80 Brocade switches connect fine, but most graphs (not all) fail to be shown and end up with the error message: "Error happened Check  - $$LPAR2RRD_HOME$$/logs/error-cgi.log - Web server error log".

The error-cgi.log file is completely empty & no errors are present in Apache error log. Furthermore, some graphs are correctly shown so it's not a generic graphing or Apache2 issue.

Where could we be looking for troubleshooting?



  • Hi,

    1.send us a screenshoot with that error
    2. tail -50 /var/tmp/stor2rrd-realt-error.log

  • Hi Pavel,
    Thanks for coming back to me so quickly. I have uploaded the screenshot.
    As for the file, it is empty.
    In fact, it didn't even exist at the beginning (and the error-cgi link was red) so I created it with "touch" (from stor2rrd user).

  • Hi,

    do not create it with touch under stor2rrd user.
    It must be owned by apache user!
    Just remove it.

    Try re-load the page?
    is the file appears with any error?

    if not then apache error_log (somewhere in/var : find /var -name error_log)
    tail error_log

  • Hi,
    I deleted it and restarted Apache, reloaded page multiple times. Still no file.
    Also, no errors on Apache error log.

    I really don't know what to think - the same config on the appliance works, so log collection is happening.

  • I noticed something: it is only "aggregated" stats that seem to fail.
    If I select something under "<Storage>-->Volume-->Items--><itemXX>, for instance, the graph is shown correctly.
    Anything above (like <Storage>-->Volume-->IO for example) gives the error.

  • what is your rrdtool version?
    rpm -qa| grep -i rrdtool
  • rrdtool-perl-1.4.8-9.el7.x86_64
  • hi,

    there must be somwthing in /var/tmp/stor2rrd-realt-error.log
    If you use private tmp feature on the Linux then the log could be here:
    ls -l /var/tmp/systemd-private*httpd*/tmp/*real*
    ls -l /var/tmp/systemd-private*apache*/tmp/*real*

    tail ...

  • Hi, I still haven't seen the logs there (I lack permissions for now) but I saw some "permission denied" errors in the VSP-related error logs. It appears that stor2rrd cannot access files starting with a dash (-). We have some of them for example in the POOL directory (-1.rrd, -1.first...). 
    Could you confirm this would be a problem?
    Here's an example of the error:

    XXX-VSPXXX:POOL:sum_io_trend:sum:0: XXX-VSPXXX:POOL:sum_io_trend failed ERROR: opening '/home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/-1.rrd': Permission denied at /opt/stor2rrd/bin/ line 4600. /opt/stor2rrd/bin/

    The file permissions are stor2rrd:stor2rrd  rw/r/r and the user can open it just fine.
  • ls -l /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/
  • Permissions seem fine everywhere. This is the directory you requested:

    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 11.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 11.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 12.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 12.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 13.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 13.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 -1.first
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 1.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 -1.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 1.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 2.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 2.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 3.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 3.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd     301 Jan 29 16:16 pools.col

  • Hello,
    Any suggestion? 
    The "error-cgi" file still doesn't exist, but if I click on the relative link on the web page I do get a log and it's full of "Permission denied" for rrd files.
    Spoiler: the permissions on the filesystem are ok (see above) and SELinux is in Permissive mode.
  • ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL
    ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX
    ls -ld /home/stor2rrd/stor2rrd/data/
    ls -ld /home/stor2rrd/stor2rrd/
    ls -ld /home/stor2rrd/
    ls -ld /home/

    All directories must be read + execute for everyone

  • Hi Pavel,
    They're all ok: 
    drwxrwxrwx. 2 stor2rrd stor2rrd 4096 Jan 29 16:16 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/

    drwxrwxrwx. 13 stor2rrd stor2rrd 4096 Feb  4 08:49 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX

    drwxr-xr-x. 89 stor2rrd stor2rrd 4096 Jan 29 16:50 /home/stor2rrd/stor2rrd/data

    lrwxrwxrwx. 1 stor2rrd stor2rrd 14 Jan 22 10:09 /home/stor2rrd/stor2rrd -> /opt/stor2rrd/

    drwxr-xr-x. 17 stor2rrd stor2rrd 4096 Feb  4 08:50 /opt/stor2rrd

    drwx------. 7 stor2rrd stor2rrd 4096 Feb  1 15:17 /home/stor2rrd

    drwxr-xr-x. 11 root root 4096 Jan 25 09:38 /home

    The only one not having it right is /home/stor2rrd, which has standard user-only configuration. But Apache does access it through /opt/stor2rrd so I fail to see how this could be relevant. 
    Plus, all the stuff that works fine is using the same data... I cannot understand what could be making a difference.

    This is a problem because we're trying to switch to stor2rrd from HDS support, but if we cannot make it work I won't be able to convince mgmt to get a license.
  • no no, apache must read /home/stor2rrd/stor2rrd sym link, it is not able

    You can workaround it by configuring direct /opt/stor2rrd in httpd.conf (instead of /home/stor2rrd/stor2rrd)
  • But apache is already pointing to /opt/stor2rrd. What I don't understand is why it is even looking into that path (/home/stor2rrd/stor2rrd/data) in the first place.

  • is even cgi-bin apache setup pointing to /opt/stor2rrd ?

  • Yes:
    Alias /stor2rrd  "/opt/stor2rrd/www/"
    <Directory "/opt/stor2rrd/www/">
        AllowOverride AuthConfig FileInfo
        Options Indexes FollowSymLinks Includes MultiViews
        AuthType basic
        AuthName "STOR2RRD authorization"
        AuthUserFile /etc/httpd/conf/htpasswd
        Require valid-user

    # CGI-BIN
    ScriptAlias /stor2rrd-cgi/ "/opt/stor2rrd/stor2rrd-cgi/"
    <Directory "/opt/stor2rrd/stor2rrd-cgi">
    AllowOverride AuthConfig FileInfo
        #AllowOverride AuthConfig
        AddHandler cgi-script .pl
        AddHandler cgi-script .sh
        SetHandler cgi-script
        Options ExecCGI Includes FollowSymLinks
        AuthType basic
        AuthName "STOR2RRD authorization"
        AuthUserFile /etc/httpd/conf/htpasswd
        Require valid-user

  • ok, can you at least try to change /home/stor2rrd right to see if it fixes the issu?
  • I've already asked the admins to do so. Will see if that works around the issue.
    I will let you know once done.
  • I believe I fixed it!  - The issue was not on the /home folder permission, but instead that I was calling the CRONTAB using the /home/stor2rrd path instead of /opt/stor2rrd !
    I copied the crontab script from the website install instructions but didn't think of it!

    I figured it out when I saw this in the script:

    # Load STOR2RRD configuration
    . `dirname $0`/etc/stor2rrd.cfg

    Then I thought, well it must only affect it when I load it manually... but then I checked the crontab and bingo! 

    I have since modified to use /opt/stor2rrd to start the scripts and now the graphs are showing!!

    Thanks for you help in this!

Sign In or Register to comment.