Most graphs missing & no error-cgi log

andyfsimon
andyfsimon
in STOR2RRD
Hello,
We had been testing the appliance with success for a few weeks. Now we moved to a linux VM and installed the tar version of Stor2rrd but we don't seem to be able to replicate the results.

All 7 storage systems (Hitachi VSPs) & 80 Brocade switches connect fine, but most graphs (not all) fail to be shown and end up with the error message: "Error happened Check  - $$LPAR2RRD_HOME$$/logs/error-cgi.log - Web server error log".

The error-cgi.log file is completely empty & no errors are present in Apache error log. Furthermore, some graphs are correctly shown so it's not a generic graphing or Apache2 issue.

Where could we be looking for troubleshooting?

Thx.

Comments

  • Pavel
    Pavel
    Hi,

    1.send us a screenshoot with that error
    2. tail -50 /var/tmp/stor2rrd-realt-error.log


  • andyfsimon
    andyfsimon
    Hi Pavel,
    Thanks for coming back to me so quickly. I have uploaded the screenshot.
    As for the file, it is empty.
    In fact, it didn't even exist at the beginning (and the error-cgi link was red) so I created it with "touch" (from stor2rrd user).

  • Pavel
    Pavel
    Hi,

    do not create it with touch under stor2rrd user.
    It must be owned by apache user!
    Just remove it.

    Try re-load the page?
    is the file appears with any error?

    if not then apache error_log (somewhere in/var : find /var -name error_log)
    tail error_log

  • andyfsimon
    andyfsimon
    Hi,
    I deleted it and restarted Apache, reloaded page multiple times. Still no file.
    Also, no errors on Apache error log.

    I really don't know what to think - the same config on the appliance works, so log collection is happening.


  • andyfsimon
    andyfsimon
    I noticed something: it is only "aggregated" stats that seem to fail.
    If I select something under "<Storage>-->Volume-->Items--><itemXX>, for instance, the graph is shown correctly.
    Anything above (like <Storage>-->Volume-->IO for example) gives the error.

  • Pavel
    Pavel
    what is your rrdtool version?
    rpm -qa| grep -i rrdtool
  • andyfsimon
    andyfsimon
    rrdtool-perl-1.4.8-9.el7.x86_64
    rrdtool-1.4.8-9.el7.x86_64
  • Pavel
    Pavel
    hi,

    there must be somwthing in /var/tmp/stor2rrd-realt-error.log
    If you use private tmp feature on the Linux then the log could be here:
    ls -l /var/tmp/systemd-private*httpd*/tmp/*real*
    ls -l /var/tmp/systemd-private*apache*/tmp/*real*

    tail ...



  • andyfsimon
    andyfsimon
    Hi, I still haven't seen the logs there (I lack permissions for now) but I saw some "permission denied" errors in the VSP-related error logs. It appears that stor2rrd cannot access files starting with a dash (-). We have some of them for example in the POOL directory (-1.rrd, -1.first...). 
    Could you confirm this would be a problem?
    Here's an example of the error:

    XXX-VSPXXX:POOL:sum_io_trend:sum:0: XXX-VSPXXX:POOL:sum_io_trend failed ERROR: opening '/home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/-1.rrd': Permission denied at /opt/stor2rrd/bin/detail-graph-cgi.pl line 4600. /opt/stor2rrd/bin/detail-graph-cgi.pl:646


    The file permissions are stor2rrd:stor2rrd  rw/r/r and the user can open it just fine.
  • Pavel
    Pavel
    ls -l /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/
  • andyfsimon
    andyfsimon
    Permissions seem fine everywhere. This is the directory you requested:

    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 11.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 11.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 12.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 12.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 13.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 13.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 -1.first
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 1.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 -1.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 1.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 2.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 2.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd      10 Jan 31 17:01 3.first
    -rw-r--r--. 1 stor2rrd stor2rrd 4770368 Feb  1 12:30 3.rrd
    -rw-r--r--. 1 stor2rrd stor2rrd     301 Jan 29 16:16 pools.col

  • andyfsimon
    andyfsimon
    Hello,
    Any suggestion? 
    The "error-cgi" file still doesn't exist, but if I click on the relative link on the web page I do get a log and it's full of "Permission denied" for rrd files.
    Spoiler: the permissions on the filesystem are ok (see above) and SELinux is in Permissive mode.
  • Pavel
    Pavel
    ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL
    ls -ld /home/stor2rrd/stor2rrd/data/XXX-VSPXXX
    ls -ld /home/stor2rrd/stor2rrd/data/
    ls -ld /home/stor2rrd/stor2rrd/
    ls -ld /home/stor2rrd/
    ls -ld /home/

    All directories must be read + execute for everyone






  • andyfsimon
    andyfsimon
    Hi Pavel,
    They're all ok: 
    drwxrwxrwx. 2 stor2rrd stor2rrd 4096 Jan 29 16:16 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX/POOL/

    drwxrwxrwx. 13 stor2rrd stor2rrd 4096 Feb  4 08:49 /home/stor2rrd/stor2rrd/data/XXX-VSPXXX

    drwxr-xr-x. 89 stor2rrd stor2rrd 4096 Jan 29 16:50 /home/stor2rrd/stor2rrd/data

    lrwxrwxrwx. 1 stor2rrd stor2rrd 14 Jan 22 10:09 /home/stor2rrd/stor2rrd -> /opt/stor2rrd/

    drwxr-xr-x. 17 stor2rrd stor2rrd 4096 Feb  4 08:50 /opt/stor2rrd

    drwx------. 7 stor2rrd stor2rrd 4096 Feb  1 15:17 /home/stor2rrd

    drwxr-xr-x. 11 root root 4096 Jan 25 09:38 /home

    The only one not having it right is /home/stor2rrd, which has standard user-only configuration. But Apache does access it through /opt/stor2rrd so I fail to see how this could be relevant. 
    Plus, all the stuff that works fine is using the same data... I cannot understand what could be making a difference.

    This is a problem because we're trying to switch to stor2rrd from HDS support, but if we cannot make it work I won't be able to convince mgmt to get a license.
  • Pavel
    Pavel
    no no, apache must read /home/stor2rrd/stor2rrd sym link, it is not able

    You can workaround it by configuring direct /opt/stor2rrd in httpd.conf (instead of /home/stor2rrd/stor2rrd)
  • andyfsimon
    andyfsimon
    But apache is already pointing to /opt/stor2rrd. What I don't understand is why it is even looking into that path (/home/stor2rrd/stor2rrd/data) in the first place.


  • Pavel
    Pavel
    is even cgi-bin apache setup pointing to /opt/stor2rrd ?

  • andyfsimon
    andyfsimon
    Yes:
    Alias /stor2rrd  "/opt/stor2rrd/www/"
    <Directory "/opt/stor2rrd/www/">
        AllowOverride AuthConfig FileInfo
        Options Indexes FollowSymLinks Includes MultiViews
        AuthType basic
        AuthName "STOR2RRD authorization"
        AuthUserFile /etc/httpd/conf/htpasswd
        Require valid-user
    </Directory>

    # CGI-BIN
    ScriptAlias /stor2rrd-cgi/ "/opt/stor2rrd/stor2rrd-cgi/"
    <Directory "/opt/stor2rrd/stor2rrd-cgi">
    AllowOverride AuthConfig FileInfo
        #AllowOverride AuthConfig
        AddHandler cgi-script .pl
        AddHandler cgi-script .sh
        SetHandler cgi-script
        Options ExecCGI Includes FollowSymLinks
        AuthType basic
        AuthName "STOR2RRD authorization"
        AuthUserFile /etc/httpd/conf/htpasswd
        Require valid-user
    </Directory>


  • Pavel
    Pavel
    ok, can you at least try to change /home/stor2rrd right to see if it fixes the issu?
  • andyfsimon
    andyfsimon
    I've already asked the admins to do so. Will see if that works around the issue.
    I will let you know once done.
    Thanks!!
  • andyfsimon
    andyfsimon
    I believe I fixed it!  - The issue was not on the /home folder permission, but instead that I was calling the CRONTAB using the /home/stor2rrd path instead of /opt/stor2rrd !
    I copied the crontab script from the website install instructions but didn't think of it!

    I figured it out when I saw this in the load.sh script:


    # Load STOR2RRD configuration
    . `dirname $0`/etc/stor2rrd.cfg

    Then I thought, well it must only affect it when I load it manually... but then I checked the crontab and bingo! 

    I have since modified to use /opt/stor2rrd to start the scripts and now the graphs are showing!!

    Thanks for you help in this!



Sign In or Register to comment.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Categories

In this Discussion