No LAN and SAN graph for one machine

mathieu_b

Hi,

I have one machine where lpar2rrd agent is installed which don't work properly.
I have memory and paging graphics but nothing for LAN and SAN.

On the directory, i have : 
drwxr-xr-x    2 lpar2rrd staff          4096 Apr  4 2014  .
drwxr-xr-x   36 lpar2rrd staff         16384 Mar 24 00:24 ..
-rw-r--r-x    2 lpar2rrd staff       3825736 Mar 24 11:58 cpu.mmm
-rw-r--r-x    1 lpar2rrd staff            12 Mar  7 2016  lan-en0.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  lan-en0.mmm
-rw-r--r-x    1 lpar2rrd staff            12 Mar  7 2016  lan-en1.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  lan-en1.mmm
-rw-r--r-x    2 lpar2rrd staff       5738264 Mar 24 11:58 mem.mmm
-rw-r--r-x    2 lpar2rrd staff       3825736 Mar 24 11:58 pgs.mmm
-rw-r--r-x    1 lpar2rrd staff            19 Mar  7 2016  san-fcs0.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  san-fcs0.mmm
-rw-r--r-x    1 lpar2rrd staff            19 Mar  7 2016  san-fcs1.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  san-fcs1.mmm
-rw-r--r-x    1 lpar2rrd staff            19 Mar  7 2016  san-fcs2.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  san-fcs2.mmm
-rw-r--r-x    1 lpar2rrd staff            19 Mar  7 2016  san-fcs3.cfg
-rw-r--r-x    1 lpar2rrd staff       3825736 Mar  7 2016  san-fcs3.mmm

On the log :
Fri Mar 24 11:54:22 2017: ********: *******_SN0610EFT:***-unx-***** Graph rrdtool error : no active dbs ETH,FCS,SEA  :

Pavel

Hi,

this info from the agent side:

tail -1 /var/tmp/lpar2rrd*txt
tail /var/tmp/lpare2rrd*err

mathieu_b

# tail -1 /var/tmp/lpar2rrd-agent-xxx-unx-xxx-adm-lpar2rrd.txt
8205-E6C*0610EFT:xxx-unx-xxxxxx:6:1490354640:Fri Mar 24 12:24:00 2017:3300000000::::mem:::16777216:13229484:3547732:3092836:8201672:5027812:pgs:::0:0:6144:1:::lan:en0:10.108.2.46:::::::lan:en1:10.184.2.89:::::::cpu:::0:16:25:0::

# tail /var/tmp/lpar2rrd-agent-xxx-unx-xxx-adm-lpar2rrd.err
Wed Feb  8 06:47:01 2017: agent timed out after : 600 seconds
Wed Feb  8 06:51:16 2017: Connect error xxx-unx-xxx-adm:8162 : Connection timed out /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Wed Feb  8 07:02:17 2017: Connect error xxx-unx-xxx-adm:8162 : Connection timed out /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Wed Feb  8 07:22:00 2017: agent timed out after : 600 seconds
Wed Feb  8 07:33:01 2017: agent timed out after : 600 seconds
Wed Feb  8 07:36:16 2017: Connect error xxx-unx-xxx-adm:8162 : Connection timed out /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Wed Feb  8 08:10:15 2017: Connect error xxx-unx-xxx-adm:8162 : Connection timed out /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Wed Feb  8 08:22:15 2017: Connect error xxx-unx-xxx-adm:8162 : Connection timed out /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Wed Mar  1 10:59:00 2017: Connect error xxx-unx-xxx-adm:8162 : Connection refused /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201
Fri Mar 17 13:50:00 2017: Connect error xxx-unx-xxx-adm:8162 : Connection refused /opt/lpar2rrd-agent/lpar2rrd-agent.pl:201

Pavel

Are there fcs adapters? Is that AIX?

lsdev -C | grep fcs
fcstat fcs0| grep Bytes

mathieu_b

# fcstat fcs0 | grep Bytes
  Input Bytes:  0
  Output Bytes: 0
  Input Bytes:  244930471192
  Output Bytes: 282077805456

# lsdev -C | grep fcs
fcs0         Available C4-T1       Virtual Fibre Channel Client Adapter
fcs1         Available C5-T1       Virtual Fibre Channel Client Adapter
fcs2         Available C6-T1       Virtual Fibre Channel Client Adapter
fcs3         Available C7-T1       Virtual Fibre Channel Client Adapter

Pavel

rpm -qa| grep lpar2rrd

is the agetn runing under root or lpar2rrd (or any other user)
If it is not under root then
su - <lpar2rrd user>
fcstat fcs0 | grep Bytes

mathieu_b

Thanks, it is a problem with fcstat command.
I have compare with another server where it run ok : there is a s bit on fcstat command.
I have add it and it's OK now :

Before :
$ fcstat fcs0 | grep Bytes
Error opening device: /dev/fscsi0
errno: 00000001

After :
$ fcstat fcs0 | grep Bytes
  Input Bytes:  0
  Output Bytes: 0
  Input Bytes:  244962127353
  Output Bytes: 282104319888

I have do the same with entstat.

Pavel

sticky bit on fcstat might remove some OS upgrade/patching.
We have already experienced several times.