SSH Key not working with Remote Authentication account

Remote LDAP user uses the username and SSH key for authentication, but queries LDAP for the user role. This is in the interest of Security - local account with a RestrictedAdmin role is more vulnerable than a remote account - remote account can have role changed/revoked from AD, outside of the Storwize.

Don't understand why Stor2RRD is failing the connection, when an SSH connection works with no issue whatsoever.

From stor2rrd.cfg:

# Perl binary

 PERL=/bin/perl

# Perl path for searching modules, it must containg path to RRDp.pm module, usually it is all discovered automatically

 PERL5LIB=/home/stor2rrd/stor2rrd/bin:/usr/share/perl5/vendor_perl:/usr/lib64/perl5:/usr/share/perl5:/usr/lib64/perl5/vendor_perl:/home/stor2rrd/stor2rrd/lib

----------------------------------------------------

[stor2rrd@corplrrdpd001 bin]$ pwd

/bin

[stor2rrd@corplrrdpd001 bin]$ perl -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Can't locate Net/OpenSSH.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).

BEGIN failed--compilation aborted.

---------------------------------------------------

^^ I believe this might be indicating the problem. If you look below, the OpenSSH.pm file is in /home/stor2rrd/stor2rrd/lib/Net -- which is not listed in this @INC path variable.

--------------------------------------------------

[stor2rrd@corplrrdpd001 /]$ find / -iname 'openssh*' 2>/dev/null

/home/stor2rrd/stor2rrd/lib/Net/OpenSSH.pm

/home/stor2rrd/stor2rrd/lib/Net/OpenSSH

/home/stor2rrd/stor2rrd-7.50/dist_storage/lib/Net/OpenSSH.pm

/home/stor2rrd/stor2rrd-7.50/dist_storage/lib/Net/OpenSSH

/usr/share/doc/openssh-7.4p1

/usr/share/licenses/openssh-7.4p1

/usr/libexec/openssh

-------------------------------------

[stor2rrd@corplrrdpd001 etc]$ rpm -qa | grep -i openssh

openssh-7.4p1-22.el7_9.x86_64

openssh-server-7.4p1-22.el7_9.x86_64

openssh-clients-7.4p1-22.el7_9.x86_64

------------------------------------------------------------

[stor2rrd@corplrrdpd001 /]$ find / -name 'RRDp.pm' 2>/dev/null

/usr/lib64/perl5/vendor_perl/RRDp.pm

[stor2rrd@corplrrdpd001 bin]$ perl -I /home/stor2rrd/stor2rrd/lib -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Version: 0.80

[stor2rrd@corplrrdpd001 bin]$ echo $PERL5LIB

/home/stor2rrd/stor2rrd/lib

Hmmm...

Well, that fixed ONE of my storage systems' remote authentication issues - the one running an older code version (8.4.0.x).

The two running the newer 8.5.0.x code versions are still having the same problem.

UPDATE:

I was mistaken that the the underlying issue was the @INC path variable not including the path to the PERL5LIB.

Your statement that the PERL5LIB location being set in the stor2rrd .cfg file is where that PERL5LIB location is normally set is, by all appearances, a correct statement and it is not necessary for the @INC path variable to include the PERL5LIB location.

My incorrect conclusion came about because the authentication started working for one of my storage systems after exporting the PERL5LIB path to the @INC path variable. TURNS OUT, however, that this was just circumstance. This one storage system was working NOT by virtue of that change, but by virtue of the fact that it's running a lower level Storwize code version (8.4.0.x)!!! There were changes made in Storwize code 8.5.0 and newer to support multi-factor authentication. And some of those changes included SSH connectivity modifications. Those modifications unfortunately broke SSH sessions that use a combination of BOTH a Remote Authentication user and SSH "Batch Mode" (which I suspect is used by stor2rrd).

IBM Level3 Support has confirmed this issue. It is slated to be fixed in Storwize code versions 8.5.0.10 (due in about 5-6 months), and in the first release of 8.6.0 code (due sometime 2nd half of 2023). Support provided an iFix (one-off problem-targeted code update) for me to use to resolve this issue in my current 8.5.0.7 code.

-----------------------------------------------------------------------------

Now, you want to REALLY help me out (along with all your other customers that monitor IBM Storwize)!??

GET RID OF THE REQUIREMENT TO USE A RESTRICED ADMIN ACCOUNT to monitor Storwize systems. IBM removed the need for this to be the case back in 8.3.1.2 code, released almost THREE YEARS AGO (August 2020):

IBM Storwize Code version 8.3.1.2 and later release notes state that data collection for Spectrum Control can now be performed by an account with Monitor permission:

The following new features have been introduced in the 8.3.1.2 release (https://public.dhe.ibm.com/storage/san/sanvc/release_notes/831_releasenotes.html):

• Monitor role available for data collection by Spectrum Control v5.3.7 or later

Our other Storage monitoring vendor (Galileo, by ATS Group) updated their storage monitoring agents to support use of a Storwize Monitor role account a full year ago (April 2022).

1. Storwize monitoring on 8.5.x firmware works fine, AS LONG AS the Restricted Admin account used is a LOCAL account on the Storwize system. If you attempt to use an account on the Storwize that's set up as a REMOTE account, Stor2rrd monitoring does not work.
2. Yes, you are correct; the cpdumps command still requires (restr)admin rights. However, beginning with 8.5.1.2 code, each node AUTOMATICALLY copies the iostats files to the config node. So there is no need to issue the cpdumps command to get the statistics files onto the config node - they're put there by Storwize code all the time without needing to use the cpdumps command. IF the Storwize is still using code prior to 8.3.1.2, then use of the cpdumps command to get the iostats files onto the config node is still required, so the admin credentials are still required for monitoring.

Correction: The cpdumps happens automatically on anything 8.3.1.2 code or newer (not incorrectly stated 8.5.1.2).