SSH Key not working with Remote Authentication account

fryer

I have a user defined on my Storwize as a "Remote Authentication" user, with an SSH keypair configured for authorization.

When I run the config check I get (IP addr. hidden):

TCP connection to "1.2.3.4" on port "22" is ok

. /home/stor2rrd/stor2rrd/etc/stor2rrd.cfg; /bin/perl /home/stor2rrd/stor2rrd/bin/ssh_test_svc.pl 1.2.3.4 svc_STG_Rstadmscript /home/stor2rrd/.ssh/swrau 30 sshtest XXXXX 0

Access restricted to authorized D&H Distributing usersPermission denied (keyboard-interactive).

Error: Unable to connect to remote machine. unable to establish master SSH connection: master process exited unexpectedly

Tue Mar 28 16:53:52 2023 - ERROR  - ssh_test.pl: Unable to connect to remote machine 1.2.3.4 - Return code: 1

But the ssh connection is working fine:

stor2rrd]$ ssh -i ~/.ssh/swrau svc_STG_Rstadmscript@1.2.3.4 "lscurrentuser"

Access restricted to authorized D&H Distributing usersname svc_STG_Rstadmscript

role RestrictedAdmin

owner_id

owner_name

password_expiry_time

multi_factor no

password_sshkey_required no

gui_disabled no

cli_disabled no

rest_disabled no

And if I change that exact same account on the storage to a Local user, instead of a Remote user, with no other changes, and with the exact same SSH keypair, it works fine - without changing a thing in Stor2RRD:

stor2rrd]$ ./bin/config_check.sh DRVDI_FS5200

=========================

STORAGE: DRVDI_FS5200: SWIZ : sample rate: 300 seconds

=========================

 TCP connection to "1.2.3.4" on port "22" is ok

. /home/stor2rrd/stor2rrd/etc/stor2rrd.cfg; /bin/perl /home/stor2rrd/stor2rrd/bin/ssh_test_svc.pl 1.2.3.4 svc_STG_Rstadmscript /home/stor2rrd/.ssh/swrau 30 sshtest XXXXX 0

Access restricted to authorized D&H Distributing usersMachine Name:      DR01IF52PD001

Machine Type-Model:   4662-6H2

etc...

And ssh connection test looks no different:

stor2rrd]$ ssh -i ~/.ssh/swrau svc_STG_Rstadmscript@1.2.3.4 "lscurrentuser"

Access restricted to authorized D&H Distributing usersname svc_STG_Rstadmscript

role RestrictedAdmin

owner_id

owner_name

password_expiry_time

multi_factor no

password_sshkey_required no

gui_disabled no

cli_disabled no

rest_disabled no

Pavel

Hi,

ok, and is a problem to use "Local user" instead of "Remote user"? Then it works, right?

I am not sure where the difference is, not sure what means remote user, LDAP ones?

fryer

Remote LDAP user uses the username and SSH key for authentication, but queries LDAP for the user role. This is in the interest of Security - local account with a RestrictedAdmin role is more vulnerable than a remote account - remote account can have role changed/revoked from AD, outside of the Storwize.

Don't understand why Stor2RRD is failing the connection, when an SSH connection works with no issue whatsoever.

Pavel

su - stor2rrd

cd /home/stor2rrd/stor2rrd

. etc/stor2rrd.cfg

$PERL -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

rpm -qa| grep -i openssh

fryer

From stor2rrd.cfg:

# Perl binary

 PERL=/bin/perl

# Perl path for searching modules, it must containg path to RRDp.pm module, usually it is all discovered automatically

 PERL5LIB=/home/stor2rrd/stor2rrd/bin:/usr/share/perl5/vendor_perl:/usr/lib64/perl5:/usr/share/perl5:/usr/lib64/perl5/vendor_perl:/home/stor2rrd/stor2rrd/lib

----------------------------------------------------

[stor2rrd@corplrrdpd001 bin]$ pwd

/bin

[stor2rrd@corplrrdpd001 bin]$ perl -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Can't locate Net/OpenSSH.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).

BEGIN failed--compilation aborted.

---------------------------------------------------

^^ I believe this might be indicating the problem. If you look below, the OpenSSH.pm file is in /home/stor2rrd/stor2rrd/lib/Net -- which is not listed in this @INC path variable.

--------------------------------------------------

[stor2rrd@corplrrdpd001 /]$ find / -iname 'openssh*' 2>/dev/null

/home/stor2rrd/stor2rrd/lib/Net/OpenSSH.pm

/home/stor2rrd/stor2rrd/lib/Net/OpenSSH

/home/stor2rrd/stor2rrd-7.50/dist_storage/lib/Net/OpenSSH.pm

/home/stor2rrd/stor2rrd-7.50/dist_storage/lib/Net/OpenSSH

/usr/share/doc/openssh-7.4p1

/usr/share/licenses/openssh-7.4p1

/usr/libexec/openssh

-------------------------------------

[stor2rrd@corplrrdpd001 etc]$ rpm -qa | grep -i openssh

openssh-7.4p1-22.el7_9.x86_64

openssh-server-7.4p1-22.el7_9.x86_64

openssh-clients-7.4p1-22.el7_9.x86_64

------------------------------------------------------------

[stor2rrd@corplrrdpd001 /]$ find / -name 'RRDp.pm' 2>/dev/null

/usr/lib64/perl5/vendor_perl/RRDp.pm

fryer

[stor2rrd@corplrrdpd001 bin]$ perl -I /home/stor2rrd/stor2rrd/lib/Net -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Can't locate Net/OpenSSH.pm in @INC (@INC contains: /home/stor2rrd/stor2rrd/lib/Net /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 .).

BEGIN failed--compilation aborted.

fryer

[stor2rrd@corplrrdpd001 bin]$ perl -I /home/stor2rrd/stor2rrd/lib -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Version: 0.80

fryer

[stor2rrd@corplrrdpd001 bin]$ export PERL5LIB=/home/stor2rrd/stor2rrd/lib

[stor2rrd@corplrrdpd001 bin]$ perl -MNet::OpenSSH -e 'print "Version: $Net::OpenSSH::VERSION\n"'

Version: 0.80

fryer

[stor2rrd@corplrrdpd001 bin]$ echo $PERL5LIB

/home/stor2rrd/stor2rrd/lib

fryer

[stor2rrd@corplrrdpd001 stor2rrd]$ export PERL5LIB=/home/stor2rrd/stor2rrd/bin:/usr/share/perl5/vendor_perl:/usr/lib64/perl5:/usr/share/perl5:/usr/lib64/perl5/vendor_perl:/home/stor2rrd/stor2rrd/lib

[stor2rrd@corplrrdpd001 stor2rrd]$ echo $PERL5LIB

/home/stor2rrd/stor2rrd/bin:/usr/share/perl5/vendor_perl:/usr/lib64/perl5:/usr/share/perl5:/usr/lib64/perl5/vendor_perl:/home/stor2rrd/stor2rrd/lib

This fixed my problem. With the user switched to a Remote user, authentication is now working. QUESTIONS

[stor2rrd@corplrrdpd001 stor2rrd]$ ./bin/config_check.sh DRVDI_FS5200

=========================

STORAGE: DRVDI_FS5200: SWIZ : sample rate: 300 seconds

=========================

 TCP connection to "172.16.100.135" on port "22" is ok

. /home/stor2rrd/stor2rrd/etc/stor2rrd.cfg; /bin/perl /home/stor2rrd/stor2rrd/bin/ssh_test_svc.pl 172.16.100.135 svc_STG_Rstadmscript /home/stor2rrd/.ssh/swrau 30 sshtest XXXXX 0

Access restricted to authorized D&H Distributing usersMachine Name:      DR01IF52PD001

Machine Type-Model:   4662-6H2

Machine Serial:     78F1BMH

Code Level:       8.5.0.7 (build 157.15.2302141801000)

Product Name:      IBM FlashSystem 5200

 connection OK

: Is this persistent? Why did I need to do this; shouldn't the PERL5LIBM in storr2rrd.cfg take care of this?

fryer

Hmmm...

Well, that fixed ONE of my storage systems' remote authentication issues - the one running an older code version (8.4.0.x).

The two running the newer 8.5.0.x code versions are still having the same problem.

Pavel

I did not understand what fixed it, different PERL5LIB?

It is normally set here

cd /home/stor2rrd/stor2rrd

grep PERL5LIB= etc/stor2rrd.cfg

fryer

UPDATE:

I was mistaken that the the underlying issue was the @INC path variable not including the path to the PERL5LIB.

Your statement that the PERL5LIB location being set in the stor2rrd .cfg file is where that PERL5LIB location is normally set is, by all appearances, a correct statement and it is not necessary for the @INC path variable to include the PERL5LIB location.

My incorrect conclusion came about because the authentication started working for one of my storage systems after exporting the PERL5LIB path to the @INC path variable. TURNS OUT, however, that this was just circumstance. This one storage system was working NOT by virtue of that change, but by virtue of the fact that it's running a lower level Storwize code version (8.4.0.x)!!! There were changes made in Storwize code 8.5.0 and newer to support multi-factor authentication. And some of those changes included SSH connectivity modifications. Those modifications unfortunately broke SSH sessions that use a combination of BOTH a Remote Authentication user and SSH "Batch Mode" (which I suspect is used by stor2rrd).

IBM Level3 Support has confirmed this issue. It is slated to be fixed in Storwize code versions 8.5.0.10 (due in about 5-6 months), and in the first release of 8.6.0 code (due sometime 2nd half of 2023). Support provided an iFix (one-off problem-targeted code update) for me to use to resolve this issue in my current 8.5.0.7 code.

-----------------------------------------------------------------------------

Now, you want to REALLY help me out (along with all your other customers that monitor IBM Storwize)!??

GET RID OF THE REQUIREMENT TO USE A RESTRICED ADMIN ACCOUNT to monitor Storwize systems. IBM removed the need for this to be the case back in 8.3.1.2 code, released almost THREE YEARS AGO (August 2020):

IBM Storwize Code version 8.3.1.2 and later release notes state that data collection for Spectrum Control can now be performed by an account with Monitor permission:

The following new features have been introduced in the 8.3.1.2 release (https://public.dhe.ibm.com/storage/san/sanvc/release_notes/831_releasenotes.html):

• Monitor role available for data collection by Spectrum Control v5.3.7 or later

Our other Storage monitoring vendor (Galileo, by ATS Group) updated their storage monitoring agents to support use of a Storwize Monitor role account a full year ago (April 2022).

Pavel

Hi,

1. do you say that storwize 8.5.x firmware does not work with our monitoring at all? What is exact problem/error?
2. restricted admin: it is all about cmd "svctask cpdumps" ( https://stor2rrd.com/storwize-rights.php ) which requires admin rights. Let us check if it can run under monitoring role under new firmware

fryer

1. Storwize monitoring on 8.5.x firmware works fine, AS LONG AS the Restricted Admin account used is a LOCAL account on the Storwize system. If you attempt to use an account on the Storwize that's set up as a REMOTE account, Stor2rrd monitoring does not work.
2. Yes, you are correct; the cpdumps command still requires (restr)admin rights. However, beginning with 8.5.1.2 code, each node AUTOMATICALLY copies the iostats files to the config node. So there is no need to issue the cpdumps command to get the statistics files onto the config node - they're put there by Storwize code all the time without needing to use the cpdumps command. IF the Storwize is still using code prior to 8.3.1.2, then use of the cpdumps command to get the iostats files onto the config node is still required, so the admin credentials are still required for monitoring.

Pavel

thanks for your info, we will test if cpdumps is done automaticly, if so, we will make necessary updates to exclude necessity of usage restr admin role

fryer

Correction: The cpdumps happens automatically on anything 8.3.1.2 code or newer (not incorrectly stated 8.5.1.2).

Pavel

you can test version without needs of restriction admin

 https://www.stor2rrd.com/download-temp/stor2rrd-7.61-12.tar

let us know in case you have time for testing

Thanks.