Sometimes when using (newer) OpenSSH 7.x or higher clients you may have multiple issues trying to connect and use public key authentication (PKI) with an OpenVMS server. The error messages will often complain about “no matching key exchange method found.” You may also get error messages related to the use of ssh-dss such as “no matching host key type found. Their offer: ssh-dss”. Finally, on some older versions of VMS, you might also see errors related to ciphers (such as aes-128-CBC) being deprecated. All of these issues are pretty easy to fix, but require some specific flags to your ssh client.
The other problem is that OpenSSH and the commercial Secure Shell server in VMS do not use the same key file format. So, we'll need to convert the key before it's ready for use.
It's crypto. Sometimes researchers find that the ways in which they were doing things no longer works because of new discoveries which put old methods at risk. Because of this, we've seen ciphers, key exchange algorithms, host key algorithms, and other areas become deprecated in newer OpenSSH clients and servers. Most of the problems with VMS seem to occur after OpenSSH version 7.4p1.
Here is a truth table of what you can work around and what you can't
|Client OS||OpenSSH v6.x Server||OpenSSH v7.x Server||OpenVMS Secure Shell 8.4 or older Server|
|OpenVMS 8.4 MultiNet||Works||2 Workarounds needed||Works|
|OpenVMS 8.4 TCP/IP Services||Works||2 Workaround needed||Works|
|OpenVMS 7.x||Works||3 workarounds needed||Works|
|Red Hat 8 or newer||Works||Works||Broken. No workaround.|
You can force newer OpenSSH clients (such as for Linux) to support backwards compatibility for VMS using a special option “-oKexAlgorithms=+diffie-hellman-group1-sha1”. This works for ssh, scp, and sftp command line programs in the same way.
You can likewise also fix the ssh-dss issue by adding the option “-oHostKeyAlgorithms=+ssh-dss” to your command line. This also works for ssh, scp, and sftp. Remember, these options only work when your Linux/Unix box is the client and VMS is the server. These options absolutely do no work for VMS clients, but then again, you don't need them in that context, either.
If you get an error from your ssh client about a missing cipher on the server side, it's probably because the client as deprecated all the algos that the server supports. However, you can usually force the client into backwards compatibility. The way to do this is with a “-c mycipher” flag to your command. The cipher name should be one that is listed in the error under “their offer”. You can see what ciphers you can support by typing ssh -Q cipher. You'll have to pick one they both agree on. I usually pick aes128-cbc for VMS servers.
You can use all the workarounds together and in many cases you will need to. Here is an example where all three workarounds are in play.
ssh -o KexAlgorithms=+diffie-hellman-group1-sha1 \ -o HostKeyAlgorithms=+ssh-dss \ -c aes256-cbc \ myuser@remotehost
If you don't want to have to type all that out every time, you may consider setting up a host entry in your local ssh client configuration. This would be your $HOME/.ssh/config file. This file might not exist if you haven't used it before (it's not created by default). Here is an example entry in the configuration file (remember this is on your Linux/Unix box, not VMS).
Host MyVMSbox KexAlgorithms +diffie-hellman-group1-sha1 HostKeyAlgorithms +ssh-dss Ciphers aes128-cbc
That will free you from having to type out all the options every time. Additionally, you can remove the “Host” line and just put those options at the top of your config file and thus that would change your default configuration.
You can use OpenSSH keys on VMS systems, but there are some gotchas. First off, you need to be using RSA keys. DSA keys don't always work depending on the version of VMS you happen to be dealing with. The procedure goes something like this:
That should do it. Now you should be able to ssh from Linux/Unix into VMS using only your key.