Passwordless SSH Access for Automation

Key-Based SSH Authentication

A common setup that DevOps engineers have to do as part of automation projects is configuring passwordless access to remote machines. In an automated job, the scripts used by it may have to access remote machines for fetching information from them or running programs on them. Such access would normally involve providing credentials in the form of a password or key to login to the remote machine. A very common example is running a script on a remote machine via SSH. Doing such tasks using SSH is not difficult when done manually, in which case the user will be prompted for a password or passphrase.

It is very obvious that such interactive sessions are not feasible in an automated transaction. SSH configuration for a user can be configured to get around that issue securely. Let’s look at some basic SSH usages first. We will look at more complex scenarios that may be needed for automation after that.

Following are the basic strategies involved (there could be any number of variations):

• Service accounts are set up to access hosts remotely without having to input password. A service account is a headless account that is not associated with any real user. Such accounts are usually labeled appropriately to point to the related automation.

• Passwordless access for the service accounts is set up using the SSH key pair.

• The SSH private keys that used for automation are managed using a secure application such as GPG or something similar.

• The SSH public keys could be checked into source code control system, a process that would facilitate the setting up of environments that require passwordless access.

SSH Into a Remote Host Using Key

If passwordless access is already set up for a user, fkafka, on auto.corp.myco.com. Here's a valid command-line to run a script or command on prague.corp.myco.com remotely from a local machine: 

$ ssh fkafka@prague.corp.myco.com "ls /tmp"

Few configuration steps should have been done beforehand to get the above command working for headless on the remote host. Let's see what they are:

• User fkafka exists on the local machine and there is a private key setup for that user under ~fkafka/.ssh/, id_rsa or id_dsa.

• On the remote machine, user fkafka exists and the ~fkafka/.ssh/authorized_keys file is populated with the public key, which corresponds to the private key located at ~fkafka/.ssh/ directory on the local machine.

• The user fkafka is authorized to execute the command line, which is the entire string after fkafka@prague.corp.myco.com on the remote machine.

• The SSH key pair for fkafka was created with no passphrase.

• fkafka had accessed prague.corp.myco.com at least once before and the public host key of prague.corp.myco.com was added to the known_hosts file on the local machine.

As you can see, there are lots assumptions made here that require some explanation. Let's look at some of the variations of last remote command execution.

If the SSH session is started as user fkafka, then command-line could be simpler:

$ ssh prague.corp.myco.com "ls /tmp"

There is no need for fkafka to be available on both the machines. If the private key for fkafka is available on the local machine, then the following SSH command-line would work:

$ ssh -i /some/location/private_key fkafka@prague.corp.myco.com "ls /tmp"

In this case, the local user who runs the command-line should own the private key, /some/location/private_key, with no access for other users (0400).

Also, the local user can SSH as fkafka without having to use the -i option as above example. In that case, the public key corresponds to the private key used by the local user must be added to ~fkafka/.ssh/authorized_keys on the remote machine. We will look at the details of such configurations later.

$ ssh fkafka@prague.corp.myco.com "ls /tmp"

In the above examples, it is also assumed that the private key used was created with no passphrase set. If the passphrase is defined for a key pair, it cannot be ignored during an SSH session. However, it is possible to cache the passphrase using ssh-agent and repeated keying-in of the passphrase could be avoided during interactive sessions. A private key with a passphrase is not very suitable for automation purposes.

The Authentication Process

In an SSH session, two basic authentications happen:

1. Verification of the authenticity of the remote machine by the local machine.

2. Authentication of user access by the remote machine.

Verification of Remote Machine

The local machine makes sure that the server it connects to is valid to avoid any man-in-the-middle attacks. The SSH host key pair created at the time of setting up SSH on a machine is used for host authentication. When a user logs in to a remote machine for the first time from another machine, the user will be prompted for verification as follows:

The authenticity of host ***** can't be established.
RSA key fingerprint is *****.
Are you sure you want to continue connecting (yes/no)?

If you proceed with connecting to the remote machine, the user's known_hosts file, ~user/.ssh/known_hosts on the local machine, will be populated with the host key of the remote machine.

You will be prompted as above only once, the first time (unless the SSH key defined for the remote_host would change later). However, in a highly automated environment where there is no manual intervention expected, the ~/.ssh/known_hosts file has to be updated with the host key automatically.

There is an SSH option to ignore this check, but that makes the environment vulnerable to the man-in-the-middle attacks.

The check is bypassed as follows:

$ ssh -o "StrictHostKeyChecking=no" -i /some/location/private_key fkafka@prague.corp.myco.com "ls /tmp"

Another option is to populate the known_hosts file of service accounts with host keys of machines that need to be accessed automatically. ssh-keys can be used to fetch public host keys that can be appended to the known_hosts file.

$ ssh-keyscan -t rsa,dsa HOST >> ~USER/.ssh/known_hosts

Passwordless User Authentication

One of the common setups needed for a service account to run programs on remote machines is configuring its passwordless access to those machines. The steps provided here are generic for any user and not only applicable to service accounts.

In general, on the client/local machine from which an SSH session is initiated, a private key has to be installed. On the remote machine, the related public key has to be installed in ~USER/.ssh/authorized_keys file for the user that will be logging in during the SSH session.

Following are steps involved on local and remote machines:

On the Local Machine

Verify that key-based authentication is enabled in /etc/ssh/ssh_config.

#This setting is the default usually, so you need to only check if it is set to 'no' explicitly.

RSAAuthentication yes #SSH1 and OpenSSH/1

DSAAuthentication yes #OpenSSH/2

Generate the SSH key pair:

$ ssh-keygen [-t rsa1] #SSH1 and OpenSSH/1

$ ssh-keygen -t dsa #OpenSSH/2

When prompted for the passphrase, enter nothing unless you plan to use this key only for interactive SSH sessions.

The execution of ssh-keygen will generate these key files in the SSH directory of user's home directory:

id_rsa and id_rsa.pub #SSH1 and OpenSSH/1

id_dsa and id_dsa.pub #OpenSSH/2

The key file with extension .pub is the public key and the other one is the private key. The key pair could be created with custom name using the -f option as follows:

$ ssh-keygen [-t rsa1] -f /path/to/KEY_FILE #SSH1 and OpenSSH/1

$ ssh-keygen -t dsa -f /path/to/KEY_FILE #OpenSSH/2

The key files generated in the directory /path/to/ will be KEY_FILE and KEY_FILE.pub.

Set up the private key for the user. If the key pair is generated locally by the same user, the private key is normally setup with correct ownership and permissions. The private key must be owned by the user and only readable by the user. Those requirements can be implemented as follows if that is not done as part of the key generation step:

$ chown USER PRIVATE_KEY_FILE

$ chmod 400 PRIVATE_KEY_FILE

Set up permissions on the user's home and the .ssh directory.

Make sure that permissions on these directories are at least restricted to the following level, on both client and the remote host:

$ chmod 755 ~USER

$ chmod 755 ~USER/.ssh

On the Remote Machine

Verify that key-based authentication is enabled in /etc/ssh/sshd_config.

#This setting is the default usually, so you need to only check if it is set to 'no' explicitly.

RSAAuthentication yes #SSH1 and OpenSSH/1

DSAAuthentication yes #OpenSSH/2

Note: If you have to change anything in /etc/ssh/sshd_config, restart the SSHD Daemon to make those changes applied to the SSH connections to that machine.

Paste the public key generated on the client machine in the ~USER/.ssh/authorized_keys file. If the doesn’t exist, create the directory structure and the file.

Set or make sure the permissions on the .ssh directory and authorized_keys file are as follows:

$ chown -R USER ~USER/.ssh

$ chmod 700 ~USER/.ssh

$ chmod 600 ~USER/.ssh/authorized_keys

Test Connectivity

These settings above will enable the user to SSH into the remote host passwordless as follows:

$ ssh REMOTE_HOST

If you SSH as a different user to login to remote machine, specify the user ID. SSH will still look for the private key under local user's .ssh directory:

$ ssh OTHER_USER@REMOTE_HOST

In this case, the public key corresponds to the local user should be added to ~OTHER_USER/.ssh/authorized_keys on the remote machine.

The private key at a different location can be specified using the SSH option IdentityFile (-i):

$ ssh -i /path/to/private_key OTHER_USER@REMOTE_HOST

In this case, the public key corresponding to /path/to/private_key should be added to ~OTHER_USER/.ssh/authorized_keys on the remote machine.

SSH User Configuration File

The SSH options set in the ~USER/.ssh/config file can override the global settings applied by /etc/ssh/ssh_config. The settings can be specific to each machine that is configured to access from the client/local machine. Using this config file, a long SSH command-line that has multiple options can be aliased be made shorter.

Let’s look at some typical examples that explain the uses of the user config file.

A simple.ssh/config file:

#Default settings
Host *
 Port 22
 Protocol 2

## host prague ##
Host prague
 HostName prague.corp.myco.com
 User fkafka
 IdentityFile /some/location/private_key


## Vagrant/CentOS instance ##
Host centos7
 HostName 127.0.0.1
 User vagrant
 Port 2222
 IdentityFile /Users/jdoe/vagrant/.vagrant/machines/centos7/virtualbox/private_key

Using the SSH user config file such as the above, host aliases can be created and the SSH options can be specified within that definition. For example, after setting up the host alias prague, user fkafka can ssh using a much shorter command-line:

$ ssh prague

The above command line will be expanded to:

$ ssh -i /some/location/private_key fkafka@prague.corp.myco.com

Following are some of the most commonly used SSH options, which could be used from command-line or in the config file.

Host: Defines host alias in the config file using which an SSH session can be started as follows:

$ ssh host_alias

The settings under Host * are applicable to all the SSH transactions the user would initiate from that local machine. Essentially, the SSH global settings on the client side, set in /etc/ssh/ssh_config, can be overridden at the user level using this option.

HostName: The real name of the machine or its IP address.

User: The user that will log into the remote machine during the SSH session.

IdentityFile: Path of the private key that is specified using option -i on command-line.

Port: The port to connect to on the remote machine. If this option is not specified, the SSH command will try to connect to port 22, the default SSH port to run the SSHD Daemon on the remote machine.

In the Vagrant example above, the instance centos7 is accessible via port 2222. By configuring like this you can ssh into the Vagrant instance from the host machine as follows:

$ ssh centos7

Such aliasing of hosts will be very handy when you have multiple Vagrant instances running on your laptop with different Linux variations.

Protocol: Specifies the protocol versions SSH should support. The possible values are 1 and 2 and both can be specified, delimited by a comma, in the order of preference.

Additional Configurations to Access Remote Machine

The configuration requirements described above are the basic SSH settings needed for passwordless access. There could be more impacting the overall access from the local machine to the remote machine.

Network Firewall

Both outbound (from a local machine) and inbound (a remote machine) network traffic might be restricted by the corporate firewalls, especially if the machines are on two different networks. ACLs may have to be opened to allow SSH traffic between two machines.

Host Level Firewall Iptables

Linux distributions come with a host-based firewall called Netfilter and that is managed using the utility iptables. Even if the SSH traffic is allowed through the corporate or network firewall, it could still be blocked at the machine level using iptables.

Normally, by stopping the related service iptables, the Netfilter can be disabled, but, if that is implemented as part some security policy, the SSH traffic to the machine should be enabled by configuring iptables.

PAM

PAM (Pluggable Authentication Module) is a user authentication mechanism available for Linux distributions. If that is enabled in your environment and made SSH aware, you will see the following setting in the /etc/ssh/sshd_config file:

UsePAM Yes

If PAM is used, settings in /etc/security/access.conf that specify whether not a user is allowed to log in to that machine could block the use access via SSH.

An entry with a prefix + specifies access and - denial. Sample entries:

+ : fkafka : ALL
- : ALL : ALL

SELinux

SELinux (Security-Enhanced Linux) is a Linux kernel security module that can be used for implementing access control policies, and it is mainly available on the RHEL-based distributions. If SELinux is enabled on a machine, the policies can prevent the key-based authentication from succeeding.

Discussion of both PAM and SELinux is beyond the scope of this article. The troubleshooting process should cover configurations of those modules also if steps provided here for setting up passwordless SSH access won’t work.

SSH-Based Tools

The SSH settings discussed here are applicable to related tools such as SCP and SFTP. Let’s look at briefly how those tools are used, piggy-backing on the same SSH configurations available on the local machine and for the user.

SCP

The SCP command is used to copy files and directories to a remote machine over SSH. Copying files passwordless is a common task in automation processes such as automated deployments and Big Data pipelines.

$ scp /path/to/a/file REMOTE_HOST:/target/path/

The above command line will work if the user can SSH as below from the same environment:

$ ssh REMOTE_HOST

If the copying has to be done using a user only available on the remote machine, the command line would look like this:

$ scp -i /some/location/private_key /path/to/a/file OTHER_USER@REMOTE_HOST:/target/path/

In the above case, the remote user will be able to SSH as follows passwordless:

$ ssh -i /some/location/private_key OTHER_USER@REMOTE_HOST

SFTP

SFTP is a secured version of FTP and it runs over SSH, as in the case of SCP. SFTP will use the existing SSH key-based configuration and it can be run in both interactive or noninteractive mode. However, if it is used in an automated job, its batch feature can be used:

$ sftp -b batch.cmds USER@REMOTE_HOST

In this case, SFTP will connect to REMOTE_HOST and run the FTP command in batch.cmds file.

RSYNC

RSYNC can be directed to transfer files between two machines securely over SSH using the -e option, as in the following example:

$ rsync -avz -e “ssh -i /some/location/private_key” /source/path/ fkafka@prague.corp.myco.com:/target/path/

Any valid SSH options can be added to ssh to control the SSH session that RSYNC would use to login to the remote machine and do file transfer.

Sudo Access on a Remote Machine

At times, for an automation task, logging into a remote machine passwordless using a service account may not be enough. It may have to run a command as another user or it might need sudo access. In both the cases, related configurations must be done using /etc/sudoers file using the Visudo utility on the remote machine.

For example, for service account fkafka to have the privilege to Sudo passwordless, the following entry must be in /etc/ sudoers:

fkafka ALL=(ALL) NOPASSWD: ALL

The above privilege could be done different ways and it should be planned with the help of the sysadmin who manages the security policy.