[Linux, OpenSSL, expect] Doing SFTP file transfers from a shell script

[Linux, OpenSSL, expect] Doing SFTP file transfers from a shell script


PRODUCT:   OpenSSH_5.4p1, OpenSSL 1.0.0a-fips 1 Jun 2010 
           expect version 5.43.0


OP/SYS:    Linux Fedora 13


COMPONENT:  SFTP, SCP 


SOURCE:     Philippe Vouters
            Fontainebleau/France


LOW-COST HIGH-TECH:  http://techno-star.fr


SYMPTOM(S) or PROBLEM(S):

The SFTP -b or the SCP -B options enables batch mode. However, under OpenSSH
version 5.4p1, these options require an empty passphrase and do not allow for
password prompting. What to do when the remote system only has port 22 opened
and only accepts OpenSSH logging using a login/password pair ? This is where
expect enables to workaround this OpenSSH restriction where you do not have the
target account RSA key with an empty passphrase. If you own the target account
RSA key with an empty passphrase then you may use the simpler

        sftp -b script user@host

or scp equivalent shell command.


SOLUTION or RESPONSE:

Use expect to synchronize with each SFTP/SCP output and to simulate their
inputs. Expect acts on terminal screens much the same way as the non-interactive
telnet tool found elsewhere in this knowledge database.


WORKAROUND or ALTERNATIVE:

In your shell script, use SFTP/SCP in interactive mode and synchronize their
output and input using expect commands.


EXPECT INFORMATION:

expect is an optional software. Under Fedora 13 and a root account, use yum to
install expect.

        # yum install expect


KSH SHELL SCRIPT NOTES:

This ksh script expects that the SFTP software is BSD based and accepts the
[-P port] option. Such an SFTP client is available under Linux Fedora 13. 
Carefully check your "man 1 sftp" before blindly using this ksh script. For 
example this sftp man at http://linux.die.net/man/1/sftp which claims BSD
compatibility is not suited for the below ksh script. The following man at
http://linuxreviews.org/man/sftp/ is fully suited and corresponds to the
author's Linux Fedora 13 computer sftp man which he based this ksh script upon.

If running a Linux SFTP client software whose man is identical to what is
described at http://linux.die.net/man/1/sftp, you would modify all occurences
of -P ${port} in the ksh script below to -oPort=${port} such as given as an
example in the http://linux.die.net/man/1/sftp Web browser displayable man.


KSH SHELL FULL SCRIPT:

#!/bin/ksh
#
#        Copyright (C) 2010 by Philippe.Vouters@laposte.net
#
#    This program is free software: you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by the
#    Free Software Foundation, either version 3 of the License, or
#    any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program.  If not, see <http://www.gnu.org/licenses/>;.
#
# Software Description:
# This software transfers the latest 3PAR (a SAN equipment) weekly*.tbz2 and
# optionally the InSplore*.tbz2 files to /tmp local directory. The InSplore
# file is transfered provided it has been produced the same day than the run of 
# this script.
#
# This script makes uses of optional expect freeware and an smtp client whose
# source code is downloadable from http://vouters.dyndns.org/zip/.
# Under Fedora 12 and a root account:
#     # yum install expect
# For smtp:
#     # unzip -aa smtp.zip
#     # export bin=/usr/local/bin
#     # cd smtp
#     # make -f Makefile_smtp
#
# Enter the IP address or the DNS name with the IP port separated with a colon
# of each Service Processor while incrementing the index.
#
SP_hostname[0]="127.0.0.1:10001" # corresponds to 10.10.31.41:22
SP_hostname[1]="127.0.0.1:10002" # corresponds to 10.10.52.25:22
#
# Indicate your email account characteristics as well as a list of remote 
# recipients with a space as the delimiter in the smtp_destinees variable.
#
smtp_server="smtp.sfr.fr"
smtp_login="Philippe.Vouters@laposte.net"
smtp_password="password"
smtp_destinees="Philippe.Vouters@laposte.fr"

# These are the 3PAR Service Processor (SP) Username and Password. They ought
# to be constant whichever the 3PAR SP.
SP_username="spvar"
SP_password="3parvar"

#
# This function prepares a notification of file transfers from the SP.
#
function send_mail_notification
{
       if [[ $1 != "" && $2 != "" ]]; then
           echo "$1 and $2 available in directory /tmp" >> /tmp/smtp_body.txt
       fi
       if [[ $1 != "" && $2 == "" ]]; then
           echo "$1 available in directory /tmp" >> /tmp/smtp_body.txt
       fi
       if [[ $1 == "" && $2 != "" ]]; then
           echo "$2 available in directory /tmp" >> /tmp/smtp_body.txt
       fi
}

#
# Function get_last_weekly
#
# This function is in charge of copying the last dated weekly report to /tmp
# The last dated weekly report has been produced on the preceeding Sunday.
# Input :
#      - Index toward the DNS or IP address stored into SP_hostname
# Output:
#      - /tmp/
#
function get_last_weekly
{
# Step 1:
# get SP's all files in weekly/ directory using sftp and 
# maintenance SSH account.
#
port=`echo ${SP_hostname[$1]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\2/g'`
hostname=`echo ${SP_hostname[$1]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\1/g'`
echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except
echo "expect -nobrace \" password: \"" >> /tmp/except
echo "send \"${SP_password}\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"ls weekly/\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"bye\n\"" >> /tmp/except
echo "wait" >> /tmp/except
echo "exit" >> /tmp/except
Weekly_files=`expect /tmp/except | gawk '{if (/weekly\//) {print $0;}}' | sed 's/\r//g' | sed 's/weekly\///g'`
for Weekly in ${Weekly_files}; do
     lastSunday=`date --date="last Sunday" +%y%m%d`
     echo ${Weekly} | sed 's/${lastSunday}//' > /dev/null
     if [[ $? == 0 ]]; then
         Weekly_file=${Weekly}
     fi
done
# Step 2:
# get SP's last dated weekly*.tbz2 in weekly/ directory using sftp and 
# maintenance SSH account.
echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except
echo "expect -nobrace \" password: \"" >> /tmp/except
echo "send \"${SP_password}\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"cd weekly\n\"" >> /tmp/except
echo "expect \"\nsftp> \"" >> /tmp/except
echo "send \"lcd /tmp\n\"" >> /tmp/except
echo "expect -nobrace \"sftp> \"" >> /tmp/except
echo "send \"get ${Weekly}\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"bye\n\"" >> /tmp/except
echo "wait" >> /tmp/except
echo "exit" >> /tmp/except
expect /tmp/except
Weekly_file=/tmp/${Weekly_file}
rm -f /tmp/except
}
#
# Function get_last_InSplore
# This function downloads to /tmp the today's Insplore file.
#
# Input :
#    - full path of SP's Insplore file
#    - Index toward the SP's IP address or DNS name.
# Output :
#    - the Insplore filename prefixed with /tmp/
#
function get_last_InSplore
{
# Step 1:
# check SP for today's InSplore file using sftp and 
# maintenance SSH account.
#
port=`echo ${SP_hostname[$2]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\2/g'`
hostname=`echo ${SP_hostname[$2]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\1/g'`
echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except
echo "expect -nobrace \" password: \"" >> /tmp/except
echo "send \"${SP_password}\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"ls $1\n\"" >> /tmp/except
echo "expect -nobrace \"\nsftp> \"" >> /tmp/except
echo "send \"bye\n\"" >> /tmp/except
echo "wait" >> /tmp/except
echo "exit" >> /tmp/except
InSplore_file=`expect /tmp/except | gawk '{if (/$$1/) {print $0;}}'`
#
# Step 2:
# get SP's today's InSplore file if it exists and using the 
# maintenance SSH account.
#
echo ${InSplore_file} | sed 's/not found//'
if [[ $? != 0 ]]; then
   echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except
   echo "expect -nobrace \" password: \"" >> /tmp/except
   echo "send \"${SP_password}\n\"" >> /tmp/except
   echo "expect -nobrace \"sftp> \"" >> /tmp/except
   echo "send \"lcd /tmp\n\"" >> /tmp/except
   echo "expect -nobrace \"sftp> \"" >> /tmp/except
   echo "send \"get $1\n\"" >> /tmp/except
   echo "expect -nobrace \"sftp> \"" >> /tmp/except
   echo "send \"bye\n\"" >> /tmp/except
   echo "wait" >> /tmp/except
   echo "exit" >> /tmp/except
   expect /tmp/except
   InSploreFile=/tmp/$(basename ${InSploreFile})
else
   InSploreFile=""
fi
rm -f /tmp/except
}
#
# Start of main script body.
#
touch /tmp/smtp_body.txt
i=0
until [ $i == ${#SP_hostname[@]} ]; do
  if [[ $SP_hostname[$i] != "" ]]; then
      #
      # First step : get last weekly
      #
      get_last_weekly $i
      #
      # Second step: Isolate the SP identification number from the weekly file.
      #
      LastSunday=`date --date="last Sunday" +%y%m%d`
      Today=`date +%Y%m%d`
      SP_num=`echo ${Weekly_file} | sed 's/\([\/A-Za-z0-9]*\)_weekly_\([0-9]*\)_'"${LastSunday}"'\.tbz2/\2/g'`
      InSploreFile=${SP_num}/insplore/InSplore.*-${SP_num}.${Today}.*.tbz2
      #
      # Third step: download if any today's InSplore file
      #
      get_last_InSplore ${InSploreFile} $i
      #
      # Fourth step: prepare notification email
      #
      if [[ ${InSploreFile} != "" ]]; then
            InSploreFile=$(basename ${InSploreFile})
      fi
      #
      # Prepare mail notification
      #
      send_mail_notification ${InSploreFile} $(basename ${Weekly_file})
  fi
  let i+=1
done
#
# send mail to recipients
#
smtp -o ${smtp_login} -server ${smtp_server} -s "Weekly and Insplore transfers over" -f /tmp/smtp_body.txt ${smtp_destinees}
# do cleanup
rm -f /tmp/smtp_body.txt
exit 0


MORE ABOUT EXPECT:

expect has been written in some compiled code (very likely C language according 
to ldd command below) to produce an executable which is linked with the Tcl
binary library. Here is the proof on Linux Fedora 13 (i686 version):

[philippe@victor ~]$ file /usr/bin/expect
/usr/bin/expect: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped

So from above a 32 bits executable

[philippe@victor ~]$ ldd /usr/bin/expect
        linux-gate.so.1 =>  (0x00737000)
        libexpect5.43.so => /usr/lib/libexpect5.43.so (0x00a03000)
        libtcl8.5.so => /usr/lib/libtcl8.5.so (0x00110000)
        libdl.so.2 => /lib/libdl.so.2 (0x009fc000)
        libm.so.6 => /lib/libm.so.6 (0x009d0000)
        libutil.so.1 => /lib/libutil.so.1 (0x06547000)
        libc.so.6 => /lib/libc.so.6 (0x00858000)
        /lib/ld-linux.so.2 (0x00836000)

So from above linked with libtcl library.


HP-UX and EXPECT:

For HP-UX, you may download expect from :
http://hpux.connect.org.uk/hppd/hpux/Tcl/expect-5.45/


REFERENCE(S):

As SFTP -b option does not prompt for any passphrase/password, this solution
is unuseable. The ksh script excerp above is based on one guest input at:
http://www.linux-bsd-central.com/index.php/content/view/26/