[Linux, OpenSSL, expect] Doing SFTP file transfers from a shell script PRODUCT: OpenSSH_5.4p1, OpenSSL 1.0.0a-fips 1 Jun 2010 expect version 5.43.0 OP/SYS: Linux Fedora 13 COMPONENT: SFTP, SCP SOURCE: Philippe Vouters Fontainebleau/France LOW-COST HIGH-TECH: http://techno-star.fr SYMPTOM(S) or PROBLEM(S): The SFTP -b or the SCP -B options enables batch mode. However, under OpenSSH version 5.4p1, these options require an empty passphrase and do not allow for password prompting. What to do when the remote system only has port 22 opened and only accepts OpenSSH logging using a login/password pair ? This is where expect enables to workaround this OpenSSH restriction where you do not have the target account RSA key with an empty passphrase. If you own the target account RSA key with an empty passphrase then you may use the simpler sftp -b script user@host or scp equivalent shell command. SOLUTION or RESPONSE: Use expect to synchronize with each SFTP/SCP output and to simulate their inputs. Expect acts on terminal screens much the same way as the non-interactive telnet tool found elsewhere in this knowledge database. WORKAROUND or ALTERNATIVE: In your shell script, use SFTP/SCP in interactive mode and synchronize their output and input using expect commands. EXPECT INFORMATION: expect is an optional software. Under Fedora 13 and a root account, use yum to install expect. # yum install expect KSH SHELL SCRIPT NOTES: This ksh script expects that the SFTP software is BSD based and accepts the [-P port] option. Such an SFTP client is available under Linux Fedora 13. Carefully check your "man 1 sftp" before blindly using this ksh script. For example this sftp man at http://linux.die.net/man/1/sftp which claims BSD compatibility is not suited for the below ksh script. The following man at http://linuxreviews.org/man/sftp/ is fully suited and corresponds to the author's Linux Fedora 13 computer sftp man which he based this ksh script upon. If running a Linux SFTP client software whose man is identical to what is described at http://linux.die.net/man/1/sftp, you would modify all occurences of -P ${port} in the ksh script below to -oPort=${port} such as given as an example in the http://linux.die.net/man/1/sftp Web browser displayable man. KSH SHELL FULL SCRIPT: #!/bin/ksh # # Copyright (C) 2010 by Philippe.Vouters@laposte.net # # This program is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by the # Free Software Foundation, either version 3 of the License, or # any later version. # # This program is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>;. # # Software Description: # This software transfers the latest 3PAR (a SAN equipment) weekly*.tbz2 and # optionally the InSplore*.tbz2 files to /tmp local directory. The InSplore # file is transfered provided it has been produced the same day than the run of # this script. # # This script makes uses of optional expect freeware and an smtp client whose # source code is downloadable from http://vouters.dyndns.org/zip/. # Under Fedora 12 and a root account: # # yum install expect # For smtp: # # unzip -aa smtp.zip # # export bin=/usr/local/bin # # cd smtp # # make -f Makefile_smtp # # Enter the IP address or the DNS name with the IP port separated with a colon # of each Service Processor while incrementing the index. # SP_hostname[0]="127.0.0.1:10001" # corresponds to 10.10.31.41:22 SP_hostname[1]="127.0.0.1:10002" # corresponds to 10.10.52.25:22 # # Indicate your email account characteristics as well as a list of remote # recipients with a space as the delimiter in the smtp_destinees variable. # smtp_server="smtp.sfr.fr" smtp_login="Philippe.Vouters@laposte.net" smtp_password="password" smtp_destinees="Philippe.Vouters@laposte.fr" # These are the 3PAR Service Processor (SP) Username and Password. They ought # to be constant whichever the 3PAR SP. SP_username="spvar" SP_password="3parvar" # # This function prepares a notification of file transfers from the SP. # function send_mail_notification { if [[ $1 != "" && $2 != "" ]]; then echo "$1 and $2 available in directory /tmp" >> /tmp/smtp_body.txt fi if [[ $1 != "" && $2 == "" ]]; then echo "$1 available in directory /tmp" >> /tmp/smtp_body.txt fi if [[ $1 == "" && $2 != "" ]]; then echo "$2 available in directory /tmp" >> /tmp/smtp_body.txt fi } # # Function get_last_weekly # # This function is in charge of copying the last dated weekly report to /tmp # The last dated weekly report has been produced on the preceeding Sunday. # Input : # - Index toward the DNS or IP address stored into SP_hostname # Output: # - /tmp/# function get_last_weekly { # Step 1: # get SP's all files in weekly/ directory using sftp and # maintenance SSH account. # port=`echo ${SP_hostname[$1]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\2/g'` hostname=`echo ${SP_hostname[$1]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\1/g'` echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except echo "expect -nobrace \" password: \"" >> /tmp/except echo "send \"${SP_password}\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"ls weekly/\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"bye\n\"" >> /tmp/except echo "wait" >> /tmp/except echo "exit" >> /tmp/except Weekly_files=`expect /tmp/except | gawk '{if (/weekly\//) {print $0;}}' | sed 's/\r//g' | sed 's/weekly\///g'` for Weekly in ${Weekly_files}; do lastSunday=`date --date="last Sunday" +%y%m%d` echo ${Weekly} | sed 's/${lastSunday}//' > /dev/null if [[ $? == 0 ]]; then Weekly_file=${Weekly} fi done # Step 2: # get SP's last dated weekly*.tbz2 in weekly/ directory using sftp and # maintenance SSH account. echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except echo "expect -nobrace \" password: \"" >> /tmp/except echo "send \"${SP_password}\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"cd weekly\n\"" >> /tmp/except echo "expect \"\nsftp> \"" >> /tmp/except echo "send \"lcd /tmp\n\"" >> /tmp/except echo "expect -nobrace \"sftp> \"" >> /tmp/except echo "send \"get ${Weekly}\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"bye\n\"" >> /tmp/except echo "wait" >> /tmp/except echo "exit" >> /tmp/except expect /tmp/except Weekly_file=/tmp/${Weekly_file} rm -f /tmp/except } # # Function get_last_InSplore # This function downloads to /tmp the today's Insplore file. # # Input : # - full path of SP's Insplore file # - Index toward the SP's IP address or DNS name. # Output : # - the Insplore filename prefixed with /tmp/ # function get_last_InSplore { # Step 1: # check SP for today's InSplore file using sftp and # maintenance SSH account. # port=`echo ${SP_hostname[$2]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\2/g'` hostname=`echo ${SP_hostname[$2]} | sed 's/\([\/A-Za-z0-9\.]*\):\([0-9]*\)/\1/g'` echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except echo "expect -nobrace \" password: \"" >> /tmp/except echo "send \"${SP_password}\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"ls $1\n\"" >> /tmp/except echo "expect -nobrace \"\nsftp> \"" >> /tmp/except echo "send \"bye\n\"" >> /tmp/except echo "wait" >> /tmp/except echo "exit" >> /tmp/except InSplore_file=`expect /tmp/except | gawk '{if (/$$1/) {print $0;}}'` # # Step 2: # get SP's today's InSplore file if it exists and using the # maintenance SSH account. # echo ${InSplore_file} | sed 's/not found//' if [[ $? != 0 ]]; then echo "spawn sftp -P ${port} ${SP_username}@${hostname}" > /tmp/except echo "expect -nobrace \" password: \"" >> /tmp/except echo "send \"${SP_password}\n\"" >> /tmp/except echo "expect -nobrace \"sftp> \"" >> /tmp/except echo "send \"lcd /tmp\n\"" >> /tmp/except echo "expect -nobrace \"sftp> \"" >> /tmp/except echo "send \"get $1\n\"" >> /tmp/except echo "expect -nobrace \"sftp> \"" >> /tmp/except echo "send \"bye\n\"" >> /tmp/except echo "wait" >> /tmp/except echo "exit" >> /tmp/except expect /tmp/except InSploreFile=/tmp/$(basename ${InSploreFile}) else InSploreFile="" fi rm -f /tmp/except } # # Start of main script body. # touch /tmp/smtp_body.txt i=0 until [ $i == ${#SP_hostname[@]} ]; do if [[ $SP_hostname[$i] != "" ]]; then # # First step : get last weekly # get_last_weekly $i # # Second step: Isolate the SP identification number from the weekly file. # LastSunday=`date --date="last Sunday" +%y%m%d` Today=`date +%Y%m%d` SP_num=`echo ${Weekly_file} | sed 's/\([\/A-Za-z0-9]*\)_weekly_\([0-9]*\)_'"${LastSunday}"'\.tbz2/\2/g'` InSploreFile=${SP_num}/insplore/InSplore.*-${SP_num}.${Today}.*.tbz2 # # Third step: download if any today's InSplore file # get_last_InSplore ${InSploreFile} $i # # Fourth step: prepare notification email # if [[ ${InSploreFile} != "" ]]; then InSploreFile=$(basename ${InSploreFile}) fi # # Prepare mail notification # send_mail_notification ${InSploreFile} $(basename ${Weekly_file}) fi let i+=1 done # # send mail to recipients # smtp -o ${smtp_login} -server ${smtp_server} -s "Weekly and Insplore transfers over" -f /tmp/smtp_body.txt ${smtp_destinees} # do cleanup rm -f /tmp/smtp_body.txt exit 0 MORE ABOUT EXPECT: expect has been written in some compiled code (very likely C language according to ldd command below) to produce an executable which is linked with the Tcl binary library. Here is the proof on Linux Fedora 13 (i686 version): [philippe@victor ~]$ file /usr/bin/expect /usr/bin/expect: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.18, stripped So from above a 32 bits executable [philippe@victor ~]$ ldd /usr/bin/expect linux-gate.so.1 => (0x00737000) libexpect5.43.so => /usr/lib/libexpect5.43.so (0x00a03000) libtcl8.5.so => /usr/lib/libtcl8.5.so (0x00110000) libdl.so.2 => /lib/libdl.so.2 (0x009fc000) libm.so.6 => /lib/libm.so.6 (0x009d0000) libutil.so.1 => /lib/libutil.so.1 (0x06547000) libc.so.6 => /lib/libc.so.6 (0x00858000) /lib/ld-linux.so.2 (0x00836000) So from above linked with libtcl library. HP-UX and EXPECT: For HP-UX, you may download expect from : http://hpux.connect.org.uk/hppd/hpux/Tcl/expect-5.45/ REFERENCE(S): As SFTP -b option does not prompt for any passphrase/password, this solution is unuseable. The ksh script excerp above is based on one guest input at: http://www.linux-bsd-central.com/index.php/content/view/26/
Tuesday, November 26, 2013
[Linux, OpenSSL, expect] Doing SFTP file transfers from a shell script
Subscribe to:
Posts (Atom)
-
During troubleshooting of issue at my customer in OneView, I have noticed that there is a SSL certificate issue with one of iLO4 of BL460ge... -
The wonderful security updates in Firefox (and Chrome) will give you the dreaded SSL_ERROR_BAD_MAC_ALERT . So no more ILO for you How to ...
-
During update from one version to another or sometimes when you reboot Zimbra you can get following error: "Error: Queue report unav...