💾 Archived View for ruario.flounder.online › gemlog › 2022-01-16_One_shot_part2.gmi captured on 2023-09-08 at 16:02:44. Gemini links have been rewritten to link to archived content
⬅️ Previous capture (2022-03-01)
-=-=-=-=-=-=-
Previously, I wrote about using sftp to simulate the behaviour of scp, on systems where scp could not be used.
"One shot" sftp (simulating scp)
Since then Szczeżuja wrote a little response on his tiny log about his own troubles getting scp working when uploading files to SDF and how he used tar pipes over ssh to transfer content. This is a good idea, especially if you want to transfer whole directory trees. It also allows you use compression, to speed up transfers and/or reduce bandwidth.
Szczeżuja's tinylog § Fri 14 Jan 2022 04:34:56 PM CET
However, I should perhaps better explain why I wrote my initial post. Often systems are setup to limit to just sftp. In such cases there is no shell (and hence tar pipes cannot work) and no scp, which on many ssh severs (like OpenSSH) is tied to shell access and cannot be enabled on its own—at least not without installing and configuring further software like scponly (to better contain the shell environment).
The reason why scp is limited and usually tied to shell access seems to relate to security. Consider the following from the OpenSSH 8.0 release announcement (from 2019-04-17).
OpenSSH 8.0 release announcement § Security
This release contains mitigation for a weakness in the scp(1) tool and protocol (CVE-2019-6111): when copying files from a remote system to a local directory, scp(1) did not verify that the filenames that the server sent matched those requested by the client. This could allow a hostile server to create or clobber unexpected local files with attacker-controlled content.
This release adds client-side checking that the filenames sent from the server match the command-line request,
The scp protocol is outdated, inflexible and not readily fixed. We recommend the use of more modern protocols like sftp and rsync for file transfer instead.
Regardless of the reasons why scp is often unavailable, the fact remains that the workflow is more convenient and faster than the default behaviour of the standard sftp client. A scp transfer is a one liner, while sftp requires the connection step, one or more 'put' commands and a disconnect step ('exit', 'quit', 'bye' or '^D'). It is just a lot of typing for a quick file transfer. To be fair you do not have to use the classic sftp client. There are plenty of others. Perhaps the nicest for me is mounting the sftp connection using sshfs and indeed that is what I do on Linux. However I also tend to flip around between systems and 'sftp' is always present, while other options may not be. Now a little shell script wrapper is technically 'extra software' but it is tiny and does not need to be properly 'installed'.
Ok, I have written a lot already, so for anyone getting bored, here is the script.
[✍ 12:10 +0100: updated to use printf instead of echo]
[✍ 16:30 +0100: removed extra quoting again as it prevented wildcards]
[✍ 18:44 +0100: added a fix for spaces in filenames]
[✍ 2022-01-17 08:43 +0100: another fix for wildcards]
[✍ 2022-01-17 11:06 +0100: stops on errors or unset variables; compacted slightly]
[✍ 2022-01-19 18:06 +0100: removed the need for 'sed' and fixed shebang]
#!/usr/bin/env -S bash -eu
if [ "${1:-}" = '-P' ]; then
p="${@:1:2}"
shift 2
fi
eval l=\${$#}
if [[ "$l" == *:* ]]; then
for f in "${@:1:$(($#-1))}"; do
printf '%s\n' "put ${f// /\\ }"
done | sftp ${p:-} "$l"
else
f="${1#*:}"
printf '%s\n' "get ${f// /\\ } \"$2\"" | sftp ${p:-} "${1%%:*}"
fi
Before you run away, if you are considering using this, read § Limitations.
For anyone sticking around, my final example from the last post had the following issues.
The first one is easy, I just forgot a couple of curly brackets (braces '{}'). The second was also obvious once I looked at it again. I just moved the sftp command out of the loop, so that it is a single process that receives all the puts together.
To handle uploads and have things work like scp, we have to do a couple of things. Firstly scp expects the file(s) to be listed first on upload and the server first on download. Secondly, the server name always has a colon (':') at the end, followed by the destination. The colon actually helps because we can look for it in the final argument and if present we know it is an upload, otherwise we assume it is a download. For downloads we simplify things as do not need to run through a loop but just pick the correct part (server or file), either side of the colon.
Personally, I need to connect to servers that use non-standard ports (e.g. 🐟flounder). For this I just collect the first couple of arguments, if the initial one is '-P'. More options could be supported using 'getopts' but I don't (generally) need them and it would more than double the length of the script. I'll add more if I ever need them or perhaps just use sftp directly.
[✍ 18:28 +0100] I am now filtering filenames with 'printf %q' to escape spaces before they are piped to sftp—I cannot hardcode quotes around all filenames or wildcards will fail.
[✍ 2022-01-17 08:43 +0100] 'printf %q' was problematic as it prevented wildcards, like '*'. Instead, I now filter filenames with sed, purely to escape 'spaces' before they are piped to sftp but not other characters that 'printf %q ' might be able to handle. The downside for this is if you had a file that actually contained an 'odd' character like '*' it would need to be double escaped.
And that is it… for now 😉. I hope this is useful to someone. If not… 🤷🏼
⁂
[✍ 16:30 +0100] Here is another version using only POSIX shell (tested against dash).
[✍ 18:44 +0100: fixed an issue with quoting problems when a filename contains spaces]
[✍ 2022-01-17 11:06 +0100: stops on errors or unset variables; compacted slightly]
#!/bin/sh -eu
if [ "${1:-}" = '-P' ]; then
p="$1 ${2:-}"
shift 2
fi
eval l=\${$#}
s () { printf '%s\n' "$1" | sed 's/ /\\ /g'; }
case "$l" in
*:*)
c=0; t="$(($#-1))"
for f in "$@"; do
if [ "$c" -lt "$t" ]; then
printf '%s\n' "put $(s "$f")"
c="$(($c+1))"
fi
done | sftp ${p:-} "$l"
;;
*)
printf '%s\n' "get $(s "${1#*:}") \"$2\"" | sftp ${p:-} "${1%%:*}"
;;
esac
⁂