Transfer files between local and remote machines 

(Updated in January 2, 2023)

There is a common need for our university researchers to securely transfer small or large files to virtual machines for various purposes. They also need to download some result files or data to their local machines. We are referring to Linux virtual machines in this post unless we explicitly mention Windows virtual machines as well. 

There are no requirements for retaining the permissions on the files being transferred. In this post, we will not go through performance considerations for each tool/method.  

I decided to write down four ways to achieve this goal.  

• SCP 

• SFTP 

• RSYNC 

• Azure File Share 

The first three methods can be used to access any virtual machines that you have ssh connection. If you are using Azure virtual machines and login with Azure Active Directory, you will get more instructions on how to connect. The last method is specific to Microsoft Azure and requires a bit of setup by Azure administrator as well as installing some software.  

SCP

SCP is a common tool for secure copy files between a local and remote machine. It uses the same authentication and security as SSH. You need a SCP client which is included in Bash shell of most Linux distributions and Mac computers and PowerShell.  

Examples 

If you are using username and password for SSH authentication, you will be prompted to enter your password. However, if you are using SSH key pair authentication, SSH will authenticate you using your private key and there is no need to enter your password (best practice). If you created your key pairs with a passphrase, you will be prompted to enter the passphrase.  

From local to remote 

scp file username@host-ip:target-directory

From remote to local 

scp username@host-ip:target-directory/file /local-directory

If you login to your Azure virtual machine using Azure Active Directory (AD), you need to create a temporary config file to use with SCP (thanks to Nigel Sim post). You are already familiar with login with AD using one of the commands below: 

az ssh vm --ip 1.2.3.4

Or

az ssh vm --resource-group myResourceGroup --name myVM

The config file can be generated for one specific virtual machine using one of the options below: 

Using VM IP address

az ssh config --ip 1.2.3.4 --file ./sshconfig

Using Resource Group and VM name

az ssh config --resource-group myResourceGroup --name myVm --file ./sshconfig

Or it can be generated as a generic config file to use with any host: 

az ssh config --ip \* --file ./sshconfig

This will generate Azure certificates and SSH keys required for connection. These config files can be used by any client that supports OpenSSH configs and certificates. Please refer to Microsoft official documentation for more details on this topic. 

Important note: The config file contains sensitive information and should be deleted when not needed. At the time of writing this post, there is a time limit for validity of the certificates and SSH keys.  

Now that the configs are ready, we can use SCP using the commands below: 

From local to remote 

scp -F ./sshconfig file 1.2.3.4:~/

From remote to local 

scp -F ./sshconfig 1.2.3.4:~/file /local-directory

Example for copying a folder from local to remote:

scp -F ./sshconfig -r folder-to-be-copied 1.2.3.4:~/

SFTP 

SFTP is another easy-to-use tool for transferring files over SSH and it replaces FTP. Like SCP, SFTP client is also included in Bash shell in most Linux and Mac computers as well as PowerShell. First, you need to connect to your remote machine using the command below: 

stfp username@host-ip

Or in the case of Azure AD login: 

sftp –F ./sshconfig 1.2.3.4 

After successful authentication, you will be inside a sftp session. You can use ‘help’ command to learn how to move files from local machine to remote and vice versa. Learn more about those commands (‘put’ and ‘get’) here.  

sftp>help

RSYNC 

RSYNC is a robust tool for transferring large files securely. It uses delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Additionally, it handles file transfer interruptions elegantly.  

Examples 

From local to remote 

rsync -avP file username@host-ip:~/ 

From remote to local 

rsync -avP username@host-ip:~/file /local-directory

And if you are using Azure AD authentication and have your SSH config file created as instructed above, use the commands below: 

From local to remote 

rsync -e 'ssh -F ./sshconfig' -avP file 1.2.3.4:~/target-directory

From remote to local 

rsync -e 'ssh -F ./sshconfig' -avP 1.2.3.4:~/file /local-directory

Azure File Share 

Azure Files is a fully managed service for file shares in the cloud which are accessible via SMB (Server Message Block) protocol or NFS (Network File System) protocol. Azure file shares can be mounted in Windows, Linux, and macOS. You can mount the file shares to virtual machines and local machines as illustrated in the image below. 

For Azure Admins 

There are many well-written tutorials on how to set up Azure Storage account and create a file share. Here are official instructions on the Microsoft website.  
 

When your file share(s) are created, follow instructions for mounting the shares in Windows or Linux from the links provided. Please note that mounting the shares on local machines and virtual machines is similar with some consideration:  

• Azure Storage Account and the virtual machine need to be in the same Azure region. Otherwise, the OS must support the encryption functionality of SMB 3.0.  
• There is a requirement for SMB 3.0 to use TCP port 445 to establish a secure connection. If this port is closed in your network, you can download free Azure Storage Explorer on your machine to upload and download files in file share. Alternatively, if you have access rights, you can use Azure portal and storage account to do file transfer operations from browser. 

Once the file shares are mounted, they can be treated like normal drives on the machine. You can start transferring files to the mounted drive.  

Further Reading

There is an open feature request on Azure-cli Github repository to add the functionality to azure-cli tool to copy files to/from virtual machines. You can follow the updates here.

AzCopy is another command-line tool for transferring files to and from Azure storage account. Azure Storage Explorer uses AzCopy under the hood for data transfer operations.

If you are interested to learn more about the difference between SCP and RSYNC, read the discussions on this Stackoverflow question.