Scp compression can speed up SCP file transfer, and sometimes not. To send a file via ssh is practical if you enjoy working inside the terminal window. Scp between Linux servers is my favorite casual file transfer tool when working in ssh.
There is a built-in compression tool into scp. This can save you a lot of bandwidth as it uses gzip to compression your files before transferring your files. Your files will be automatically decompressed on the other end. So it is an easy and convenient way to save bandwidth and speed up SCP file transfer. But not always. Be aware of that.
If you are on a high-speed connection to your remote machine. Either it could be at home or in the same datacenter as you, or maybe between virtual machines on the same host. The SCP compression is more likely to slow down your SCP file transfer instead of speeding it up. The reason is SCP uses the CPU to compress your files. Unless you got a high-end CPU it might take longer to compress the files, than it takes to send them. Or is that true?
I did a few tests on my home network. I got to large files. A 2.5GB Iso file and a 1GB text file I wanted to measure the transmission time to see which method was fastest. My home network runs on a 1Gbit Cisco Switch, and the servers all have 1Gbit network cards. Let us test some secure copy Linux!
Local CPU: Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz
Remote CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
Test 1 – 2.5GB Iso file with no compression
Transfer speed was 89.0MB/s and the time used was 31 seconds.
Test 2 – 2.5GB Iso file with compression
Transfer speed was 89.0MB/s and the time used was 31 seconds.
So exact speed and time in both tests. Buy maybe the iso file was already compressed to a level where it couldn’t be more compressed? That could mean the nothing for SCP to compress. Let us find out with a large text file. I created a random text file on a total of 1GB. A text file can usually be compressed quite a bit.
Test 3 – 1GB text file with no compression
Transfer speed was 83.3MB/s and time used was 12 seconds
Test 4 – 1GB text file with compression
Transfer speed was 83,3MB/s and time used was 12 seconds
Identical results again. How can this be? The computers were not virtual machines on the same host, and the result was identical in both tests. I even tried to scp to a remote host in a data center in another country. Since my internet connection is very limited I didn’t transfer the largest files. There were finally a small difference.
Test 5 – 100MB text file with no compression
Transfer speed was 1.2MB/s and time used was 1 minute and 24 seconds.
Test 6 – 100MB text file with compression
Transfer speed was 1.4MB/s and time used was 1 minute and 13 seconds.
It makes sense that a compressed text file should take less. But not much of a difference, and it could very well be caused by a lot of things. Like connection, somewhere between here and Germany or my son downloading some crap. Let us run the test again to be sure.
Test 7 – 100MB text file with no compression
Transfer speed was 1.4MB/s and time used was 1 minute and 14 seconds.
Test 8 – 100MB text file with compression
Transfer speed was 1.7MB/s and time used was 1 minute and 0 seconds.
The difference in time is the same, but the speed was a little higher this time. So it looks like you can save some time compressing text files on a very limited connection.
Let us look a little bit on how to use SCP. To SCP multiple files you add the parameter -r (for recursive), that will copy the entire folder. Scp example on how to SCP copy directory:
Scp -r mydirectory user@myhost
You can also use wildcard (*) to copy multiple files at ones. Just use the wildcard like in any other Linux command involving wildcard. An little example SCP to remote using wildcards to copy multiple files. This example will transfer all file starting with myfiles:
Scp myfiles* user@myhost
Another quick example on how to scp from remote to local. There is no need, even though it is quick and simple with ssh, to login to the remote host to send files to your local machine. Just download the file with SCP. Another SCP command example:
Scp user@host:/file-path
If you liked my scp compression article check out my other ssh articles linked above for more information. Or read my article about slow ssh login on Debian systems. Or read more about secure copy.
That it about scp compression.
Posted in Debian Tagged with: scp compression