Scp compression can speed up SCP file transfer, and sometimes not. To send a file via ssh is practical if you enjoy working inside the terminal window. Scp between Linux servers is my favorite casual file transfer tool when working in ssh.
Scp compression
There is a built-in compression tool into scp. This can save you a lot of bandwidth as it uses gzip to compression your files before transferring your files. Your files will be automatically decompressed on the other end. So it is an easy and convenient way to save bandwidth and speed up SCP file transfer. But not always. Be aware of that.
If you are on a high-speed connection to your remote machine. Either it could be at home or in the same datacenter as you, or maybe between virtual machines on the same host. The SCP compression is more likely to slow down your SCP file transfer instead of speeding it up. The reason is SCP uses the CPU to compress your files. Unless you got a high-end CPU it might take longer to compress the files, than it takes to send them. Or is that true?
I did a few tests on my home network. I got to large files. A 2.5GB Iso file and a 1GB text file I wanted to measure the transmission time to see which method was fastest. My home network runs on a 1Gbit Cisco Switch, and the servers all have 1Gbit network cards. Let us test some secure copy Linux!
Local CPU: Intel(R) Xeon(R) CPU E3-1240 V2 @ 3.40GHz
Remote CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz
Scp compression testing
Test 1 – 2.5GB Iso file with no compression
Transfer speed was 89.0MB/s and the time used was 31 seconds.
Test 2 – 2.5GB Iso file with compression
Transfer speed was 89.0MB/s and the time used was 31 seconds.
So exact speed and time in both tests. Buy maybe the iso file was already compressed to a level where it couldn’t be more compressed? That could mean the nothing for SCP to compress. Let us find out with a large text file. I created a random text file on a total of 1GB. A text file can usually be compressed quite a bit.
Test 3 – 1GB text file with no compression
Transfer speed was 83.3MB/s and time used was 12 seconds
Test 4 – 1GB text file with compression
Transfer speed was 83,3MB/s and time used was 12 seconds
Identical results again. How can this be? The computers were not virtual machines on the same host, and the result was identical in both tests. I even tried to scp to a remote host in a data center in another country. Since my internet connection is very limited I didn’t transfer the largest files. There were finally a small difference.
Test 5 – 100MB text file with no compression
Transfer speed was 1.2MB/s and time used was 1 minute and 24 seconds.
Test 6 – 100MB text file with compression
Transfer speed was 1.4MB/s and time used was 1 minute and 13 seconds.
SCP compression – Last test
It makes sense that a compressed text file should take less. But not much of a difference, and it could very well be caused by a lot of things. Like connection, somewhere between here and Germany or my son downloading some crap. Let us run the test again to be sure.
Test 7 – 100MB text file with no compression
Transfer speed was 1.4MB/s and time used was 1 minute and 14 seconds.
Test 8 – 100MB text file with compression
Transfer speed was 1.7MB/s and time used was 1 minute and 0 seconds.
The difference in time is the same, but the speed was a little higher this time. So it looks like you can save some time compressing text files on a very limited connection.
SCP usage
Let us look a little bit on how to use SCP. To SCP multiple files you add the parameter -r (for recursive), that will copy the entire folder. Scp example on how to SCP copy directory:
Scp -r mydirectory user@myhost
You can also use wildcard (*) to copy multiple files at ones. Just use the wildcard like in any other Linux command involving wildcard. An little example SCP to remote using wildcards to copy multiple files. This example will transfer all file starting with myfiles:
Scp myfiles* user@myhost
Another quick example on how to scp from remote to local. There is no need, even though it is quick and simple with ssh, to login to the remote host to send files to your local machine. Just download the file with SCP. Another SCP command example:
Scp user@host:/file-path
If you liked my scp compression article check out my other ssh articles linked above for more information. Or read my article about slow ssh login on Debian systems. Or read more about secure copy.
PHP ZipArchive is a file archive for zip compression used by many web applications and plugins. How to install PHP ZipArchive for Debian is an easy task. Let’s see how we install it.
PHP ZipArchive Library is missing or disabled
PHP ZipArchive Library is missing or disabled is the usual message you get if it is missing or disabled. So let’s assume it is missing, as in not installed. You only need to run one line of commands.
sudo apt-get install php7.0-zip
Now you should have installed PHP ZipArchive. Don’t forget to restart your web server too. If you are running Apache2, just run:
sudo service apache2 restart
PHP ZipArchive should be ready for your web application or plugin now.
The fastest Debian mirror or repository can be found by using a small tool called netselect apt (or netselect-apt as the package name). Unless you run your own local apt repository, check this article on how to find the fastest Debian apt mirror for you. This will, of course, vary depending on where you live or connect to the internet from.
Find the fastest Debian mirror for you
You could do a manual mirror check of all the mirror archive you find on the Debian mirror list. That would take like forever. Or you could use netselect that will check the Debian mirrors and find the fastest one for you. It is not always the official Debian mirror in your country that will be fastest for you.
You install netselect-apt by typing (as root or sudo):
apt-get install netselect-apt
Once the netselect-apt is installed, we fire it up to find our fastest Debian mirror. Before we run it, let’s look at the arguments to use with netselect-apt.
Netselect-apt options
-a (Use mirrors containing ARCH)
-s (Include the deb-src lines to get the Debian source packages)
-i (Use INFILE instead of downloading mirror list as a temporary file)
-o (Outfile instead of sources.list)
-n (Include the non-free section in sources.list or Outfile)
-f (Use FTP instead of HTTP mirrors)
How to use netselect-apt
The syntax is: netselect-apt [options] [release]
So if we want to find mirror for Debian 9 (stretch) including source and the non-free section we run the command like this. You need to use the name of Debian and not the version number. To find out the name of your version, check out this list.
netselect-apt -s -n stretch
For me here in Norway on Altibox network, the output will be like this. The program finds the best and fast mirror, at the moment, of all Debian mirrors. The speed of a mirror server can change from day to day, depending on a lot of factors. Usage and network speed is only a few of them.
If you want to see in a video how to find the fastest Debian mirror, check out the video below.
If you like this find the fastest Debian mirror and need to read up on Debian 9 Static IP, check out that link.
Setting Debian 9 static IP is not too different from the previous version. You might find the network cards named a bit differently now. Apart from that, it is mostly the same. And we will look at an alternative at the end, and an issue while configuring Debian 9 static IP.
Debian 9 Static IP
Edit network interfaces
Let’s look at how to give your Debian 9 a static IP address. The first thing you need is to edit the network interfaces. Launch your favorite text editor and edit /etc/network/interfaces. It will look something like below. In this example, I’m using a wired network interface. It will now be called ens3 instead of eth0 like in earlier versions. Your interface might have a different name.
You need to change DHCP to static. Then you need to add the address, gateway, and netmask. In this example, I used 192.168.2.86 for address and 1921.68.2.1 for the gateway (because that is the address of my gateway). Most likely your settings will be different. Change them to fit your network. After editing the network interfaces my config looks like this after setting Debian 9 static IP.
Apply the Debian 9 static IP
To apply the changes we need to restart the network service. If you are connected to the computer via SSH, you will need to reboot. You will lose connection when restarting the network service if you are on SSH. If you are local or a KVM console restart your interface like below.
To restart the network service just type into the terminal or console: service networking restart (typing /etc/init.d/networking restart will also do). If you at this point check your network settings with ifconfig or ip addr, the network will be unreachable. To fix this run: ifup ens3 (or whatever your interface is called). Now run: ip addr, to verify your new settings.
Here I noticed my new static IP had become a secondary IP. I don’t want that. I only need 1 IP for this machine. At this point, I can use both 192.168.2.86 and 192.168.2.59 (which was the IP given by the DHCP server). I just rebooted to remove the DHCP assigned IP address.
Another option to static IP addresses
There is another option to a static IP address. Your router, if it is not too old and simple, has an option to assign IP addresses to specific mac addresses. On my wifi router, also used as a DHCP server it looks like this.
This is an inexpensive Asus RT-N66U router. But most wifi routers have this option to assign mac addresses to IP addresses. I prefer using Debian 9 static IP addresses instead of assigning them to the router.
Debian ifconfig command not found. Can’t find ifconfig on Debian? You have probably installed a minimum installation without installing the standard system utilities. But do you really need ifconfig? If it just for checking the ip addresses, check out a Debian ifconfig alternative below.
Debian ifconfig
First, let us install the ifconfig command. After all, I assume that is how you found this article. I assume you got superuser or admin rights on the computer. Without the standard system utilities installed you need to install almost everything you need yourself. The package you need to install is net-tools.
apt-get install net-tools
Now you have the ifconfig command available. If you don’t mind Debian ifconfig missing, there is an option built-in. The IP command should be in the sbin folder no matter if you installed the system tools or not. The complete command to display the same info at ifconfig is:
ip addr show
This command can be shortened to:
ip a
It is exactly the same as the ip addr show command. So if Debian doesn’t have ifconfig, try to use the ip command instead. No installation needed. It will be in all installation. However, the net-tools package is only about 500-600kB. So it will hardly use any space on your hard drive. And I guess it comes down to preference if you want to use ifconfig or ip.
Some prefer not to install the Standard System Utilities. If you chose not to install it, your security updates in the apt config (repository) might not be set. And I would check if the security repository is added to the repository list.
Debian 9 change hostname is done very similarly to the previous Debian versions. With one major difference. It is how you apply the changes. If you don’t apply the changes, the changes will not be taken into effect until you reboot or restart your system. The same way if you use the hostname commands to change the hostname, the hostname is reverted when you reboot. Let us see how to Debian 9 change hostname permanently.
In the following example, let us assume the server name is example.server.com.
Debian 9 change hostname permanently
If you run the command hostname, it will return example. That is the hostname of your system.
root@example:~# hostname
example
Domain Name
When adding the parameter -d to the hostname command it will give you the domain name currently set for your system.
root@example:~# hostname -d
server.com
FQDN / Fully Qualified Domain Name
Change the parameter to -f and you get the FQDN or Fully Qualified Domain Name instead. Here is an example.
root@example:~# hostname -f
example.server.com
Change the hostname
To change the hostname you need to edit the /etc/hostname file. If you use the hostname command, it will not be permanent. The hostname file will only contain the hostname. The picture below is from one of my database servers.
Change the Domain or FQDN
Any changes to the domain name or the FQDN has to be done in the /etc/hosts file. Here is an example of how a hosts file looks. Again from my database server.
Apply the changes
As mentioned above the changes needs to be applied, like restarting or reloading the hostname service. If not the changes will not be applied until you reboot or restart your system.
This is how you apply the settings on a Debian 9 system:
SSH slow login can be a result of recent network changes. If your SSH connection was ok before, and now takes a lot longer. The problem is either with your network or the terminal you try to using SSH protocol from. You can also try to login to the terminal from the localhost. Both can be tedious to fault find.
SSH slow login
Here is what happened to me, and how I fixed it. One day all my servers at home suddenly took 10-15 seconds or longer to connect to with SSH. My external servers behaved normally. So I knew it wasn’t my laptop. Also, all my servers, both at home and externally, is running Debian. Various versions of Debian, but only Debian.
My local internet provider has these really crappy fiber cable modems. Extremely limited settings and general bad quality with its built-in wifi too. I eventually got a decent Asus wifi router but had to move my servers at home to a different subnet. That resulted in ssh login takes very long time. It was not a slow ssh connection, just login to ssh was slow.
I configured the network interface changes, but that did not change anything (except the internet connection). I checked the OpenSSH log. That didn’t give anything away either. Then I noticed the nameserver address in the resolv.conf file. It was set to the old subnet. Here is how my resolv.conf file looks:
cat /etc/resolv.conf
domain home
search home
nameserver 192.168.2.1
After I changed the nameserver over to the new subnet for the Asus router it works like before. Lightning quick login to ssh. Just like before I installed the new router. Because of some stupid limitations in the fiber cable modem, I really had to create a new subnet for the home servers.
Wrong nameserver setting is not the only reason for ssh slow login. There can be other reasons for ssh slow login too. It can also be a good idea to check the configuration for the host in the hosts file.
UseDNS
Some have fixed the issue by setting UseDNS setting to no in the ssh config (sshd_config). Like this:
sudo vim /etc/sshd_config
UseDNS no
That will just disable the DNS for the secure shell server. I think just setting the correct nameserver address will be a better solution. Especially if a slow ssh login is a problem when connecting from an ssh client.
GSSAPI authentication
Other have solved slow ssh connection issues by disabling the GSSApi authentication. GSSAPI is a framework for authentication. You could try that too if none of the above worked out for your server. You disable the GSSAPI authentication in the sshd_config file.
sudo vim /etc/sshd_config
GSSAPIAuthentication no
ssh -v
If none of the tips above helps. Try adding the -v argument when trying to connect to your ssh server. It will print out everything that is going on, and you can see where it hangs. If it is the nameserver settings that is wrong, it will look like the picture below.
After doing these changes a restart is recommended.
If you enjoyed my ssh slow login article you can check out my SSH login message change here. Or how to use the SCP command. An excellent method to transfer files with an encrypted connection on port 22.
Debian Apache log files contain tons of useful information about your web server and websites. Here I will show how to install GoAccess to help you get data and statistics on a lot of information. Here is what data you will easily visualize with GoAccess.
– Statistics and bandwidth usage
– Visitors data, top visitor, referring sites and links, 404
– Ip location, hosts, reverse DNS
– Operating systems and browser statistics
– Other Http status codes
– Geo Location
You can go to their website to check out more of this amazing Debian Apache log file tool.
How to Install GoAccess for Debian
GoAccess supports several Linux distributions, but I will only cover how to install it for Debian. By the way, it is the same installation for Ubuntu. GoAccess has been available since Debian 6 (Squeeze).
apt-get install goaccess
That will install the latest version of GoAccess made available to Debian package manager.
How to use GoAccess on Debian
Type the following and hit enter: goaccess
This will list all available parameters and arguments for GoAccess.
The argument we are interested in is -f. To specify the path to the log files. If we test it on this site, it would look something like this:
You might get a fatal error message where GoAccess complaints about missing time format. Just go to /etc/goaccess.conf and uncomment the time, the date and log format you want to use.
This will generate an HTML report like the one below. But you can have a look your self by clicking the report link above.
So this is how you install and some basic usage of GoAccess on a Debian system with Apache web server and Debian Apache log file analyzing.
Swift, Apples new programming language, to take over for Objective C has been open source for some time now. It only took at short time before it was available for various Linux Distros. The latest Swift version in the official Debian repos are 2.3.1. If you want version 3.x of swift Debian you need download and install it your self. Here is how to do it.
Use sudo or root to create directories and install Swift.
We will install Swift Debian in the /opt/swift/build folder. So first: cd /opt
Then create the new folder for our Swift build. mkdir -p swift/build
Change to the new directory before we download Swift. cd swift/build
Download the swift build from swift.org wget https://swift.org/builds/swift-3.0.2-release/ubuntu1404/swift-3.0.2-RELEASE/swift-3.0.2-RELEASE-ubuntu14.04.tar.gz
The Swift Debian is still compressed, so we need to extract it. Make sure you downloaded the file to /opt/swift/build directory. tar zxvf swift-3.0.2-RELEASE-ubuntu14.04
Now we need to make sure our system knows where the swift compiler is stored. For immediate effect you can paste this line. export PATH=/opt/swift/build/swift-3.0.2-RELEASE-ubuntu14.04/usr/bin:"${PATH}"
But once you logged out, that setting will be gone. To make that permanent go to your home folder. Once back in your home folder (/home/what-ever-your-user-is-named), you want to edit the .profile file. vim .profile
And paste this in at the end of the file. PATH=/opt/swift/build/swift-3.0.2-RELEASE-ubuntu14.04/usr/bin:"${PATH}"
To make sure it works, you can disconnect or log out (if it is a local system) and log back in. Try to type in swift and press enter. The swift console should now open. You might run into some errors if you try to compile anything now. So we need to install a bunch of packages. Install these packages by typing (or copy paste) apt-get install curl gcc python-dev python-pip libffi-dev python-setuptools sqlite3 git-core git cmake ninja-build clang uuid-dev libicu-dev icu-devtools libbsd-dev libedit-dev libxml2-dev libsqlite3-dev swig libpython-dev libncurses5-dev pkg-config
Now your system should be able to compile and make executable files from swift source codes. However if you are new to swift there is one thing you need to know. Create a folder for each project you make. Because your main source file needs to be called main.swift. If not the compiler will return error code 1.
You are now ready to use both the swift console and compile your files. Give it a test.
Create a folder and create main.swift And here is short sample file you can try to compile. let message = “Hello Debian World”
print(message)
Here I will explain Debian add hard drive. It is a little more to it than just connect the hard drive. Not much more, but a little bit more.
After you have connected the new hard drive to your Debian system and switched it on, your Debian system doesn’t know how to use your new hard drive. Debian knows it is there, but can’t use it for anything.
I have divided the process into four steps.
Debian add hard drive – Step 1
Partition your new hard drive. Even if you don’t need several partitions, you will need to create at least one partition.
Find all detected hard drives. sudo fdisk -l | grep ‘Disk’
That will output something like this. Disk /dev/sda: 500.1 GB, 500107862016 bytes Disk identifier: 0x00000000 Disk /dev/sdb: 60.0 GB, 60022480896 bytes Disk identifier: 0x7d09ee5f
sdb is new hard drive here. A small SSD I want to run MySQL databases on.
So we know the hard drive is there. Lets create that partition.
sudo fdisk /dev/sdb
Type n to create a new partition. And accept all default values. Once all the values are accepted, type w to write the new partition table and exit. If you messed up something and don’t want to write the new partition table, type q to exit without saving any changes. Everything you do in fdisk is just saved in memory until you hit w .
Debian add hard drive – Step 2
Format the new hard drive.
Here we decided we want to use ext3 partition on our new drive.
Type in: sudo mkfs.ext3 /dev/sdb1
(sdb1 is the new partition you created in the previous step)
Debian add hard drive – Step 3
Mount the new hard drive
We still can’t use the new hard drive until we mount it. That means we make it useable in our Debian System.
First we create a folder to mount our new hard drive to.
sudo mkdir /disk1
Then we mount sdb1 to the new folder.
sudo mount /dev/sdb1 /disk1
Now your new hard drive should be usable by the system. To verify this, type in df -h
You should find something like this:
/dev/sdb1 60G 22G 35G 39% /disk1
If you restart your system at this point, you will not see your new hard drive. That’s because we haven’t told the system to mount it automatically on boot.
Debian add hard drive – Step 4
Make the new hard drive automatically mounted on boot.
Open your /etc/fstab file. You can do that with vim, nano or some other text editor. I like vim. sudo vim /etc/fstab
Add this to the bottom of the file. /dev/sdb1 /disk1 ext3 defaults 1 2
Save and close the file. In vim you do that by first pressing the ESC key, and type :wq
That is how you add a new hard drive to a Debian System.
