Linux teardown checklist
With virtualisation it's very easy to quickly spin up new Linux hosts for development. Unfortunately this also means it's very easy to have a large number of unused hosts and old VM images. This post is going to go through places to check before completely deleting a Linux host.
Assuming the host is headless, shell history is one of the best places to look for information on what a host was used for. The history shell built-in can be used to show recently run commands:
$ history 1 shutdown -h now 2 python 3 cat /etc/yum.repos.d/CentOS-Base.repo 4 vim /etc/yum.repos.d/CentOS-Base.repo 5 yum search python-cv 6 yum search opencv 7 yum install opencv-python ...
Note: if multiple users have been set up, make sure you check the history for each user account.
Services and listening ports
ss can be used to quickly identify listening services which might have associated data you want to backup:
$ ss --numeric --tcp --udp --listening --process Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port udp UNCONN 0 0 *:38680 *:* users:(("dhclient",pid=649,fd=20)) udp UNCONN 0 0 *:68 *:* users:(("dhclient",pid=649,fd=6)) udp UNCONN 0 0 :::38602 :::* users:(("dhclient",pid=649,fd=21)) tcp LISTEN 0 128 *:22 *:* users:(("sshd",pid=927,fd=3)) tcp LISTEN 0 100 127.0.0.1:25 *:* users:(("master",pid=1023,fd=13)) tcp LISTEN 0 128 :::22 :::* users:(("sshd",pid=927,fd=4)) tcp LISTEN 0 100 ::1:25 :::* users:(("master",pid=1023,fd=14))
$ systemctl list-unit-files --state enabled UNIT FILE STATE accounts-daemon.service enabled auditd.service enabled bluetooth.service enabled crond.service enabled dbus-org.bluez.service enabled dbus-org.freedesktop.NetworkManager.service enabled dbus-org.freedesktop.nm-dispatcher.service enabled display-manager.service enabled ...
System configuration can normally be found in
/etc/. The exact files you want
to backup will obviously depend on the system, however below are a few
locations you may want to check for configuration:
/etc/fstab: file system mount points.
/etc/network/: on Debian systems network configuration is normally kept here.
/etc/sysconfig/network-scripts/: on Red Hat based systems network configuration is normally kept here.
/etc/hosts: static host entries.
/etc/resolv.conf: DNS configuration.
/etc/sysconfig/iptables: on Red Hat base systems, persistent iptables rules are stored here. You may also want to run iptables-save to dump the rules from memory.
/etc/sudoers.d/: sudo rules.
/etc/profile.d/: global shell login scripts.
/etc/apt/sources.list.d/: repository configuration for Apt on Debian based systems.
/etc/yum.repos.d/: Yum configuration on Red Hat based systems.
User accounts and home directories
Home directories are an obvious place to check for files which you might want
to recover before deleting a system. Initially just listing the directories in
/home is a good way to work out where you need to check:
$ ls /home/ alice
It's also worth checking entries in
/etc/passwd for any non-standard home
Once you've got a list of user home directories, have a look through each directory. When looking through the files, make sure you don't forget to look at hidden files. Below are a few hidden files you might want to consider backing up:
.ssh/: SSH configuration including private keys is normally stored in
~/.ssh. It's also worth making a note of any accounts using
~/.ssh/authorized_keysto allow remote users to connect.
.bash_profile: If you've setup any useful custom aliases you might want to back them up. Bash configuration may also point you at software being used on the system. For example if
PGDATAis being set, there is a good chance there may be a PostgreSQL database you might need to back up.
.vim/: If you use Vim, check for useful configuration you might want to carry over to another system.
.gnupg/: Check the GPG configuration directory for any keys you might want to keep.
Besides users home directories, the following places may contain data that might be worth backing up:
/var/lib/: this directory is used to hold variable state information. It's often used by services to store data, for example PostgreSQL often stores database files under
/srv/: data for services is sometimes kept here.
/var/www/html: This is the default document root for Apache.
/var/spool/mail/: user emails are normally stored here.
/var/log: you many want to check system logs for useful info.
Before removing a system, it's often useful making a list of packages which were installed. This can be helpful when trying to locate configuration you might want to backup, or if you decide to re-create the system.
Yum and RPM
For CentOS systems
yum history can be used to review Yum transactions:
$ yum history Loaded plugins: fastestmirror Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast ID | Login user | Date and time | Action(s) | Altered ------------------------------------------------------------------------------- 7 | root <root> | 2016-12-11 19:46 | Install | 4 6 | root <root> | 2016-12-11 19:32 | Install | 1 5 | root <root> | 2016-12-11 19:28 | Install | 194 4 | root <root> | 2016-12-11 19:04 | Install | 1 3 | root <root> | 2016-12-11 18:28 | I, U | 178 EE 2 | root <root> | 2016-12-11 18:24 | Install | 63 1 | System <unset> | 2016-09-29 14:06 | Install | 312
You can also use
yum history info to get more information on a specific
$ yum history info 4 Loaded plugins: fastestmirror Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast Transaction ID : 4 Begin time : Sun Dec 11 19:04:06 2016 Begin rpmdb : 378:21fccfb42d3b1e74979111e732852b5061a2c6ed End time : 19:04:07 2016 (1 seconds) End rpmdb : 379:75a7237da62e0b9a892be970216af98ad488fb76 User : root <root> Return-Code : Success Command Line : install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm Transaction performed with: Installed rpm-4.11.3-17.el7.x86_64 @base-local Installed yum-3.4.3-132.el7.centos.0.1.noarch @base-local Installed yum-plugin-fastestmirror-1.1.31-34.el7.noarch @base-local Packages Altered: Install epel-release-7-8.noarch @/epel-release-7-8.noarch history info
Yum will also log transaction information to
$ grep epel /var/log/yum.log Dec 11 19:04:06 Installed: epel-release-7-8.noarch
For more information on using
yum refer to the Yum man page.
You may also want to produce a list of installed packages, this can be done with the rpm command:
$ rpm -qa | sort abattis-cantarell-fonts-0.0.16-3.el7.noarch accountsservice-0.6.35-9.el7.x86_64 accountsservice-libs-0.6.35-9.el7.x86_64 acl-2.2.51-12.el7.x86_64 adwaita-cursor-theme-3.14.1-1.el7.noarch adwaita-gtk2-theme-126.96.36.199-2.el7.x86_64 adwaita-icon-theme-3.14.1-1.el7.noarch aic94xx-firmware-30-6.el7.noarch ...
As well as listing packages, the
rpm command can also be used to find
modified files. This is useful for identifying configuration that may have been
$ rpm -Va .......T. /lib/modules/3.10.0-229.el7.x86_64/modules.devname .......T. /lib/modules/3.10.0-229.el7.x86_64/modules.softdep S.5....T. c /etc/yum.repos.d/CentOS-Base.repo missing /run/gdm .M....G.. /var/log/gdm ....L.... c /etc/pam.d/fingerprint-auth ....L.... c /etc/pam.d/password-auth ....L.... c /etc/pam.d/postlogin ....L.... c /etc/pam.d/smartcard-auth ....L.... c /etc/pam.d/system-auth missing /var/run/wpa_supplicant S.5....T. c /etc/plymouth/plymouthd.conf S.5....T. c /etc/sysconfig/authconfig ...
The characters at the start of each line denote the following:
S: file Size differs
M: Mode differs (includes permissions and file type)
5: digest (formerly MD5 sum) differs
D: Device major/minor number mismatch
L: readLink(2) path mismatch
U: User ownership differs
G: Group ownership differs
T: mTime differs
P: caPabilities differ
Apt and dpkg
Debian based systems like Ubuntu use the Apt package management system.
Unfortunately Apt doesn't have a direct equivalent to
yum history, however
the following log files should provide details on when packages were installed,
updated, and removed:
To produce a list of installed Debian packages the dpkg command can be used:
$ dpkg -l Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=============================-==============================-============-=============================================================================== ii acl 2.2.52-3+b1 amd64 Access control list utilities ii adduser 3.115 all add and remove users and groups ii apache2 2.4.25-3+deb9u3 amd64 Apache HTTP Server ii apache2-bin 2.4.25-3+deb9u3 amd64 Apache HTTP Server (modules and other binary files) ii apache2-data 2.4.25-3+deb9u3 all Apache HTTP Server (common files) ii apache2-utils 2.4.25-3+deb9u3 amd64 Apache HTTP Server (utility programs for web servers) ii apt 1.4.7 amd64 commandline package manager ii base-files 9.9+deb9u1 amd64 Debian base system miscellaneous files ...
Have a quick look in the following directories for any cron jobs you might have set up:
SSH keys and config
As well as checking in
~/.ssh for each user, you may also want to backup the
host's SSH keys. If you're rebuilding a host these can be reused to keep the
same host fingerprint. This will prevent conflicting
known_hosts entries when
connecting to the system. The public and private keys can normally be found in
$ ls -l /etc/ssh/ssh_host* -rw-r-----. 1 root ssh_keys 227 Sep 29 2016 /etc/ssh/ssh_host_ecdsa_key -rw-r--r--. 1 root root 162 Sep 29 2016 /etc/ssh/ssh_host_ecdsa_key.pub -rw-r-----. 1 root ssh_keys 387 Sep 29 2016 /etc/ssh/ssh_host_ed25519_key -rw-r--r--. 1 root root 82 Sep 29 2016 /etc/ssh/ssh_host_ed25519_key.pub -rw-r-----. 1 root ssh_keys 1679 Sep 29 2016 /etc/ssh/ssh_host_rsa_key -rw-r--r--. 1 root root 382 Sep 29 2016 /etc/ssh/ssh_host_rsa_key.pub