[WITNESS server - part I] Setup & Tuning Ubuntu

In this first part of a series of 3 posts about setting up a Witness server we will see how to install the prerequisites and do some tuning.

Hardware

• CPU: Intel Xeon E3-1230v6 4 Cores/8 Threads 3.50 GHz
• RAM: 32GB DDR4 ECC 2133MHz
• HDD: 1 To SSD NVMe

How-To: Secure Ubuntu server (18.04)

Increase the security and usability of your Ubuntu server is very important. There are few configuration/install that you should take early on as part of the basic setup.

Root Login

Never use directly the user root and prefer to create a new user. For our case we will use an account named witness with sudo power

As root, run this command to add your new user

root@witness:~$ adduser witness

The system will ask you for a password, use a secure one! (Upper, Lower, Number, Special)

Adding user `witness' ...
Adding new group `witness' (1000) ...
Adding new user `witness' (1000) with group `witness' ...
Creating home directory `/home/witness' ...
Copying files from `/etc/skel' ...
Enter new UNIX password: 

When done, you just need to fill the Full Name and to write y to validate.

Retype new UNIX password: 
passwd: password updated successfully
Changing the user information for witness
Enter the new value, or press ENTER for the default
    Full Name []: witness
    Room Number []: 
    Work Phone []: 
    Home Phone []: 
    Other []: 
Is the information correct? [Y/n] y

Not forgot to run this command to add your new user to the sudo group

root@witness:~$ usermod -aG sudo witness

Add Public Key Authentication (SSH)

Set up public key authentication for your new user. Setting this up will increase the security of your server by requiring a private SSH key to log in.

Generate The Key Pair for SSH

If you haven’t an SSH key pair already you can create it by following this process. To generate a new key pair, enter the following command (use the option -b 4096 for higher security) in your terminal.

witness@witness:~$ ssh-keygen -b 4096

Assuming your local user is witness, you will see the following output:

Generating public/private rsa key pair.
Enter file in which to save the key (/home/witness/.ssh/id_rsa):

Hit return to accept and securing your keys with a strong passphrases!

Created directory '/home/witness/.ssh'
Enter passphrase (empty for no passphrase):
Enter same passphrase again:

At the end you will have an output like this

Your identification has been saved in /home/witness/.ssh/id_rsa.
Your public key has been saved in /home/witness/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:XXXXXXXXXXXXXXXXXXX5GcBMBXXXXXXXXXXM witness@xxx
The key's randomart image is:
+---[RSA 4096]----+
|XXXX             |
|XX               |
|X X XX XXX       |
|XXX X XXX        |
|X XXXX X  XX     |
| XX XX           |
|   X             |
| X               |
|                 |
+----[SHA256]-----+

You have now 2 files in the directory /home/witness/.ssh/ a private key id_rsa and a public key id_rsa.pub

Rename the Public Key

if you generate the keys directly on the server rename the public key id_rsa.pub in authorized_keys like this

witness@witness:~/.ssh$ sudo mv id_rsa.pub authorized_keys

And retrieve the private key id_rsa on your computer (not let it on the server!)

Putty case

Putty users, you need to load the private key id_rsa in PuTTYgen then save the private key for have it in .ppk format

Disabling Password Authentication and change the default TCP port

If you were able to login to your account using SSH with the private key then you have successfully configured SSH key-based authentication to your account. We can now remove the authentication with password only in the ssh config file (not hesitate to change also the default port number 22).

Edit the ssh config file

witness@witness:~/.ssh$ sudo vim /etc/ssh/sshd_config

Uncomment the # Port 22 line and change the number to 8022.

Port 8022

Change the value of PermitRootLogin to no

PermitRootLogin no

Change the value of PasswordAuthentication to no

PasswordAuthentication no

Save and close the file when you are finished. To actually implement the changes we just made, you must restart the service.

witness@witness:~/.ssh$ sudo service ssh restart 

Basic firewall rules

The default firewall configuration tool for Ubuntu is ufw. It’s an interface to iptables.

Allow our specific TCP port for ssh

To add our TCP port 8022 for ssh you can use the command below

witness@witness:~$ sudo ufw allow proto tcp from any to any port 8022

Allow other ports needed for a Witness server

a Witness server use per default the TCP port 2001 and optionally need TCP port for running ws, http and rpc plugins. To allow them, you can use the command below as exemple (the port 443 is for https!)

witness@witness:~$ sudo ufw allow proto tcp from any to any port 2001,443 

Activate ufw

To enable ufw, use this command:

witness@witness:~$ sudo ufw enable

Answer y to the question for proceed

Command may disrupt existing ssh connections. Proceed with operation (y|n)?

To list the active rules you can use the command

witness@witness:~$ sudo ufw status

The output will be something like this

Status: active

To                         Action      From
--                         ------      ----
8022/tcp                   ALLOW       Anywhere                  
2001/tcp                   ALLOW       Anywhere 
443/tcp                    ALLOW       Anywhere                 
8022/tcp (v6)              ALLOW       Anywhere (v6)             
2001/tcp (v6)              ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)

Fail2ban

Fail2ban scan the log files of the server and bans IPs that show the malicious signs. Like for exemple too many password failures, seeking for exploits, etc.. It work as a service and create rules that automatically alter iptables configuration. All based on a predefined number of unsuccessful login attempts. This will allow the server to respond to illegitimate access attempts without manual intervention.

Install

Fail2ban is in package list of Ubuntu. To install it from a command prompt do like this (update first).

witness@witness:~$ sudo apt-get update
witness@witness:~$ sudo apt-get install fail2ban

Configure

Configuration files are in the /etc/fail2ban directory. Stop the service

witness@witness:~$ sudo service fail2ban stop

Duplicate the config file jail.conf to keep default options inside (this file can be overwriten when update applied). Put all the specific settings in jail.local

witness@witness:~$ sudo cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local

Now we can modify the jail.local file to adjust to our server case

witness@witness:~$ sudo vim /etc/fail2ban/jail.local

The base

Setup the base consist to add one or more source @ip to ignore, the bann time and the number of retry allowed. To do this we need to modify the variables:

• ignoreip = @ip to ignore (separated by a space).
• bantime (in second) = parameter for banned client (default 10 minutes).
• findtime (in second) = a window of time to find a specific number of tries (see below).
• maxretry = number of tries before being banned.

By default ban 10 minutes a client after 5 tries in 10 minutes.

Email alerts

Configure email alerts with the variable destemail, sendername, and mta. To use it you need the Ubuntu package sendmail!

SSH

To activate a service juste need a line enabled = true in the appropriate section [ssh] and modify the port (remember we use specific!)

[sshd]
enable  = true
# To use more aggressive sshd modes set filter parameter "mode" in jail.local:
# normal (default), ddos, extra or aggressive (combines all).
# See "tests/files/logs/sshd" or "filter.d/sshd.conf" for usage example and details.
#mode   = normal
port    = ssh,8022
logpath = %(sshd_log)s
backend = %(sshd_backend)s

After editing, start the fail2ban service

sudo service fail2ban start

If you go want deeper in fail2ban I suggest this excellent post from Linode

Fail2ban Client

use the command fail2ban-client with one of these command to action/check information:

• start: Starts the server and jails.
• reload: Reloads configuration files.
• reload JAIL: Replaces JAIL with the name of a Fail2ban jail; this will reload the jail.
• stop: Stop the server.
• status: Show the status of the server, and enable jails.
• status JAIL: Show the status of the jail, including any currently-banned IPs.

Replace JAIL by the service you want to check, exemple with ssh

witness@witness:~$ sudo fail2ban-client status
Status
|- Number of jail:  1
`- Jail list:   sshd

Or

witness@witness:~$ sudo fail2ban-client status sshd
Status for the jail: sshd
|- Filter
|  |- Currently failed: 0
|  |- Total failed: 0
|  `- File list:    /var/log/auth.log
`- Actions
   |- Currently banned: 0
   |- Total banned: 0
   `- Banned IP list:   

UTILITIES and TUNING

NMAP

Install and use the port scanning NMAP to check your open ports

witness@witness:~$ sudo apt-get install nmap

To scan the server use the command

witness@witness:~$ nmap -sV -p 1-65535 localhost

Starting Nmap 7.60 ( https://nmap.org ) at 2020-03-31 05:06 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.000070s latency).
Not shown: 65532 closed ports
PORT      STATE SERVICE VERSION
53/tcp    open  domain
953/tcp   open  rndc?
8022/tcp  open  ssh     OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel

Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 12.89 seconds

Where:
-sV = service identification
-p = list of port to scan (range separate by -)

Time Synchronization with NTP

For a better time synchronization, we will use the NTP packet.

Before installing ntpd, we should turn off timesyncd (the default manager for time synchronization):

witness@witness:~$ sudo timedatectl set-ntp no

Verify that timesyncd is off:

witness@witness:~$ timedatectl
                      Local time: Tue 2020-03-31 05:32:52 UTC
                  Universal time: Tue 2020-03-31 05:32:52 UTC
                        RTC time: Tue 2020-03-31 05:32:52
                       Time zone: Etc/UTC (UTC, +0000)
       System clock synchronized: yes
systemd-timesyncd.service active: no
                 RTC in local TZ: no

We can see that systemd-timesyncd.service active is set to no

To install the NTP packet run the command

witness@witness:~$ sudo apt install ntp

ntpd will be started automatically after install. You can check the version with the command

witness@witness:~$ sntp --version
sntp 4.2.8p10@1.3728-o (1)

We need to edit the ntp.conf file to replace the default ubuntu.pool with something closer. To do this, you first need to check here https://www.ntppool.org/zone/@ which are the closest pool servers.

Edit the ntp.conf file

witness@witness:~$ sudo vim /etc/ntp.conf

Replace

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
pool 0.ubuntu.pool.ntp.org iburst
pool 1.ubuntu.pool.ntp.org iburst
pool 2.ubuntu.pool.ntp.org iburst
pool 3.ubuntu.pool.ntp.org iburst

By your nearest NTP pool servers (fr in my case)

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
pool 0.fr.pool.ntp.org iburst
pool 1.fr.pool.ntp.org iburst
pool 2.fr.pool.ntp.org iburst
pool 3.fr.pool.ntp.org iburst

Restart the service

witness@witness:~$ sudo service ntp restart

Check if everything OK

witness@witness:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.fr.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 1.fr.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 2.fr.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000
 3.fr.pool.ntp.o .POOL.          16 p    -   64    0    0.000    0.000   0.000

Increase the max open files limit

To avoid the problem of Too many open files we need to make a few changes to the ulimit

Edit the limits.conf file

witness@witness:~$ sudo vim /etc/security/limits.conf

Add the following lines

root            soft     nproc          999999
root            hard     nproc          999999
root            soft     nofile         999999
root            hard     nofile         999999
witness         soft     nproc          999999
witness         hard     nproc          999999
witness         soft     nofile         999999
witness         hard     nofile         999999

Edit the common-session file

witness@witness:~$ sudo vim /etc/pam.d/common-session

Add the following line

session required        pam_limits.so

Reboot the server for the change to take effect.

Check the open files limit (need to be connected with the right user!)

witness@witness:~$ ulimit -n
999999

Install SCREEN and LNAV

Install the package screen (a terminal multiplexer) which will continue to run either if the window is not visible or if you get disconnected if not already installed.

witness@witness:~$ sudo apt install screen

Install the package lnav a log file navigator, streamer, regex searchable

witness@witness:~$ sudo apt install lnav

---

We are now ready for the second part and the installation of Hive
[WITNESS server - part II] HIVE - The manual way

---

If you liked Upvote, Follow, Reblog or Cross post are welcome - @mintrawa