content.md

### Table of Contents

* Networking & NFtables basics
* How to debug network problems
* Wireguard: easy to setup vpn
* Ansible: how to manage many computers
* Why have servers running at home
* Internet connection technologies
* How to build a linux home router

---

## Expected end result
![](assets/talosiannn.jpg)

---

## Enough joking

A lot of what we are going to
cover today has been done in depth
in previous Linux courses  
(links in the relevant sections)  
This talk will focus more on how
all these topics can come together

----

# Networking

---

## First, a microshot of Networking basics

[Slides](https://slides.poul.org/2019/corsi-linux/3a_networking/)  
[Video](https://youtu.be/jSZSIM4sRuU)

---

<img src="assets/tcpip_stack.png" style="min-height: 90%" />

---

## Network Layer

Transmits the actual 1s and 0s over the wire,
has routing but it is mostly Point-to-Point

Main technologies: Ethernet (802.3) and WiFi (802.11)  
Addressing information: MAC address  
Software Stack: ARP & NDP - `ip neigh`  

---

## Internet Layer

Deals mainly with routing packets and managing fragmentation

Main technologies: IPv4 & IPv6  
Addressing information: IP address  
Software stack: `ip {address,route}`

---

## Transport Layer

Works to provide a usable implementation (socket) to the
program, it provides important feature such as:  

* Ports
* Packet reliability

Main technologies: TCP & UDP  
Addressing information: Port

---

## Application Layer

Everything above __Transport__ is application-specific

Main technologies: HTTP, HTTPS, IMAP, gRPC

---

## What is important to remember

* Every computer needs an **IP address** to communicate to
  other computers
* IPs have a __subnet mask__ that indicates which other IPs
  can be contacted directly through the link layer
  * Computers use ARP/NDP to have a **IP <-> MAC** translation
  * Also there is a ARP/NDP record that tells us
    the **MAC address** relative to the IP we want to contact

---

<h2 style="min-width: 110%">What is important to remember /2</h2>

* For any other IP (eg. `176.31.102.216`) we need to have
  a **default gateway** to which we deliver our IP packets to have
  them sent to the Internet
* Computers that act as default gateways are called **routers**
  (since they **route** packets) and usually take the form of
  those boxes that ISPs give to you

---

## How to retrieve ARP information

```bash
$ ip neigh
192.168.0.1 dev wlp2s0 lladdr c4:6d:1f:fa:41:a0 REACHABLE
^ ip            ^ device      ^ MAC address     ^ status
```

Can we manually add entries? yes, with `ip neigh add` but if you have
to do that then you might have some network problems that need to be fixed

---

## How to retrieve IP-related information

```bash
$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 <clipped>
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 <clipped>
    link/ether d8:cb:8a:ef:80:44 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 <clipped>
    link/ether 9c:b6:d0:06:87:35 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.103/24 brd 192.168.0.255 scope global
       valid_lft 6028sec preferred_lft 6028sec
    inet6 fe80::b3b6:2ab6:2dcf:3b42/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
```

---

## Important information

```bash
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500
```

* **wlp2s0**: network interface name
* **UP**: we (or our network manager) have turned the interface on
* **LOWER_UP**: the interface is actually working
* **mtu 1500**: max packet size before we need to fragment

---

## Important information /2

```bash
link/ether 9c:b6:d0:06:87:35
```

Our interface's MAC address

```bash
inet 192.168.0.103/24
inet6 fe80::b3b6:2ab6:2dcf:3b42/64
```

Our interface's IP addresses

---

## Important information /3

```bash
$ ip route show
default via 192.168.0.1 dev wlp2s0 <clipped>
192.168.0.0/24 dev wlp2s0 <clipped>
```

* Second line: if we want to contact a `192.168.0.X` ip we need to
  do an ARP request, and then send our packet to that MAC address
* First line: if we want to send the packet elsewhere we need to send
  it to `192.168.0.1` and it will take care of delivering it for us

----

# NFtables

(Partially overlaps with Iptables)  
[Slides](https://slides.poul.org/2019/corsi-linux/3b_firewall/)  
[Video](https://youtu.be/ut50w2q6VEw)

---

## What is NFtables?

NFtables is a program that allows us to control Linux's internal
packet filter (firewall) allowing us to block out unwanted traffic

---

<h2 style="min-width: 110%">NFtables vs IPtables vs firewalld (& co.)</h2>

* **iptables**: frontend for `xt_*` kernel modules, old (1998)
* **nftables**: frontend for `nf_*` kernel modules, "new" (2014)
* **firewalld & co.**: allow easier configuration, under the hood 
  they all use iptables

---

## Overview of Linux packet filtering

<img src="assets/Netfilter-packet-flow.svg" style="background-color: white; min-width: 100%" />

Source: [Wikipedia](https://commons.wikimedia.org/wiki/File:Netfilter-packet-flow.svg)

---

## How to read this graph

<img src="assets/nf-hook-cell.png" style="background-color: white; min-width: 100%" />

---

## How to read this graph /2

<img src="assets/nf-logic-cell.svg" style="min-width: 30%" />

* **Bridge check**: check if the packet needs to be routed on a bridged interface
* **Routing decision**: decide if we want a program to receive the packet (input)
  or if we need to __forward__ it to somewhere else
* **Conntrack**: linux checks if this packet is related to others (eg. TCP connection)

---

### Important things to know

* **bridging**: we instruct the linux kernel to forward MAC packets between
  2 separate interfaces (like a switch)
* The **routing decision** step is also what decides the output interface,
  the __forward__ hook is the first to know the output interface
* There are **4** chains: **filter** and **nat** (common), `mangle` and `raw` (rarely used)
* There are **5** hooks:
  * filter: **input**, **forward**, **output**
  * nat: **prerouting**, **postrouting**

---

## nftables hierarchy

* **Rule**: some form of filtering based on the packet (eg. source ip)
* **Chain**: a collection of rules that are applied to a **hook** with
  optionally a default policy (default open)
* **Table**: collects chains together, determines the **family** to use
  * families: **ip** (IPv4), **ip6** (IPv6), **inet** (IPv4&6),
    **arp** (ARP), **bridge** (MAC)

---

## Getting started with nftables

```bash
# nft add table inet my_table
# nft add chain inet my_table my_chain \
    '{ type filter hook input priority 0; policy accept; }'
# nft add rule inet my_table my_chain iifname eth0 tcp dport 22 drop
```

1. we create a table **my_table**
2. we add a chain **my_chain** that uses the **input** hook in 
   the **filter** chain, by default we **accept** packets
3. we add a rule that if a packet comes from **eth0** and has
   destination port **22** we drop it

---

## Do I need to use the CLI?

No, nft natively supports loading and saving rules to a file,
the example in the previous slide can be written as

```
table inet my_table {
    chain my_chain {
        type filter hook input priority 0; policy accept;
        iifname eth0 tcp dport 22 drop
    }
}
```

---

## Some important filtering rules

* `{iifname|oifname} <name>`: input/output interface name
* `{tcp|udp} {sport|dport} <number>`: TCP/UDP source/destination port
* `ip {saddr|daddr} <ip>`: source/destination ip
* `ct state {new|established|related|invalid}`: conntrack status

----

# Debug network problems

---

## You can't connect to the Internet, now what?

When setting up a network it might happen that everything
breaks, we will go through some tools that might help debug
network configuration problems

---

### Network checklist

* Can the network route to another address?
* Does the network resolve domain names?
* Are response packets coming back?
* Is everything ok in upper layers?

---

### Quick routing check: ping & traceroute

* **ping** is useful to check that we have working routing
  a quick `ping 1.1.1.1` can tell us if we can reach a known
  ip (in this case cloudflare DNS)

* **traceroute** can help with identifying routing
  nodes that are dropping packets

---

### DNS check: ping (again) & dig

* Again **ping** can be used to quickly check if DNS resolution
  is working as intended, `ping poul.org` will return an ip
  address if DNS is working

* **dig** can help to check more complex problems,
  using `dig poul.org` will tell us if the system DNS
  (found in `/etc/resolv.conf`) is working, whereas
  `dig @1.1.1.1 poul.org` will tell us if there is some
  external problem related to DNS

---

### Checking for responses

* Tools like **tcpdump** and **Wireshark** can help debug
  problems related to responses not coming back, just point to
  the right interface and filter for the traffic (usually __icmp__)

* Only have ssh but want to have a gui? No problem!<br>
  ```
  ssh root@<ip> "tcpdump -i eth0 -s0 -w - 'not port 22'" | wireshark -k -i -
  ```

---

### Upper Layer Problems

So far we only talked about L1-3 problems, but there can be just as many
in L4-7, however these usually require specific tools to debug

* **curl** can be used to detect a [TCP Blackhole](https://en.wikipedia.org/wiki/Path_MTU_Discovery#Problems) and a whole other lot of problems
  ```
  curl -vv https://example.org
  ```

----

# Wireguard
[Link to Slides](https://slides.poul.org/2019/corsi-linux/3c_wireguard.pdf)  
[Link to Video](https://youtu.be/Ppw6XxbmlWA)

---

### What is a VPN?

A VPN is a piece of software that allows us to access a **Network**
that is not publicly available (**Private**) and which we have
no physical connection to (**Virtual**)

---

### Wireguard

Project started in 2016 to replace OpenVPN, in terms
of security and ease of configuration

Strong Points:
* Minimal codebase, ~40000 loc (originally 4000) vs 600000 for OpenVPN
* Minimal configuration
* Fail-open behavior, if misconfigured no packets are routed

---

### Step 1: Generating a key pair

```bash
wg genkey > privatekey
cat privatekey | wg pubkey > publickey
```

This needs to be done for every __host__ that wants to join the vpn

---

### Write server configuration

```
[Interface]
Address = 192.168.10.1/24
ListenPort = 51820
PrivateKey = kIbuAUUbNZeC18onuKDtUui2Oa+l4/RrsU/sjcVKgmU=

PostUp = nft add rule ip filter FORWARD iifname %i counter accept
PostUp = nft add rule ip nat POSTROUTING oifname eth0 counter masquerade
PostDown = nft del rule ip filter FORWARD iifname %i counter accept
PostDown = nft del rule ip nat POSTROUTING oifname eth0 counter masquerade

[Peer]
PublicKey = yL4ajtwU9a2zP9vyVa5hdB5cSl/deLXv0Ldck1Y/FSU=
AllowedIPs = 192.168.10.2/32
```

---

### Write client configuration

```
[Interface]
Address = 192.168.10.2/24
PrivateKey = CCSq5ngQcIGjKS3qu5woC7tYVQM2zJhJVR4jQ9xrXUY=
DNS = 192.168.10.1

[Peer]
PublicKey = xopK6ZfGT0CAS8g8SXmEZf4Ppp3al5XkDJPCYl5Z8So=
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = <server public ip>:51820
```

---

```
wg-quick up server.conf
```

```
wg-quick up client.conf
```

---

# Done!

---

## Well not so fast

Due to the fail-open nature, it is sometimes difficult to debug
Some debugging techniques are shown in the previous sections.  
Ping tends to report an error if wireguard is misconfigured.  
The linked slides offer a more in-depth explaination of the
configuration that might help to debug problems.

----

# Ansible
[Link to Slide](https://slides.poul.org/2019/corsi-linux/4b_ansible/)  
[Link to Video](https://youtu.be/Cc396Em2BuM)

---

## What is Ansible?  

Ansible is a declarative configuration managment tool

Strong points:  
* Only Python and SSH need to be installed to have a machine
  be controlled by Ansible
* Idempotent (multiple runs have the same result as one)
* Declarative/Stateless - what is on the config files dictates the end result,
  not how to get there

---

## What can you do with ansible?

* Maintain consistent configurations across machines
* Have set of installed packages on all your machines
* Manage installation/upgrade of services in a reproducible way

---

## Basic ansible terms

* **inventory**: defines hosts and their grouping
* **playbook**: describes what operations to do on the hosts
* **task**: defines the single operation to execute

---

## Anatomy of an inventory

```ini
[webservers]
web.example.org
web[1-4].example.org

[motd]
192.168.0.1

[motd:children]
webservers
```

---

## Anatomy of a task

```yaml
name: "Install webserver"
apk:
  name: lighttpd
  state: latest
```

---

## Anatomy of a playbook

```yaml
---
- hosts: webservers
  remote_user: root
  tasks:
  - name: install webserver
    package:
      name: lighthttpd
      state: latest
  - name: start webserver
    service:
      name: lighthttpd
      state: started
      enabled: yes
```

----

# Beyond the cloud

----

## Why have a server at home?

---

### Server@Home

* We have full control on the hardware
* Advanced setups possible
* Is better suited in some scenarios (eg. backup)
* Usually cheaper than a server __*__  
  (eg. storage)

---

### Cloud Server

* Usually has better uptime (much closer to 100%)
* Storage has magic underneath
* Usually has higher network bandwidth

---

### What to consider to do cost/benefit-wise

* On cloud servers the entire infrastructure
  is redundant, and if a component breaks the
  cloud provider replaces it free of charge,
  at home you have to pay for replacements
* Magic ratio `kWh/Wy` -> `8.76`
    * High power consumption can lead to an
      expensive bill

----

# How to pick the hardware

---

### Storage technologies summary

|   | €/GB | R/W Speed | Lifetime |
| - |:---- |:--------- |:-------- |
| HDD | Low | Medium | Medium<br>(3-5 years) |
| SSD | Medium | High | High<br>(10+ years) |
| SD/eMMC | High | Medium-Low | Short<br>(1-2 years)|
| Bluray | Low | Medium (RO) | Extreme (100+ years) |
| Cloud | - | Depends* | Infinite* |

---

## ARM SBC/Router

<div style="display: block; margin: 0 auto">
    <img src="assets/rpi4.jpg" style="max-width: 27%">
    <img src="assets/tinkerboard.png" style="max-width: 30%">
    <img src="assets/rockpro64.png" style="max-width: 25%">
</div>

* Cost: 20-100€
* Computing power: Medium-Low
* Storage: scarce/decent
* Power consumption: 5-15W

---

## Things to consider before buying

* Ethernet maximum speed (100 vs 1000)
* How the ethernet chip is connected to the SoC
* How the USB ports are connected to the SoC
* CPU computing power
* RAM quantity
* Bootable storage (SD/eMMC vs USB 3/SATA)
* Mainline linux support

---

## Why you shouldn't buy a RPi

* 1-3 have usb ports behind a hub
* 1-3 have ethernet behind an usb adapter<br>(guess where it is attached?)
* Wifi/BT share an antenna, better result achievable with
  an u.FL connector and a proper antenna
* Computing power is generally lower than
  competitors for the same price point
* RPi linux distributions tend to come with
  a lot of proprietary broadcom blobs
* Generally poor power supply circuitry

---

## Laptop/Entry-level desktop

<div style="display: block; margin: 0 auto">
    <img src="assets/laptop_n.jpg" style="max-width:40%" />
    <img src="assets/comp_antisgamo.jpg" style="max-width:40%" />
</div>

* Cost: 50-200€
* Computing power: Medium-High
* Storage: usually quite decent
* Power consumption: 20-50W

---

## High-End Desktop/Server

<div style="display: block; margin: 0 auto">
    <img src="assets/server_1.png" style="max-width:15%" />
    <img src="assets/server_2.png" style="max-width:35%" />
</div>

* Cost: 100-2000€
* Computing power: High
* Storage: sufficent-plenty (NAS)
* Power Consuption: 200-2000W
* Pros: [no house heating costs](https://hardware.slashdot.org/story/17/09/13/002250/french-company-plans-to-heat-homes-offices-with-amd-ryzen-pro-processors)

---

## I need to crunch numbers, what should I do?

* Use a low-power device to trigger [Wake on LAN](https://en.wikipedia.org/wiki/Wake-on-LAN) for a more powerful device
* Use a combination of Home+Cloud
* Use spot/preemptible cloud VMs<br>(this depends on the workload)

---

# Is there such a thing as too much hardware?

---

# NO

<div style="display: block; margin: 0 auto">
    <img src="assets/rack_1.png" style="max-width:35%" />
    <img src="assets/rack_2.jpg" style="max-width:22%" />
</div>

----

# Internet connection technologies

---

How do we connect to the internet?  

3 main technologies:  

* *DSL
* GPON (fiber)
* Ethernet

---

## *DSL

<img src="assets/RJ11.jpg" style="max-height: 30%" />

* Encompasses various technologies (ADSL/VDSL/VDSL2)
* All of them use a copper medium (telephone line) to transmit data
* Phone line length and EMF interference *matter* on transmission speed

---

## Router options for *DSL

* Lantiq-based routers ( [list](https://openwrt.org/docs/techref/targets/lantiq) )
* No one else since the DSL chipset is usually custom-made and manufacturers don't relase the source code
* Some routers allow enough configuration to be ISP-neutral, but that comes at a
  cost of old software
* In theory _Modem Libero_ should allow the use of old ISP routers, but I wouldn't hold my breath on that

---

## GPON (fiber)

<img src="assets/fiber.jpg" style="max-height: 30%" />

* New technology used for FTTH (1 Gb/s internet)
* There is a direct optical link between the end user and the ISP
* Since the transmission is optical, no interference!
* Optic fiber crimping has some tooling costs (~200€)

---

## ONTs

<div style="display: block; margin: 0 auto">
    <img src="assets/ont.jpg" style="max-width:40%" />
    <img src="assets/sfp_ont.jpg" style="max-width:40%" />
</div>

A ONT is a device that bridges a fiber link to ethernet  
(allowing us to use any device we want as a router)  
There are only couple of ONTs with WIP OpenWRT support

---

## Raw Ethernet

<img src="assets/ethernet.jpg" style="max-height: 30%" />

Some providers (eg. Eolo) use alternative transmission mediums,
usually these will have an ordinary ethernet cable attached
to a mundane wifi router

----

# Deploying a FOSS router at home

---

## Materials needed:

* A device to interface with your internet
* Some ethernet cables (crimping tools recommended)
* A device with 2+ ethernet ports (1 works on some setups but don't)
* WiFi device

---

## Small digression: Modem Libero

At the end of 2018 the AGCOM (italian telecomunications agency) approved
a set of rules that forces ISPs to provide appropiate means to allow consumers
to choose their own router (ONTs are still considered part of the ISP's network)  
Obtaining the various settings for connecting to the ISP's network varies
by difficulty and amount of call centers you have to contact based on the ISP you pick  
_(there was also a clause that allowed consumers to have a vendor-neutral firmware
at the end of the contract, but everyone seems to have forgotten about that)_

---

## Connecting to the ISP

There are 2 main ways in which you connect to ISPs

* PPPoE over VLAN: used by basically everyone
* DHCP over raw ethernet: used only by Fastweb

Obviously neither methods is within spec, both use some *slight*
change in protocol parameters, which may need some fiddling

---

## IPv6

Currently (April 2020), only 2 ISPs provide IPv6 connectivity

* Fastweb: via [6rd](https://en.wikipedia.org/wiki/IPv6_rapid_deployment)
* Telecom Italia: via a separate PPPoE connection

However do note that both these tunnels might not support the full bandwidth

---

## What software to use?

* ppp for connecting to PPPoE
* Systemd-networkd for managing connections  
  (It is the only network manager that supports most ISPs configurations fully)

----

## Special Thanks to

* Federico Amedeo Izzo for NAND info
* Nicolo Izzo for his [original talk](https://slides.poul.org/2017/corsi-linux-avanzati/servers/) and images
* Davide Depau for help with NFtables and photos
* Lorenzo Ribis for some photos
* POuL members in general for being always helpful