howto/Talos-Linux.md
... ...
@@ -0,0 +1,337 @@
1
+# Talos Linux
2
+
3
+This guide sets up a DN42 router on a [Talos Linux](https://www.talos.dev/) node, using WireGuard tunnels and BIRD. It covers what's specific to Talos; for BIRD itself, see the [Bird2](/howto/Bird2) guide.
4
+
5
+Talos is different from a normal Linux distribution: there is no shell, no SSH and no package manager. You configure everything with YAML documents, sent to the node with `talosctl`. Software that isn't part of Talos, like BIRD, comes in as a *system extension* baked into the Talos image.
6
+
7
+Tested with Talos v1.15.0-alpha.2 and the `siderolabs/bird2` 2.18 extension.
8
+
9
+## Before you start
10
+
11
+You need:
12
+
13
+* A registered DN42 AS, prefix and ROA, see [Getting Started](/howto/Getting-Started). This guide uses **AS4242421234** and **fd00:1234:5678::/48** as examples; replace them with your own everywhere.
14
+* A Talos node you can manage with `talosctl`, with a public IPv6 address peers can reach.
15
+* `wg` (wireguard-tools) on your workstation, to generate keys.
16
+* At least one peer, see [Wireguard](/howto/wireguard) for what to exchange with them.
17
+
18
+The steps:
19
+
20
+1. Add BIRD to the Talos image
21
+2. Create a WireGuard tunnel to your peer
22
+3. Configure BIRD
23
+4. Check that it works
24
+5. Optional, for Kubernetes nodes: move DN42 into its own routing table (VRF)
25
+
26
+## 1. Add BIRD to the Talos image
27
+
28
+Talos images are built by the [Image Factory](https://factory.talos.dev/) from a *schematic* that lists the extensions. Create `schematic.yaml` (keep any extensions you already use):
29
+
30
+```yaml
31
+customization:
32
+ systemExtensions:
33
+ officialExtensions:
34
+ - siderolabs/bird2
35
+```
36
+
37
+Register it; the response contains the schematic ID:
38
+
39
+```
40
+curl -X POST --data-binary @schematic.yaml https://factory.talos.dev/schematics
41
+```
42
+
43
+Upgrade the node to an installer image built from that ID. This reboots the node. Use your Talos version, and the installer type that matches your node (`metal-installer` for bare metal):
44
+
45
+```
46
+talosctl -n <node> upgrade --image factory.talos.dev/metal-installer/<schematic id>:v1.15.0-alpha.2
47
+```
48
+
49
+After the reboot, `talosctl -n <node> get extensions` lists `bird2`. BIRD itself starts once it has a config (step 3).
50
+
51
+## 2. Create a WireGuard tunnel
52
+
53
+Generate a key pair. Send the public key to your peer, together with your endpoint (`[<node IPv6>]:<port>`) and your link-local address:
54
+
55
+```
56
+wg genkey | tee private.key | wg pubkey > public.key
57
+```
58
+
59
+Create `wg-example.yaml`, one per peer. The link-local addresses are agreed with the peer; WireGuard endpoints must be IP addresses, not hostnames.
60
+
61
+```yaml
62
+apiVersion: v1alpha1
63
+kind: WireguardConfig
64
+name: wg-example
65
+privateKey: <contents of private.key>
66
+listenPort: 21234
67
+addresses:
68
+ - address: fe80::1234/64 # your side of the tunnel
69
+peers:
70
+ - publicKey: <peer public key>
71
+ endpoint: "[2001:db8::1]:21234" # the peer's endpoint
72
+ persistentKeepaliveInterval: 25s
73
+ allowedIPs:
74
+ - fe80::/10
75
+ - fd00::/8
76
+---
77
+# Don't let the peer configure this node through router advertisements or redirects.
78
+machine:
79
+ sysctls:
80
+ net.ipv6.conf.wg-example.accept_ra: "0"
81
+ net.ipv6.conf.wg-example.accept_redirects: "0"
82
+```
83
+
84
+Apply it:
85
+
86
+```
87
+talosctl -n <node> patch machineconfig --patch @wg-example.yaml
88
+```
89
+
90
+The private key is now in your machine config. If you keep your config in git, encrypt it, for example with SOPS.
91
+
92
+Check the tunnel with `talosctl -n <node> get links wg-example`, and by pinging the peer's link-local address from your peer's side.
93
+
94
+## 3. Configure BIRD
95
+
96
+BIRD's config is passed in an `ExtensionServiceConfig` document. The config below is a trimmed-down version of the one in the [Bird2](/howto/Bird2) guide, which explains the filters and options in more detail. Create `bird.yaml`:
97
+
98
+```yaml
99
+apiVersion: v1alpha1
100
+kind: ExtensionServiceConfig
101
+name: bird2
102
+configFiles:
103
+ - mountPath: /usr/local/etc/bird.conf
104
+ content: |
105
+ define OWNAS = 4242421234;
106
+ define OWNNETv6 = fd00:1234:5678::/48;
107
+ define OWNNETSETv6 = [fd00:1234:5678::/48+];
108
+
109
+ # Any unique IPv4 address. Use your DN42 IPv4 address if you have one.
110
+ router id 10.12.34.1;
111
+ log stderr all;
112
+
113
+ protocol device {
114
+ scan time 10;
115
+ }
116
+
117
+ function is_self_net_v6() -> bool {
118
+ return net ~ OWNNETSETv6;
119
+ }
120
+
121
+ function is_valid_network_v6() -> bool {
122
+ return net ~ [ fd00::/8{44,64} ];
123
+ }
124
+
125
+ # ROAs, see "ROA validation" below
126
+ roa6 table dn42_roa6;
127
+
128
+ protocol rpki rtr {
129
+ roa6 { table dn42_roa6; };
130
+ remote 2001:db8::8282 port 8282;
131
+ refresh 600;
132
+ retry 30;
133
+ expire 7200;
134
+ }
135
+
136
+ # Announce your own prefix.
137
+ protocol static {
138
+ route OWNNETv6 unreachable;
139
+ ipv6 {
140
+ import all;
141
+ export none;
142
+ };
143
+ }
144
+
145
+ # Install learned routes into the kernel.
146
+ protocol kernel {
147
+ scan time 20;
148
+ learn off;
149
+ ipv6 {
150
+ import none;
151
+ export filter {
152
+ if source = RTS_STATIC then reject;
153
+ accept;
154
+ };
155
+ };
156
+ }
157
+
158
+ template bgp dnpeers {
159
+ local as OWNAS;
160
+ path metric on;
161
+ ipv6 {
162
+ # re-check received routes when the ROAs change
163
+ import table on;
164
+ import filter {
165
+ if !is_valid_network_v6() || is_self_net_v6() then reject;
166
+ if roa_check(dn42_roa6, net, bgp_path.last) != ROA_VALID then {
167
+ print "[dn42] ROA check failed for ", net, " ASN ", bgp_path.last;
168
+ reject;
169
+ }
170
+ accept;
171
+ };
172
+ export filter {
173
+ if is_valid_network_v6() && source ~ [RTS_STATIC, RTS_BGP] then accept;
174
+ reject;
175
+ };
176
+ import limit 9000 action block;
177
+ };
178
+ }
179
+
180
+ # One per peer: their link-local address and AS, over their tunnel.
181
+ protocol bgp example from dnpeers {
182
+ neighbor fe80::4321 as 4242424321;
183
+ interface "wg-example";
184
+ }
185
+```
186
+
187
+Apply it:
188
+
189
+```
190
+talosctl -n <node> patch machineconfig --patch @bird.yaml
191
+```
192
+
193
+BIRD can't reload its config: every change to this document restarts it, which briefly drops all BGP sessions.
194
+
195
+### ROA validation
196
+
197
+BIRD checks routes against the DN42 ROAs, which it fetches from an *RTR server* (`remote` in the `rpki` block). Talos can't run a cron job to download a ROA file, so run a small RTR server somewhere your node can reach, for example [stayrtr](https://github.com/bgp/stayrtr) with the DN42 feed:
198
+
199
+```
200
+stayrtr -cache=https://dn42.burble.com/roa/dn42_roa_46.json -checktime=false
201
+```
202
+
203
+It listens on port 8282. Until BIRD has ROAs, every route fails the check and is rejected.
204
+
205
+### Reaching DN42 from your LAN
206
+
207
+Your LAN router needs a route for `fd00::/8` to the Talos node, and your LAN hosts need addresses from your DN42 prefix.
208
+
209
+## 4. Check that it works
210
+
211
+```
212
+talosctl -n <node> service ext-bird2
213
+talosctl -n <node> logs ext-bird2
214
+```
215
+
216
+The service should be `Running`. The log shows rejected routes as `ROA check failed` lines, which is normal: not every route in DN42 has a valid ROA.
217
+
218
+For the full picture, use `birdc`. Its control socket is `/var/run/bird/bird.ctl` on the node. On a Kubernetes node, from a debug pod:
219
+
220
+```
221
+kubectl debug node/<node> -it --image=alpine --profile=sysadmin
222
+apk add bird2
223
+birdc -s /host/var/run/bird/bird.ctl show protocols
224
+```
225
+
226
+The BGP protocol should show `Established`.
227
+
228
+## 5. Optional: a VRF on Kubernetes nodes
229
+
230
+With the setup above, DN42 routes go into the node's main routing table. That's fine on a dedicated router. On a Kubernetes node, the node and all its pods then reach DN42 directly. And because pod and service networks are often ULAs too (`fd00::/8`), a peer announcing an overlapping prefix could pull cluster traffic into a tunnel.
231
+
232
+A VRF gives DN42 its own routing table, so the node and its pods never see DN42 routes. It isn't sealed off: it has exactly two doors, and both lead to your LAN router.
233
+
234
+```
235
+LAN host
236
+ │ default route
237
+ ▼
238
+LAN router ── route fd00::/8 → Talos node
239
+ │
240
+ ▼ arrives on eth0
241
+Talos node ── rule: "came in on eth0, going to fd00::/8 → use the DN42 table" ← way in
242
+ │
243
+ ▼ DN42 table: routes from BIRD
244
+wg-example ──► DN42
245
+
246
+reply from DN42
247
+ │
248
+ ▼ arrives on wg-example → DN42 table
249
+Talos node ── DN42 table: "your /48 → LAN router via eth0" ← way out
250
+ │
251
+ ▼
252
+LAN router ──► LAN host
253
+```
254
+
255
+For background, see the Linux kernel's [VRF documentation](https://docs.kernel.org/networking/vrf.html), Talos's [VRFConfig](https://docs.siderolabs.com/talos/v1.15/reference/configuration/network/vrfconfig) and [RoutingRuleConfig](https://docs.siderolabs.com/talos/v1.15/reference/configuration/network/routingruleconfig) references, and BIRD's [`vrf` option](https://bird.network.cz/?get_doc&v=20&f=bird-3.html#proto-vrf).
256
+
257
+Anything in the DN42 table that isn't your prefix or a DN42 route is dropped. So DN42 can't reach the node or its pods, and your LAN router's firewall sees all DN42 traffic in both directions. Pods on the node reach DN42 like any other LAN host: via the LAN router, which sends it back through the way in.
258
+
259
+Add these documents, replacing `eth0`, the router's link-local address and the pod range with yours. Talos only accepts table numbers 1–252.
260
+
261
+```yaml
262
+apiVersion: v1alpha1
263
+kind: VRFConfig
264
+name: vrf-dn42
265
+table: "42"
266
+links:
267
+ - wg-example # every tunnel
268
+---
269
+# Way out: your prefix goes back to the LAN router.
270
+apiVersion: v1alpha1
271
+kind: LinkConfig
272
+name: eth0
273
+routes:
274
+ - destination: fd00:1234:5678::/48
275
+ gateway: fe80::1 # LAN router
276
+ table: "42"
277
+---
278
+# Way in: DN42 traffic from the LAN router uses the DN42 table...
279
+apiVersion: v1alpha1
280
+kind: RoutingRuleConfig
281
+name: "210"
282
+iifName: eth0
283
+dst: fd00::/8
284
+table: "42"
285
+---
286
+# ...except traffic for the cluster's own ULA ranges (lower number = checked first).
287
+apiVersion: v1alpha1
288
+kind: RoutingRuleConfig
289
+name: "200"
290
+iifName: eth0
291
+dst: fd10:1111:2222::/48 # pod range
292
+table: main
293
+---
294
+# Drop everything else in the DN42 table. Without this, lookups that miss it
295
+# fall through to the main table. Talos can't add "unreachable" routes to a
296
+# VRF table, so the default route points into a dummy link, which drops it.
297
+apiVersion: v1alpha1
298
+kind: DummyLinkConfig
299
+name: dn42-null
300
+routes:
301
+ - gateway: fe80::1
302
+ metric: 4278198272
303
+ table: "42"
304
+```
305
+
306
+Then tell BIRD to use the VRF: add these lines to the `protocol kernel` block and to the `template bgp dnpeers` block, and reject the cluster ranges in the import filter.
307
+
308
+```
309
+protocol kernel {
310
+ vrf "vrf-dn42";
311
+ kernel table 42;
312
+ ...
313
+}
314
+
315
+template bgp dnpeers {
316
+ vrf "vrf-dn42";
317
+ ...
318
+}
319
+```
320
+
321
+```
322
+# the cluster's pod and service ranges
323
+function is_cluster_net_v6() -> bool {
324
+ return net ~ [ fd10:1111:2222::/48+, fd10:3333:4444::/112+ ];
325
+}
326
+
327
+ if !is_valid_network_v6() || is_self_net_v6() || is_cluster_net_v6() then reject;
328
+```
329
+
330
+Leave `net.ipv4.tcp_l3mdev_accept` and `udp_l3mdev_accept` at their default of `0`, so services on the node can't be reached from the VRF.
331
+
332
+## Why not Talos's built-in BGP?
333
+
334
+Talos has its own BGP support (`BGPInstanceConfig`), but it doesn't work for DN42 yet:
335
+
336
+* it has no route filters or ROA validation, so it accepts anything a peer sends ([siderolabs/talos#14252](https://github.com/siderolabs/talos/issues/14252));
337
+* it can't peer over a WireGuard tunnel. The tunnel itself is fine, but a neighbor is either a `link` or an `address`, not both like BIRD's `neighbor fe80::… interface "wg-…"`. With `link:` Talos waits for neighbor discovery, which WireGuard doesn't do; with a link-local `address:` the session comes up, but routes with a link-local next hop aren't installed.