Skip to content

docs(networking): add VLAN 30 storage network to address plan and design #23

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jmgilman merged 1 commit into from
Aug 22, 2026
Merged

docs(networking): add VLAN 30 storage network to address plan and design #23

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 14 additions & 6 deletions docs/docs/designs/lab-v2-core-network.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ flowchart LR
HOME[Home networks] --> RTR[rtr01]
RTR -->|10.0.0.0/30 transit| GW[gw01]
GW -->|VLAN 10 and 40 trunk| CORE[sw-core01]
CORE -->|SFP+ links, roles assigned later| HOSTS[lab01–03 and nas01]
CORE -->|VLAN 30 storage: SFP+ LAGs and 10GbE| HOSTS[lab01–03 and nas01]
GW -->|VLAN 10 and 70 trunk| MGMT[sw-mgmt01]
MGMT -->|10GbE RJ45 management| NODES[lab01–03]
MGMT -->|2.5GbE RJ45 AMT| AMT[lab01–03 AMT]
Expand All @@ -85,18 +85,19 @@ untagged access to the upper 10GbE RJ45 port on each MS-02 and VLAN 70 as
untagged access to the lower 2.5GbE vPro/AMT port. Both VLANs use a tagged
uplink to `gw01`.

The two MS-02 SFP+ interfaces and the `nas01` 10GbE interface remain physically
connected to `sw-core01`. Their instance, cluster, storage, and aggregation
roles are deferred. Initial IncusOS installation and management do not depend
on those links.
The two MS-02 SFP+ interfaces form an 802.3ad LAG per node on `sw-core01`, and
the `nas01` 10GbE interface connects to port 7. These links carry the
non-routed VLAN 30 storage network (session 010); instance and aggregation
VLANs join them when their first consumer arrives. Initial IncusOS
installation and management do not depend on those links.

## Device Responsibilities

| Device | Responsibilities |
| --- | --- |
| `rtr01` | Home routing, internet edge, upstream transit endpoint, and route to `10.10.0.0/16` |
| `gw01` | Lab gateways, static routing, firewall policy, DHCP, DNS forwarding, Tailscale subnet routing, source NAT, and local `glab.lol` mirror |
| `sw-core01` | Layer 2 transport for management and future compute-facing VLANs |
| `sw-core01` | Layer 2 transport for management, the storage VLAN, and future compute-facing VLANs |
| `sw-mgmt01` | Layer 2 separation of host management and AMT/OOB traffic |
| `sandbox01` | General-purpose tests and spikes on the isolated sandbox/workload VLAN |

Expand Down Expand Up @@ -161,6 +162,13 @@ The authoritative `gw01` source is the tracked
configuration template and CoreDNS assets in `GilmanLab/networking`; encrypted
inputs remain in `GilmanLab/secrets` and are rendered in memory.

The storage-network configuration on the IncusOS nodes (bonds, VLAN 30
addresses, and storage pools) has its authoritative source in the
[`cluster/`](https://github.com/GilmanLab/fleet/tree/master/cluster) pyinfra
project in `GilmanLab/fleet`, which converges the nodes through the Incus and
IncusOS APIs; the node seeds mirror the converged network state for
reinstalls.

The authoritative `sw-core01` source is the
[`routeros/sw-core01/`](https://github.com/GilmanLab/networking/tree/master/routeros/sw-core01)
OpenTofu root in `GilmanLab/networking`. An operator runs `just plan` and
Expand Down
44 changes: 27 additions & 17 deletions docs/docs/reference/networking/address-plan.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ addresses.
| VLAN | Name | Prefix | Gateway | Use |
| --- | --- | --- | --- | --- |
| `10` | Management | `10.10.10.0/24` | `10.10.10.1` | IncusOS management, network-device management, and `nas01` management |
| `30` | Storage | `10.10.30.0/24` | None (not routed) | Incus node storage network on the compute-facing `sw-core01` links |
| `40` | Sandbox/workload | `10.10.40.0/24` | `10.10.40.1` | `sandbox01` and future explicitly attached workload endpoints |
| `70` | OOB | `10.10.70.0/24` | `10.10.70.1` | MS-02 AMT, `pikvm01`, `kvm01`, and management-switch administration |

Expand All @@ -44,6 +45,11 @@ The management and OOB VLANs remain separate. The sandbox/workload VLAN cannot
initiate connections to management or OOB endpoints. Firewall policy permits
required administration flows explicitly and permits established replies.

The storage VLAN is Layer 2 only. It has no gateway interface, is not carried
on the `gw01` trunk, and is therefore unreachable from every routed network by
construction. Its addresses are static IncusOS runtime configuration converged
by the `GilmanLab/fleet` `cluster/` project and mirrored in each node's seed.

## Address allocations

### Infrastructure and services
Expand All @@ -59,15 +65,15 @@ required administration flows explicitly and permits established replies.

### Hosts

| Device | Management | OOB | Notes |
| --- | --- | --- | --- |
| `lab01` | `10.10.10.11` | `10.10.70.11` | 10GbE RJ45 management; 2.5GbE RJ45 AMT |
| `lab02` | `10.10.10.12` | `10.10.70.12` | 10GbE RJ45 management; 2.5GbE RJ45 AMT |
| `lab03` | `10.10.10.13` | `10.10.70.13` | 10GbE RJ45 management; 2.5GbE RJ45 AMT |
| `nas01` | `10.10.10.14` | — | 5GbE RJ45 management link through `sw-mgmt01` |
| `sandbox01` | `10.10.40.10` | — | Direct untagged sandbox/workload attachment to `gw01` |
| `pikvm01` | — | `10.10.70.20` | Direct untagged attachment to `gw01` |
| `kvm01` | — | `10.10.70.21` | Direct untagged attachment to `gw01` |
| Device | Management | Storage | OOB | Notes |
| --- | --- | --- | --- | --- |
| `lab01` | `10.10.10.11` | `10.10.30.11` | `10.10.70.11` | 10GbE RJ45 management; SFP+ LAG storage; 2.5GbE RJ45 AMT |
| `lab02` | `10.10.10.12` | `10.10.30.12` | `10.10.70.12` | 10GbE RJ45 management; SFP+ LAG storage; 2.5GbE RJ45 AMT |
| `lab03` | `10.10.10.13` | `10.10.30.13` | `10.10.70.13` | 10GbE RJ45 management; SFP+ LAG storage; 2.5GbE RJ45 AMT |
| `nas01` | `10.10.10.14` | `10.10.30.14` | — | 5GbE RJ45 management link through `sw-mgmt01`; 10GbE storage |
| `sandbox01` | `10.10.40.10` | — | — | Direct untagged sandbox/workload attachment to `gw01` |
| `pikvm01` | — | — | `10.10.70.20` | Direct untagged attachment to `gw01` |
| `kvm01` | — | — | `10.10.70.21` | Direct untagged attachment to `gw01` |

`gw01` supplies DHCP on every client VLAN. Dynamic clients use `.200` through
`.250` within each client VLAN. DHCP-served named hosts (`sandbox01`,
Expand All @@ -78,9 +84,11 @@ configuration.
Infrastructure endpoints do not depend on DHCP: gateway and managed-switch
interface addresses and the local DNS mirror address are static interface
configuration, IncusOS node management addresses are static in each node's
seed (bound to the management NIC's hardware MAC in `GilmanLab/fleet`), and
lab-node AMT addresses are static in MEBx so out-of-band access survives a
gateway outage. The AMT interfaces have no DHCP reservations.
seed (bound to the management NIC's hardware MAC in `GilmanLab/fleet`),
storage-network addresses are static IncusOS runtime configuration (converged
by the fleet `cluster/` project and mirrored in the seeds), and lab-node AMT
addresses are static in MEBx so out-of-band access survives a gateway outage.
The AMT interfaces have no DHCP reservations.

## Gateway interface mapping

Expand Down Expand Up @@ -117,11 +125,13 @@ the `nas01` 5GbE NIC therefore negotiate no faster than 2.5Gbps on this switch.
### `sw-core01`

Port 8 is the 802.1Q trunk to `gw01` and carries VLANs 10 and 40. VLAN 10
provides the switch management path. Ports 1 through 6 connect the two SFP+
interfaces from each MS-02, and port 7 connects the `nas01` 10GbE interface.
Those compute-facing links receive instance, cluster, or storage VLAN
membership only after the compute-network design assigns their roles; they are
not required for initial IncusOS management boot.
provides the switch management path. Ports 1 through 6 form three 802.3ad
LAGs, one per MS-02 SFP+ pair: `bond-lab01` (ports 1–2), `bond-lab02` (ports
3–4), and `bond-lab03` (ports 5–6). Port 7 connects the `nas01` 10GbE
interface. VLAN 30 is tagged on the three LAGs and port 7; those links carry
no other VLAN membership yet and are not required for IncusOS management
boot. Instance/workload VLANs join these links when their first consumer
arrives.

## DHCP and DNS ownership

Expand Down