My PCIe switch is a Broadcom PEX88096 in the board’s one x16 slot, with four AMD Radeon AI PRO R9700s and the two RTX 3090s from my PCIe splitter post behind it, and the build post has the whole box.

The switch gives each card eight lanes and lets the cards copy to each other without going through the CPU. AMD’s ROCm calls that peer-to-peer (P2P), and it only turns it on when each card’s whole 32 GB is mapped. On Linux, that means a 32 GB Resizable BAR on every card. A base address register, or BAR, maps a range of the CPU’s address space onto a device’s memory. Resizable BAR is the BIOS option that lets a card’s BAR be the full 32 GB instead of the default 256 MB.