The problem #
As the number of peers increases, the likelihood of packets sent via dn42-fw1 having their replies come back via dn42-fw2 (and vice versa) increases. This breaks stateful firewalling because dn42-fw2 has no knowledge of connections initiated from dn42-fw1 (and vice versa) so cannot let return packets through on the basis of established / related connection tracking.
The solution #
A solution is to switch to stateless firewalling:
set firewall ipv4 name DN42_IN default-action drop
set firewall ipv4 name DN42_IN description 'DN42 to internal'
set firewall ipv4 name DN42_IN rule 10 action accept
set firewall ipv4 name DN42_IN rule 10 description 'Forward DN42 transit traffic - i.e. anything not to/from us'
set firewall ipv4 name DN42_IN rule 10 destination address !172.23.38.64/27
set firewall ipv4 name DN42_IN rule 10 source address !172.23.38.64/27
set firewall ipv4 name DN42_IN rule 20 action accept
set firewall ipv4 name DN42_IN rule 20 description 'Allow icmp'
set firewall ipv4 name DN42_IN rule 20 protocol icmp
set firewall ipv4 name DN42_IN rule 30 action accept
set firewall ipv4 name DN42_IN rule 30 description 'Allow udp'
set firewall ipv4 name DN42_IN rule 30 protocol udp
set firewall ipv4 name DN42_IN rule 40 action accept
set firewall ipv4 name DN42_IN rule 40 description 'Allow established tcp'
set firewall ipv4 name DN42_IN rule 40 protocol tcp
set firewall ipv4 name DN42_IN rule 40 tcp flags not syn
set firewall ipv4 name DN42_IN rule 41 action accept
set firewall ipv4 name DN42_IN rule 41 description 'Allow handshake tcp syn-ack'
set firewall ipv4 name DN42_IN rule 41 protocol tcp
set firewall ipv4 name DN42_IN rule 41 tcp flags syn
set firewall ipv4 name DN42_IN rule 41 tcp flags ack
set firewall ipv4 name DN42_IN rule 50 action accept
set firewall ipv4 name DN42_IN rule 50 description 'Allow DNS to ns1.f333.dn42'
set firewall ipv4 name DN42_IN rule 50 destination address 172.23.38.68
set firewall ipv4 name DN42_IN rule 50 destination port 53
set firewall ipv4 name DN42_IN rule 50 protocol tcp_udp
set firewall ipv4 name DN42_IN rule 60 action accept
set firewall ipv4 name DN42_IN rule 60 description 'Allow web access to Caddy on www.f333.dn42'
set firewall ipv4 name DN42_IN rule 60 destination address 172.23.38.69
set firewall ipv4 name DN42_IN rule 60 destination group port-group CADDY_PORTS
set firewall ipv4 name DN42_IN rule 60 protocol tcp_udp
set firewall ipv4 forward filter rule 5 action 'jump'
set firewall ipv4 forward filter rule 5 inbound-interface name 'wg*'
set firewall ipv4 forward filter rule 5 jump-target 'DN42_IN'