4500 and 500 are the typical ports used for IPSec VPN (specifically ISAKMP is 500/UDP, IPSec NAT traversal is 4500 - usually UDP first then TCP second). If Smoothwall has any settings in that regard that they label as IPSec/VPN related then that would be a good place to tweak as well.
The Cisco IPSec/VPN client also uses port 10000/UDP, not sure if the BB does. Additionally, if you can also filter by raw protocols (other than TCP/UDP) then also prioritize protocols 47, 50 and 51. While not strictly necessary, if you're gonna do it might as well cover all the bases top to bottom.