If I'm reading NEC 430 Part III correctly, a 5 HP, 230 V, 3-phase motor with 12.8 A full-load current needs a 16 A circuit (125% of full-load current). So, 12 ga wire with a 20 A breaker should do. The 12 ga THHN wire has an ampacity of 30 A, derated for 7-9 conductors makes it 21 A, so that should be more than adequate, right?
But my service is 208 V 3-phase. Does that change anything?