I went to Lowes and bought a 6" 3/8" drive 4mm ball end hex driver and it worked perfectly! about $8 for the tool.
With the tool I described NO, they are not a PITA I've done this three times.
Quit embellishing how hard it can be without the right tool, a simple thing done with the right tool.
My bad, sorry for coming off as a dick.