Im of a similar height and build to you, Dan. Ive tried a large size on and I found it a little too close fitting (this was a newer version that had, according to the manufacturer, 25% more fabric in it than a previous version). The XL size has just the right amount of extra room that suits me. I can get a bulky mid-layer on underneath it if I needed more insulation in cold weather, and I find that the slightly looser fit when its worn without a bulky insulating mid-layer, isnt problematic as the neck, wrists and waist cord can all be closed to prevent heat loss, or conversely loosened to vent warm moist air if Ive been wearing it during strenuous activity.
From my experience with other Swazi gear, the general rule of thumb would be, if you like a close fit buy the size as indicated on the manufacturers chart that fits your body measurements, and if you prefer a looser fit, then buy the next size up.
Hope this helps.
Best regards,
Paul.