I personally would steer away from down - yes it is great for warmth but, in cyclable conditions, you are much more likely to experience cold, damp conditions.
I have an old(ish) Rohan insulated jacket (a predecessor to the Icepack-Jacket
which they have in their "Souk" clearance at the mo). Windproof, light & surprisingly warm - I've not yet had to use it up a mountain pass but it does get used sitting around outside on cool evenings & looks reasonably smart for evening wear off the bike. It packs into it own chest pocket & I have pressed it into service as a pillow while camping too.
Other makes do similar things - it may be worth going to an outdoors type shop (or browsing websites & see what they have to balance your preferences of cost, weight, warmth & packability. Gilets/vests of similar construction would work too to give core insulation but at the expense of insulation on the arms.