I'm never getting gauge

Here is the sweater that I was knitting along with my friend Thuy... (Haven't shortened the zipper yet)

Yummy, yummy yarn.  I love the tonal brown.  "Fig" is the shade.

We converted the pattern to be a more seamless pattern.  Not the easiest thing, since the pattern wasn't exactly the easiest written pattern.   I normally knit quite loosely and with going down a couple needle sizes I seemed to roughly get gauge.  But the other tricky thing about sizing was that it was sizing the sweater according to your bust size, and I don't have a bust anymore.  So I tried to hedge it and make the next size up from my chest circumference.

 It was a top down pattern and everything seemed to be going swimmingly .... until I put in the raglan sleeves, and it was too tight right at the seams.  Aaagh!!

Pattern: Ribby cardi
Yarn: madelinetosh merino vintage