Working stuff out from a recording is a good way to improve. You are doing that already.
Dont try too hard to figure notes out as you play along.
Instead repeatedly listen to the part you want to get , get it into your head and then turn it off.
In silence, try to find the notes guided by memory. If you still dont get it. Repeat the procedure. I used to do this a lot, and became proficient at ear learning.
I was better than I am now. I hardly do this at all anymore, and get frustrated sometimes when I try (like you are finding).