The easy cop-out answer is 'you need a good ear'. But before you can trust your ears you need to start somewhere. You need to know the key 1st. If you can 'hear' the key, then that is ideal.
If not, the note (or chord) that ends a tune is usually what gives it away. So if that chord is 'C major' then the key is C major. If you can hear the root note of C but are unsure of whether its major or minor, then play them both to see what fits. One will sound right & the other will sound wrong.
Sticking with C major. If you play that scale (all the white notes on a piano starting and ending with C) it will sound familiar. If you then play the white notes starting & ending with A (so A is the root note) then the notes it will sound sad. That key is A minor. A is the the 5th note of the C major scale. So start and end using the 5th note.
Still sticking with all the white notes, begin with the 2nd note of that scale, D. Play from D and end with D. This is the Dorian mode in the key of D.
The 4th note is F. Start & end with F will give you the Mixolydian scale in F.
The point is you are using all the exact same notes each time, but beginning and ending in different places.
To extract these examples from any key use the scale numbers explained.
5th = (relative) Minor,
2nd = Dorian
4th = Mixolydian
There are other scales/modes of course, but the method is the same for finding them all. So to find 'E Dorian' you would work this backwards. E is the 2nd note of the major scale of D. So the notes from D major will be the correct notes to use for playing E Dorian.
For the riff you mentioned. You need to recognise the note pattern of the scale/mode in relation the the key. These note patterns are all unique. So it can be done, but you need to put in some scale/mode practice, that will also serve as ear training.
Hope this helps.