There are a lot of influencing factors, the vocalist's range being one of the primary ones. If the vocalist doesn't sound good on the song in that key, you've got an issue. At the same time, if the accompanying instrument - such as guitar or piano - doesn't sound right with the chord progression or riff in that key, you also might need to take that into consideration. A really solid singer with a big range might sound good in a variety of keys, especially ones that are close to each other. A half step difference on guitar can make a big difference in the overall sound because of different voicings available to the player. A capo doesn't always solve this problem, though many times it does. Something might also sound too low and muddy or too high and thin in certain keys on piano or guitar (just picking out those two examples). This doesn't always work out and help show off the song or singer.
At the same time, you need to consider and accept the fact that not every song is going to sound great with every vocalist or accompanying instrument. Most songs seem to have a key (or maybe two) that sound "best" with the progression and melody. Many songwriters and composers feel strongly that every song has a "right" key and performing the song in a different key takes something away. This is subjective and sometimes I agree, sometimes I don't. Many times it has to do with the arrangement, etc. But if the vocalist doesn't sound "right" in that key, you either have to make a sacrifice somewhere, consider getting a different vocalist for the song, or consider rearranging the song to try and make it work.
There are plenty of songs I would love to do as a solo acoustic singer/player but just can't pull off because of my vocal range, the phonetics not sounding right, the arrangement not sounding right in the key that works for my voice, or a combination of reasons. With all of the songs to be written and all of the songs already written, it's not worth fretting about (no pun intended). Just move on to the next, or if you really love the song, find another vocalist for it, rearrange it, etc. There are no limitations with music and collaborations.