To be honest, a great deal depends on the genre. Country genres, for example tend to be less tolerant of mismatched syllables because that affects the melody and rhythm of the song. Where melody and vocal rhythm is less key syllable counts can greatly vary. Either way, it affects the melody and vocal rhythm.
Most pop songs build it in to a small degree to add that essential variance between verses, to help keep the song interesting. It is a balancing act, between the expected rhythm, variation and memorability. A couple of syllables is often okay, too many syllables different and it becomes more of a challenge. Scansion will also dictate where the added or removed syllables will go within any given line (scansion being that natural spoken rhythm of syllables and words).
Often writers think they absolutely must match syllables (or don't care at all). Certainly closer to matched syllables tends to lend a song to portability across genres, but it's not fixed in stone. I think as long as you are close-ish I wouldn't worry unnecessarily.