I think Drive purposely spared down any unnecessary dialogue or character development because it's trying to make it clear the black-and-white worldview the Driver has. He's the hero saving the damsel in distress (in his mind). And without the internal narration, we're not privy to his motives, but we don't need to be. He's a blank slate, a guy who only knows how to drive. That's his identity. If this were going for any trace of realism, it failed, but I think it's more of a fable.
Taxi Driver takes place in a recognizable New York City. It's Scorsese's vision of NYC. Drive is a glossy, unspecific view of Los Angeles. It's similar to Taxi Driver in that the leads are both delusional anti-heros (but Drive is a bit more ambiguous with the audience about the Driver's mental illness). Other than that, they have very different motives and means of telling their stories.
But if it didn't work for you, it didn't work.