This is a pretty routine point, but I get the same kind of pleasure from reading Borges (or Calvino to a lesser extent) as I get from really good sci-fi. The way they just take some weird idea and run with it. Bad sci-fi authors write about dumb ideas poorly, and of course some sci-fi is just adventure stories in space.
Where Borges seems to take his ideas from history, literature, theology, and philosophy, your Iain Banks takes his from science. (Social science is included in here are the best sci-fi always has visions of weird societies. Ken MacLeod is great at projecting things like left libertarianism into the future where humans interact with space lizards.)