I forgot to mention the obvious thing for tourists: the Old City, basically the northeast portion of downtown on the river. It is very pretty but as I remember it mostly consists of fancy restaurants and shops selling t-shirts and maple syrup.
If you're interested in art, I would skip the Museum of Fine Arts and the Contemporary Art Museum, and just go to the Canadian Centre for Architecture, which in my opinion is the city's only world-class museum. It only has temporary exhibitions which rotate every four months or so, but they're usually great if you're interested in design:
http://www.cca.qc.ca/
