There are a few interpretations on what the law is in these kinds of situations.
In
Gaylord .v. US Postal Service a photograph of a sculpture was deemed to be fair use.
A later lawsuit over a similar issue (Mackie v. Hipple) ended up being resolved in mediation.
You mentioned it being a travel memoir. It is worth remembering that the US Law is a bit different to most other countries.
In Australia,
Section 65 specifically permits the kind of photography you are talking about - it is 100% clear that it isn't a copyright infringement.
In Canada Section 32.2(1) of the Canada Copyright Act states that it is NOT an infringement of copyright for a person to reproduce in photo or film a building or 'a sculpture or work of artistic craftsmanship that is permanently situated in a public place or building'.
However the US equivalent leaves out the sculpture references - it only applies to buildings.
Obviously I'm not a lawyer - this is just my understanding as a layperson.
Mac
(PS:
Rogers .v. Koons is an amusing reversal - and a pretty pathetic attempt to claim a 'parody' exception)