This book came to mind because I'm currently breathing New England atmosphere in my rereading of Moby-Dick. I read it eons ago, but can't remember much of it, and I drove by the house in Salem once, without having a chance to visit it.
I also read The Scarlet Letter, but the Hawthorne works that left the biggest impression on me were a short story, Rappaccini's Daughter, and two books, A Wonder Book and Tanglewood Tales, retelling the Greek Myths for children.