It turns out the Boost_LIBRARYDIR was defined when I call cmake. Removing the CMakeCache.txt file, the problem goes away. I guess I might have set Boost_LIBRARYDIR when I was trying to make it work and it got cached. FindBoost.cmake will exit when Boost_LIBRARYDIR has been defined....
If you have all of that and it's still giving you errors, just go specify the BOOST_LIBRARYDIR variable like any other cmake variable. I assume you can just do it in the gui, but you could also do it on the command line with like so:cmake -DBOOST_LIBRARYDIR:PATH=D:/tmp/boost_...