A Second Date Years Later

This seems to always happen.  I broke up with that guy and a guy I had dinner with years ago reached out to me.  I went out with him again (a second date years later) last night.  I can't see him as anything more than "just a friend."  I have zero attraction towards him.  He really wants to take me on a day trip to Napa or anywhere over the weekend.  I do not want to go.  I can either tell him, I'm not interested or tell him I'm busy (so I can give it some more time before I completely rule him out).  I hear a lot of women say, "I wasn't into him at all when we first met."  I don't think I'm one of those though.