This experience is not at all uncommon, and the dynamics of the situation will happen every time.
Instead of talking about the weather, talk about what matters to you. What do you love about your life? What are you challenged by? What’s happening that’s upsetting you? What’s your actual deal? And listen to your friends reveal themselves too.
The next time he brings up moving in if I were in your shoes, I’d respond with…
When that first meeting is definitely going to be awkward, learn how to deal with it head-on.
If his lack of knowledge/interest in intellectual topics cause you to lose respect for him, you don’t stand much of a chance together.