<human>: My boyfriend and I have know each other about a year and have been together roughly six months. We get along great, have a great sex life, and genuinely care about each other. Neither one of us have been in really long relationships and we both agreed that even though we've only been together six months, it feels like longer. 

We both work for the same company (there are no rules against dating). Last night we were talking about work, began talking about the physical attractiveness (or lack there of in our store) and he said, "No one here is really attractive but I guess if I hadn't met you I would've gone after X."

This rubbed me the wrong way but I'm not sure that it should. He's a very affectionate and loving person, but I don't know if I should speak up or be worried.

Summarize the given text in a few sentences.
<bot>: My boyfriend told me that if he hadn't met me he would probably go after another girl we work with. I don't know if I should speak up and be worried or think nothing of it and let it go.