<human>: Can you condense the text into three sentences or less?

This isn't a huge deal but it is something new and I don't understand it, I am hoping someone can shine some light on it for me as I don't even know what to respond.

Lately my SO says something often, he says " I don't have to justify it". 

I only have 2 examples so far, first, I was spending the weekend at his place (he invited me),  Sunday night comes along and he says "maybe its best I drive you back home"  I was a bit sad because I wanted to spend more time with him and I just go " dawww", so he says " yeah I just need a bit of time for myself and do my stuff" so I say "Ok thats fine, you can just tell me that, I understand" and he says " I shouldnt have to justify it"  I just kinda stood there a bit shocked, it struck me as super odd, I never asked him to justify anything or I don't know, I don't even feel like the sentence belonged in our convo all I meant is that he can be straight forward with me instead of beating around the bush.

Second time was yesterday, we were playing a video game and at the end he says "one more ?" to which I answer "hmmm I am going to go to bed.." I was still talking and he interrupted me saying " Its fine you don't have to justify it" lol what? I enjoy communicating and I just wanted to finish my sentence with "...I am tired and I am attempting to get to work early lately" for the sake of talking and sharing my thought.

What gives with the whole justifying thing?! I don't understand and what can I answer to that when he says it? I just want to have a normal conversation.

I must also note that he is really bad at communicating, will avoid it at any cost and run away from serious talk, but thats another issue although could be linked.
<bot>: My SO always says " I dont have to justify it/ you dont have to justify it", I don't understand why he says that