Do you work? A man that works monday through friday until 7pm is not always going to be up for sex every night no matter how hard he tries or how young he is. It has nothing to do with him not being attracted to you, he's just tired. If it makes you feel better, you should make an attempt to talk to him and come to a happy medium where you were to get more sex (on weekends, perhaps) and he were to get his rest during the week.
Try not to let this make you paranoid, it's normal. Unless he is giving you a real reason to be jealous and paranoid--don't be.