A lot of angles to take on this one, so I'll only cover a couple.
Negotiate a rate of return that is fair to both of you. Let free market principles govern as much as possible. If you think you can do better elsewhere, then you can walk. If he thinks he can do better elsewhere, then he can take his dollars elsewhere.
You do gain a fair benefit by having your loan held by a benevolent benefactor. You state that you are not in a position to pay additional lump sum mortgage reduction payments. I'd think hard before giving up this advantage, since you don't have a lot of flexibility in your budget.
Also remember that your dad probably could have put that 150k in a 5-year GIC three years ago for 3-4% (I'm guessing here for argument's sake). He chose to tie up that money with you instead.
Remember to keep things friendly but fair.