I really try to avoid this phrase. It's total bullshit. You didn't have time? Everyone has time. You should instead say "I didn't make time for [whatever]."
No matter what it was, you could have chosen to spend your time on it and you didn't. Don't cop out and say, "I didn't have time" when what you really mean is, "After I did everything that I wanted to do and that was either important to me or urgent, there was no time left over for the unimportant thing you asked me to do" or more succinctly, "I chose not to."
You could have slept a little bit less. You could have watched a little less TV. You could have left work a little earlier or stayed a little later. You could have cancelled that dinner with your friends.
But you chose not to. Own it. Time is the one resource you have that cannot be replaced. It is precious, and how you spend your time defines who you are. You are right to be stingy with your time. Deciding how to spend your time involves making tradeoffs. It's impossible to do everything. People understand this intuitively.
So think again the next time you're about to say "I didn't have time."