Is this rude and intrusive ?

Don't think it's rude if you gave him your number and haven't told him you don't like early and/or late 'calls'. If you didn't give him your number, I'd have mive problems with that.