How much do they really know about you?

I got called by one a couple of days ago and I told him I knew what he was up to.

I called him a nasty little criminal and a few other names, but I wasn't vulgar, nor did I swear.

His response was that he knew where I lived and he was going to rape me.

I hung up but he rang back twice more, I let it go to answer phone.

The number on my caller-ID is 023597628.

Should I be worried?