My avatar is out of date, as I recently decided to grow a goatee and moustache - or a circular beard - and have the same thoughts as you: some days I look in the mirror and think it suits me, other days I think "nah, shave it off". I put this self-doubt down to never having had a beard before and thus it's something totally different that I'm still getting used to. People have either been complimentary or just seem to not have noticed (I was never clean shaven for periods longer than a couple of days anyway, so really it was a logical step to shape it), and not one person has asked me to get rid of it.
I saw it as a big decision to grow a beard, because it makes you look so different from what you and other people are used to. That said, I'd definitely say keep it, because eventually you'll grow accustomed to having a beard and I'm sure you won't have the doubt that you sometimes seem to have about it at the minute.