Is it bad to not accept an interview at the program you rotate at? Could it harm you in anyway? I feel like an externship is done to see if you see yourself fitting in. So if you realize you don't, then you know it's not for you.
This depends on a lot of factors:
1) Do you have enough other interviews?
2) Did you have a bad vibe during your month there?
3) Did you get negative feedback while you were there?
What I would worry about is that god forbid you don't match (which I'm sure will not be the case!) would you look back and regret not having gone to that interview.
Ultimately if you don't want (or need) to interview probably makes sense to give that spot to someone else who really is interested in that program.