i have heard of people leaving their fellowship for multiple reasons ... i think it depends on what is in you contract ... if you have a true contract, the terms may be spelled out ...
People do get out of fellowship commitments (whether they have signed contracts or not I am not aware) but keep in mind that if you embark upon that path it can have negative repercussions for both you and your residency.
That being said, if it is something you feel you must do I would make sure that you enlist your Department Chairman and Program Director (or any specific sub-specialty specific Attendings who helped you get the fellowship) as soon as possible to help support you. If this is for the upcoming year then please do so as soon as possible so the fellowship can try and recruit a fellow for your position.