Welcome aboard; hope we can help with your problem. I have not personally made any soft goat’s milk cheese; but in my experience a bitter taste is usually due to poor sanitation during the make. Go back over your procedure and analyze everything - and I mean everything - that has any possibility of coming into contact with your milk. I have even gone so far as to sanitize the outside of the milk containers prior to pouring into my cheese pot, just in case something might drop off it into the milk. Being absolutely scrupulous in your make will likely take care of your bitterness problem.
As far as what to do with the cheese - if you don’t like the taste of it - get rid of it.
So, try again, and let us know how it comes out.