JustUs2- I would recommend keeping mum towards your husband about your hesitation, because it sounds like he's already heard it all from you. I think you should pray and be open to what our Heavenly Father may have to say to you. Like Julieb said, I had moments in my walk, when my Heavenly Father corrected me in how I treated my husband, and idols that I placed in front of my Heavenly Father. It may be that you are not meant to be in a poly marriage, but for sure you are meant to be with your husband, whether he decides to have one wife or more.
There was a time early on in my marriage, when I wondered if my husband was backsliding because he accepted a can of beer from the neighbor lady while working outside. She was not the most virtuous of ladies, and that may have added fuel to the fire, but the fire started because I had nearly sworn off alcohol having had a childhood with an alcoholic step father. I would literally cringe at the sound of a beer can or bottle opening. I almost left. Went to a hotel to think and pray. I heard my Heavenly Father say, "Is this how you are going to treat the husband I gave you?" You see, I had been wondering if marrying him was a mistake. Had I married an unequal yoke? But Yah's reply gave me peace, He assured me that this husband I had was given me by my Heavenly Father. He was an equal yoke, and I couldn't find a Scripture saying our marriage was forbidden in the first place. That meant my vows stood. Even if he was backsliding, and he wasn't, as I was totally overreacting, but even if he was, the vow I made was for better or worse, for richer or poorer, sickness and in health, to love and to cherish, until death do us part. Backsliding is not death parting us, and certainly a beer on a hot day while working on the house is not. I now have no issue with my hubby drinking alcohol a few times a week, and I partake of some with him occasionally myself, because I know me, my husband, and my Bible better than I did 12 years ago, and I know we are not in danger of becoming drunken or alcoholics.
Be careful about questioning your husband's motives about his openness about or even desire to have another wife. Only our Father in Heaven really knows our hearts. Don't think you do, even if your husband sinned in the past with fornication a few years ago while you were married. Maybe his motives were wrong originally, which may have led to sexual sin, and maybe he realizes that now and sees he needs to go about it a different way. If your husband sexually sinned with a woman, has he at any point since then confessed and repented? He must have if he's not jumped into another sexual relationship, right? Sometimes, our Father in Heaven just wants us to be open to a possibility, because like Julieb said, your public image, job, or church may have become an idol in your life that is preventing you from making just and Biblical judgements about polygynists, your own husband, and though you may not realize it, our Heavenly Father. Each one of us is a bride of Christ. Yes, we are one body, but as individuals, we each have our own relationship with our Heavenly Husband. Yah has more than one wife. If you can cope with that, maybe you can start there in your struggle with the idea of plural wives.
By the way, I'm an only wife, too, and my husband is open to more than one wife, but I was the first to bring up to my husband just over 5 years ago that maybe the Old Testament concept of plural wives was never made illegal in the New Testament, after a long night of Scripture study. My husband is in no hurry to take another wife, though he would like to have more children, and I cannot have anymore than the 2 I gave him. Every once in a while we talk about adoption, or surrogacy, or adding another wife, but we haven't recently actively been seeking either of those roads because we have been focusing on becoming better spouses to each other and better parents to the children we have been blessed with. We just remember that if we put the Kingdom first, all the rest will be added unto us, and it really is coming to pass in our lives. Feel free to PM me. Not sure if I can send PMs...