The accusation levelled against him is that he is promoting smoking a joint to God, or words to that effect. In reality he says nothing of the sort, nor even hints it. On the contrary, he says if you WERE smoking (over Sat night / Sun morning), and now have the munchies, come to church and we'll feed you. If anything he's offering to help them sober up, and while they're doing it, hear about God. The smoke-filled car is a symbol of where someone was, before church, and through the video the smoke disappears - even the symbolism is of the smoke going away.
He's trying to reach people in a certain culture and turn them towards God. Whether his method is effective or not is another matter. It's not promoting anything bad even if marijuana was a sin.
And this really is so much like how people react to polygamy. You mentioned multiple wives so you must be abusive, condone adultery, be interested in little girls, be a Mormon or Muslim, a wife-beater... All sorts of slander, none of which actually follows from polygamy - but have sadly each been truly practiced in that context by someone in the past. There's a reason for the association - but it's still an unfair and slanderous association to make.
In the same way some pastors do condone drug use, so there's a reason for the association being made by people in this conversation also - yet it's still unfair and potentially slanderous to assume of this pastor.
People complain about being unfairly judged by others but can be equally quick to jump to similar assumptions about others on another issue.