Imagine you lived in 1760. You have a sweet tooth. You buy sugar to put in your tea. Sugar is in itself not intrinsically harmful or addictive. But the sugar industry is based largely on slavery and the sugar brand you buy is made by slaves. Are you doing something wrong?
I see a similar parallel to people who buy drugs today. The main argument for legalizing at least certain drugs such as weed is that they’re less addictive and harmful then already legal drugs such as tobacco and alcohol. I think that argument is sound. I still think that buying drugs from criminals is deeply immoral because those drugs are often grown by slaves and the industry leads to violence and horror for both criminals and ordinary citizens.
My views on how far individuals morally ought to follow states laws are conflicted. But even in the absence of any law, I think buying a good is wrong when the act of purchase contributes to such evil.
TLDR: If you want to do drugs you should grow them yourself, not buy them.