Criminalized by opponents
Aside from the slightly increased risk associated with medication use, the biggest drawback is that all psychedelics have been criminalized in the past. This was done to continue suppressing the masses. By portraying psychedelics as dangerous and legally prohibited, we can all be conditioned. If you mention somewhere now that you're going to eat magic mushrooms as medicine, you might get strange looks. Acceptance is low, and that's the only real drawback at the moment.
View here the safety of medication and magic mushrooms
Explanatory video
Want to know why and how we've been lied to into thinking magic mushrooms are strange when they're actually a natural medicine? Then check out this Youtube video.