Criminalized by opponents
Apart from the slightly increased risk in combination with medication use, the biggest disadvantage is that all psychedelics have been criminalized in the past. This happened in order to continue to oppress the masses. By dismissing psychedelics as something dangerous and legally prohibited, we can all be conditioned. If you say somewhere that you are going to eat magic mushrooms as medicine, you may be looked at strangely. Acceptance is low and that is now the only real disadvantage.
View here safety of medication and magic mushrooms
Video with explanation
Do you want to know why and how we have been lied to to find magic mushrooms strange while it is a natural medicine? Then check out this one Youtube video.