Growing prohibited plants

I accidentally discovered several poppy bushes on my property. Will I be held responsible for growing them if they appeared there without my knowledge?