We study the metric fluctuations from primordial magnetic fields which are generated during inflation by a coupling to the inflaton field. We find that even thuogh the metric fluctuations grow large in longitudinal gauge, the induced Weyl tensor is still much smaller than the background Ricci tensor, and perturbation theory remains valid. However, after inflation, a part of the Weyl tensor remains constant while the background curvature decays. For all interesting magnetic field amplitudes generated in this way, this leads to untenable large fluctuatiuons in the geometry of the Universe. We believe that this problem rules out a large class of mechanismes for the inflationary generation of magnetic fields.