By Ted Sherman and Kelly Heyboer/The Star-Ledger NEW BRUNSWICK — Rutgers athletic director Tim Pernetti, who hired and then opted not to fire the coach at the center of the growing scandal that has all but engulfed Rutgers University, was forced to resign today. Saying he did so reluctantly, Pernetti…