When I put the battery in, the lead going from the battery to the plug was fully extended. You need to fold it back on itself - look at the shape of the battery you just removed.
Unfortunately, I didn't realize this until AFTER I clipped the new battery in. I tried to align the battery to where it should go while it was still connected, in the process I broke one of the soldered connections on the battery connector. Wouldn't boot up and did the continuous battery recycling.
Luckily, I was able to put the old battery back in and all was working again. I don't think I'll attempt a new battery at this point...