Gilbey never wanted to leave Dons for Charlton