Set-delayed cement compositions comprising pumice and associated methods