Device and method for controlling motor system of green car