You have been told the letter of the law. Once that visa is cancelled, yes, you SHOULD get out of the country at the first possible moment, or you would be subject to deportation. If you get deported from a country, that has repercussions for any visa for anywhere that you later want to apply for - having to declare yourself as a law-breaker - and being banned from applying for another NZ visa for (I think) five years.
It seems to vary how quickly INZ cancel a visa once the worker leaves a job. In some cases, it has sounded as though it may be that the employer hasn't immediately reported them as having left. EGoodhue and ChrisMwn, is there a set procedure, please?
This next part is just me thinking, nothing official. If you had some holiday entitlement from your job, not used, MAYBE you could get your employer to agree that you could take that time for your little break and sorting out time, after you finish going in to work, but still nominally employed???