The last straw is when, through a comic mishap, they get the misimpression their owner is trying to throw them away. They feel this existential threat releases them from their duty of loyalty. Note, however, that it’s only the misimpression that removes the duty. When they learn (from Mrs. Potato Head) that they were mistaken and Andy only planned to put them in the attic, they rush to return to him. The attic is not an existential threat and thus does not remove the duty.
Warning, spoilers.