He gave you your answer. An officer is only allowed to detain you for a “reasonable” amount of time to conduct his investigation into the original offense and to check the status of your permit. If they lack a “reasonable suspicion” that crime is afoot and if the investigation can’t be done in a “reasonable” time, they have to cut you lose.