Borrowers ages 50 to 61 have the highest average balance of any age group at $48,203, according to federal data.