You don’t need to delete unsubscribed contacts; we automatically remove unsubscribes from your list(s) for you.
You are not charged for any contact that is in your Unsubscribed folder.
The reason you cannot remove the contact completely is because you must maintain a record of when or if someone unsubscribed from your list.
If this record were removed, then it would be possible for unsubscribes to get re-added to your list, which would be bad for obvious reasons!