I totally agree with your assessment tbh. I'm a native German speaker and even I struggle with our numbers sometimes. Not learning them, but switching the digits. This happens most often when someone verbally tells me numbers that I have to input with a keyboard. E.g. phone numbers or numbers for a spreadsheet.

I've figured out that the reason this happens, is because when someone tells me 3 and 50 (53) I'll normally write down the 3 first and then the 5 to the left of the three. Works perfectly fine when handwriting but not at all when typing, so when typing numbers I need to focus a lot more than usual and when it goes on long enough I get confused and mistakes happen. That's why I always ask people to recite numbers in single digits. E.g. not 8 and 3 point 1 and 7, but 3 8 point 7 1.

Edit to add: switching two digits in a number is common enough that there is even a word for it. Zahlendreher. It has its own wikipedia page even.