The expression is mostly used in informal contexts in American English. When you say that something cost someone a pretty penny, what you mean is that it cost the individual a considerable sum of money. In other words, it was very expensive.
*That new car of his must have cost him a pretty penny.
‘Pretty’ in this context has nothing to do with looks; it means ‘considerable’. A ‘penny’ is a one cent coin, and in the expression, it is used to represent money. So when you say ‘pretty penny’ you mean ‘considerable money’. The expression ‘cost a pretty penny’ was first used in 1768, and at that time you could buy a lot of things with a penny.