The problem becomes more acute when it is not the physical credit card itself which is stolen, but merely its details.
These cases are often described as 'identity theft', which I think is a sneaky way of banks transferring the problem onto the credit card holder. But the problem should not be the credit card holder's. What happens thereafter is fraud: the thief uses the card pretending to be the authorized holder when he is not. That fraud is perpetrated on (i) the credit card company, and (ii) the shop (say) which takes the card in exchange for goods. The credit card holder is entitled to say to the credit card company, because it is true, you have deducted from my account something which I did not authorize, nor given me anything in exchange, so I will not pay that bill.
What about the allegation that the PIN was not secure?
First, how can the credit card company prove that? Just because the card was used, I do not think it follows that the PIN was not secure. Very sophisticated technology exists to obtain PIN numbers. My credit card was skimmed once and used successfully, and my PIN is written down nowhere, and known only to me. (Fortunately the bank blocked the transaction.)
Second, any allegation of tortious negligence on the part of the credit card holder is irrelevant; when A perpetrates a fraud on B, any negligence by C is irrelevant to that claim.
Third, perhaps the proper analogy is with someone who writes a blank cheque then used by a rogue to draw down more money than the writer anticipated. That, I think, is a breach of contract by the cheque writer to the bank. So perhaps an insecure PIN is a breach of contract to the credit card company which, if they can prove it, would allow them to authorize the deduction.
Finally, how can the credit card holder recover from the thief? The thief stole from the credit card company. The credit card holder has to indemnify the company through its breach of contract. The credit card holder gets subrogated to the position of the credit card company and can sue the thief in fraud.
Nathan