Transaction malleability
While transactions are signed, the signature does not currently cover all the data in a transaction that is hashed to create the transaction hash. Thus, while uncommon, it is possible for a node on the network to change a transaction you send in such a way that the hash is invalidated. Note that this just changes the hash; the output of the transaction remains the same and the bitcoins will go to their intended recipient. However this does mean that, for instance, it is not safe to accept a chain of unconfirmed transactions under any circumstance because the later transactions will depend on the hashes of the previous transactions, and those hashes can be changed until they are confirmed in a block (and potentially even after a confirmation if the block chain is reorganized). In addition, clients must always actively scan for transactions to them; assuming a txout exists because the client created it previously is unsafe.
Signature Malleability
The first form of malleability is in the signatures themselves. Each signature has exactly one DER-encoded ASN.1 octet representation, but OpenSSL does not enforce this, and as long as a signature isn't horribly malformed, it will be accepted.[1] In addition for every ECDSA signature (r,s), the signature (r, -s (mod N)) is a valid signature of the same message.[2]
As of block 363724[3], the BIP66 soft fork has made it mandatory for all new transactions in the block chain to strictly follow the DER-encoded ASN.1 standard. Further efforts are still under way to close other possible malleability within DER signatures.
Signatures can still be changed by anyone who has access to the corresponding private keys.
scriptSig Malleability
The signature algorithm used in Bitcoin does not sign any of the scriptSig to create the signature. While signing the whole scriptSig would be impossible - the signature would be signing itself - this does mean that additional data can be added such that it will be pushed on the stack prior to the required signatures and public keys. Similarly OP_DROP can be added to leave the stack exactly as before prior to scriptPubKey execution.
Preventing scriptSig malleability is being considered as well. Currently transactions with anything other than data push operations in their scriptSig are considered non-standard and are not relayed, and eventually this rule may extend to enforcing that the stack have exactly one item after execution. However doing that may interfere with later extensions to Bitcoin.
BIP62
BIP_0062 was a bitcoin improvement proposal from early 2014 which aimed to deal with malleability. It aimed to find all possible methods of malleability and fix them one-by-one. The BIP was withdrawn because that was found to be not sufficient for the use cases which malleability prevents (e.g payment channels).[4][5][6] The BIP document itself contains all malleability methods that the authors could think of.
Segwit
Segregated Witness was an update to bitcoin had as one of its aims to fix all forms of malleability. Transactions which only spend segwit outputs are not vulnerable to malleability.
References
- https://bitcointalk.org/index.php?topic=8392.msg122410#msg122410
- https://bitcointalk.org/index.php?topic=8392.msg1245898#msg1245898
- When did BIP66 switch from activation to enforcement?, answer by StephenM347, 6 July 2015, Bitcoin.StackExchange
- https://bitcoin.stackexchange.com/questions/35904/how-much-of-bip-62-dealing-with-malleability-has-been-implemented
- https://github.com/bitcoin/bips/commit/916142e742b4256686c26b15a0bf943aea3f5ef9
- https://old.reddit.com/r/Bitcoin/comments/3tgmch/bip62_dealing_with_transaction_malleability_has/ archived link