Dust is often used colloquially to refer to any small amount UTXO. In contrast, Bitcoin Core’s precise definition of dust is used in mempool policy to define a minimum for output amounts in standard transactions.
Bitcoin Core considers a transaction output to be dust, when its value is lower than the cost of creating and spending it at the dustRelayFee rate. The default value for dustRelayFee is 3,000 sat/kvB¹, which results in the same dust values as the prior dust definition used before Bitcoin Core 0.15.0. The previous dust definition tied the dust limit to the minRelayTxFee rate and the spending cost of an output exceeding 1/3 of its value.
dust = (input_vsize + output_size) × 3 sat/vB
At the default dustRelayFee of 3,000 sat/kvB, given the size of a P2PKH input being 148 bytes and the size of a P2PKH output being 34 bytes, P2PKH outputs worth less than 546 satoshis are considered dust by Bitcoin Core (546 satoshi being the smallest non-dust value). For P2WPKH outputs, Bitcoin Core considers outputs worth less than 294 satoshis² dust. Since the input sizes of scripthash and taproot outputs are unknowable, Bitcoin Core combines their actual output sizes with the input sizes of the corresponding keyhash types for dust thresholds of 540 satoshis for P2SH, and 330 satoshis for P2WSH and P2TR.
The dustRelayFee can be set with the hidden command line option -dustrelayfee.
Additionally, Bitcoin Core 0.15.0 added a discard_rate. The discard_rate matches the longest target fee rate estimate (currently the 1,000 block fee rate estimate), but is bounded to at least dustRelayFee and at most 10,000 satoshi/kvB. Any change output that would be dust at the discard_rate, i.e. would cost more to spend than its value at the discard_rate, will automatically be dropped to the fee instead of being created as a change output.
Other uses of “dust”
Note that the term “dust” is also colloquially used to refer to a) any UTXOs of vanishingly low value and b) for UTXOs that are uneconomical to create or spend at the current feerate.
¹ The transaction size was measured in bytes before segwit. Since segwit, the relevant measure is transaction weight which is either given in virtualbytes (vbyte, vB) or weight units (wu) where 4 wu = 1 vB. For non-segwit transactions, their size is equal to their virtualsize: 1 B = 1 vB. For segwit transactions, their virtualsize is smaller or equal to their size. You can read more about these units on Is there a difference between bytes and virtual bytes (vbytes)?.
² The attentive reader might expect the dust limit for P2WPKH outputs to be 297 satoshis as P2WPKH outputs are 31 bytes and P2WPKH inputs are about 68 vbytes, but it appears that when the dust limit for P2WPKH outputs was implemented, the code contained a rounding error in the vsize of the input that caused it to be assumed one vbyte smaller than it actually is. (Hat tip to Crypt-iQ for pointing this out.)