replace by fee – Losing replaced transactions when watching the mempool


I am trying to capture all the replaced transactions. I have complied Bitcon Core with tracing enabled and listen to the mempool:replaced as suggested in this question.

This method works as expected and I can get the replaced transactions. However, in some cases (about 20% of all the transactions I capture) the tracepoint alerts me of the transaction being replaced, but then I am not capturing the replacement of that transaction.

I can think of two reasons why this is happening:

  1. It could be that some of the replacements do not enter my mempool. But if that is the case, how does the tracepoint knows that the transactions is being replaced? Also, it is weird that this would happen with so many transactions (20% as explained earlier).
  2. The second option is that the script that process the transaction is not considering some cases, which ends up in losing transactions. The script is a modified version of this example. I have modified it so I can capture the children of the replaced transactions as well. The data I gather with this script is later saved in a pkl file, which is why it has a shared list.

I don’t know why I am losing so many transactions, and I was wondering if someone could help see what I am doing wrong.

Here are the modified functions:

       def handle_replaced(_, data, size):
          # Establish a new RPC connection for this event
          rpc_connection_replaced = connection()
          event = bpf["replaced_events"].event(data)
          hash_replaced = bytes(event.replaced_hash)[::-1].hex()
          hash_new = bytes(event.replacement_hash)[::-1].hex()
          tx_time = get_timestamp()
          
          hex_tx = rpc_connection_replaced.getrawtransaction(hash_new)
          new_tx = rpc_connection_replaced.decoderawtransaction(hex_tx)
          new_tx['hex'] = hex_tx
          # Determine parent transactions that are still in the mempool and are not the transaction itself
          parents = set([x['txid'] for x in new_tx['vin'] if x['txid'] in mempool.keys() and x['txid'] != new_tx['txid']])
          # Retrieve the old transaction info from the mempool 
          old = mempool.get(hash_replaced, None)
          if old is not None:  # Some transactions may be missing initially
            mempool.pop(hash_replaced, None)
            # Share the replaced event details via list_shared queue
            list_shared.put((old, [new_tx, tx_time, parents]))
            # Find all child transactions that reference the new transaction as a parent
            childs = [i[0]['txid'] for i in mempool.values() if new_tx['txid'] in i[2]]
            
            # Recursive function to collect child transaction IDs
            def child(ll):
              if len(ll) == 0:
                return []
              new_childs = [i[0]['txid'] for i in mempool.values() if ll - i[2] != ll]
              return list(ll) + new_childs + child(set(new_childs))
            
            if len(childs) > 0:
              childs = child(set(childs))
              # Share the connection between the new transaction and the first child
              list_shared.put(([new_tx, tx_time, parents], mempool[childs[0]]))
              # Share connections between subsequent child transactions
              for i in range(len(childs)-1):
                list_shared.put((mempool[childs[i]], mempool[childs[i+1]]))

          # Add the new transaction to the mempool with its timestamp and parent set
          mempool[new_tx['txid']] = [new_tx, tx_time, parents]
          logger.info('-----------')
          logger.info('New RBF!')        
          
      def handle_added(_, data, size):
          rpc_connection_added = connection()
          event = bpf["added_events"].event(data)
          hash_new = bytes(event.hash)[::-1].hex()
          hex_tx = rpc_connection_added.getrawtransaction(hash_new)
          tx_raw = rpc_connection_added.decoderawtransaction(hex_tx)
          tx_raw['hex'] = hex_tx
          parents = set([x['txid'] for x in tx_raw['vin'] if x['txid'] in mempool.keys()])
          mempool[tx_raw['txid']] = [tx_raw, get_timestamp(), parents]
          
      def handle_removed(_, data, size):
        event = bpf["removed_events"].event(data)
        if event.reason != b'replaced':
            txid_rem = bytes(event.hash)[::-1].hex()
                    
            keys = mempool.keys()
            if txid_rem in keys:
              mempool.pop(txid_rem) 
              logger.info('-----------')
              logger.info(f'Removed. Reason:{event.reason}')

Leave a Reply

Your email address will not be published. Required fields are marked *