btrfs: fix return value check of btrfs_start_transaction()