Running TigerBeetle without a control plane database. Locking.

Of course, TigerBeetle does not provide anything for locking, but I don’t want to introduce a control-plane database or add Redis to the mix, so let’s build some new nails for the TigerBeetle hammer.

First, we will need a dummy account to use as the second leg for the double-entry accounting.

account = tb.Account(
    id=43,
    ledger=42,
    code=42,
)
errors = client.create_accounts([account])

Then we will need a new account for each lock (mutex) with flags that prevent a negative balance:

mutex = tb.Account(
    id=42,
    ledger=42,
    code=42,
    flags=tb.AccountFlags.DEBITS_MUST_NOT_EXCEED_CREDITS,
)
errors = client.create_accounts([mutex])

Each mutex must be initialized with an initial balance of 1. To make it fail-proof, we can use the account identifier as the transfer identifier to enforce idempotency. We will get CreateTransferResult.EXISTS when the mutex account has been initialized before.

init = tb.Transfer(
    id=42,
    debit_account_id=43,
    credit_account_id=42,
    amount=1,
    ledger=42,
    code=42,
)

errors = client.create_transfers([init])
if errors and errors[0].result != tb.CreateTransferResult.EXISTS:
    raise ValueError(errors)

Now we can implement the actual locking. We will implement lock expiration (or Time-To-Live) to avoid crashed or stalled processes from halting the system. TigerBeetle has a timeout feature that will come in handy.

Locking will transfer an amount of 1 from the mutex account back to the dummy account with a timeout that will automatically cancel the transfer:

lock = tb.Transfer(
    id=tb.id(),
    debit_account_id=42,
    credit_account_id=43,
    amount=1,
    ledger=42,
    code=42,
    flags=tb.TransferFlags.PENDING,
    timeout=ttl,
)

TigerBeetle does not provide a tool to wait for the account balance, so we need a busy-loop that will try to acquire the lock for some time. We will retry the locking transfer when it fails with CreateTransferResult.EXCEEDS_CREDITS. The curious detail is that the failing transfer is still persisted (?!) - we need to assign a new transfer ID for each try, or it will fail with an error saying that the transfer already exists. And to avoid polluting TigerBeetle with failed locking attempts, I added a small delay after each failure. But just imagine all our locking attempts stored durably for audit purposes :D

deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
    lock.id = tb.id()
    errors = client.create_transfers([lock])
    if errors:
        if errors[0].result == tb.CreateTransferResult.EXCEEDS_CREDITS:
            time.sleep(0.1)
        else:
            raise ValueError(errors)
    else:
        break
else:
    raise TimeoutError()

And now the easy part - releasing the lock is just voiding the pending lock transfer. The only interesting part here is handling the expired transfers - when you have held the lock for a longer time than the Time-To-Live. We might want to report these cases to better adjust the timeout values.

unlock = tb.Transfer(
    id=tb.id(),
    pending_id=lock.id,
    flags=tb.TransferFlags.VOID_PENDING_TRANSFER,
)
errors = client.create_transfers([unlock])
if errors:
    if errors[0].result == tb.CreateTransferResult.PENDING_TRANSFER_EXPIRED:
        pass
    else:
        raise ValueError(errors)

Is it crazy? Yes. But does it work? Yes.