LITD: Received critical error from subsystem, shutting down: payment isn't initiated

[gStart9]

Hi, we encountered a user running into this issue today on v0.10.5:

2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.804 [INF] LITD: LiT is running in mainnet, the taproot assets subserver do not support the `mainnet` network yet, disabling taproot assets subserver
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.805 [INF] LITD: LiT version: 0.10.5-alpha commit=v0.10.5-alpha
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.806 [INF] LITD: Listening for http_tls on: 127.0.0.1:8443
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.807 [INF] LITD: Listening for http on: 172.18.0.10:8443
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.859 [INF] SESS: Checking for schema update: latest_version=0, db_version=0
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.860 [INF] LITD: Dialing lnd gRPC server at lnd.embassy:10009
2023-09-02T20:19:21+02:00  ----------------------------------------------------------
2023-09-02T20:19:21+02:00   Lightning Terminal (LiT) by Lightning Labs
2023-09-02T20:19:21+02:00  
2023-09-02T20:19:21+02:00   LND Operating mode remote
2023-09-02T20:19:21+02:00   LND Node status online
2023-09-02T20:19:21+02:00   LND Alias xxxxxxxxxxxxxxxxxxxx
2023-09-02T20:19:21+02:00   LND Version 0.16.4-beta commit=v0.16.4-beta
2023-09-02T20:19:21+02:00   LiT Version 0.10.5-alpha commit=v0.10.5-alpha
2023-09-02T20:19:21+02:00   Web interface 127.0.0.1:8443, lightning-terminal.embassy:8443 (open https://127.0.0.1:8443 or http://lightning-terminal.embassy:8443 in your browser)
2023-09-02T20:19:21+02:00  ----------------------------------------------------------
2023-09-02T20:19:21+02:00  2023-09-02 18:19:21.905 [INF] LITD: Connecting basic lnd client
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.116 [INF] LITD: Connecting full lnd client
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.116 [INF] LNDC: Creating lnd connection to lnd.embassy:10009
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.116 [INF] LNDC: Connected to lnd
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.123 [INF] LNDC: Waiting for lnd to unlock
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.125 [INF] LNDC: Wallet state of lnd is now: Lnd main server is ready for requests
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.161 [INF] LNDC: lnd version: v0.16.4-beta, build tags 'autopilotrpc,signrpc,walletrpc,chainrpc,invoicesrpc,watchtowerrpc,neutrinorpc,monitoring,peersrpc,kvdb_postgres,kvdb_etcd,kvdb_sqlite'
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.161 [INF] LNDC: Using network mainnet
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.161 [INF] LNDC: Waiting for lnd to be fully synced to its chain backend, this might take a while
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.190 [INF] LNDC: lnd is now fully synced to its chain backend
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.365 [INF] LOOPD: Protocol version: MuSig2
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.365 [INF] LOOPD: Swap server address: swap.lightning.today:11010
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.365 [INF] LOOPD: Opening sqlite3 database at: /root/.loop/mainnet/loop_sqlite.db
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.498 [INF] LOOPD: Starting liquidity manager
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.498 [INF] LOOPD: Waiting for updates
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.498 [INF] POOL: Version: 0.6.4-beta commit=v0.6.4-beta
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.498 [INF] LOOPD: Starting swap client
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.499 [INF] LOOP: Connected to lnd node 'xxxxxxxxxxxxxxxxxxxx' with pubkey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx (version v0.16.4-beta, build tags 'autopilotrpc,signrpc,walletrpc,chainrpc,invoicesrpc,watchtowerrpc,neutrinorpc,monitoring,peersrpc,kvdb_postgres,kvdb_etcd,kvdb_sqlite')
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.503 [INF] LOOP: Wait for first block ntfn
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.504 [INF] LOOP: Starting event loop at height 805924
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.634 [INF] POOL: Auction server address: pool.lightning.finance:12010
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.650 [INF] CLDB: Checking for schema update: latest_version=1, db_version=1
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.699 [INF] POOL: Using batch version 42 for connecting to auctioneer
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.718 [INF] POOL: Using batch version 42 for connecting to auctioneer
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.718 [INF] RPCS: Starting trader server
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.735 [INF] RPCS: Connected to lnd node xxxxxxxxxxxxxxxxxxxx with pubkey xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.774 [INF] FNDG: Starting funding manager
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.774 [INF] FNDG: Funding manager is now active
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.775 [INF] RPCS: Trader server is now active
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.793 [INF] LITD: Starting LiT macaroon service
2023-09-02T20:19:22+02:00  2023-09-02 18:19:22.973 [INF] AUTO: Starting Autopilot Client
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.620 [INF] LITD: Starting LiT session server
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.648 [INF] SESS: Mailbox RPC server listening on mailbox:xxxxxxxxx1@mailbox.terminal.lightning.today:443
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.703 [INF] SESS: Mailbox RPC server listening on mailbox:xxxxxxxxx2@mailbox.terminal.lightning.today:443
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.729 [INF] LITD: Starting LiT account service
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.730 [INF] SESS: Mailbox RPC server listening on mailbox:xxxxxxxxx3@mailbox.terminal.lightning.today:443
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.808 [INF] LITD: Starting LiT middleware manager
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.816 [INF] LITD: Internal sub server startup complete
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.816 [ERR] LITD: Received critical error from subsystem, shutting down: payment isn't initiated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.817 [INF] POOL: Received shutdown signal, stopping server
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.817 [INF] AUCT: Shutting down auctioneer client
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.817 [INF] RPCS: Trader server stopping
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.817 [INF] AUCT: Shutting down auctioneer client
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.817 [INF] FNDG: Funding manager stopping
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] FNDG: Stopped funding manager
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] RPCS: Stopped trader server
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] LOOPD: Stopping gRPC server
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] LOOPD: Stopping REST server
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [ERR] LNDC: Error closing lnd connection: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] LOOPD: Liquidity manager stopped
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [INF] LOOP: Swap client terminating
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.819 [INF] LOOP: Swap client terminated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.819 [INF] LOOPD: Swap client stopped
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [ERR] RMID: Error in interceptor: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [ERR] RMID: Error in interceptor: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [ERR] RMID: Error in interceptor: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.818 [ERR] RMID: Error in interceptor: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.892 [INF] LOOPD: Daemon exited
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LNDC: Error closing lnd connection: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: payment isn't initiated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: payment isn't initiated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: payment isn't initiated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: payment isn't initiated
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error while stopping litd: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:24+02:00  2023-09-02 18:19:24.909 [ERR] LITD: Error shutting down: RPC middleware receive failed: rpc error: code = Canceled desc = grpc: the client connection is closing
2023-09-02T20:19:29+02:00  2023-09-02 18:19:29.649 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: Try again". Reconnecting...
2023-09-02T20:19:29+02:00  2023-09-02 18:19:29.649 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: Try again". Reconnecting...
2023-09-02T20:19:29+02:00  2023-09-02 18:19:29.649 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: Try again". Reconnecting...
2023-09-02T20:19:32+02:00  2023-09-02 18:19:32.818 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: no such host". Reconnecting...
2023-09-02T20:19:32+02:00  2023-09-02 18:19:32.818 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: no such host". Reconnecting...
2023-09-02T20:19:32+02:00  2023-09-02 18:19:32.818 [WRN] GRPC: [core] grpc: addrConn.createTransport failed to connect to {mailbox.terminal.lightning.today:443 mailbox.terminal.lightning.today:443 <nil> 0 <nil>}. Err: connection error: desc = "transport: Error while dialing dial tcp: lookup mailbox.terminal.lightning.today: no such host". Reconnecting...

Any idea what the issue could be?

[guggero]

Thanks for the report. Is the user using the "custodial accounts" feature of Lit?

[gStart9]

@guggero Yes, they were using a custodial account via Create a new session > Permissions > Custom > Custodial Account

[guggero]

@gStart9 thanks for the info. I have created a fix now: #640
I guess it's not very easy to try out a custom build of LiT on Start9, but if you could test the PR that would be very helpful, as I wasn't able to reproduce this locally.

[gStart9]

Hrm, well I did successfully build it, but as you indicated, now comes the "not very easy" part to package it into a .s9pk. For this, our wrapper was pulling the docker image:
FROM lightninglabs/lightning-terminal:v0.10.5-alpha

Would there be an easy way for me to pull a docker build of that branch with the fix? Or I could just manually copy the new binaries over the binaries in the docker image after the image gets pulled in. Would it be enough to just copy /bin/litd?

[guggero]

Yeah, I think just copying over the /bin/litd should work.