Upgrade the World

[tnull]

Closes #195.

... we update LDK, lightning-liquidity, BDK, rust-bitcoin, rust-esplora-client,
rust-electrum-client, etc.

Notably, we mostly maintain the functionalities here, any feature upgrades and refactors (e.g., adding additional chain data sources which we now can finally do) will happen in follow-up PRs.

Should be ready-for-review. The last commit can be dropped if BDK ships their next release during the review process, otherwise we'll (partially) revert it afterwards (see #359).

[tnull]

VSS CI failure is unrelated and should be fixed by #357.

[jkczyz]

Almost through the LDK upgrade commits.

[jkczyz]

the channel is always treated as outbound (and thus this value is never zero).

Not sure what is meant by this. Is because LDK Node only allowed outbound channels? If so, why do we need to say this?

[tnull]

No, it has nothing to do with LDK Node per se. This too is directly copied from the LDK docs to account for the behavior change:

Note that if this channel is inbound (and thus our counterparty pays the commitment transaction fee) this value will be zero. For ChannelMonitors created prior to LDK 0.0.124, the channel is always treated as outbound (and thus this value is never zero).

[jkczyz]

Should this mention other amounts no longer included here?

[tnull]

Mh, I left the docs to mirror LDK's docs, and there they seem unchanged?

[jkczyz]

What went into picking this number out of curiosity?

[tnull]

Mhh, not much tbh. For the maximum I mostly put a number that's higher in relation to the prior confirmation target, as I'm not super sure how to estimate these fallbacks best. That said, note that we a) current should never use these number in the first place as we enforce/block on updating fee rate estimations on startup (could even consider putting a debug_assert here for the time being and b) potentially should re-evaluate all fee-rate related magic numbers in a follow-up eventually to make sure they still reflect the current fee market. Although, the latter should probably go hand-in-hand with updates to LDK to re-enable interop with other implementations on regtest/signet/testnet which currently seems more or less broken for many users (cf. lightningdevkit/rust-lightning#2925).

[tnull]

Rebased to resolve minor conflict.

[jkczyz]

Reviewed up to but not including the first persistence commit.

[tnull]

Made a few more changes (in the respective fixup commits) to make cfg(vss) build:

diff --git a/src/builder.rs b/src/builder.rs
index 97c34f0f..123f9a0d 100644
--- a/src/builder.rs
+++ b/src/builder.rs
@@ -343,4 +343,6 @@ impl NodeBuilder {
        #[cfg(any(vss, vss_test))]
        pub fn build_with_vss_store(&self, url: String, store_id: String) -> Result<Node, BuildError> {
+               use bitcoin::key::Secp256k1;
+
                let logger = setup_logger(&self.config)?;

@@ -352,12 +354,11 @@ impl NodeBuilder {
                let config = Arc::new(self.config.clone());

-               let xprv = bitcoin::bip32::ExtendedPrivKey::new_master(config.network.into(), &seed_bytes)
-                       .map_err(|e| {
-                               log_error!(logger, "Failed to derive master secret: {}", e);
-                               BuildError::InvalidSeedBytes
-                       })?;
+               let xprv = bitcoin::bip32::Xpriv::new_master(config.network, &seed_bytes).map_err(|e| {
+                       log_error!(logger, "Failed to derive master secret: {}", e);
+                       BuildError::InvalidSeedBytes
+               })?;

                let vss_xprv = xprv
-                       .ckd_priv(&Secp256k1::new(), ChildNumber::Hardened { index: 877 })
+                       .derive_priv(&Secp256k1::new(), &[ChildNumber::Hardened { index: 877 }])
                        .map_err(|e| {
                                log_error!(logger, "Failed to derive VSS secret: {}", e);
diff --git a/src/io/vss_store.rs b/src/io/vss_store.rs
index d7308ae6..474f7dbc 100644
--- a/src/io/vss_store.rs
+++ b/src/io/vss_store.rs
@@ -138,5 +138,12 @@ impl KVStore for VssStore {
                // unwrap safety: resp.value must be always present for a non-erroneous VSS response, otherwise
                // it is an API-violation which is converted to [`VssError::InternalServerError`] in [`VssClient`]
-               let storable = Storable::decode(&resp.value.unwrap().value[..])?;
+               let storable = Storable::decode(&resp.value.unwrap().value[..]).map_err(|e| {
+                       let msg = format!(
+                               "Failed to decode data read from key {}/{}/{}: {}",
+                               primary_namespace, secondary_namespace, key, e
+                       );
+                       Error::new(ErrorKind::Other, msg)
+               })?;
+
                Ok(self.storable_builder.deconstruct(storable)?.0)
        }

[jkczyz]

Should we use the TLV macros instead? Or at least write a length for each field? That way we can accommodate removed fields.

[tnull]

Should we use the TLV macros instead?

I don't think we can, at least not simply impl_writeable_tlv_based if I'm not mistaken? I don't want to write even more boilerplate newtypes for all the individual types that the ChangeSets are composed of (many of which we don't use and hence don't have serialization logic for in LDK), but we're also unable implement Writeable/Readable directly for them. So it's more convenient to implement Readable/Writeable manually and using the De/SerWrappers where needed?

Or at least write a length for each field? That way we can accommodate removed fields.

Hmm, yeah, we could. I now added a fixup that replaces the total length with per-field lengths using some new read/write macros to DRY it up a bit. Note that all this might be over the top as BDK committed to make only additive, backwards compatible changes to the ChangeSet types in form of Options. But I'd rather be safe than sorry here.

[tnull]

Mh, nevermind the "I don't think we can" part. Went down the rabbit hole to clean up the serialization based on macros, until I realized that everything I wrote was already available in LDK. While I still don't think we can just use the high-level macros like impl_writeable_tlv_based, I now switched to use encode_tlv_stream/decode_tlv_stream.

[jkczyz]

So this is saying if we ever failed to write one of these, then we will start with a fresh wallet? Presumably this is safe because the data below would never have been written?

[tnull]

Presumably this is safe because the data below would never have been written?

Right, descriptor/change_descriptor/network are mandatory and we'll always write them before any of the other data. I now added a debug_assert to check this invariant in persist.