High-level overview of current 'wallet2' functionality

[rbrunner7]

I went through the code of wallet2 and made the following high-level overview of all the functionality that is implemented there, in over 14,000 lines of code. The idea was to start somehow with getting an overview what we have to implement, because sooner or later we will have to decide on the "modules" of the Seraphis wallet, how they will interact, and in which order we will best implement those modules, and that starts with the total of the functionality to implement, IMO.

It's not yet very systematic; so far I put emphasis on completeness and not yet in a good hierarchical ordering / structuring.

And I also dump this here early to get feedback from my fellow team members :)

- manage wallet files
  - construct wallet file names, using some conventions
  - manage ".keys" file, read and write keys and options from and to it
  - set option defaults for new wallets
  - handle file encryption and decryption based on the password
  - auto-upgrade wallet files from older formats
  - deal with "portable" versus "unportable" file formats (legacy? "rotten" code?)
  - make a copy of a wallet by copying all files under new names
- manage wallet
  - restore normal wallets from secret keys or from seed
  - handle or restore view-only wallets
  - handle wallet passwords
    - handle input at all the right times
    - check validity by checking whether wallet file decrypts properly
  - evaluate wallet options and create wallet object accordingly
    - options from config file
    - options on command line
  - care for the 3 possible networks mainnet, stagenet and testnet
  - encrypt and decrypt private keys to hold them in memory only for short times
  - manage the connection to the daemon
  - handle subaddresses and accounts
  - manage the list of outputs that the wallet owns
    - manage their state, spent or not
    - manage freezing and thawing to influence their spending order
  - manage the list of payments made
  - handle the address book
  - handle transaction notes, account tags, wallet comment
  - handle wallet attributes
  - calculate various balances, e.g. per account, unlocked versus locked
- handle short, encrypted payment ids
- parse URIs with receiver address and amount
- prune a transaction
- manage the "ring database" to deal with chain forks like MoneroV
  - manage "blackballing" of outputs based on ring database data
- construct new transactions
  - estimate transaction size, weight and fee
  - handle fee multipliers
  - auto-pick fee multiplier based on an estimate of transaction backlog size
  - pick outputs to spend, try hard to make "good" choices when doing so
    - calculate "output relatedness" for doing so
  - pick decoys with "gamma distribution"
  - add fake outputs as needed
  - handle tx_extra data
  - actually submit transactions to the daemon
  - split into multiple transactions if "things get too big"
  - calculate key images
  - calculate transaction keys
  - care about pre-RCT outputs to keep them spendable
    - handle "unmixable" outputs for that
  - support "sweeping"
- scan the blockchain
  - deal with different rules according to blockheight and forks
  - listen for new blocks, look through them for transactions destined for the wallet
  - do periodic automatic refresh in a separate thread
  - manage subaddress scanning with look-ahead
  - adjust all internal data for any reorgs that happened
  - parallelize scanning of blocks with multiple threads for more speed
  - calculate and compare view tags
  - rescan the blockchain and refresh all internal walet data in various ways
  - check for double-spends using key images
- handle communication with a connected hardware wallet
- handle sent and incoming transaction in the pool
- communicate with a light wallet server (obsolete? "rotten" code?)
- manage RPC-Pay funds to spend for daemon operations
- imports, exports
  - construct and check spend proofs
  - construct and check transaction proofs
  - construct and check reserve proofs
  - export and import key images
  - export and import outputs
  - save and load transactions to and from files e.g. for cold signing
    - sign transactions loaded from files
- multisig
  - handle initial key echanges for setting up a multisig wallet
  - create a multisig wallet and its address after all key exchange rounds are through
  - load and write multisig transactions from and to files for transporting them between signers
  - sign a multisig transaction

[UkoeHB]

Commented:

- manage wallet files **(possible approach: construct wallet using data extracted from file [file handle is NOT passed into constructor], save wallet data to file with a callback [wallet format etc. is an injected dependency instead of baked in])**
  - construct wallet file names, using some conventions
  - manage ".keys" file, read and write keys and options from and to it
  - set option defaults for new wallets
  - handle file encryption and decryption based on the password
  - auto-upgrade wallet files from older formats
  - deal with "portable" versus "unportable" file formats (legacy? "rotten" code?)
  - make a copy of a wallet by copying all files under new names
- manage wallet  **(some of these things seem like the proper domain for a wallet)**
  - restore normal wallets from secret keys or from seed
  - handle or restore view-only wallets
  - handle wallet passwords
    - handle input at all the right times
    - check validity by checking whether wallet file decrypts properly
  - evaluate wallet options and create wallet object accordingly  **(danger: config variations introduce behavior complexity)**
    - options from config file
    - options on command line
  - care for the 3 possible networks mainnet, stagenet and testnet  **(daemon stuff can be dependency injected)**
  - encrypt and decrypt private keys to hold them in memory only for short times
  - manage the connection to the daemon
  - handle subaddresses and accounts  **(modularize this)**
  - manage the list of outputs that the wallet owns  **(see EnoteStoreMockV1)**
    - manage their state, spent or not
    - manage freezing and thawing to influence their spending order
  - manage the list of payments made  **(modularize this)**
  - handle the address book  **(modularize this)**
  - handle transaction notes, account tags, wallet comment  **(modularize this)**
  - handle wallet attributes
  - calculate various balances, e.g. per account, unlocked versus locked  **(see EnoteStoreMockV1::get_balance())**
- handle short, encrypted payment ids  **(modularize this)**
- parse URIs with receiver address and amount  **(modularize this)**
- prune a transaction  **(why is this even in wallet2?)**
- manage the "ring database" to deal with chain forks like MoneroV  **(modularize this)**
  - manage "blackballing" of outputs based on ring database data
- construct new transactions  **(most of this is implemented in src/seraphis)**
  - estimate transaction size, weight and fee
  - handle fee multipliers  **(better approach than fee multipliers [module]: estimate mining delay at different fee/weights [big statistics research/design project for @Rucknium], produce multiple tx proposals at different fee/weights, user selects fee/weight for delay that they accept)**
  - auto-pick fee multiplier based on an estimate of transaction backlog size
  - pick outputs to spend, try hard to make "good" choices when doing so  **(need to implement: InputSelectorV1)**
    - calculate "output relatedness" for doing so
  - pick decoys with "gamma distribution"  **(need to implement: LegacyDecoySelector for legacy inputs, SpRefSetIndexMapper for seraphis inputs)**
  - add fake outputs as needed  **(see get_additional_output_types_for_output_set_v1())**
  - handle tx_extra data  **(modularize)**
  - actually submit transactions to the daemon  **(submit via callback of some kind)**
  - split into multiple transactions if "things get too big"  **(might be able to split this into free functions)**
  - calculate key images  **(handled by enote records)**
  - calculate transaction keys  **(not sure what this means/entails)**
  - care about pre-RCT outputs to keep them spendable  **(pre-RCT outputs are treated equivalent to RCT outputs; see LegacyEnoteVariant)**
    - handle "unmixable" outputs for that
  - support "sweeping"  **(maybe use free function(s); it's a standard part of 'I want to make a tx' interface)**
- scan the blockchain  **(need to implement: EnoteScanningContextLedger, EnoteFindingContextLedger, EnoteStoreUpdaterLedger)**
  - deal with different rules according to blockheight and forks
  - listen for new blocks, look through them for transactions destined for the wallet
  - do periodic automatic refresh in a separate thread  **(I question if this needs to be inside the wallet proper)**
  - manage subaddress scanning with look-ahead  **(modularize?)**
  - adjust all internal data for any reorgs that happened  **(see SpEnoteStoreMockV1)**
  - parallelize scanning of blocks with multiple threads for more speed  **(EnoteFindingContextLedger)**
  - calculate and compare view tags
  - rescan the blockchain and refresh all internal walet data in various ways  **(see refresh_enote_store_ledger())**
  - check for double-spends using key images
- handle communication with a connected hardware wallet  **(modularize if possible)**
- handle sent and incoming transaction in the pool  **(baked into refresh_enote_store_ledger(); can specialize EnoteScanningContextLedger to only look at the unconfirmed pool if you want to ignore blocks)**
- communicate with a light wallet server (obsolete? "rotten" code?)  **(should be implemented as an EnoteFindingContextLedger backend)**
- manage RPC-Pay funds to spend for daemon operations  **(consider deprecating this)**
- imports, exports  **(modularize; all the meat of this stuff should definitely be in free functions)**
  - construct and check spend proofs
  - construct and check transaction proofs
  - construct and check reserve proofs
  - export and import key images
  - export and import outputs
  - save and load transactions to and from files e.g. for cold signing
    - sign transactions loaded from files
- multisig  **(multisig wallet should NOT be baked into the normal wallet; see unit_tests/seraphis_multisig::seraphis_multisig_tx_v1_test() for all [currently non-legacy] multisig operations)**
  - handle initial key echanges for setting up a multisig wallet
  - create a multisig wallet and its address after all key exchange rounds are through
  - load and write multisig transactions from and to files for transporting them between signers
  - sign a multisig transaction

[UkoeHB]

Went through this again, this time separating the existing functionality based on how I see it being modularized.

// misc admin/orchestration: TBD
- manage wallet files
  - construct wallet file names, using some conventions
  - manage ".keys" file, read and write keys and options from and to it
  - set option defaults for new wallets
  - handle file encryption and decryption based on the password
  - auto-upgrade wallet files from older formats
  - deal with "portable" versus "unportable" file formats (legacy? "rotten" code?)
  - make a copy of a wallet by copying all files under new names
- manage wallet
  - restore normal wallets from secret keys or from seed
  - handle or restore view-only wallets
  - handle wallet passwords
    - handle input at all the right times
    - check validity by checking whether wallet file decrypts properly
  - evaluate wallet options and create wallet object accordingly
    - options from config file
    - options on command line
  - care for the 3 possible networks mainnet, stagenet and testnet
  - manage the connection to the daemon
  - handle wallet attributes
- prune a transaction
- imports, exports
  - construct and check spend proofs
  - construct and check transaction proofs
  - construct and check reserve proofs
  - export and import key images
  - export and import outputs
  - save and load transactions to and from files e.g. for cold signing
    - sign transactions loaded from files

// balance recovery: enote store + enote scanning framework + customizable find-received backends (legacy and seraphis backends are separate)
- scan the blockchain
  - deal with different rules according to blockheight and forks
  - listen for new blocks, look through them for transactions destined for the wallet
  - adjust all internal data for any reorgs that happened
  - parallelize scanning of blocks with multiple threads for more speed
  - calculate and compare view tags
  - rescan the blockchain and refresh all internal walet data in various ways
  - check for double-spends using key images
  - CROSS-PROCESS TBD: manage [legacy] subaddress scanning with look-ahead
- FEED TX HISTORY: handle sent and incoming transaction in the pool
- ??: handle communication with a connected hardware wallet

// balance recovery process management: TBD
- scan the blockchain
  - do periodic automatic refresh in a separate thread
  - CROSS-PROCESS TBD: manage [legacy] subaddress scanning with look-ahead

// detailed enote store: TBD (connect to core enote store)
- handle short, encrypted payment ids
- manage wallet
  - manage the list of outputs that the wallet owns
    - manage their state, spent or not
    - manage freezing and thawing to influence their spending order
  - calculate various balances, e.g. per account, unlocked versus locked

// core transaction builder: seraphis lib methods and types
- construct new transactions
  - NEW: deterministic tx size/weight/fees
  - add fake outputs as needed
  - handle tx_extra data
  - calculate key images
  - calculate transaction keys

// normal wallet tx builder: TBD
- manage wallet
  - encrypt and decrypt private keys to hold them in memory only for short times
- parse URIs with receiver address and amount
- construct new transactions
  - NEW: fee estimator based on recent network activity
  - INPUT SELECTOR CONNECTED TO DETAILED ENOTE STORE (unified legacy + seraphis): pick outputs to spend, try hard to make "good" choices when doing so
    - calculate "output relatedness" for doing so
  - REFERENCE SET SELECTORS (legacy and seraphis): pick decoys with "gamma distribution"
  - actually submit transactions to the daemon
  - split into multiple transactions if "things get too big"
  - support "sweeping"
- ??: handle communication with a connected hardware wallet

// normal wallet tx history: TBD
- CONNECT TO ENOTE STORE: handle sent and incoming transaction in the pool
- manage wallet
  - manage the list of payments made
  - handle transaction notes

// normal wallet misc??: TBD
  - handle subaddresses and accounts
  - handle the address book
  - handle account tags, wallet comment

// multisig account setup utility: TBD
- multisig
  - handle initial key echanges for setting up a multisig wallet

// multisig wallet transaction builder: TBD
- parse URIs with receiver address and amount
- multisig
  - CONSUME SETUP UTILITY: create a multisig wallet and its address after all key exchange rounds are through
  - load and write multisig transactions from and to files for transporting them between signers
  - sign a multisig transaction

// multisig transaction history: TBD
- CONNECT TO CORE ENOTE STORE: handle sent and incoming transaction in the pool

// deprecated
- communicate with a light wallet server (obsolete? "rotten" code?)
- manage RPC-Pay funds to spend for daemon operations
- construct new transactions
  - care about pre-RCT outputs to keep them spendable
    - handle "unmixable" outputs for that
- manage the "ring database" to deal with chain forks like MoneroV
  - manage "blackballing" of outputs based on ring database data
- construct new transactions
  - estimate transaction size, weight and fee
  - handle fee multipliers
  - auto-pick fee multiplier based on an estimate of transaction backlog size