[WIP] add functions to wallet API

[SNeedlewoods]

Do not merge, this is still in development and just PRd for easier discussion.

This is an attempt to make the wallet API "feature complete" and it's part of this proposal.

The approach was to search for every wallet2 function in simplewallet and wallet_rpc_server, if it is used in either of those, but is not implemented in the API I considered it missing and it was added in this PR.
I believe wallet2_api.h now contains all missing functions, but some are not implemented yet. I'm waiting on feedback on those, if you want to have a look, things I need help with are marked as QUESTION, you can also comment on things marked TODO (those may become QUESTIONs if I'm unable(/unsure how) to solve them) or anything else.

Also gathered this collection of (mostly minor) issues found during API work.

[rbrunner7]

As we can see from the table above, there are quite some things that wallet2 provides that are missing from current Wallet API. The additions will therefore be quite substantial.

I think we should not just rush headlong into adding those many things. After all, if we are successful, this API will be very important, will live for a long time, and will be used by many people. If the new resulting interface is not well designed, it will haunt us for years to come.

Problem is that there are too many ways how you could design it, it's not "naturally clear" what types and methods there have to be. My feeling is that you could do almost everything in a hundred different ways.

To contribute something useful in these quite difficult circumstances I try to come up with some general rules to follow when defining the API that will hopefully guide us towards a good solution.

[rbrunner7]

Make it complete

This rule is pretty self-evident and clear, but I repeat it here nevertheless in the sake of completeness.

Goal is to provide a viable successor to the "API" that the public things in wallet2.h provide and eventually be able to completely remove that header file from the codebase. Each and every user of it, e.g. the CLI wallet and the wallet RPC server, will have to switch and use the Wallet API exclusivelly.

Because of this, we have to make sure that the Wallet API is feature complete. We must not lose any wallet functionality that is needed and in actual use.

[rbrunner7]

Make it small

We should try, while respecting the first rule of completeness of course, to make the API small. The smaller it is, the faster will new devs be able to read into it, the easier it will be to find the right methods to use, the less work to document it, etc.

Where do we have potential to reduce the number of methods? One way I see is this:

Do not include methods from wallet2.h that would be trivial to implement by clients using other methods in the interface themselves with only a few lines of code. I would speak of essential methods, that being methods that do provide something that is not available through any other method and that clients can't build themselves because of this. So, include only such essential methods.

I have mentioned an example of such method in this earlier issue about wallet related methods: There is a variant of wallet2::get_payments that filters for payment ID. This method is not "essential" in the sense that I propose here because clients can easily use the general get_payments method and search themselves.

Of course use common sense: It may be essential that already the code of the Wallet API filters or sorts something because otherwise too much data must get transferred, or doing things in the client would be much, much slower.

[rbrunner7]

Be careful about terminology

A good API uses a consistent terminology when naming things, and uses terms that are "expressive", hard to misunderstand and hard to confuse.

IMHO quite a number of things in wallet2 have sub-optimal names, maybe because not enough importance was given to terminology when originally naming things. Prime example for me is the mixup of the terms transaction, payment and transfer. I detailed some examples in this already mentioned issue.

This motivated me to propose to ban the term transfer outright exactly because it's so easy to misunderstand and easy to confuse. I detailed my proposal here.

Of course many things in the Wallet API already have the names they have, and we must let them stand and are not completely free how to name the new things we add. Still I see quite a degree of freedom left.

[rbrunner7]

Limit the influence of existing clients

We have 2 main clients of wallet2.h in the existing codebase: The CLI wallet and the wallet RPC server. According to the plan we will have to migrate them to the Wallet API.

Of course, striclty only seen from the point of view of that migration, the more similar the Wallet API is to the "old" wallet2.h, the less work it will take. The question arises: It this important, and is this something that should guide us when designing the new API?

I propose "No" as answer. I think having a good API is much more important than trying to optimize towards migration work on the CLI wallet and the wallet RPC server being small. That migration happens once and is then over for all times, whereas the Wallet API will go indefinitely into the future and (hopefully) shape tons of Monero related software that together trump any migration work.

In detail that means we don't try to give each method of wallet2.h a more or less one-to-one counterpart in the parts we add to the Wallet API. We mostly design the API independently, following the other rules that I propose here, and then migrate.

[rbrunner7]

Keep the style consistent

I write this down more for the sake of completeness, like the very first rule Make it complete:

There is a clash in naming style between the Wallet API and most of the rest of the Monero codebase: The first uses casing to form names like TransactionInfo and getUnlockedBalance while the rest uses "snake casing", like so: device_derivation_path_option, unconfirmed_transfer_details.

For the things we add we basically have two choices: We keep the naming style consistent with the things that are already there in the Wallet API, or we use the snake casing that is much more widely used in the codebase. Both choices are well possible, and both have advantages as well as disadvantages. (Renaming existing things as a third option and a way to achieve overal naming style consistency is probably not on the table.)

We dicussed this in a Seraphis wallet workgroup meeting, and it seemed that more people voted for keeping the style consistent. I myself was also in that camp: To continue with the "non-standard" naming style is IMHO the lesser evil than mixing styles within a single API.

[rbrunner7]

Now my opinion about one the more important design questions:

wallet2.h has a number of separate methods to get various types of payments / transactions, e.g. get_payments (incoming confirmed payments), get_unconfirmed_payments (incoming unconfirmed payments), get_payments_out (outgoing confirmed transactions), and get_unconfirmed_payments_out (outgoing unconfirmed transactions).

As I explained here, neither the Wallet API, nor the RPC interface, nor Woodser's wallet API have such separate calls. They basically work with a single call that get payments / transactions of all types, using a struct that is "encompassing" i.e. is able to store all details of all these types. I hope I got that right, the Wallet API call for this seems to be history.

According to the rules Make it small and Limit the influence of existing clients I propose to continue with this one call and not create direct equivalents of these methods in the Wallet API.

Probably the type TransactionInfo needs some extensions to be able to give back every last bit of info that all those wallet2 methods provide, but I see this as a quite natural extension of the interface.

[rbrunner7]

Can you give some possible arguments for pushing this down to the account level (if I understand you correctly)? I don't see any right now, but you have probably give this much more thought than I :)

[SNeedlewoods]

Quick update:
There is no need to move createOneOffSubaddress to the account level. Since this is an exotic function, maybe we should add a note/warning that this function should only be used in special cases?
(moneromoo: "It was intended to help people who generated a subaddress far into the random index values.")
e.g. something like:

* brief: createOneOffSubaddress - Create a new account or subaddress for given index, without adding it to known accounts and subaddresses.
*                                 Use `SubaddressAccount::addRow()` or `Subaddress::addRow()` instead, to make sure the wallet adds accounts/subaddresses consecutively and keeps track of accounts and subaddresses.

WalletImpl::addSubaddress & WalletImpl::addSubaddressAccount also do not refresh the known accounts/subaddresses, if that's not for a good reason (that I don't see) I would either add refresh at the end of those functions, or at least give the same warning as proposed above.
Seems the GUI only uses the addRow functions (see here) not the addSubaddress* ones (see here).

[rbrunner7]

Here we probably enter a whole region of things that we won't need in this form if my proposal to make a single encompassing "get payments / transactions" methods really flies.

[rbrunner7]

Can go. We don't have the possibility to choose ringsize anymore for years, and well, soon we won't have any rings at all. Maximum that I see here is an info "ring size" in the new WalletState.

[tobtoht]

Would require something like this: tobtoht@894adf0. Can PR.

[rbrunner7]

I am not sure yet what possibilities of "pool info update" the Wallet API has to offer, or should offer, but I am pretty sure that this method in this form is on too low a level of abstraction, like the following processPoolState as well.

[SNeedlewoods]

This is too complex for me right now to fully dive into, maybe someone with more experience in this area can comment?

[jeffro256]

I try to follow the general principle of YAGNI, especially when making APIs. Don't add it unless someone asks for it. A use case for processing abstractions of the pool state might pop up in the future at some point, but it also might not. Until then, these functions are internal business logic and shouldn't be exposed to the API.

[SNeedlewoods]

If we don't add them to the API, how would the wallet-rpc & wallet-cli call this function, if the goal is to get rid of all the wallet2 function calls? Sorry if I'm missing something.

[rbrunner7]

I checked, and it's used indeed, so it's not a case of YAGNI like @jeffro256 suspected, and after I saw that @moneromooo-monero fixed an information leak a few years back when using remote daemons where somehow this method and the next for processing the pool state were involved, I think it's best to just add those to the interface and keep them "public".

[rbrunner7]

I wonder whether it would make sense to group all the multisig related methods together in the source, and not e.g. put non-multisig and multisig versions of methods that do similar things together. Thus, "write normal tx to file" and "write multisig transaction to file" would not be next to each other, but the first in the group of "normal" methods and the second in the multisig method group.

My argument: Many wallet apps will probably continue to not offer multisig, and their devs may welcome that all those multisig methods do not "clutter" the API for them.

There also have been thoughts that ideally, "multisig wallet" would be a completely separate wallet type, with its own API, and maybe even with the code in a separate repository. That multisig stuff is so highly special, so different, and so complicated ...

I also ask myself wether we can't make some of such methods "smart" anyway so they decide themselves what there is to do, based on internal knowledge whether a transaction is multisig or not.

[SNeedlewoods]

In regards to grouping multisig stuff together, that's already done in another PR (can be seen e.g. here or here). If I remember correctly for this PR I was asked to add everything new in one place, if the "organize functions" get merged this can get based onto that (actually there is still another one for in between in the pipeline I didn't PR yet, the "add comments" found here).

A separate multisig wallet really sounds ideal, not sure how many clients currently use the multisig functionality from the API, the GUI afaik doesn't. But that's out of scope for now I'd say!?

I'll look into if I can come up with such "smart" methods.

[rbrunner7]

Seems to me, if we "go with the pattern", we have a method somewhere that gives back all existing accounts with info, using an account_info struct. The tag would be part of that struct, and thus this call here not needed because not "essential".

Or maybe a call giving back all subaddresses, with info about them, and info how they group into accounts - and any tags of course.

Methods like the following setAccountTag stay of course even if we use this pattern whenever we can, because changing things is something fundamentally different and altogether impossible if there is no method, after all.

[SNeedlewoods]

I think that tags would fit into SubaddressAccountRow, but returning the "i-th account's tag" is just one part of this function, it also returns all tag descriptions (used by wallet-cli and wallet-rpc). So we probably need at least a getAccountTagDescriptions() method anyways, or alternatively we could store the tag description in SubaddressAccountRow, too.

[rbrunner7]

As far as I could see in the code of the CLI wallet that uses this method it's only there to support that super-special case where you could lock a transaction not by giving a blockheight, but a time span, e.g. "10 years". Something nobody ever did in earnest. We anyway removed already all CLI commands to lock.

I am pretty sure we don't need this.

[rbrunner7]

Maybe we should have a list somewhere of public wallet2 methods that do not get a counterpart in the Wallet API, with a short reason per method? Otherwise people in the future may wonder, or worse still, assume an oversight, and get tempted to add it ...

[SNeedlewoods]

A list like the following? Just needs a better place then in the comments of this PR. So far we have:

List of public wallet2 methods that do not get a counterpart in the wallet API

wallet2 method	source	reason
bool unset_ring(const std::vector<crypto::key_image> &key_images)	source
bool unset_ring(const crypto::hash &txid)	source
bool find_and_save_rings(bool force = true)	source
bool is_output_blackballed(const std::pair<uint64_t, uint64_t> &output) const	source
bool is_transfer_unlocked(uint64_t unlock_time, uint64_t block_height)	source
uint64_t get_min_ring_size()	source	Rings will become deprecated soon with FCMPs.
uint64_t adjust_mixin(uint64_t mixin)	source	Rings will become deprecated soon with FCMPs.
bool is_deprecated() const	source	This information is now part of WalletState.
std::string get_daemon_address() const	source	This information is now part of WalletState.

[rbrunner7]

Yes!

[rbrunner7]

Not sure what you mean with "overlap". I think you have to be very careful to not become victim of the confusion in wallet2 regarding the term transfer. If only people would not use terminology confusingly and inconsistently, things could be much easier :)

If you look at the big picture, we have enotes and we have transactions, and these two are two clearly distinct and different things. They are as different as ships and containers. A ship transports containers, that's it already. And a transaction has some enotes as inputs and some other enotes as outputs. Really simple.

Nobody in their right mind would ever confuse ships and containers, that's why everytime I look at some definition in wallet2.h, or check a command in the CLI wallet, and it's not immediately clear whether I am looking at ships (transactions) or at containers (enotes), I ask myself, am I really so dumb, or are these definitions and command names simply not as clear as they should, and could, be?

Anyway, enough ranting, back to the questions at hand: Because enotes and transactions are different things, there are different infos to be had about those two. Any "overlap" would immediately rise my suspicion of a confusion somewhere.

Of course you can cross-reference things, and we should pass on such cross-references whenever wallet2 provides them: A) at a transaction, enotes that were the inputs of it, B) at a transaction, enotes that where the outputs of it, C) at an enote, the transaction it arrived with, D) at an enote, if spent, the transaction that it was spent with.

From some earlier investigation I am moderately sure that info of type D) is not available because wallet2 does not store it anywhere. Maybe it would be a good exercise for you to get a clearer overview to confirm (or refute) that.

[SNeedlewoods]

rbrunner and I already had a chat about this, but for completeness I post my conclusion here.

With overlap I meant what we currently have with "tx_id". To stay with the analogy:
The "tx_id" is an unique identifier for each ship (e.g. crypto::hash m_tx_hash; in payment_details, for ships that deliver containers into our wallet) and we store this identifier alongside each container that we received (as crypto::hash m_txid; for transfer_details). So it's possible to receive multiple containers from the same ship and you can later check which ship delivered which containers and the other way around, looking at the containers, which container got delivered by which ship.

AFAIUI this is the current state of A)-C) (I'm still struggling with cryptonote::transaction_prefix and all the things that come along, so this part may be the most wrong.)

Scenario	Type	Struct	Cross-Reference
A)	outgoing transaction	confirmed_transfer_details & unconfirmed_transfer_details	key_offsets and k_image in std::vector<txin_v> vin; in cryptonote::transaction_prefix m_tx; if the variant txin_v is a txin_to_key
B)	incoming transaction	payment_details	m_tx_hash (loop over wallet2::m_transfers and compare txids)
C)	-	transfer_details	m_txid

I came to the following result for scenario D):
TLDR: It's not easily possible.
Long story: It seems the necessary information is very close to be available in three out of four cases:
m_spent only gets set here
set_spent() only gets called from 4 different places (here 1, here2, here 3, here 4) which all have access to the tx_id (except #2 wallet2::rescan_spent()), so it's not as easy as just adding tx_id_out to the function parameter of set_spent() and m_tx_id_out to transfer_details.

Slightly offtopic, but I assume the k_image in txin_to_key is the reason for this 10 year old TODO?
Also AFAIUI in confirmed_transfer_details & unconfirmed_transfer_details we also store the key image twice, in m_tx.vin[i] and m_rings[i].first.
And in confirmed_transfer_details we store the unlock time in m_tx.unlock_time and m_unlock_time.

After further investigation into cryptonote::transaction_prefix it looks like tx*_to_script and tx*_to_scripthash are not actually used anywhere.
In conclusion so far I tend towards structuring it the following way:

transaction_prefix members	TransactionInfo replacements
size_t version;	std::size_t m_tx_version (new)
uint64_t unlock_time;	uint64_t m_unlock_time;
std::vector<txin_v> vin;	see table below for "txin_v variant"
std::vector<tx_out> vout;	see table below for "txout_target_v variant"
vout[i].amount	std::vector<Transfer> m_transfers; m_transfers[i].amount
std::vector<uint8_t> extra;	QUESTION : Do we really need tx_extra in TransactionInfo? If yes, I'd add: std::string m_tx_extra; (new)

txin_v variant	variant members	TransactionInfo replacements	EnoteDetails replacements
txin_gen (coinbase)	size_t height;	uint64_t m_blockheight	-
txin_to_key	uint64_t amount;	uint64_t m_amount*¹	std::uint64_t m_amount
-	std::vector<uint64_t> key_offsets;	-	QUESTION : Couldn't find a good reason why we would need this in EnoteDetails, any other opinion?
-	crypto::key_image k_image;	-	std::string m_key_image

*¹ Actually this is the sum of all ammounts in the vector vin.

txout_target_v variant	variant members	TransactionInfo replacements	EnoteDetails replacements
txout_to_key (before HF_VERSION_VIEW_TAGS)	crypto::public_key key;	-	std::string m_onetime_address
txout_to_tagged_key (after HF_VERSION_VIEW_TAGS)	crypto::public_key key;	-	std::string m_onetime_address
-	crypto::view_tag view_tag;	-	std::string m_view_tag

Is it correct that key in txout_to_key and txout_to_tagged_key is an output onetime address?

[rbrunner7]

Tricky naming conflict that we have here. We would like to switch from output to enote as term, and with that the method would get a name of exportEnotesToStr, yet the method exportOutputs exists, should probably continue to exist under that name, and sets a precedent of course.

Just now, in the mood that I am right now, I would probably choose exportEnotesToStr. If we are not a little bit aggressive in our switch to enote it will probably fail from pure inertia ...

[rbrunner7]

Anyway, I think if the name stays at least the description should use already enote as term.

[rbrunner7]

Difficult question. I would tend to keep the "style" of using getters, to have things consistent, even if we probably agree that the whole programming pattern is overkill.

Maybe something to bring up in the next meeting?

[rbrunner7]

I would vote for removing. That whole tx_extra mechanism is very controversial, problematic privacy-wise, and we once were on the verge of deleting it altogether. We sure don't want to give people ideas to use it.

Where did you check for use?

[rbrunner7]

Little detail: I think these things are usually merely called seed, not seed phrase.

Anyway, I don't know whether you ever had a look at such a multisig seed: It's not words, but a long hex (or base58?) string.

[rbrunner7]

I don't understand the question. Can you please elaborate?

[rbrunner7]

II think this "interface draft" will soon reach a state where we should consider to try to get feedback about the draft in the form it reached from as many wallet app writers and maintainers as we can reach. Not sure how to best do that, however.