Extendend whitelist options #1644
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
b1ff17f
to
de26071
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
3 times, most recently
from
cf3fcab
to
cc0f1a7
Compare
Cmdv
requested review from
sgillespie and
kderme
and removed request for
sgillespie
sgillespie
reviewed
Mar 14, 2024
sgillespie
reviewed
Mar 14, 2024
cardano-db-sync/src/Cardano/DbSync/Era/Universal/Insert/Other.hs
Outdated
Show resolved
Hide resolved
sgillespie
reviewed
Mar 14, 2024
sgillespie
previously approved these changes
Mar 14, 2024
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
cc0f1a7
to
ffd68e0
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
72083ce
to
ea4514c
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
ea4514c
to
78418bf
Compare
sgillespie
previously approved these changes
Apr 4, 2024
|
Run into following error message with executable built from this branch, on
and it is hanging like that for quite some time, even though I used Longer log: |
|
Another question would be - should we have validator for stake addresses in config whitelisting ? I used this and no errors on |
what version of cardano-node are you using at the moment? I can't even get things to start syncing locally. I can see in the code where the issue is happening. Will try to get sanchonet working today 👍
The problem with that is if you add the stake address before starting a fresh sync, there is no way to validate if it's existence as it's not been yet created. |
@Cmdv not sure if this is right - it seemed to work with this version. |
|
I am using... node: db-sync: Network: Network Configs: wget https://book.play.dev.cardano.org/environments/sanchonet/config.json \
https://book.play.dev.cardano.org/environments/sanchonet/db-sync-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/submit-api-config.json \
https://book.play.dev.cardano.org/environments/sanchonet/topology.json \
https://book.play.dev.cardano.org/environments/sanchonet/byron-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/shelley-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/alonzo-genesis.json \
https://book.play.dev.cardano.org/environments/sanchonet/conway-genesis.jsondb-sync config: "insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plutus": {
"enable": true
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},Logs: So If I am reading it correctly - it breaks on querying / inserting first record in
Query on database synced without whitelisting: sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | |
2 | 1948 | 0 | | 1000000000 | 1024 | 252 | 2 | InfoAction | {"tag": "InfoAction"} | | | | |
3 | 4116 | 0 | | 1000000000 | 1044 | 253 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 51000000]], null]} | | | | |
(3 rows) |
|
OK, I think I got it something more. So basically adding sancho=# select * from gov_action_proposal limit 3;
id | tx_id | index | prev_gov_action_proposal | deposit | return_address | expiration | voting_anchor_id | type | description | param_proposal | ratified_epoch | enacted_epoch | dropped_epoch | expired_epoch
----+-------+-------+--------------------------+------------+----------------+------------+------------------+---------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------+----------------+---------------+---------------+---------------
1 | 1878 | 0 | | 1000000000 | 1024 | 251 | 1 | TreasuryWithdrawals | {"tag": "TreasuryWithdrawals", "contents": [[[{"network": "Testnet", "credential": {"keyHash": "248f556b733c3ef24899ae0609d3796198d5470192304c4894dd85cb"}}, 10000000000000]], null]} | | | | | in this case sancho=# select * from stake_address where id=1024;
id | hash_raw | view | script_hash
------+--------------------------------------------------------------+------------------------------------------------------------------+-------------
1024 | \xe0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef | stake_test1uqvj4r4h0fh5qhvpjmwedkcnj5xysvq2plutdapw0yq69mccrcnd7 |
(1 row)to "stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2","e0192a8eb77a6f405d8196dd96db13950c48300a0ff8b6f42e7901a2ef"]Moved error past epoch So if stake address for |
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
2772d9f
to
efa38c2
Compare
|
@Cmdv With latest version on this branch I am left stuck at I see that tables listed below are empty while the non whitelisted version of DB have records for them ( the common thing is that they have select * from constitution;
id | gov_action_proposal_id | voting_anchor_id | script_hash
----+------------------------+------------------+-------------
(0 rows)
select * from new_committee_info;
id | gov_action_proposal_id | quorum_numerator | quorum_denominator
----+------------------------+------------------+--------------------
(0 rows)
select * from new_committee;
id | gov_action_proposal_id | deleted_members | added_members | quorum_numerator | quorum_denominator
----+------------------------+-----------------+---------------+------------------+--------------------
(0 rows)
select * from new_committee_member;
id | gov_action_proposal_id | committee_hash_id | expiration_epoch
----+------------------------+-------------------+------------------
(0 rows)
select * from voting_procedure;
id | tx_id | index | gov_action_proposal_id | voter_role | drep_voter | pool_voter | vote | voting_anchor_id | committee_voter
----+-------+-------+------------------------+------------+------------+------------+------+------------------+-----------------
(0 rows)Also do not see records in those tables ( contrary to non whitelisted DB version ) EDIT: "shelley": {
"enable": true,
"stake_addresses": ["e083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2"]
},sancho_regular=# select * from stake_address where id=5;
id | hash_raw | view | script_hash
----+--------------------------------------------------------------+------------------------------------------------------------------+-------------
5 | \xe083d27f03b197750ee41b7dca02c806718164b934b0fbfa747349e8f2 | stake_test1uzpaylcrkxth2rhyrd7u5qkgqeccze9exjc0h7n5wdy73usd0pxpl |
(1 row)
It seems like there might be also issue with sancho_regular=# select * from tx_out where stake_address_id=5;
id | tx_id | index | address | address_has_script | payment_cred | stake_address_id | value | data_hash | inline_datum_id | reference_script_id
----+-------+-------+---------+--------------------+--------------+------------------+-------+-----------+-----------------+---------------------
(0 rows)
sancho_regular=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
6210
(1 row)and on whitelisted by sancho_test_config_shelley_whitelist=# select count(*) from tx_out;
count
--------
103980
(1 row)
sancho_test_config_shelley_whitelist=# select count(*) from tx_out where stake_address_id IS NOT NULL;
count
-------
0
(1 row)So we should get rid off all those records with NULL |
|
This is related to Assuming that config below is correct: "insert_options": {
"tx_out": {
"value": "enable"
},
"ledger": "enable",
"shelley": {
"enable": true
},
"multi_asset": {
"enable": true
},
"metadata": {
"enable": true
},
"plututs": {
"enable": true,
"script_hashes":["34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518","67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656", "bccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611", "1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349"]
},
"governance": "enable",
"offchain_pool_data": "enable",
"json_type": "text"
},I am seeing entries in sancho_test_config_shelley_whitelist_plutus=# SELECT id, tx_id, hash, type, json
FROM script
WHERE hash NOT IN (
'\x34250edd1e9836f5378702fbf9416b709bc140e04f668cc355208518', -- hash for 'timelock' - 1st entry in script_hashes
'\x67f33146617a5e61936081db3b2117cbf59bd2123748f58ac9678656', -- hash for 'plutusV1' - 2nd entry in script_hashes
'\xbccf2a430c016bc960fbf31b02694011cd399d20da8882aac9d33611', -- hash for 'plutusV2' - 3rd entry in script_hashes
'\x1b539b68e5389bdc3fe511c7694583e3755f58d4b38aef50384f4349' -- hash for 'plutusV3' - 4th entry in script_hashes
)
ORDER BY type, id;
id | tx_id | hash | type | json
----+-------+------------------------------------------------------------+----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6 | 17132 | \xeeb006239ad2562556b736c3990f20968fcf89bb6d4f7292aaa0c30a | timelock | {"type": "atLeast", "scripts": [{"type": "sig", "keyHash": "ff750767ac2496a659b8325bf7064f92d969672d931ce0bc0290f6ba"}, {"type": "sig", "keyHash": "b7b9594ef50295739c95144238654413221df009f388cbf9b8ca73b7"}, {"type": "sig", "keyHash": "0253cc2bc1ed8176c675f454dd730fae5bfaa147b73924bde70d786a"}], "required": 2}
24 | 28937 | \x6b0261483fa1fe471c6f941c2a917d75164cbd96a93a9e42f62efd1c | timelock | {"type": "sig", "keyHash": "6c6fe0b26f5bfa67ece9221683ea4e7ad19bdf14db8a9fae7b294b5e"}
9 | 19924 | \x56b0f0b597150e619c76bed60683f3b1e42d7bc0685ed951b882bfc5 | plutusV2 |
10 | 19924 | \x86bff95ba20e9d1d1b34899a56d86bbacc9fed999260b27dcc92d128 | plutusV2 |
11 | 19927 | \x0689163fa0280c34d6e3d78657e2b06980be7ac77854f0dd5201bf05 | plutusV2 |
12 | 21241 | \x8275a6621fdb65e60c117fddd5f162d43adf6364c2645cd865bf5482 | plutusV2 |
14 | 24872 | \x04b8c921d534b5debae418ae679a83ec5e628b8e680af1a241788790 | plutusV2 |
... |
|
@ArturWieczorek thanks a lot for the very thorough reports 🙏
I wonder if it has something with stopping and restarting as I got to 301 last week and that was the tip but now we're on 306. So this morning I restarted and like you it was just not moving forward. I can't fully tell it's it's node not providing more data or something is going on on our side. Will investigate more.
So it's doesn't just whitelist against script hash, it also checks payment creds. Which was the existing functionality. I'm unsure as to the reasoning but it's functionality was kept. New code here |
kderme
reviewed
May 14, 2024
kderme
requested changes
May 14, 2024
There was a problem hiding this comment.
I think for multiassets and plutus, there needs to be a better distinction on the checks that happen before trying to insert anything (this checks if there is a whitelist) and the check that happens during inserting (this checks if the feauture is completely disabled). I left some comments that may help
kderme
reviewed
May 14, 2024
kderme
reviewed
May 14, 2024
kderme
reviewed
May 14, 2024
kderme
reviewed
May 17, 2024
Cmdv
force-pushed
the
1600-extend-whitelist
branch
3 times, most recently
from
ec10eac
to
31b0fcd
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
31b0fcd
to
f2b8e01
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
2 times, most recently
from
3d2319e
to
98c7b03
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
98c7b03
to
35014b1
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
2 times, most recently
from
37b3270
to
911028b
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
911028b
to
e285b01
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
2 times, most recently
from
6b3e580
to
1fc6558
Compare
Cmdv
force-pushed
the
1600-extend-whitelist
branch
from
1fc6558
to
c4f7e83
Compare
sgillespie
approved these changes
Jul 19, 2024
|
@ArturWieczorek could you double check this PR for me when you next get the chance? 👍 |
|
On "insert_options": {
"preset": "only_gov"
}with: I ran into [db-sync-node:Info:77] [2024-07-25 11:23:36.81 UTC] Starting epoch 393
[db-sync-node:Info:77] [2024-07-25 11:23:36.81 UTC] Insert Conway Block: epoch 393, slot 33955211, block 1698926, hash 2bbcce48b0580dd58831de1fb1349133ae130b37b0462a3d72728e07892119b6
[db-sync-node:Info:85] [2024-07-25 11:23:36.85 UTC] Asynchronously wrote a ledger snapshot to ledger-state/sancho_only_gov/33955151-163957d025-392.lstate in 0.039110436s.
[db-sync-node:Info:77] [2024-07-25 11:23:37.60 UTC] Inserted 0 EpochStake for EpochNo 394
[db-sync-node:Info:77] [2024-07-25 11:23:38.84 UTC] Insert Conway Block: epoch 393, slot 33976998, block 1700000, hash 48490c35b1cbd36546ec51ccd8b89954b23be572455cf4e2857b4797f550bb60
[db-sync-node:Info:77] [2024-07-25 11:23:45.53 UTC] Removing old epoch boundary file ledger-state/sancho_only_gov/33436789-a577f01423-386.lstate
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Removing 0 orphaned rewards
[db-sync-node:Warning:77] [2024-07-25 11:23:45.54 UTC] validateEpochRewards: rewards spendable in epoch 394 expected total of 3 but got 0
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Persistant SQL Statement Cache size is 44
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC]
Cache Statistics:
Stake Addresses: cache sizes: 0 and 0, hit rate: 0%, hits: 0, misses: 97
Pools: cache size: 0, hit rate: 0%, hits: 0, misses: 1703194
Datums: cache capacity: 250000, cache size: 0, hits: 0, misses: 0
Multi Assets: cache capacity: 250000, cache size: 0, hits: 0, misses: 0
Previous Block: hit rate: 49%, hits: 1703193, misses: 1703194
TxId: cache size: 1390, cache capacity: 300000, hits: 0, misses: 0
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Starting epoch 394
[db-sync-node:Info:77] [2024-07-25 11:23:45.54 UTC] Insert Conway Block: epoch 394, slot 34041601, block 1703194, hash 2588a19fd69700dae6edaf56ee64116ccb19508b66faff27b8b840f096d042f1
[db-sync-node:Error:77] [2024-07-25 11:23:45.57 UTC] Error SNErrDefault: "resolveGovActionProposal.queryGovActionProposalId missing GovAction (TxKey {unTxKey = SqlBackendKey {unSqlBackendKey = 1283}}, 0)"
[db-sync-node:Info:77] [2024-07-25 11:23:45.57 UTC] Shutting down DB thread |
|
@Cmdv Looks like
sancho_new_test_config_full=# select * from tx_cbor;
id | tx_id | bytes
----+-------+-------
(0 rows)EDIT: @Cmdv Looks like it is correct behaviour. According to docs:
|
This was referenced Jul 26, 2024
Closed
|
@Cmdv This is my config - is it correct ? "insert_options": {
"multi_asset": {
"enable": true,
"policies": [
"6c969320597b755454ff3653ad09725d590c570827a129aeb4385526",
"065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404"
]
}
}and here are queries that show issue with filtering not happening. preview_13_3_0_0_ma_filter=# SELECT count(*) FROM multi_asset WHERE policy IN ('\x6c969320597b755454ff3653ad09725d590c570827a129aeb4385526', '\x065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404');
count
-------
34
(1 row)
preview_13_3_0_0_ma_filter=# SELECT count(*) FROM multi_asset WHERE policy NOT IN ('\x6c969320597b755454ff3653ad09725d590c570827a129aeb4385526', '\x065270479316f1d92e00f7f9f095ebeaac9d009c878dc35ce36d3404');
count
-------
1719
(1 row) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes #1600 adding more whitelists to the configs
The most complex whitelist was the Shelley Stake Address as it stored in the table
stake_address.The functionality is if the stake address doesn't match in the whitelist on any of the inserts into the following tables, then we don't insert at all:
Checklist
fourmoluon version 0.10.1.0 (which can be run withscripts/fourmolize.sh)Migrations
If there is a breaking change, especially a big one, please add a justification here. Please elaborate
more what the migration achieves, what it cannot achieve or why a migration is not possible.