-
Notifications
You must be signed in to change notification settings - Fork 122
Data Extraction
Loopring protocol is a layer 2 solution using zkRollup. As such, extracting trade or other information from Ethereum is harder than layer 1 solutions where most data is directly accessible on-chain, likely via easy to parse events. However, even though we are on layer 2, all data is still available on Ethereum.
There are couple of ways Loopring exchange data can be accessed. How you want to access the data depends on who you trust and how the Ethereum data is stored.
We provide an API to get the data directly from our servers, no need to parse any Ethereum data. For overall exchange level data, it is here: https://github.com/Loopring/protocols/wiki/Loopring-Exchange-Data-API This is likely the easiest way to access the exchange data, but it means you'll have to trust exchange to provide correct data.
For the actual Loopring Relayer API (for which to build trading bots, or query individual layer 2 account balances) see here: https://docs3.loopring.io/en/
If you have questions, please email us, or join our Discord.
If the Ethereum data is parsed and stored into a database, you can easily extract most data from Ethereum. As an example, you can visit our Dune Analytics dashboard. By creating an account you can also view the source of each query to learn how this is achieved. There is also a Loopring subgraph you can use to read Loopring data from Ethereum.
Here's some example code that shows how to run over all blocks for an exchange and list all transactions that ever happened, but please check our Dune queries code directly so you always have the latest code. If you're not using Dune the data is likely stored a bit differently, but the main process functions which decodes all trade information should be usable as is.
DROP FUNCTION dune_user_generated.fn_decode_float_24;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_decode_float_24(data bytea) RETURNS double precision AS $$
DECLARE
exponent integer;
mantissa integer;
value integer;
BEGIN
value = get_byte(data, 0) * 65536 + get_byte(data, 1) * 256 + get_byte(data, 2);
exponent = value / 524288;
mantissa = value - (exponent * 524288);
return mantissa * POW(10, exponent);
END; $$
LANGUAGE PLPGSQL;
DROP FUNCTION dune_user_generated.fn_decode_float_16;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_decode_float_16(data bytea) RETURNS double precision AS $$
DECLARE
exponent integer;
mantissa integer;
value integer;
BEGIN
value = get_byte(data, 0) * 256 + get_byte(data, 1);
exponent = value / 2048;
mantissa = value - (exponent * 2048);
return mantissa * POW(10, exponent);
END; $$
LANGUAGE PLPGSQL;
DROP FUNCTION dune_user_generated.fn_process_block;
DROP TYPE dune_user_generated.transaction_struct;
DROP TYPE dune_user_generated.deposit_struct;
DROP TYPE dune_user_generated.withdraw_struct;
DROP TYPE dune_user_generated.transfer_struct;
DROP TYPE dune_user_generated.spot_trade_struct;
DROP TYPE dune_user_generated.account_update_struct;
DROP TYPE dune_user_generated.amm_update_struct;
DROP TYPE dune_user_generated.signature_verification_struct;
CREATE TYPE dune_user_generated.deposit_struct AS (
toAddress bytea,
toAccount integer,
token integer,
amount double precision
);
CREATE TYPE dune_user_generated.withdraw_struct AS (
fromAddress bytea,
fromAccount integer,
token integer,
amount double precision,
feeToken integer,
fee double precision
);
CREATE TYPE dune_user_generated.transfer_struct AS (
token integer,
amount double precision,
feeToken integer,
fee double precision,
fromAccount integer,
toAccount integer,
toAddress bytea,
fromAddress bytea
);
CREATE TYPE dune_user_generated.spot_trade_struct AS (
accountA integer,
accountB integer,
tokenA integer,
tokenB integer,
amountA double precision,
amountB double precision
);
CREATE TYPE dune_user_generated.account_update_struct AS (
ownerAddress bytea,
ownerAccount integer,
feeToken integer,
fee double precision,
publicKey bytea
);
CREATE TYPE dune_user_generated.amm_update_struct AS (
ownerAddress bytea,
ownerAccount integer,
token integer,
feeBips integer,
weight double precision
);
CREATE TYPE dune_user_generated.signature_verification_struct AS (
ownerAddress bytea,
ownerAccount integer
);
CREATE TYPE dune_user_generated.transaction_struct AS (
block_timestamp timestamptz,
blockIdx integer,
txIdx integer,
txType integer,
deposit dune_user_generated.deposit_struct,
withdraw dune_user_generated.withdraw_struct,
transfer dune_user_generated.transfer_struct,
spot_trade dune_user_generated.spot_trade_struct,
account_update dune_user_generated.account_update_struct,
amm_update dune_user_generated.amm_update_struct,
signature_verification dune_user_generated.signature_verification_struct
);
DROP FUNCTION dune_user_generated.fn_to_uint96;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_to_uint96(data bytea) RETURNS double precision AS $$
DECLARE
result double precision;
i integer;
BEGIN
result = 0;
FOR i IN 1 .. 12
LOOP
result = result + get_byte(data, (i-1)) * POWER(2,(96-i*8));
END LOOP;
return result;
END; $$
LANGUAGE PLPGSQL;
DROP FUNCTION dune_user_generated.fn_to_uint32;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_to_uint32(data bytea) RETURNS integer AS $$
BEGIN
return (get_byte(data, 0) & 127) * 16777216 + get_byte(data, 1) * 65536 + get_byte(data, 2) * 256 + get_byte(data, 3);
END; $$
LANGUAGE PLPGSQL;
DROP FUNCTION dune_user_generated.fn_to_uint16;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_to_uint16(data bytea) RETURNS integer AS $$
BEGIN
return get_byte(data, 0) * 256 + get_byte(data, 1);
END; $$
LANGUAGE PLPGSQL;
CREATE OR REPLACE FUNCTION dune_user_generated.fn_process_block(blockSize integer, _data bytea, block_timestamp timestamptz, blockIdx integer)
RETURNS dune_user_generated.transaction_struct[]
AS $$
DECLARE
data bytea;
i integer;
transaction dune_user_generated.transaction_struct;
transactions dune_user_generated.transaction_struct[];
BEGIN
FOR i IN 0 .. blockSize-1
LOOP
data = (substr(_data, 99+i*29, 29)||substr(_data, 99+blockSize*29+i*39, 39));
SELECT
block_timestamp,
blockIdx,
i,
get_byte(substr(data, 1, 1), 0) as txType,
(
substr(data, 2, 20),
dune_user_generated.fn_to_uint32(substr(data, 2 + 20, 4)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 20 + 4, 2)),
dune_user_generated.fn_to_uint96(substr(data, 2 + 20 + 4 + 2, 12))
) as deposit,
(
substr(data, 2 + 1, 20),
dune_user_generated.fn_to_uint32(substr(data, 2 + 1 + 20, 4)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 1 + 20 + 4, 2)),
dune_user_generated.fn_to_uint96(substr(data, 2 + 1 + 20 + 4 + 2, 12)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 1 + 20 + 4 + 2 + 12, 2)),
dune_user_generated.fn_decode_float_16(substr(data, 2 + 1 + 20 + 4 + 2 + 12 + 2, 2))
) as withdraw,
(
dune_user_generated.fn_to_uint16(substr(data, 2 + 1 + 4 + 4, 2)),
dune_user_generated.fn_decode_float_24(substr(data, 2 + 1 + 4 + 4 + 2, 3)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 1 + 4 + 4 + 2 + 3, 2)),
dune_user_generated.fn_decode_float_16(substr(data, 2 + 1 + 4 + 4 + 2 + 3 + 2, 2)),
dune_user_generated.fn_to_uint32(substr(data, 2 + 1, 4)),
dune_user_generated.fn_to_uint32(substr(data, 2 + 1 + 4, 4)),
substr(data, 2 + 1 + 4 + 4 + 2 + 3 + 2 + 2 + 4, 20),
substr(data, 2 + 1 + 4 + 4 + 2 + 3 + 2 + 2 + 4 + 20, 20)
) as transfer,
(
dune_user_generated.fn_to_uint32(substr(data, 2 + 4 + 4, 4)),
dune_user_generated.fn_to_uint32(substr(data, 2 + 4 + 4 + 4, 4)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 4 + 4 + 4 + 4, 2)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 4 + 4 + 4 + 4 + 2, 2)),
dune_user_generated.fn_decode_float_24(substr(data, 2 + 4 + 4 + 4 + 4 + 2 + 2, 3)),
dune_user_generated.fn_decode_float_24(substr(data, 2 + 4 + 4 + 4 + 4 + 2 + 2 + 3, 3))
) as spot_trade,
(
substr(data, 2 + 1, 20),
dune_user_generated.fn_to_uint32(substr(data, 2 + 1 + 20, 4)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 1 + 20 + 4, 2)),
dune_user_generated.fn_decode_float_16(substr(data, 2 + 1 + 20 + 4 + 2, 2)),
substr(data, 2 + 1 + 20 + 4 + 2 + 2, 32)
) as account_update,
(
substr(data, 2, 20),
dune_user_generated.fn_to_uint32(substr(data, 2 + 20, 4)),
dune_user_generated.fn_to_uint16(substr(data, 2 + 20 + 4, 2)),
get_byte(substr(data, 2 + 20 + 4 + 2, 1), 0),
dune_user_generated.fn_to_uint96(substr(data, 2 + 20 + 4 + 2 + 1, 12))
) as amm_update,
(
substr(data, 2, 20),
dune_user_generated.fn_to_uint32(substr(data, 2 + 20, 4))
) as signature_verification
INTO transaction;
transactions = array_append(transactions, transaction);
END LOOP;
RETURN transactions;
END;
$$
LANGUAGE PLPGSQL;
WITH transactions AS (
SELECT unnest(dune_user_generated.fn_process_block(
CAST(t.block ->> 'blockSize' AS INT),
decode(substring(t.block ->> 'data', 3, char_length(t.block ->> 'data') - 2), 'hex'),
c.call_block_time,
blockIdx::integer
)) as transaction,
call_tx_hash as tx_hash
FROM loopring."ExchangeV3_call_submitBlocks" c,
jsonb_array_elements(c."blocks") with ordinality as t(block, blockIdx)
)
If you have access to Ethereum data via a web3 provider you can use our JS data extraction library. This library allows you to collect all exchange states of all Loopring exchanges by running over all Ethereum transactions which impact the exchanges. Afterwards all data is available in an easy to use API. The source code can also be used as a simple example how to decode our on-chain data.
// Initialize the explorer with a web3 provider
const explorer = new Explorer();
await explorer.initialize(web3, universalRegistry.address);
// Fetch all data from the web3 provider
await explorer.sync(await web3.eth.getBlockNumber());
// Get a specific exchange
const exchange = this.explorer.getExchangeById(exchangeId);
// Get account state (owner, balances, ...)
const numAccounts = exchange.getNumAccounts();
const account = exchange.getAccount(accountID);
// Blocks
const numBlocks = exchange.getNumBlocks();
const block = exchange.getBlock(blockIdx);
// Processed requests/transactions
const numRequests = exchange.getNumProcessedRequests();
const requestsInRange = getProcessedRequests(startIdx, count);
// ... and much more
Loopring Foundation
nothing here