Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Helper functions to simplify writing social indexers. #381

Closed
wants to merge 2 commits into from
Closed

Helper functions to simplify writing social indexers. #381

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e5d00dc
Helper functions to simplify writing social indexers.
gabehamilton Nov 9, 2023
0f1ccfb
Handle near-primitives type mismatch, provide type for FunctionCall w…
gabehamilton Nov 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 55 additions & 1 deletion runner/src/indexer/indexer.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,28 @@ interface Dependencies {
parser: Parser
};

// '@near-lake/primitives/receipt' FunctionCall but with base64 decoded args
declare class DecodedFunctionCall {
readonly methodName: string;
readonly args: string;
readonly gas: number;
readonly deposit: string;
constructor (methodName: string, args: string, gas: number, deposit: string);
}

interface Context {
graphql: (operation: string, variables?: Record<string, any>) => Promise<any>
set: (key: string, value: any) => Promise<any>
log: (...log: any[]) => Promise<void>
fetchFromSocialApi: (path: string, options?: any) => Promise<any>
db: Record<string, Record<string, (...args: any[]) => any>>
base64Decode: (base64EncodedString: string) => string
getFunctionCalls: (
block: Block,
contract: string,
method: string,
) => DecodedFunctionCall[]
getSocialOperations: (block: Block, operation: string) => Array<Record<string, unknown>>
}

interface IndexerFunction {
Expand Down Expand Up @@ -140,6 +156,26 @@ export default class Indexer {
const functionNameWithoutAccount = functionName.split('/')[1].replace(/[.-]/g, '_');
const schemaName = functionName.replace(/[^a-zA-Z0-9]/g, '_');

const base64DecodeFunction = (base64EncodedString: string): string => {
const buff = Buffer.from(base64EncodedString, 'base64');
return JSON.parse(buff.toString('utf-8'));
};
const getFunctionCallsFunction = (block: Block, contract: string, method: string): any[] => {
return block
.actions()
.filter((action) => action.receiverId === contract)
.flatMap((action) =>
action.operations
// @ts-expect-error block data is an object keyed by type
.map(({ FunctionCall }) => FunctionCall)
.filter((operation) => operation?.methodName === method)
.map((functionCallOperation) => ({
...functionCallOperation,
args: base64DecodeFunction(functionCallOperation.args),
Copy link
Collaborator

@darunrs darunrs Nov 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be possible for the decode to fail depending on the structure of args. I vaguely recall a block which failed there. I would wrap this in a try catch and return an empty result. The user is not able to control the contents of a block or this function code. So, if it throws errors, they need to be aware that some blocks will crash their code and stop their indexer if they don't handle the error. Or we can just handle it, which I feel is the better option.

})),
);
};

return {
graphql: async (operation, variables) => {
console.log(`${functionName}: Running context graphql`, operation);
Expand All @@ -164,7 +200,25 @@ export default class Indexer {
fetchFromSocialApi: async (path, options) => {
return await this.deps.fetch(`https://api.near.social${path}`, options);
},
db: this.buildDatabaseContext(account, schemaName, schema, blockHeight)
db: this.buildDatabaseContext(account, schemaName, schema, blockHeight),
base64Decode: base64DecodeFunction,
getFunctionCalls: getFunctionCallsFunction,
getSocialOperations: (block, operation) => {
Copy link
Collaborator

@darunrs darunrs Nov 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fortunately, I do actually have a test block for this piece of code. The block 100505912 has social posts with malformed data. As a result, the below filter and map functions will actually fail, throwing an uncaught error that will stop your indexer. You can run that block against the indexer you made and see that it does in fact fail. The fix I did was to simply wrap the calls in a try catch. You can see the fix here: https://github.com/near/near-discovery-components/blob/develop/indexers/social_feed/social_feed.js#L271-L283

const contract = 'social.near';
const method = 'set';
return getFunctionCallsFunction(block, contract, method)
Comment on lines +207 to +209
Copy link
Collaborator

@darunrs darunrs Nov 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can't we simply put the contract and method in the function call, since they're constants anyway? Otherwise, if we want to keep the variables to declare the constants, we should place them as a class level constant with the names SOCIAL_CONTRACT and SET_METHOD. Or at least rename the two you have here to that.

.filter((functionCall) => {
const accountId = Object.keys(functionCall.args.data)[0];
return functionCall.args.data[accountId][operation];
})
.map((functionCall) => {
const accountId = Object.keys(functionCall.args.data)[0];
return {
accountId,
data: functionCall.args.data[accountId][operation],
};
});
},
};
}

Expand Down