Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dApp- Gamefied price forecasting + NEO advanced analytics dashboard (with 3D visualizations) #46

Closed
grantshares-dapp bot opened this issue Dec 1, 2022 · 61 comments
Labels
request-for-funding GrantShares: Request for Funding

Comments

@grantshares-dapp
Copy link

grantshares-dapp bot commented Dec 1, 2022

Abstract

For the sake of sharing advanced forecasting and data analysis techniques amongst the NEO community, increasing NEO blockchain usage, and increasing familiarity amongst community members, develop a web-based dApp/game that allows entrants to make predictions on the price of NEO at a future time. Payouts/ prizes would be 100% given back to the community. A complimentary analytics dashboard would be developed which features additional tooling (3D rendering of global node locations with ThreeJS, regression/ seasonal decomposition, daily transaction history, etc.) and gives users a snapshot of where the price is headed.

Proposal Information

Description

How the game works:

At the start of each month, after connecting their wallets, users can place a bet on what the actual price of an asset will be (for now NEO/ GAS) will be at the end of the month (or some future date). As the purpose of this dApp is primarily educational/recreational, buy-in amounts would be low and capped at around 1-3 gas per user (for now).

Note that the target asset does not necessarily have to be NEO, we can also consider related assets such as the price of Bitcoin, ETH, for the future.

All bets will be automatically compiled into a neo-python or neow4j smart contract. The cut-off point for new entries would be about 2 weeks before month- end. At month end, whichever user’s price is closest to the actual price will win the prize pool. The payout structure will be 100% of all submissions received and will be as follows: 1st place: 70%, 2nd place: 20%, 3rd place: 10%.

The backbone of the dApp will be a smart contract that will automatically execute after the prediction date; I will not be able to touch the pot or subsequent dissemination as it will be handled automatically.

Again, the primary purpose of this game is for educational and recreational purposes. It will repeat on a continual, monthly basis and I plan to run it for perpetuity while supporting its dissemination with articles about data science/ forecasting/ economics.

Dashboard description:

The N3 ecosystem already has a small number of dashboards available. They include: neo-dashboard, ndapp, Dora, neodepot, and OneGate. They are quite useful for querying historical information on the NEO blockchain.

Rather than recreate what other dashboards have already done, my dashboard will focus more on analytics/ forecasting/ non-traditional things users may not normally consider. For example, it will feature:

  • Use of ThreeJS to create a 3D rendering of global location of NEO's nodes
  • Prominent use of seasonal decomposition (i.e. long-term cycle forecasting) to help the community visualize where the price may be going based on previous cyclical patterns
  • Use of multiple linear regression to see what basket of assets the price of NEO is correlated to
  • Creation and installation of a neural network to make forecasts for NEO based on estimates of prices of aforementioned correlated assets
  • Chart and linear regression of NEO/GAS price to NEO transactions per day/GAS burned per day (currently getting some data assistance from kokahunter on this)

Motivation

Prior to coding, I worked as a CPA with a Big 4 firm and then as a principal research analyst in the energy industry for a number of years. As part of our responsibilities, we had to come up with predictions for a number of important indicators, including oil prices, as well as demand for certain oil extraction equipment. I found that oftentimes, our official oil price forecasts were way off. I wanted to know why, and how our forecasts could improve in accuracy and timing. Thus, I became fascinated with the skill of forecasting- what are the drivers, and how does one know if one has included all of the necessary factors? What are the common errors being made by those in the field? I’ve studied many of the great forecasters, and came to realize that you can count on one hand the number that are consistently accurate.

As the NEO community grows, it is important that its members have a good grasp of cutting edge forecasting and data analytics skills. The gamifying of the forecasting process using very small bets could be a way for NEO users to better understand how forecasting/ data science works and improve their trading and investing skills, while also increasing interaction amongst members. Lastly, developing this dApp will allow me to build my own skill-set in NEO smart contract development.

Goals

  1. Enhance the understanding and real-world usage of data analytics (forecasting) and data science techniques amongst the NEO community.

  2. Drive increased usage of NEO/ GAS

  3. Increase participation / community engagement amongst NEO members

  4. Have fun while doing all the above

Deliverables & Roadmap

Specify deliverables in detail, including the following info for each:

  • Total Estimated Duration: 30-40 days (deliver within 1.5-2 months)
  • Full-Time Equivalent (FTE): 1
  • Total Budget: $3200 or 485 NEO

Roadmap:

Backend (neo-python, Python, GraphQL, WalletConnect)

  • Smart contract which takes all user bets and holds them in escrow. Then within 6 hours
    after prediction date is passed, automatically release 100% of prize pool to participants, in accordance with waterfall payout structure described above. (8 days)
    - Back-end logic tied to live data updates from Pandas Datareader
    - Smart contract to be coded with neo-python or neow4j
  • Build an additional SQL/GraphQL database to handle metadata such as user profile, competition history, winners history, etc.  (3 days)

Frontend (ReactJS/MaterialUI/Render):

  • Eye-catching, beautiful homepage and 3D visualization dashboard (15 days)
    • Live dashboard to feature long-term price history and data analysis tools (see recent example here))
    • 3D rendering of locations of global nodes on homepage
  • Scoreboard showing all entrants and bets placed (3 days)
  • History and feedback page showing winners of previous months’ competition, including feedback from winners on general techniques they used to make their forecasts (2 days)
  • WalletConnect/ Github integration (2 days)
  • About page (1 day)
  • Design a logo (.5 day)

Other

  • Deployment/ Final Configurations (3 days)
  • Marketing- publicize app on Reddit, Discord, Twitter, etc. (3 days)
    • Write NEO-focused/forecasting/data science articles to drive usage of the app
    • Integration with Discord?

Total working days (~35)

Deliverables Verifiability

1) dApp is successfully launched with features as described above

  • Includes functioning live dashboard featuring data analysis and forecasting tools

2) Ads and promotions are placed throughout Reddit, Discord, and Twitter to attract users

  • I will likely write a series of accompanying articles to explain how to use the dashboard tools and also to promote the dApp itself

3) First monthly competition is held successfully

  • Includes at least 10-13 paying participants.
  • 100% payouts are automatically and successfully made to winners via smart contract.
  • Winners and their techniques are listed on the history/feedback page

Budget Plan$

  • For one FTE and ~38-days of work- $3200 or 485 NEO
    • Note- total amount includes annual costs for domain name hosting ($10/yr), backend hosting on Render.com ($84/yr), as well as the use of Tiingo's financial data API, which will cost $120 per year (Yahoo Finance's free API is currently unavailable)

About You / Your Organization

  • Individual/Entity Name: Rob Liou
  • Website: https://robliou.github.io

Short-Bio

My name is Rob Liou. I started coding/ data science at my previous job about 3 years ago and have been hooked ever since. Previously, I worked for a Big 4 Accounting Firm as a CPA, and then after my MBA, at a large consulting firm as a principal research analyst in the upstream oil and gas industry, and later at IMA as a research manager/ data engineer. I have an MBA from the University of Michigan and did my undergraduate in economics, accounting, and programming at UCLA.

Portfolio of Projects / Past Experience

After participating in the NEO Frontier Hackathon in 2021 with DDR Store (an "Amazon Store for Market Intelligence"), I was invited to give a demo on the first day of DemoWeek. After the successful demonstration, my project, DDR Store, was later enrolled and launched via the NEO Ecoboost Incubator.

In addition, I have built projects to serve the NEO community in different ways, including:

NEO price regression engine

NEO cycle/ seasonality discovery engine

I've done other analysis of NEO on my personal blog that has been shared with the community to positive feedback.

The NEO analytics dashboard I plan to create would be based on a oil price forecasting engine I created during the Amazon AWS API hackathon held last month.


Proposal Info 📋

  • Proposal Type: request-for-funding

  • Amount Requested: 485

  • Token: 0xef4073a0f2b305a38ec4050e4d3d28bc40ea63f5 (NEO)

  • Receiver Address: NW3eqonPrRueuWi85cpHgqDrRnxP8Xzyta (0xb83c34adfaa1b364c79939142046b508d545256f)

  • Created by: @robliou 🚀

  • Raw Intents: 👀

[ {
  "target_contract" : "0x6276c1e3a68280bc6c9c00df755fb691be1162ef",
  "method" : "releaseTokens",
  "params" : [ {
    "type" : "Hash160",
    "value" : "ef4073a0f2b305a38ec4050e4d3d28bc40ea63f5"
  }, {
    "type" : "Hash160",
    "value" : "b83c34adfaa1b364c79939142046b508d545256f"
  }, {
    "type" : "Integer",
    "value" : "485"
  } ],
  "call_flags" : 15
} ]

👇 React with 👍 if you liked it, or 👎 if you think this proposal can be enhanced!

@grantshares-dapp grantshares-dapp bot added the request-for-funding GrantShares: Request for Funding label Dec 1, 2022
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

3 similar comments
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp grantshares-dapp bot changed the title dApp- gamefied monthly NEO price forecasting competition dApp- gamefied NEO price forecasting + analytics dashboard Dec 1, 2022
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@ioannistsil
Copy link

ioannistsil commented Dec 2, 2022

this is interesting . definitely NEO need this kind of dapp

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@robliou
Copy link

robliou commented Dec 3, 2022

this is interesting . definitely NEO need this kind of dapp

Thanks! Hope to hear more feedback on this project soon.

@DylanNNT
Copy link

DylanNNT commented Dec 6, 2022

I tested the initial application Rob had put together a few months ago, after he shared a link on the Neo sub-Reddit. You can see our back and forth here. It was a pretty cool tool to test NEO's price movement against other asset classes, and to play around with other assets to see if there might be any correlative relationships.

As a principal, when talking about about NEO and the ecosystem (especially so in NNT coverage) I've avoided discussing price, because it's so speculative and I don't want to fuel any hopium in readers or folks who follow my socials. However, I think with a tool like the one proposed in this GrantShares application, it could be a great education opportunity to learn about how assets interact with broader markets. Many in the community like to trade, and I think this could serve as an educational tool to foster more focussed conversations on that subject-matter. I certainly walked away with a little more knowledge after reading Rob's post and testing the initial version of the tool.

I appreciate that Rob took feedback offered a few months ago about a) making a game that allowed all ecosystem participants easily use and b) not take any fees for himself from folks who staked GAS/NEO in the game.

Plus, I think the ecosystem would generally benefit from more websites/pages dedicated to dashboards, as they're simple ways to digest various types of data and statistics about network usage and activity.

Should Rob be granted funding, I believe he'd be able to deliver the dApp. He's already built out an initial concept, and successfully delivered another dApp in the Neo Frontier Launchpad. Further, the request for funding isn't absurd, but also respects the time an applicant has to put in to make this project come to life. As GrantShares voting members, we should be supporting projects that have realistic funding requests and well-defined deliverables.

@robliou
Copy link

robliou commented Dec 7, 2022

Hi Dylan, muchas gracias for the positive and encouraging feedback!

I have learned, while being in the incubator, what causes a deFi project to be successful or not. In the deFi world, given peoples' short attention spans, it needs to be useful, simple, and easily accessible. You can rest assured I will do my best to make this project successful and attractive, meet the timelines for the deliverables, etc.

That said, if anyone has additional constructive criticism, feedback, or suggestions, etc., I'm all ears.

@robliou
Copy link

robliou commented Dec 10, 2022

As a principal, when talking about about NEO and the ecosystem (especially so in NNT coverage) I've avoided discussing price, because it's so speculative and I don't want to fuel any hopium in readers or folks who follow my socials.

Hey @DylanNNT , I think that's a fair point. Building an app that would appear to users to be speculating on the price of NEO could possibly be divergent from NEO's stated goals.

I was thinking that I could expand the analytics dashboard to also cover other important assets within the digital world, such as Bitcoin, Ethereum, etc.? For example, the dashboard could list the top coins/ assets which are currently out-of-favor versus in-favor. This would be done by deriving and visualizing the long-term cyclical/seasonal patterns of said coins, and then listing, say, the top 5 coins of each. This would assist an investor in discovering which coins may be on the verge of a break-out versus sell-off.

In addition, rather than making forecasts for the price of NEO, the game would instead allow users to forecast the price of other assets such BTC or ETH (or really any other crypto or asset class, for that matter). Of course, pay-ins/ payouts would still be done using GAS.

Thoughts?

@csmuller
Copy link
Member

csmuller commented Jan 4, 2023

I'm in favor of this proposal. It's scope and budget are reasonable and it is a playful addition to the ecosystem. I'm not interested in price predictions myself but i'm sure many other users are.

@robliou
Copy link

robliou commented Jan 4, 2023

Hi @csmuller ,

Thanks for your comments and vote of support!

The past few weeks, I've been using Reddit and Discord to gauge the community's needs and wants as it pertains to an advanced analytical dashboard. I'm also looking at what's already been done, as I have a few ideas as well about things. For sure, the dashboard can host some of the forecasting tooling I've already created, which would help to cut down on some of my hosting costs.

I'll update my submission in the next few days to include the results of this new research I've gathered, so that the game/dashboard is more cohesive and likely to succeed with the community.

Thanks again and have a happy new year,

@grantshares-dapp grantshares-dapp bot changed the title dApp- gamefied NEO price forecasting + analytics dashboard dApp- Gamefied price forecasting + NEO advanced analytics dashboard Jan 6, 2023
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

3 similar comments
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@robliou
Copy link

robliou commented Jan 6, 2023

@DylanNNT @csmuller @ioannistsil

Based on your (and the community's) input, I've made a few minor edits to this proposal. However, it is mostly the same as before, just with more detail and clearer objectives. Please have a look and let me know your thoughts, if any.

I plan to put it on-chain soon.

Thanks,

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

1 similar comment
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp grantshares-dapp bot changed the title dApp- Gamefied price forecasting + NEO advanced analytics dashboard dApp- Gamefied price forecasting + NEO advanced analytics dashboard (with 3D visualizations) Jan 7, 2023
@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@csmuller
Copy link
Member

csmuller commented Jan 9, 2023

Those are good edits IMO. I'm still in favor of endorsing the proposal.

@grantshares-dapp
Copy link
Author

🚨 This proposal was edited by the proposer.

@grantshares-dapp
Copy link
Author

⛓ This proposal was created on-chain! 🔥🚀🎉

➡️ Now, waiting for a GrantShares Member to endorse it... ⏰


🚨IMPORTANT🚨
Pay attention to the following deadlines:

  • Expiration (should be endorsed until): Feb 8, 2023 2:13:53 PM UTC

General info:

@robliou
Copy link

robliou commented Jan 18, 2023

Hi Everyone,

Just to add some more context to this discussion, I've gone ahead and built out a portion of the 3D analytics component of the dashboard. I posted the site on Reddit, and have already gotten some pretty good feedback. You can see the site and description here.

Please feel free to let me know your thoughts, or any additional suggestions. Thanks.

@grantshares-dapp
Copy link
Author

⛓ This proposal was endorsed on-chain! 📄🔑

➡️ Voting period starts NOW!


🚨IMPORTANT🚨
Pay attention to the following deadlines:

  • Voting period deadline: Jan 25, 2023 6:12:59 PM UTC
  • Timelock period deadline: Jan 28, 2023 6:12:59 PM UTC
  • Expiration (executable until): Feb 27, 2023 6:12:59 PM UTC

General info:

@gsmachado
Copy link
Member

Please, @shargon @lllwvlvwlll @JohndeVadoss @steven1227 @igormcoelho @deanragnarok @hacfox @roman-khimov cast a vote (either "accept", "reject", or "abstain") if you haven't done yet for this proposal.

@gsmachado
Copy link
Member

gsmachado commented Jan 25, 2023

@robliou Congrats! It seems the proposal was accepted! 🎉

You can now start the hard working part! 💪

Please, post here your progress and keep the community connected to you! 🙏

You'll be able to execute the proposal once the Time Lock phase is over. 🔜

@robliou
Copy link

robliou commented Jan 26, 2023

@gsmachado , thanks, this is great news, and thanks for your and @csmuller 's support! I am excited to get this moving forward into something useful.

An important follow-up question:

According to the Grantshares Parameters, there is an expiration time of "30 days after the proposal is ready for execution. In both cases, the proposal expires if no actions occur for the period defined by this parameter."

What does that mean, exactly? What are "actions" referring to?

In my proposal, I stated I need about 35-40 working days to complete this project. Looking at the other projects that were accepted, none seemed to 'expire' after 30 days of execution..? If I have to complete this in 30 days from now, that could be a bit challenging..

Thanks again,

@csmuller
Copy link
Member

@robliou, the "execution" term might be a bit confusing here. With execution we mean executing the proposal on-chain. You will need to do that once the time lock is over. Thereby, you will claim the funds that have been granted to you. If you don't do that within 30 days, the proposal will expire and you'll not be able to execute anymore. For the actual "project execution" you can stick to your estimated efforts.

@robliou
Copy link

robliou commented Jan 26, 2023

@csmuller , OK, thanks for the clarification.

I'll plan to keep everyone posted as this progresses. @DylanNNT will also help me with publicizing when the time is right.

By the way, I can also think of using this forum, as well as Reddit/Discord to share progress with the community. Are there any other channels I should consider using?

Thanks,

@csmuller
Copy link
Member

@robliou, Since you already have some traction on Reddit, you can def continue posting there.
It's just important that you also report your progress here sometimes such that the community know this grant is being worked on. But links to, e.g., NNT or Discord posts suffice 👍

@robliou
Copy link

robliou commented Jan 27, 2023

@csmuller , understand and make sense. Will definitely post here to show my progress to the community, as well as post to NNT/ Discord/ Reddit, etc. Thanks.

@robliou
Copy link

robliou commented Jan 28, 2023

@robliou, the "execution" term might be a bit confusing here. With execution we mean executing the proposal on-chain. You will need to do that once the time lock is over. Thereby, you will claim the funds that have been granted to you. If you don't do that within 30 days, the proposal will expire and you'll not be able to execute anymore. For the actual "project execution" you can stick to your estimated efforts.

For the good of the community, I've been thinking about this further. I can understand maybe 50% of the request being released after the lockout period, but 100%? Shouldn't there be some further incentive for the proposer to actually achieve what they proposed? (i.e. 50% released after lockout, then another 50% released after project is completed and accepted by committee members)? I understand this introduces a further round of review and effort for the committee (or community), but it just seems more appropriate than giving out everything at the start. I do understand that the proposal and acceptance phase is quite rigorous as well.

Anyways, I'll probably open a separate issue for this...

@grantshares-dapp
Copy link
Author

🎉 Outcome: this proposal was accepted! 🚀

  • Accept: 3
  • Reject: 2
  • Abstain: 1
  • Required quorum: 50%
  • Acceptance ratio: 50%

➡️ Now, waiting for someone to execute it... ⏰


General info:

  • On-Chain ID: 26

@grantshares-dapp
Copy link
Author

⛓ This proposal was executed on-chain! ✅ 💚


General info:

@csmuller
Copy link
Member

@robliou, Your suggestions about splitting a grant into a payout schedule is something we desire too. We already ran an internal workshop on such a "milestone" feature. Currently, execution is halted by the question about resources and funding.

@robliou
Copy link

robliou commented Jan 30, 2023

@csmuller

That's a good point. Also, as mentioned elsewhere, splitting up the grants already appears to be taking place. And I guess since many of these grants aren't for very large amounts, maybe it's not time-effective to nitpick on smaller details/ deliverables.

@robliou
Copy link

robliou commented Feb 20, 2023

Hi NEO friends,

Just an FYI- I've been working hard on the analytics dashboard and hope to have a first demo ready later this week. I will likely share it first with @DylanNNT to get his thoughts and input and then maybe drop a link here to a demo after the bugs are worked out. Looking forward to hearing your feedback on it.

Be in touch!

@csmuller @gsmachado @ioannistsil @ngd

@gsmachado
Copy link
Member

Hey @robliou
Any news about the progress of this project? Please, report your progress here in the discussion thread.
Thanks in advance!

@robliou
Copy link

robliou commented Mar 7, 2023

Hi @gsmachado @csmuller @steven1227

Yes, thank you for reaching out! Below is link to the demo and update I sent to @DylanNNT last week. It appears he was attending an important conference over the weekend, so hasn't been able to respond yet:

"I would like to share with you the pre-release of the NEO forecasting dashboard here, I can hope you can take a look and offer some feedback:

https://www.dashboard.robliou.com/

Some points to note:

  1. There are 4 charts displayed on the dashboard, but only the two on the right (Seasonality and Adjuster) are editable and affect the final price calculation.

  2. Seasonality allows the user to visualize the one-year, two-year, and three-year cycles which impact the price of NEO. It also includes the option to look at the cycles for Bitcoin, which is interesting because Bitcoin has a longer history and thus, more developed cycles.

The cycle you select will impact the final forecast (Note that this dashboard and game currently only forecasts the price of NEO to the end of the current month). Of the three periods, I would say the two-year cycle clearly has the highest balance of accuracy and impact on the price of NEO.

  1. The Regression tab shows 3 baskets of assets which have a high correlation to NEO. They are all nearly 80%, which is quite high. This means that, the prices of the three assets in each basket, when combined, can explain nearly 80% of the price of NEO.

  2. On the Adjuster tab, you can select which basket you want to use and then adjust the month-end price of each asset using the slider bar. Hit 'Generate' after you are done making your inputs.

  3. After clicking the 'Generate' button, a special algorithm is launched which calculates the forecasted price of NEO at month-end.

As we know that Observed values = Seasonal + Trend + Residual, given that we already know the Seasonal variable, I attempted to derive the Trend and Residual variables by converting them using the predicted values from the regression basket.

Combining these factors has proved to be a bit more difficult than I expected, involving the use of special equations, curved lines, etc. It seems to be more useful than just using the Regression variables alone, however, the Seasonal impact can sometimes be too strong, especially during the down cycles (i.e. forecasted price of NEO drops substantially). Therefore, I am still working on getting these variables to play nicely together.

In the meantime, I think the dashboard still has special value due to the ability to quickly and easily make a fairly accurate forecast just by adjusting the 3 variables in the Regression basket. Also, the Seasonal visualizations are quite useful. For example, if anyone is wondering why the price of NEO has doubled since December, they can simply look at the one-year and two-year Seasonal cycles to see that the price of NEO always rises around this time of year. Secondly, they also give a strong indication how much longer this cycle will continue for (i.e. the two-year cycle indicates this bull run should last into May).

  1. I will continue to sort out some of the kinks in the dashboard, as well as to find ways to improve the final algorithm. I will also buy a domain name to register this app. At this point, I will also move onto developing the game and smart-contract that this dashboard is meant to support. In the meantime, if you have any ideas, comments or suggestions, or know of anyone else that would like to work with me on this, please feel free let me know!! "

@robliou
Copy link

robliou commented Mar 7, 2023

Further to the app, I am now working on the smart contract that supports the forecasting game. The dashboard itself is not perfect, and will not generate a perfectly accurate forecast, but hopefully it helps the user to consider some things he/she may not normally consider, while giving him/her confidence to participate in the forecasting game.

Also, I've gotten some good feedback from kokahunter on the dashboard and game itself, as well as a volunteer to help with the SC. In the meantime, if you guys have any feedback or additional ideas for the dashboard/game, that would be wonderful to hear and I'd be all ears.

@gsmachado
Copy link
Member

@robliou thanks for the update!

@csmuller
Copy link
Member

csmuller commented Mar 9, 2023

Looking good :)

@robliou
Copy link

robliou commented Mar 28, 2023

Hello guys,

I'm having issue compiling and deploying my smart contract due to type-related errors.

Specifically, I'm trying to understand what is the point of using Byte prefixes in the smart contract? For example, in the sample NEP17 Boa contract, there are these byte-related initializing variables that are set at the top of the contract:

PERSON_B: bytes = b'person b'
ADDRESS_PREFIX: bytes = b'address'
NOT_INITIALIZED: bytes = b'not initialized'
FUNDED_PREFIX: bytes = b'funded'
AMOUNT_PREFIX: bytes = b'amount'
TOKEN_PREFIX: bytes = b'token'

I'm having trouble understanding why are these byte prefixes needed? Are they required for certain variables in a NEO smart contract? If so, which cases (i.e. addresses)?

For example, to use storage.put, this signature is used:
storage.put(b'X' + from_address.to_bytes(), amount)

I've looked through various tutorials online but can't seem to find a good answer to this question. I've also asked in Discord but didn't get a clear answer.

Thanks,

@gsmachado @csmuller

@EdgeDLT
Copy link

EdgeDLT commented Mar 28, 2023

I can't say if it's the only reason, but I can give you at least one justification for it. Using prefixes for dynamically created storage keys gives predictability. That predictability tends to be convenient.

Say for example, I want to look into the storage of a NEP-17 token and view all user balances at some point in time. Let's say the NEP-17 token in question stores balances like this:

BALANCE_PREFIX = b'\x01'

storage.put(BALANCE_PREFIX + from_address.to_bytes(), amount)

Now if I pull the storage records over RPC, I can select all the records prefixed with AQ (they are base64 encoded), drop the prefixes, and have myself a full list of addresses and balances in an instant.

If you did this on the other hand:

storage.put(from_address.to_bytes(), amount)

While that also works for your contract internally, you have made my life much more difficult. Now when I grab the key-values from your contract's storage, I have no idea which keys correlate to balance values and which are irrelevant to me. I'll have to iterate through each entry and check if it is an address.

Two other things to note. First up, putting data into storage costs GAS. More data costs more. In short, shorter key prefixes are better. Secondly, I don't see why a NEP-17 contract would require as many prefixes as you have listed above, those look like they belong in a swap contract to me.

Just to add on to that last one, think about what kind of data your contract stores about a given address. A token contract only stores a balance for each address, that's all it needs to handle transfers. So you could get away with just using the address as a key (even though I'd advise against it, as described above). But if you needed to store say, an auction ID, a holding balance, and a reputation value for each user in a marketplace contract, you can't store them all under that key (unless you wanted to use objects of course, but that would probably be inefficient depending on the use case). So a prefix in that case is a convenient way to store those different values independently while still using the address to derive the key.

@robliou
Copy link

robliou commented Mar 29, 2023

Thanks for the explanation, @EdgeDLT .

So essentially the point of prefixes are to help us to classify, sort, and retrieve data that is put into storage, correct?

As for the Byte class itself, are they a necessity when working with types like addresses? What doesn't string suffice for getting/ storing addresses?

For example, given the method below to return an address; we initialize the from_address with a type of UInt160. However, when retrieving it from storage, we need to first convert it to bytes and then convert it to int:

    return storage.get(from_address.to_bytes()).to_int() 

Why the double conversion?

Similarly, when putting from_address into storage,


    storage.put(from_address.to_bytes(), amount)

Even though it's initialized as type UInt160, we have to convert it to bytes before storing it.

Any idea why this is?

Thanks,

@EdgeDLT
Copy link

EdgeDLT commented Mar 29, 2023

So essentially the point of prefixes are to help us to classify, sort, and retrieve data that is put into storage, correct?

It's one reason for it, I can't say if it's the only reason because I don't know what I don't know.

As for the Byte class itself, are they a necessity when working with types like addresses? What doesn't string suffice for getting/ storing addresses?

I think you're talking about something mostly specific to Boa here really, you'll get the best answers for that on Discord or the Boa Github.

Again, I don't know the "full" answer to your question, but I'll make an observation based on my admittedly limited knowledge. As far as NeoVM is concerned, everything you store is bytes. Whether those bytes resolve to a string, an integer, or some other type is a matter of interpretation. You convert a retrieved value to the type that is useful and correct for your use case, because raw bytes are not usually helpful.

Why the double conversion?

I think there are a couple of misunderstandings here.

First, I don't believe it's actually necessary for you to create a key with from_address.to_bytes(). from_address alone should work just fine (though ideally with an added prefix).

Secondly:

 storage.put(from_address, amount)

Does not put from_address into storage. It puts amount value into storage under key from_address. When you retrieve amount with

storage.get(from_address)

You get bytes back. Since it's an amount, it's probably an integer you are looking for. So actually the line should actually be:

storage.get(from_address).to_int()

So we are not double converting anything. We are retrieving the amount value's bytes, stored under the key from_address, and then interpreting those bytes as an integer with to_int().

@cschuchardt88
Copy link

@EdgeDLT The reason it works like this is because Neo uses a Key-Value Store Database to store all their data. And the type for this is as follows

put(byteArray key, bytesattray value)
get(byteArray key, bytesattray value)
seek(byteArray KeyOrPrefix)

@robliou
Copy link

robliou commented Apr 19, 2023

Hey guys,

Just an update from my end. I posted this forecast on reddit the other day (based on the dashboard portion of the project), and it's gotten some positive feedback so far.

As for the game portion of the project, I'm still struggling with creating the smart contract. Luckily, I'm getting some assistance from another user in the community on this (kokahunter).

Cheers,

@csmuller
Copy link
Member

Thanks for the update @robliou and good post on Reddit 💪

@robliou
Copy link

robliou commented Jun 18, 2023

Hey guys! @gsmachado @csmuller @DylanNNT

It's been awhile! Just wanted to give an update here on our project.

So, I started a new job last month, and this has significantly cut into my time to work on this project. But, I am still determined to complete and execute it well. I also got a lot of help from the community, especially from kokahunter who has been outstanding for the smart contract (would not have been able to get this far without his help).

You can see a preliminary demo of our forecasting app/ game here. This is the figma for how the backend logic works. Basically, the most difficult parts of the app are all completed. Namely, which is that NFT's for each season can easily be minted, the smart contract NFT can take in transfers/ metadata from the front-end successfully, the front-end can retrieve all transfers/ metadata from the NFT and display them, and a winner can be selected from the NFT at season's end with all assets in the NFT transferred to him (in addition to the analytics dashboard that was completed in April).

There is still a number of design/ small issues we need to clear up before deploying to the community. We also need to come up with a marketing/ promotional plan (seasons will likely last only a week or so as opposed to a month). But, hopefully we can complete them in the next few weeks. We are also open to any suggestions you may have.

In the meantime, we have a question: do our smart contract methods need to mirror those of the NEP-11 standard exactly (i.e. use the exact same method name, and fulfill all of the standard's recommendations), or can we take/use from the NEP-11 standard as we see fit while adding/ subtracting things we believe are more relevant to our use case?

As an example, when trying to retrieve tokens from storage, this is the standard NEP-11 method:

public static UInt160 OwnerOf(ByteString tokenId)
{
    StorageMap tokenMap = new(Storage.CurrentContext, Prefix_Token);
    TokenState token = (TokenState)StdLib.Deserialize(tokenMap[tokenId]);
    return token.Owner;
}

Whereas, this is what our custom implementation looks like:

  public static BigInteger RecordCountOf(ByteString name)
  {
      if (name is null) throw new Exception("The argument \"name\" is invalid.");
      StorageMap recordCountMap = new(Storage.CurrentContext, Prefix_RecordCount);
      return (BigInteger)recordCountMap[GetKey(name)];
  }

Is it OK to deploy with the latter method, or do we have to use the former?

Thanks,

@csmuller
Copy link
Member

Hey @robliou, Thanks for the update. Great work so far!
About the NEP-11 standard: You contract does not have to mirror the NEP-11 standard even if you call the tokens produced by it "NFT". But, if you adhere to the standard it makes your contract or NFTs accessible to any other project that works with the standard. If you don't adhere to it, your NFTs will not be consumable by other software that work with the NEP-11 standard. So, if you don't need your NFTs to be interoperable with other projects than it's fine to deviate from the standard.

@roman-khimov
Copy link

I'd say if you're doing some kind of NFT, you better stick to NEP-11 unless you have some very good reasons to deviate. NEP-11 compliance is something expected of a Neo-based NFT, once you're NEP-11 all the other tooling around (like explorers or wallets) can easily work with your contract/tokens via this standard interface, it's a very powerful integration. And it should be pretty easy to do, take NNS contract for example, it handles domain names which have a lot of specifics, yet at the same time any domain is a proper NEP-11 NFT.

@robliou
Copy link

robliou commented Jun 20, 2023

@csmuller @roman-khimov

Appreciate your inputs on this topic! We have discussed internally, and will plan to adhere to the NEP-11 standard as much as possible. Shouldn't take too much time from our end.

Also, we hope to push out a promotional version of this app in the next week or so. Hope you can help support us by trying out and partcipating in the first season!

Thanks,

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
request-for-funding GrantShares: Request for Funding
Projects
None yet
Development

No branches or pull requests

8 participants