Table of Contents
- π Get Started
- π¦ Environment variables
- π¦ Data Sources
- π Web3 Integration
- π©βπ€ Storybook
- π€ Testing
- β¨ Code Style
- π³ Production
- β¬οΈ Deployment
- π Contributing
- π΄ Forking
- π° Pricing Options
- β GDPR Compliance
- π License
The app is a React app built with Next.js + TypeScript + CSS modules and will connect to Ocean remote components by default.
To start local development:
git clone [email protected]:oceanprotocol/market.git
cd market
# when using nvm to manage Node.js versions
nvm use
npm install
# in case of dependency errors, rather use:
# npm install --legacy-peer-deps
npm start
This will start the development server under
http://localhost:8000
.
If you prefer to connect to locally running components instead of remote connections, you can spin up barge
and use a local Ganache network in another terminal before running npm start
:
git clone [email protected]:oceanprotocol/barge.git
cd barge
# startup with local Ganache node
./start_ocean.sh
Barge will deploy contracts to the local Ganache node which will take some time. At the end the compiled artifacts need to be copied over to this project into node_modules/@oceanprotocol/contracts/artifacts
. This script will do that for you:
./scripts/copy-contracts.sh
Finally, set environment variables to use this local connection in .env
in the app:
# modify env variables
cp .env.example .env
npm start
To use the app together with MetaMask, importing one of the accounts auto-generated by the Ganache container is the easiest way to have test ETH available. All of them have 100 ETH by default. Upon start, the ocean_ganache_1
container will print out the private keys of multiple accounts in its logs. Pick one of them and import into MetaMask.
To fully test all The Graph integrations, you have to run your own local Graph node with our ocean-subgraph
deployed to it. Barge does not include a local subgraph so by default, the subgraphUri
is hardcoded to the Rinkeby subgraph in our getDevelopmentConfig
function.
Cleaning all Docker images so they are fetched freshly is often a good idea to make sure no issues are caused by old or stale images:
docker system prune --all --volumes
The app.config.js
file is setup to prioritize environment variables for setting each Ocean component endpoint. By setting environment variables, you can easily switch between Ocean networks the app connects to, without directly modifying app.config.js
.
For local development, you can use a .env
file:
# modify env variables, Rinkeby is enabled by default when using those files
cp .env.example .env
All displayed data in the app is presented around the concept of one asset, which is a combination of:
- metadata about an asset
- the actual asset file
- the NFT which represents the asset
- the datatokens representing access rights to the asset file
- financial data connected to these datatokens, either a fixed rate exchange contract or a dispenser for free assets
- calculations and conversions based on financial data
- metadata about publisher accounts
All this data then comes from multiple sources:
All initial assets and their metadata (DDO) is retrieved client-side on run-time from the Aquarius instance, defined in app.config.js
. All app calls to Aquarius are done with 2 internal methods which mimic the same methods in ocean.js, but allow us:
- to cancel requests when components get unmounted in combination with axios
- hit Aquarius as early as possible without relying on any ocean.js initialization
Aquarius runs Elasticsearch under the hood so its stored metadata can be queried with Elasticsearch queries like so:
import { QueryResult } from '@oceanprotocol/lib/dist/node/metadatacache/MetadataCache'
import { queryMetadata } from '@utils/aquarius'
const queryLatest = {
query: {
// https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
query_string: { query: `-isInPurgatory:true` }
},
sort: { created: 'desc' }
}
function Component() {
const { appConfig } = useMarketMetadata()
const [result, setResult] = useState<QueryResult>()
useEffect(() => {
if (!appConfig.metadataCacheUri) return
const source = axios.CancelToken.source()
async function init() {
const result = await queryMetadata(query, source.token)
setResult(result)
}
init()
return () => {
source.cancel()
}
}, [appConfig.metadataCacheUri, query])
return <div>{result}</div>
}
For components within a single asset view the useAsset()
hook can be used, which in the background gets the respective metadata from Aquarius.
import { useAsset } from '@context/Asset'
function Component() {
const { ddo } = useAsset()
return <div>{ddo}</div>
}
Most financial data in the market is retrieved with GraphQL from our own subgraph, rendered on top of the initial data coming from Aquarius.
The app has Urql Client setup to query the respective subgraph based on network. In any component this client can be used like so:
import { gql, useQuery } from 'urql'
const query = gql`
query TopSalesQuery {
users(first: 20, orderBy: totalSales, orderDirection: desc) {
id
totalSales
}
}
`
function Component() {
const { data } = useQuery(query, {}, pollInterval: 5000 })
return <div>{data}</div>
}
Publishers can create a profile on 3Box Hub and when found, it will be displayed in the app.
For this our own 3box-proxy is used, within the app the utility method get3BoxProfile()
can be used to get all info:
import get3BoxProfile from '@utils/profile'
function Component() {
const [profile, setProfile] = useState<Profile>()
useEffect(() => {
if (!account) return
const source = axios.CancelToken.source()
async function get3Box() {
const profile = await get3BoxProfile(account, source.token)
if (!profile) return
setProfile(profile)
}
get3Box()
return () => {
source.cancel()
}
}, [account])
return (
<div>
{profile.emoji} {profile.name}
</div>
)
}
Based on list-purgatory some assets get additional data. Within most components this can be done with the internal useAsset()
hook which fetches data from the market-purgatory endpoint in the background.
For asset purgatory:
import { useAsset } from '@context/Asset'
function Component() {
const { isInPurgatory, purgatoryData } = useAsset()
return isInPurgatory ? <div>{purgatoryData.reason}</div> : null
}
For account purgatory:
import { useWeb3 } from '@context/Web3'
import { useAccountPurgatory } from '@hooks/useAccountPurgatory'
function Component() {
const { accountId } = useWeb3()
const { isInPurgatory, purgatoryData } = useAccountPurgatory(accountId)
return isInPurgatory ? <div>{purgatoryData.reason}</div> : null
}
All displayed chain & network metadata is retrieved from https://chainid.network
on build time and integrated into NEXT's GraphQL layer. This data source is a community-maintained GitHub repository under ethereum-lists/chains.
Within components this metadata can be queried for under allNetworksMetadataJson
. The useWeb3()
hook does this in the background to expose the final networkDisplayName
for use in components:
export default function NetworkName(): ReactElement {
const { networkId, isTestnet } = useWeb3()
const { networksList } = useNetworkMetadata()
const networkData = getNetworkDataById(networksList, networkId)
const networkName = getNetworkDisplayName(networkData, networkId)
return (
<>
{networkName} {isTestnet && `(Test)`}
</>
)
}
Web3 Integration is a critical component of Ocean Marketplace, enabling seamless interaction with blockchain technology and smart contracts. It facilitates functions like asset creation, encryption, publishing, and metadata management while ensuring secure and transparent data transactions on the Ocean Protocol network.
The PublishPage
component in this project is a key part of the data publishing workflow, and it heavily relies on the integration with the Web3 ecosystem. Below, we'll explore the functions and processes involved in this component:
The createAlbum
function is responsible for creating NFTs (Non-Fungible Tokens) and datatokens, as well as defining the pricing schema for the published data.
async function createAlbum(values: FormPublishData): Promise<{
erc721Address: string
datatokenAddress: string
}> {
setFeedback((prevState) => ({
...prevState,
'1': {
...prevState['1'],
status: 'active',
errorMessage: null
}
}))
try {
const config = getOceanConfig(chainId)
LoggerInstance.log('[publish] using config: ', config)
const { erc721Address, datatokenAddress, txHash } =
await createTokensAndPricing(
values,
accountId,
config,
nftFactory,
web3
)
const isSuccess = Boolean(erc721Address && txHash && datatokenAddress)
if (!isSuccess) throw new Error('No Token created. Please try again.')
LoggerInstance.log('[publish] createTokensAndPricing tx', txHash)
LoggerInstance.log('[publish] erc721Address', erc721Address)
LoggerInstance.log('[publish] datatokenAddress', datatokenAddress)
setFeedback((prevState) => ({
...prevState,
'1': {
...prevState['1'],
status: 'success',
txHash
}
}))
return { erc721Address, datatokenAddress }
} catch (error) {
LoggerInstance.error('[publish] error', error.message)
if (error.message.length > 65) {
error.message = 'No Token created. Please try again.'
}
setFeedback((prevState) => ({
...prevState,
'1': {
...prevState['1'],
status: 'error',
errorMessage: error.message
}
}))
}
}
Here's what it does:
- It uses the provided form data (
values
) to create NFTs and datatokens. - This function communicates with the Ocean Protocol contracts to create these tokens.
- If successful, it updates the state with the created ERC721 (NFT) and datatoken addresses.
The encrypt
function takes the form data, along with the ERC721 (NFT) and datatoken addresses, and encrypts the DDO (Decentralized Data Object).
async function encrypt(
values: FormPublishData,
erc721Address: string,
datatokenAddress: string
): Promise<{ ddo: DDO; ddoEncrypted: string }> {
setFeedback((prevState) => ({
...prevState,
'2': {
...prevState['2'],
status: 'active',
errorMessage: null
}
}))
try {
if (!datatokenAddress || !erc721Address)
throw new Error('No NFT or Datatoken received. Please try again.')
const ddo = await transformPublishFormToDdo(
values,
newCancelToken(),
datatokenAddress,
erc721Address,
albumPrice
)
if (!ddo) throw new Error('No DDO received. Please try again.')
setDdo(ddo)
LoggerInstance.log('[publish] Got new DDO', ddo)
const ddoEncrypted = await ProviderInstance.encrypt(
ddo,
values.services[0].providerUrl.url,
newAbortController()
)
if (!ddoEncrypted)
throw new Error('No encrypted DDO received. Please try again.')
setDdoEncrypted(ddoEncrypted)
LoggerInstance.log('[publish] Got encrypted DDO', ddoEncrypted)
setFeedback((prevState) => ({
...prevState,
'2': {
...prevState['2'],
status: 'success'
}
}))
return { ddo, ddoEncrypted }
} catch (error) {
LoggerInstance.error('[publish] error', error.message)
setFeedback((prevState) => ({
...prevState,
'2': {
...prevState['2'],
status: 'error',
errorMessage: error.message
}
}))
}
}
Here's what it does:
- It transforms the form data into a DDO using the provided addresses.
- The DDO is then encrypted using the Ocean Protocol's
ProviderInstance
. - The encrypted DDO is updated in the component's state.
The publish
function is responsible for writing the DDO and associated metadata into the NFT metadata.
async function publish(
values: FormPublishData,
ddo: DDO,
ddoEncrypted: string
): Promise<{ did: string }> {
setFeedback((prevState) => ({
...prevState,
'3': {
...prevState['3'],
status: 'active',
errorMessage: null
}
}))
try {
if (!ddo || !ddoEncrypted)
throw new Error('No DDO received. Please try again.')
const res = await setNFTMetadataAndTokenURI(
ddo,
accountId,
web3,
values.metadata.nft,
newAbortController()
)
if (!res?.transactionHash)
throw new Error(
'Metadata could not be written into the NFT. Please try again.'
)
LoggerInstance.log('[publish] setMetadata result', res)
setFeedback((prevState) => ({
...prevState,
'3': {
...prevState['3'],
status: res ? 'success' : 'error',
txHash: res?.transactionHash
}
}))
return { did: ddo.id }
} catch (error) {
LoggerInstance.error('[publish] error', error.message)
setFeedback((prevState) => ({
...prevState,
'3': {
...prevState['3'],
status: 'error',
errorMessage: error.message
}
}))
}
}
Here's what it does:
- It takes the form data, the DDO, and the encrypted DDO as input.
- The function writes the metadata into the NFT, ensuring that it is associated with the published data.
- If successful, it returns the DID (Decentralized Identifier) of the published data.
The setNewMetadata
function is responsible for updating the metadata of songNFTs that are part of the album.
async function setNewMetadata(
values: FormPublishData,
albumDid: string,
albumName: string
) {
setFeedback((prevState) => ({
...prevState,
'4': {
...prevState['4'],
status: 'active',
errorMessage: null
}
}))
const txs: Promise<TransactionReceipt>[] = []
for (let i = 0; i < values.metadata.songs.length; i++) {
const { did } = values.metadata.songs[i]
const tx = updateMetadata(did, albumDid, albumName)
txs.push(tx)
}
await Promise.all(txs)
.then((txs) => {
console.log('txs', txs)
})
.then((txs) => {
setFeedback((prevState) => ({
...prevState,
'4': {
...prevState['4'],
status: 'success',
txs
}
}))
})
.catch((error) => {
LoggerInstance.error('[publish] error', error.message)
setFeedback((prevState) => ({
...prevState,
'4': {
...prevState['4'],
status: 'error',
errorMessage: error.message
}
}))
})
}
Here's what it does:
- It iterates through the list of songs in the form data and updates their metadata.
- The function returns a list of transaction receipts for the updates.
The handleSubmit
function orchestrates the entire publishing process. It ensures that each step (createAlbum, encrypt, publish, setNewMetadata) is executed in sequence.
async function handleSubmit(values: FormPublishData) {
try {
const [isOwned, song] = await hasSongsPermission(selectedSongs)
if (!isOwned) {
throw new Error(`You don't own the song ${song}`)
}
let _erc721Address = erc721Address
let _datatokenAddress = datatokenAddress
let _ddo = ddo
let _ddoEncrypted = ddoEncrypted
let _did = did
if (!_erc721Address || !_datatokenAddress) {
const { erc721Address, datatokenAddress } = await createAlbum(values)
_erc721Address = erc721Address
_datatokenAddress = datatokenAddress
setErc721Address(erc721Address)
setDatatokenAddress(datatokenAddress)
}
if (!_ddo || !_ddoEncrypted) {
const { ddo, ddoEncrypted } = await encrypt(
values,
_erc721Address,
_datatokenAddress
)
_ddo = ddo
_ddoEncrypted = ddoEncrypted
setDdo(ddo)
setDdoEncrypted(ddoEncrypted)
}
if (!_did) {
const { did } = await publish(values, _ddo, _ddoEncrypted)
_did = did
}
await setNewMetadata(values, _did, _ddo.metadata.name)
setDid(_did)
} catch (error) {
console.log(error)
}
}
Here's what it does:
- If the ERC721 (NFT) and datatoken addresses are not available, it calls the
createAlbum
function to create these tokens. - If the DDO or encrypted DDO is missing, it calls the
encrypt
function to generate and encrypt the DDO. - Then, it calls the
publish
function to write the DDO into the NFT metadata. - Finally, it calls the
setNewMetadata
function to update the metadata of songNFTs.
These functions collectively enable the seamless publishing of data assets on the Ocean Protocol network, integrating with Web3 technologies.
Note: This explanation provides a high-level overview of the functions. Detailed implementation and code can be found in the PublishPage
component in this repository.
The BuyAlbum
function represents a React component responsible for handling the purchase of an album containing multiple data assets on the Ocean Protocol network. This component integrates with the Web3 ecosystem and utilizes various utility functions to facilitate the purchase process.
async function handleSubmit(e: FormEvent) {
e.preventDefault()
console.log('clicked buy album')
try {
const orders = []
for (let i = 0; i < dids.length; i++) {
const asset = await retrieveAsset(dids[i], newCancelToken())
const accessDetails = await getAccessDetails(
asset.chainId,
asset.services[0].datatokenAddress,
asset.services[0].timeout,
accountId
)
const fullAsset = { ...asset, accessDetails }
const orderPriceAndFees = await getOrderPriceAndFees(
fullAsset,
ZERO_ADDRESS
)
orders.push({
asset: fullAsset,
orderPriceAndFees,
hasDatatoken: true
})
}
const res = await startMultipleOrders(
web3,
nftFactory,
chainId,
accountId,
orders
)
if (!res?.transactionHash)
throw new Error('MultiOrders failed. Please try again.')
console.log('Succesfull purchase!', res)
} catch (error) {
console.log(error)
}
}
Here's what it does:
-
It accesses essential Web3-related context variables such as
accountId
,web3
, andchainId
using theuseWeb3
hook. -
The function defines an array of
dids
(Decentralized Identifiers) representing the data assets to be purchased. These DIDs are essential for identifying the assets. -
Within the
handleSubmit
function, which is invoked when a form is submitted, the following steps are performed:-
It prevents the default form submission behavior and logs a message to indicate that the "Buy Album" button has been clicked.
-
It initializes an empty
orders
array, which will be used to store order details for each data asset in the album. -
It iterates through the array of
dids
and performs the following actions for each data asset:-
Retrieves the asset information by calling the
retrieveAsset
function, providing the DID and a cancel token to potentially abort the request. -
Obtains access details for the asset, including the datatoken address and access timeout, by invoking the
getAccessDetails
function, passing the asset's information and the user'saccountId
. -
Combines the retrieved asset information and access details into a
fullAsset
object. -
Calculates the order price and fees for the asset using the
getOrderPriceAndFees
function, with theZERO_ADDRESS
as the recipient address. -
Pushes an object representing the order, including the
fullAsset
, order price and fees, and a flag indicating that the asset has a datatoken, into theorders
array.
-
-
After processing all data assets, it invokes the
startMultipleOrders
function, passing theweb3
,nftFactory
,chainId
,accountId
, and theorders
array. -
If the
startMultipleOrders
function is successful and returns a transaction hash (res?.transactionHash
), it logs a success message indicating that the purchase was successful. -
In case of any errors during the purchase process, it logs an error message to help identify and troubleshoot any issues.
-
The BuyAlbum
component serves as a crucial part of the application's user interface, allowing users to initiate the purchase of an album containing multiple data assets, all while seamlessly interacting with Web3 technologies.
Storybook helps us build UI components in isolation from our app's business logic, data, and context. That makes it easy to develop hard-to-reach states and save these UI states as stories to revisit during development, testing, or QA.
To start adding stories, create a index.stories.tsx
inside the component's folder:
src ββββ components β ββββ @shared β ββββ β β index.tsx β β index.module.css β β index.stories.tsx β β index.test.tsx
Starting up the Storybook server with this command will make it accessible under http://localhost:6006
:
npm run storybook
If you want to build a portable static version under storybook-static/
:
npm run storybook:build
Test runs utilize Jest as test runner and Testing Library for writing tests.
All created Storybook stories will automatically run as individual tests by using the StoryShots Addon.
Creating Storybook stories for a component will provide good coverage of a component in many cases. Additional tests for dedicated component functionality which can't be done with Storybook are created as usual Testing Library tests, but you can also import exisiting Storybook stories into those tests.
Executing linting, type checking, and full test run:
npm test
Which is a combination of multiple scripts which can also be run individually:
npm run lint
npm run type-check
npm run jest
A coverage report is automatically shown in console whenever npm run jest
is called. Generated reports are sent to CodeClimate during CI runs.
During local development you can continously get coverage report feedback in your console by running Jest in watch mode:
npm run jest:watch
Code style is automatically enforced through ESLint & Prettier rules:
- Git pre-commit hook runs
prettier
on staged files, setup with Husky - VS Code suggested extensions and settings for auto-formatting on file save
- CI runs a linting & TypeScript typings check as part of
npm test
, and fails if errors are found
For running linting and auto-formatting manually, you can use from the root of the project:
# linting check
npm run lint
# auto format all files in the project with prettier, taking all configs into account
npm run format
To create a production build, run from the root of the project:
npm run build
# serve production build
npm run serve
Every branch or Pull Request is automatically deployed to multiple hosts for redundancy and emergency reasons:
A link to a preview deployment will appear under each Pull Request.
The latest deployment of the main
branch is automatically aliased to market.oceanprotocol.com
, where the deployment on Netlify is the current live deployment.
We welcome contributions in form of bug reports, feature requests, code changes, or documentation improvements. Have a look at our contribution documentation for instructions and workflows:
We encourage you to fork this repository and create your own data marketplace. When you publish your forked version of this market there are a few elements that you are required to change for copyright reasons:
- The typeface is copyright protected and needs to be changed unless you purchase a license for it.
- The Ocean Protocol logo is a trademark of the Ocean Protocol Foundation and must be removed from forked versions of the market.
- The name "Ocean Market" is also copyright protected and should be changed to the name of your market.
Additionally, we would also advise that your retain the text saying "Powered by Ocean Protocol" on your forked version of the marketplace in order to give credit for the development work done by the Ocean Protocol team.
Everything else is made open according to the apache2 license. We look forward to seeing your data marketplace!
If you are looking to fork Ocean Market and create your own marketplace, you will find the following guides useful in our docs:
To allow publishers to set pricing as "Fixed" you need to add the following environmental variable to your .env file: NEXT_PUBLIC_ALLOW_FIXED_PRICING="true"
(default).
To allow publishers to set pricing as "Free" you need to add the following environmental variable to your .env file: NEXT_PUBLIC_ALLOW_FREE_PRICING="true"
(default).
This allocates the datatokens to the dispenser contract which dispenses data tokens to users for free. Publishers in your market will now be able to offer their assets to users for free (excluding gas costs).
Ocean Market comes with prebuilt components for you to customize to cover GDPR requirements. Find additional information on how to use them below.
Feel free to adopt our provided privacy policies to your needs. Per default we cover four different languages: English, German, Spanish and French. Please be advised, that you will need to adjust some paragraphs in the policies depending on your market setup (e.g. the use of cookies). You can easily add or remove policies by providing your own markdown files in the content/pages/privacy
directory. For guidelines on how to format your markdown files refer to our provided policies. The pre-linked content tables for these files are automatically generated.
Additionally, Ocean Market provides a privacy preference center for you to use. This feature is disabled per default since we do not use cookies requiring consent on our deployment of the market. However, if you need to add some functionality depending on cookies, you can simply enable this feature by changing the value of the NEXT_PUBLIC_PRIVACY_PREFERENCE_CENTER
environmental variable to "true"
in your .env
file. This will enable a customizable cookie banner stating the use of your individual cookies. The content of this banner can be adjusted within the content/gdpr.json
file. If no optionalCookies
are provided, the privacy preference center will be set to a simpler version displaying only the title
, text
and close
-button. This can be used to inform the user about the use of essential cookies, where no consent is needed. The privacy preference center supports two different styling options: 'small'
and 'default'
. Setting the style propertie to 'small'
will display a smaller cookie banner to the user at first, only showing the default styled privacy preference center upon the user's customization request.
Now your market users will be provided with additional options to toggle the use of your configured cookie consent categories. You can always retrieve the current consent status per category with the provided useConsent()
hook. See below, how you can set your own custom cookies depending on the market user's consent. Feel free to adjust the provided utility functions for cookie usage provided in the src/utils/cookies.ts
file to your needs.
import { CookieConsentStatus, useConsent } from '@context/CookieConsent'
import { deleteCookie, setCookie } from '@utils/cookies'
// ...
const { cookies, cookieConsentStatus } = useConsent()
cookies.map((cookie) => {
const consent = cookieConsentStatus[cookie.cookieName]
switch (consent) {
case CookieConsentStatus.APPROVED:
// example logic
setCookie(`YOUR_COOKIE_NAME`, 'VALUE')
break
case CookieConsentStatus.REJECTED:
case CookieConsentStatus.NOT_AVAILABLE:
default:
// example logic
deleteCookie(`YOUR_COOKIE_NAME`)
break
}
})
The privacy preference centre has two styling options default
and small
. The default view shows all of the customization options on a full-height side banner. When the small
setting is used, a much smaller banner is shown which only reveals all of the customization options when the user clicks "Customize".
The style can be changed by altering the style
prop in the PrivacyPreferenceCenter
component in src/components/App.tsx
. For example:
<PrivacyPreferenceCenter style="small" />
Copyright 2022 Ocean Protocol Foundation Ltd.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.