Provider Incentives Pilot 2 (PIP02) for GPUs

[lechenghiskhan]

Background & Wins

The first Provider Incentive Pilot passed in February 2024 in proposal 246 has brought over 300 total GPUs onto Akash Network and is now fully committed. The commitments are broken down by model as follows:

GPU model	Count	Committed Cost	Term (months)
H100	136	$3,437,009.76	12
A100-80	64	$969,907.20	12
RTX 4090	56	$268,622.40	12
A6000	64	$330,777.60	12
V100	8	$3,387.20	12
Storage		$65,126.40	12

With these resources, the Overclock Labs Core Team has attracted several premier tenants including Venice.ai, Brev.dev, Prime Intellect, Nous Research, Flock.io, Neural AI, UT Austin, RIT, Morpheus, Thumper, and Nodeshift.

In addition, Akash has integrated models like Meta’s Llama-3.1-405B and Nous Research’s Hermes 2 LLMs, which have been made available for free on Akash Chat.

Last but not least, the open beta of Akash Chat API launched in August, has served over 100 million daily average tokens.

These integrations and milestones have driven Akash Network’s daily network revenue from $1.2k in February to a new peak of $5k on August 21.

Expected vs Actual

The first Provider Incentive Pilot originally proposed three approaches. To quickly recap, these were the approaches and their respective motivations

1. Committed Pool: provide high-quality computing resources by engaging professional providers for at least one year. The program was open to anyone with strict requirements for quality, quantity, reliability, compliance, and support.
2. Liquidity Mining Pool: this model proposed creating homogeneous GPU mining pools based on specific configurations where “mining” rewards would be distributed pro-rata to all participants.
3. Research & Development Pool: This provides a budget that would create a buffer for next-generation GPUs and rapidly evolving AI models

As we near the 9-month mark of proposal 246, the Overclock Labs’ Core Team proposes the following:

1. Hold off on the Liquidity Mining Pool pending the development of a robust verifiable compute solution like the one proposed here.
2. Merge the R&D pool into future Committed Pool proposals to reduce overhead and give the Core Team more flexibility to act quickly to meet demand.

Reporting on PIP01

Link to the public spending report

Average utilization across A100s and H100s has stayed consistently over 50% with peak usage cresting 80%.

Approach

We propose continuing the Committed Pool approach to incentivize providers, all administered by the Pilot Program Administrator (PPA). Anyone can apply to be the PPA, provided they meet the following criteria, which were outlined in the previous pilot:

• Should have extensive experience in managing the community.
• Must possess deep expertise in managing Akash providers.
• Should be publicly known and respected within the Akash Community.
• Should have contributed to the Akash open-source repositories.

Responsibilities:

• Commit time and resources towards administering the program.
• Disclose all payments to the public and maintain a strict record of all payments that are subject to 3rd party audit if necessary.
• Provide server management and support to participants.
• Provide responsible settlement services (AKT to USD) for participants if needed.
• Builds tools for verifying compute.

Provider Incentives Pilot 2

The first Committed Pool under Provider Incentive Pilot 1 has been a success. It brought high-quality computing resources to the Akash Network by engaging 6 high-quality professional compute providers supplying over 300 GPUs.

Provider Incentives Pilot 2 will follow in the footsteps of its predecessor.

Budget & Distribution

We propose allocating $10,000,000 of the Provider Incentive Pilot 2 budget according to, approximately, the following:

GPU model	Est cost/hr	Quantity	Budget
H100-80	$2.40	256	$5,375,000
L40s | A100-80	$1.30	256	$2,910,000
MI300	$3.50	8	$230,000
H200-141	$5.75	8	$400,000
RTX 4090	$0.45	204	$805,000
RTX A6000	$0.50	64	$280,000
Total			$10,000,000

Taking into account community feedback, between 5-10% of the total budget will be made available to community providers who meet the following requirements for quality, quantity, reliability, compliance, and support:

• Akash Community Member: Must be a technical member of the Akash Community with experience managing providers on the network for at least 6 months.
• Quality: should operate in a DC or Colocation Center with at least partial power and cooling redundancy.
• Quantity: Must have a minimum of one 8 GPU node of the referenced GPU models and must have persistent storage.
• Reliability: Must have a yearly uptime of at least 99.741%, equivalent to no more than 22 hours of downtime per year. Must have redundancy for storage.
• Support: Must be serviceable without affecting the critical environments. Must be responsive to issues 24/7.

Given the pace of commitment and deployment of funds from the first GPU incentive pilot, this pilot could take 6-9 months to fully deploy in annual contracts. This proposal allocates funds to existing GPU models carried over from PIP01 and either keeps or adds more GPUs in response to demand

Limited Market Impact & Transparent Reporting

Limited Market Impact

Overclock Labs will custody the requested funds in a new, distinct wallet so that funds from any other source are not commingled.

All funds will be liquidated and managed in a manner that ensures minimal impact on the market. These funds will be managed with the same care and attention as all previous Community Funding Proposals with liquidations done in a fashion that will not adversely affect the market. In practice, the effort of this liquidation will add depth to the AKT market for buyers looking to enter.

Transparent Reporting

All costs and records will be made publicly available through reports to ensure maximum transparency and accountability.

Support & Expenses

The PPA will be responsible for technical and logistical support ranging from server management and settlement services (AKT to USD) to building and managing tools necessary for verifying compute resources.

Expenses incurred will include, but are not limited to: transaction and settlement fees, support and bounties paid to insiders or vendors, applicable taxes, and software.

At the end of this program, just as was done with PIP01, all revenue collected by the Akash Network Core Team, net of expenses, will be returned to the Community Pool.

[Zblocker64]

How does one apply to be apart of this program and what is the minimum requirements to apply?

[lechenghiskhan]

Hey @Zblocker64
Please feel free to contact the core team: @anilmurty and @gosuri will be able to help vet.

Minimum requirements at this point include, but are not limited to:

1. Tier 3 data center or better
2. High density (number) of the required GPU models cited in the discussion
3. Has the requisite external and internal connectivity

[anilmurty]

One correction - T2 or higher DC as outlined in this prop https://github.com/orgs/akash-network/discussions/448 -- the primary reason being that we want to make sure the initial set of client applications (a few of which are called out in this proposal) are successful -- which means high availability, reliability, quality.

Note that the original prop also called out a second pool (called the "liquidity mining pool" at the time but could be renamed to a "community provider pool") that was intended to be specifically for community providers and any types of GPUs (not just high end ones).

[0x0factor]

How would you structure your team to propose a Supplier Incentive Pilot 2 on Akash Network based on the success of PIP01?

I would suggest the following:

1. Clarify your success metrics
   Utilization: While the average utilization of the A100 and H100 is over 50%, it would be helpful to explain how this compares to your initial goals. Were you expecting higher, lower, or equal usage? Clear metrics of success can help guide future adjustments and help the community see the impact.
   Revenue Growth: The increase in daily revenue from 1.2k to 5k is impressive. Consider detailing how this aligns with your long-term financial projections for Akash Network. Including year-end or future revenue targets can help show community members the path to growth.
2. Justify your budget allocation
   You allocated $10M to this pilot, but some readers will want to understand how you arrived at this specific GPU quantity and cost. This may help:
   Mention your GPU demand forecast based on current or future tenant requirements. Include insights from PIP01 on which GPUs Akash tenants have performed best or demanded the most.
   Detail cost-effective strategies or insights gained in the past by using these specific GPUs.
3. Plan for unused resources
   You mentioned that PIP01 GPUs are well utilized, but what happens if a specific GPU type in PIP02 is underutilized? Do you have any plans to reallocate resources or provide incentives for their use?
4. Timeline for liquidity mining pools
   While it is understandable that liquidity mining pools have put a hold on verifiable computing solutions, providing a tentative timeline for when they will resume, even if subject to change, is a rough estimate that builds trust and confidence among stakeholders.
5. Promote tenant success stories
   While it is nice to include names like Venice.ai, Brev.dev, Nous Research, and others, highlighting simple success stories or testimonials from these partners — how they have benefited from Akash’s resources — provides a concrete example of the value of the platform and retains future partners. 6. Community Engagement
   While the Pilot Program Administrator (PPA) role is important, I think it would be ideal to create more avenues for community engagement in the process, allowing the community to vote on certain aspects of GPU allocation.
6. Risk Management
   Given the volatility of the GPU and cryptocurrency markets, there should be a section on risk management.
7. Scalability and Future Outlook
   I also think it would be important to the success of this project to briefly outline how this pilot lays the foundation for long-term growth.
   For example, what’s next for Akash after PIP02? Will Akash continue to scale its GPU capacity, or will it diversify into other types of compute resources? A roadmap for the next few years could help illustrate how this fits into Akash’s broader vision.
8. Streamline Financial Transparency

I think it would be a good idea to create a simple visual representation (chart or infographic) that shows how funds are allocated, spent, and reported over time.
I think it would be a good idea to provide real-time updates or dashboards that allow the community to monitor progress and more easily use the funds.

[Zblocker64]

Chat GPT much?

[deathlessdd]

Is this only for datacenter or can any provider apply for this?

[lechenghiskhan]

Hey @deathlessdd!

See the requirements I posed to @Zblocker64's question above.
Having said that, please feel free to speak to the core team about your thoughts or share them here!

The biggest things Akash Network needs to ensure are (@gosuri or @anilmurty to opine):

1. Reliability
2. Performance
3. Density of GPUs
4. Responsiveness

[deathlessdd]

Well Id like to share my thoughts here, The reason why Im asking because it appears to me that the overclock team is monopolizing the compute on the network not really giving other providers a chance. "aka" Taking over the network because the prices are competing to other providers is much cheaper. And this proposal would further increase this even more not giving other providers that have been on the network for a really long time a chance. What are your thoughts on this?

[lechenghiskhan]

I completely understand your concern. Please let me clear up a few things:

1. OCL is not monopolizing the incentives since all proceeds are going directly to providers, not to OCL. Furthermore, any revenue generated by these providers are subsequently sent BACK to the community pool. Please refer to the reports repo to see on-chain validation of this.
2. Secondly, as I referenced in the original post, a robust way to validate and verify compute resources from a highly diverse array of community providers would be required to protect against bad actors and outright fraud. See other depin networks for examples of unbridled abuse without the ability to validate contributions.
3. Please put yourself in the shoes of an AI company looking to choose Akash. Ensuring the best possible user experience, density, and reliability is paramount to win adoption. For example: will you be available nearly 24/7 to answer questions and address potential issues if your provider received PIP incentives? This is an essential requirement of high tier data centers.
4. Looking forward, on chain incentives will be pushed as we scale. It has to. I ask that you understand the priorities related to early stage adoption right now.
5. Please join us in steering committee and sig economics discussions to further explore your perspectives.

[cosmonought]

I would like more context around the numbers in the Budget and Distribution table.

In particular, allocating $5,375,000 for 256 H100-80s comes to about $21,000 per H100-80. This seems to be anywhere from 50-100% of the cost of an H100.

On the other hand, $300,000 is allocated to 8 MI300s, which comes to about $37,500 per MI300. This seems to be 150-300% the cost of an MI300.

Given that MI300s are significantly cheaper than H100s, why is there more allocated to them?

Zooming out a bit:

• is the purpose of 12 months of provider incentives supposed to cover the cost of this equipment (and then some, in the case of MI300s)? If so, does the program risk farming/abuse without more longitudinal commitments? I know the verification process is meant to address this somewhat, but what happens at the end of the 12 months once a provider has recouped the cost of the equipment?
• If PIP is going to completely or significantly subsidize the cost of equipment, is it important that PIP participants agree to provide compute for the network beyond the approved PIP funding phase (i.e., are there contracts to bind providers to Akash)?
• Will those who participated in a previous PIP phase be eligible for subsequent PIP rounds?
• Finally, what is the proportion of PIP incentives to actual usage revenue for providers? This may need to be broken out by GPU model, but my overarching question is whether you think usage revenue is on track to eventually eclipse PIP incentives?

Thank you!

[brewsterdrinkwater]

This proposal passed on chain November 5th, 2024: https://www.mintscan.io/akash/proposals/273