[Tutorial] Split the Markets Subsystem

[jennijuju]

[WIP]

See suggested practice:

• Running All Storage Provider Systems on the Same Box
• Running Storage Provider Systems in the Different Boxes

[TippyFlitsUK]

Running All Storage Provider Systems on the Same Box

[Shekelme]

Hmm. Somehow I started this splitted setup. Wondering why lotus-miner --call-on-markets storage list sees all storages attached to mining node (i.e. sealing storages from workers), by all logic, only the stores specified in the personal markets node's storage.json should be displayed there...

[TippyFlitsUK]

I believe that is the intended behaviour. On my single machine my markets storage.json is exactly the same as my miner storage.json anyway. It only needs to be changed on a multiple machine setup.

If you had customized storage locations for your monolith miner, consider either:

1. copying the ~/.lotusminer/storage.json file to the ~/.lotusmarket
   directory, if both miner and markets are running on the same machine and have
   access to the same paths.
2. adjusting ~/.lotusmarket/storage.json to point to network mounts shared
   with the miner, if both processes are running on separate machines.

[Shekelme]

I wonder what was the intention of the developers when they implemented this. The market node does not need to know anything about sealing workers' storage...
But okay, the question now is different. If I attach additional storage to the miner, should I repeat the same procedure for the market node? How should it look like?
lotus-miner storage attach --init --store /media/filecoin-storage-11/
lotus-miner --call-on-markets storage attach --init --store /media/filecoin-storage-11/
?
It's a little scary to experience this on a working miner, even simply attaching a new storage to the miner required it to be restarted (now with v1.11.1)...

[TippyFlitsUK]

Adding your storage.json to the markets repo is a suggestion rather than a requirement. It will still work absolutely fine without it. With the storage.json added the requests for data will no longer be routed through the miner as the markets instance can access the data directly. It would ultimately reduce miner load but on a single box setup I would imagine that the reduction is negligible.

[Shekelme]

Anyway it looks that lotus-miner --call-on-markets storage attach --store /media/filecoin-storage-11/ simply doesn't work. I do not pass the --init flag since the storage path is already initialized by lotus-miner.

[benjaminh83]

Running Storage Provider Systems in the Different Boxes

[benjaminh83]

Running the lotus-miner market node on a separate machine does involve a bit more reconfiguration of API tokens. If you coming from a one box daemon/miner, then there is a good chance that everything is talking over 127.0.0.1, and lotus picks up the tokens automatically. This is not the case if running on separate machines.

Lotus-miner market node will need the following API tokens loaded to initialize and run:

export APISEALER=xxxx:/ip4/{your main miner node IP}/tcp/{Port}/http
export APISECTORINDEX=xxxx:/ip4/{your main miner node IP}/tcp/{Port}/http
export FULLNODE_API_INFO=xxxx:/ip4/{your daemon IP}/tcp/{Port}/http

If you have been running the lotus daemon together with the lotus-miner, then you need to change the .lotus/config.toml, so its not listening on 127.0.0.1. I personally don't like to set it to 0.0.0.0, so I put in my actual IP that it should be listening on:

[API]
  ListenAddress = "/ip4/{your daemon IP}/tcp/{Port}/http"
  RemoteListenAddress = "/ip4/{your daemon IP}/tcp/{Port}/http"

After you done so, you need to restart the daemon, and create a new API token that will include the new IP. This will also be needed on the local main miner, and it now cannot connect on 127.0.0.1.

lotus auth api-info --perm admin

Then you can export this API before you start up the local main miner (otherwise it basically cannot find the daemon, and wont start)
You can similarily create the API tokens from the main miner, that needs to be used for APISEALER and APISECTORINDEX. To my understanding, this is the same token - just the one you get when:
lotus-miner auth api-info --perm admin
On the main miner.

Now you can transfers the content of the API tokens - I just put them in a text file and SCP them over to the market node. Also you need to SCP over the backup, which is created on the main miner node, but will be needed on the market miner, to initialize the lotus-miner market node.

I think this basically covers the biggest changes from splitting locally to running on separate servers over the network.

You probably also want to do the lotus-miner auth api-info --perm admin on the market node, once running, and transfer this token back on the main miner (or wherever) - this is mainly for running commands from one place, as you as an example, won't be able to gather the content in lotus-miner info without having API access tokens loaded to both main miner node and market node.

Good luck

[stuberman]

For remote listen I use this syntax "{IP of daemon}:{port of daemon}"

RemoteListenAddress = "192.168.1.95:1234"

https://docs.filecoin.io/build/lotus/enable-remote-api-access/#setting-the-listening-interface-for-the-api-endpoint

[dayou5168]

How to separate the storage markets to different machines

Benefits of separating storage markets

• Separate operations can run independently of each other, whether you have different machines, or the same machine running two programs
• Reduces port exposure and improves security

Our documentation on Separate Storage Marketplace documentation is more than complete. You can refer to the existing documentation to make sure you understand the separation process.

1. Mining/sealing/verifying node operations

Please make sure that all sealing jobs have been finished before starting the operation.

step 1. stop mining/sealing/verifying nodes and configure environment variables, here we use backup directory as /backup

lotus-miner stop
export LOTUS_BACKUP_BASE_PATH=/backup

step 2. Change the mining/sealing/proving node configuration

The subsystem area mainly changes EnableMarkets to false.

[subsystem]
EnableMining = true
EnableSealing = true
EnableSectorStore = true
EnableMarkets = false

Comment out the following fields

#[Libp2p]
#libp2p request endpoints (public)
#ListenAddresses = ["/ip4/0.0.0.0/tcp/24001", "/ip6/::/tcp/24001"] .

Step 3. Start lotus-miner and perform a backup

nohup lotus-miner run &
lotus-miner backup /backup/backup.cbor
cp ~/.lotusminer/dagstore /backup/ #If you have already initialized the dagstore that was created   
cp ~/.lotusminer/config.toml /backup/  

Step 4. echo the variable values for the marketplace node

echo APISEALER=`lotus-miner auth api-info --perm=admin`
echo APISECTORINDEX=`lotus-miner auth api-info --perm=admin` 
lotus-miner auth api-info --perm=admin

Record these three values and import them into the environment variables of the markets node.

step 5 Copy the /backup folder to the root of the markets node.

2. markets node operations

step 1. Edit /backup/config.toml

[subsystem]
Enable mining = false
Enable sealing = false
Enable sector storage = false
Enable Marketplace = true

step2. Import environment variables

The main environment variables are APISEALER, APISECTORINDEX, MINER_API_INFO (these three environment variables are echoed in 1.step4)
Another one is FULLNODE_API_INFO, which you can get by executing this command in your lotus daemon node

lotus auth api-info --perm admin  

APISEALER, APISECTORINDEX is used by the markets node to send encapsulated information to lotus-miner after receiving the order.
MINER_API_INFO is used for lotus-miner client to interact with mining/sealing/proving.
fullnode_api_info is used for on-chain interaction
Here is my example

export FULLNODE_API_INFO=xxx:/ip4/192.168.0.105/tcp/1234/http
export APISECTORINDEX=MINER_API_INFO=XXXX:/ip4/192.168.0.105/tcp/2345/http  
export APISEALER=MINER_API_INFO=xxxx:/ip4/192.168.0.105/tcp/2345/http  
export MINER_API_INFO=XXXX:/ip4/192.168.0.105/tcp/2345/http  

step3. Initialize the node and modify the configuration

1 Execute the initialization command

lotus-miner --markets-repo=~/.lotusmarkets init service --type=markets 
                                                        --api-sealer=$APISEALER /-api-sealer=$APISEALER
                                                        --api-sector-index=$APISECTORINDEX (--api-sealer=$APISECTORINDEX)
                                                        --config=/backup/config.toml /backup/config.toml
                                                        /backup/backup.cbor

This step is mainly to initialize the markets node and does not directly run　　

2 Restore the dagstore file (optional, if you initialized the dagstore).

cp /backup/dagstore ~/.lotusmarkets/ # if you have initialized the dagstore

3 Edit the configuration file and export environment variables

Change the corresponding address section to the address of your current host

ListenAddresses = ["/ip4/0.0.0.0/tcp/0", "/ip6/::/tcp/0" ]
AnnounceAddresses = []

Exporting environment variables.

export LOTUS_MINER_PATH=~/.lotusmarkets

Step 4.　Start the node

nohup lotus-miner run &

Step 5. How to confirm successful separation

Execute the following two commands

lotus-miner information  
lotus-miner stores the list of transactions  

If there are no errors reported, you can basically be sure that it is successful.
Finally, it is recommended that you initiate a storage deal test for your miner to ensure that the functions are working properly after the separation.

step 6 Exporting environment variables and other configurations

If you end up changing your extranet address for accepting deals, run the following command to update

lotus-miner actor set-addrs<NEW_MULTIADDR>.

Get the API connection information for the markets node

lotus-miner --call-on-markets auth api-info --perm admin

The output is similar to.

MARKETS_API_INFO=eyJhXXX:/ip4/192.168.0.105/tcp/2345/http

Please export this variable into your mining/sealing/proving node, otherwise you will encounter the following error when executing the command with lotus-miner info.

ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin: invalid memory address or nil pointer dereference

step 7 update the Addresses section of the markets config.toml,Update this section as same as mining/sealing/proving node

[Addresses]
#PreCommitControl = ["f00XX1"]
# CommitControl = ["f00XX2"]
TerminateControl = []
# DealPublishControl = ["f00XX3"]
DisableOwnerFallback = false
DisableWorkerFallback = false

Normally, if you initialize the markets node from the config.toml used in the mining/sealing/proving node, the settings in this section will be the same as in the mining/sealing/proving node, but if you are already running a subsystem and have changed the configuration in this area midway through the mining/sealing/ proving node, please make the same changes in the markets node, otherwise the DealPublishControl configuration will not take effect and DealPublish costs will be deducted from the worker wallet.see this issue

Caution.

This tutorial is a test based on the nerpa network. As we operate in different environments, this tutorial cannot cover all scenarios and the storage provider should adapt the operation to the specific situation

version information

lotus-miner version 1.11.1-rc2+git.40449f1cc.dirty
Linux nerpa-miner 5.11.0-25-generic #27-Ubuntu SMP Fri Jul 9 23:06:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

[Angelo-gh3990]

Make sure that you update your miner config.toml (RemoteListenAddress = "ip-a-d-d-r-e-s-s:2345") with the IP address of your miner so the market node knows which IP address to connect to because it reads the IP address from the miner through the API. If you had the market node running on the same box before it could be still 0.0.0.0:2345 which the remote market node will not know how to connect to :-) .

If you want the market node to pull everything through the miner node's API, then make sure your storage.json looks like :

{
  "StoragePaths": null
}

of course if your market node runs in the same network and is not divided by a firewall for security reasons and it is able to access the storage files where the sectors are located directly, then make sure the storage.json file reflects that. Because else you will be pulling and storing all deals through the miner API and not directly from/to the storage location.

And the same goes for the lotus daemon node when it comes to the RemoteListenAddress (like @benjaminh83 mentions above), make sure you specify the IP addresses in the config files and try to avoid using 0.0.0.0 as a RemoteListenAddress anywhere when separating roles.

Further my advice would be to restore and initialize the market node on an other machine and rebuild the dagstore, it is also a good indication/practice to see that everything is working well.

Also, update your firewall with the new NAT address if you moved the market node to a different machine.