task: added delete task handler #160

gioelecerati · 2023-02-28T18:45:46Z

No description provided.

codecov · 2023-02-28T18:50:25Z

Codecov Report

Merging #160 (56b2073) into main (aba5f6e) will decrease coverage by 0.58142%.
The diff coverage is 0.00000%.

Additional details and impacted files

@@                 Coverage Diff                 @@
##                main        #160         +/-   ##
===================================================
- Coverage   17.46835%   16.88693%   -0.58142%     
===================================================
  Files             16          17          +1     
  Lines           1975        2043         +68     
===================================================
  Hits             345         345                 
- Misses          1605        1673         +68     
  Partials          25          25

Files	Coverage Δ
task/runner.go	`22.67206% <ø> (ø)`
task/delete.go	`0.00000% <0.00000%> (ø)`

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aba5f6e...56b2073. Read the comment docs.

Files	Coverage Δ
task/runner.go	`22.67206% <ø> (ø)`
task/delete.go	`0.00000% <0.00000%> (ø)`

victorges

Requesting changes due to lack of pagination and the whole error handling story

task/delete.go

victorges · 2023-03-01T21:53:17Z

task/delete.go

+		glog.Errorf("Error listing files in directory %v", directory)
+	}
+
+	totalFiles := files.Files()


This is not total files, it's only 1 page! You have to handle pagination. e.g.: https://github.com/livepeer/livepeer-infra/pull/1214/files#diff-71a8aa1d4b064228d72ee7fbf0ba12affc84118bac0b08da9ad06125cd5f86deR178 (you dont need the rate limiting)

task/delete.go

victorges · 2023-03-01T21:59:10Z

task/delete.go

+	totalFiles := files.Files()
+
+	accumulator := NewAccumulator()
+	tctx.Progress.TrackCount(accumulator.Size, uint64(len(totalFiles)), 0)


We have a problem here due to pagination. The total is not really the total if we do the pagination concurrently with the deletion. One alternative, only to get a right progress number, would be listing all the files beforehand accumulating them in memory until there are no more pages, then start the deletion process. It will make the deletion take a little longer if there are too many pages, but I think it may be interesting anyway.

Otherwise could try something smarter... using the asset size or duration and approximating the number of files... but idk, sounds like overengineering so i'd prefer either the pre-listing idea, having progress go back and forth once there are multiple pages, or even not have any progress at all.

WDYT?

Note: maybe the ideal long term solution might be to store the total size of the asset files in the asset object, when they are created or modified. Then here we could just measure the progress based on that total size. Not a change for this PR tho as it involves a lot other code paths.

victorges · 2023-03-01T22:00:26Z

task/delete.go

+				glog.Errorf("Error deleting file %v: %v (retrying...)", filename, err)
+				time.Sleep(retryInterval)
+			}
+			return fmt.Errorf("failed to delete file %v after %d retries", filename, maxRetries)


If you return an error the whole error group will cancel. You should always return nil and only log the errors instead (if you want to delete as much as possible before failing)

victorges · 2023-03-01T22:04:27Z

task/delete.go

+	if err := eg.Wait(); err != nil {
+		glog.Errorf("Error deleting files: %v", err)
+	}
+
+	if ipfs := asset.AssetSpec.Storage.IPFS; ipfs != nil {
+		err = UnpinFromIpfs(*tctx, ipfs.CID, "cid")
+		if err != nil {
+			glog.Errorf("Error unpinning from IPFS %v", ipfs.CID)
+		}
+		err = UnpinFromIpfs(*tctx, ipfs.NFTMetadata.CID, "nftMetadataCid")
+		if err != nil {
+			glog.Errorf("Error unpinning metadata from IPFS %v", ipfs.NFTMetadata.CID)
+		}


Actually, thinking further about it, if you never return errors, this task will always succeed and we will never even retry deleting the files. In order to simplify this whole logic (since you have some complexity missing like pagination), I'd suggest you to just return errors for now.

If you do want the failures to be as much partial successes as possible, you can accumulate the intermediate errors you get on a local slice here, and in the end you create a single error in case you found any errors during the process. I think that could be premature optimization for now tho, so I'd rather invest more in the additional retry logics you implemented here for file deletion, and could delete for IPFS unpinning as well.

Can even create a generic helper function to retry another function a few times :D
Geppetto gave me this:

func retry(maxRetries int, f func() error) error { var err error for i := 0; i <= maxRetries; i++ { if err = f(); err == nil { return nil } } return fmt.Errorf("after %d attempts, last error: %s", maxRetries, err) }

WDYT?

victorges · 2023-03-01T22:06:42Z

task/delete.go

+	}
+
+	return &TaskHandlerOutput{
+		TaskOutput: &data.TaskOutput{},


Probably still create the Delete output field (empty object), otherwise we have some code in the API that can explode if the task type output is missing.

victorges · 2023-03-01T22:07:14Z

task/delete.go

+func UnpinFromIpfs(tctx TaskContext, cid string, filter string) error {
+	assets, _, err := tctx.lapi.ListAssets(api.ListOptions{
+		Filters: map[string]interface{}{
+			filter: cid,


Weird go syntax, it looks like the field is called filter lol

victorges · 2023-03-01T22:07:41Z

task/runner.go

@@ -35,14 +35,15 @@ var (
 	// without including extraneous error information from Catalyst
 	errInvalidVideo = UnretriableError{errors.New("invalid video file codec or container, check your input file against the input codec and container support matrix")}
 	// TODO(yondonfu): Add link in this error message to a page with the input codec/container support matrix
-	errProbe   = UnretriableError{errors.New("failed to probe or open file, check your input file against the input codec and container support matrix")}
+	errProbe = UnretriableError{errors.New("failed to probe or open file, check your input file against the input codec and container support matrix")}


I've fixed this on main already so you may get some conflicts

gioelecerati requested a review from a team as a code owner February 28, 2023 18:45

victorges requested changes Mar 1, 2023

View reviewed changes

task: added delete task handler

b39dc44

gioelecerati force-pushed the gio/TD/delete branch from ec5e3aa to b39dc44 Compare December 4, 2023 14:19

rebase fix

56b2073

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

task: added delete task handler #160

task: added delete task handler #160

gioelecerati commented Feb 28, 2023

codecov bot commented Feb 28, 2023 •

edited

Loading

victorges left a comment

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

victorges Mar 1, 2023

task: added delete task handler #160

Are you sure you want to change the base?

task: added delete task handler #160

Conversation

gioelecerati commented Feb 28, 2023

codecov bot commented Feb 28, 2023 • edited Loading

Codecov Report

victorges left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 28, 2023 •

edited

Loading