Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove large io.ascii files from master branch #23

Open
astrofrog opened this issue Dec 6, 2017 · 7 comments
Open

Remove large io.ascii files from master branch #23

astrofrog opened this issue Dec 6, 2017 · 7 comments

Comments

@astrofrog
Copy link
Member

astrofrog commented Dec 6, 2017

It would be better to generate these on-the-fly in the benchmark setup functions/methods

@bsipocz
Copy link
Member

bsipocz commented Jul 29, 2018

so these are the reason cloning the repo takes forever?

@MSeifert04
Copy link

If they are really problematically big we might need to rewrite history to avoid keeping them as part of the clone.

@astrofrog
Copy link
Member Author

I think the biggest part of the repo is actually the results branch - it has tens of thousands of files.

@pllim
Copy link
Member

pllim commented Jul 30, 2018

Yeah! Let's squash master into 1 commit! results branch won't be affected and should not be fetched blindly.

@bsipocz
Copy link
Member

bsipocz commented Jul 30, 2018

😱

Actually if it's only the small files, then cloning only master would solve the issue anyway. And while it indeed takes a while, I can very well live with the long cloning time, we don't do it in CI so local times doesn't really matter on the long run.

@pllim
Copy link
Member

pllim commented Jul 30, 2018

Results used to be in master until @astrofrog moved them to results, so that might be the reason why it is slow too? (I still don't understand the git clone magic completely.)

@saimn
Copy link
Contributor

saimn commented Jul 30, 2018

There is the --single-branch option which is quite useful if ones want to get only master (10s / 36Mb vs 2m 51s / 639Mb for the full clone):

~/lib 
(base) ❯ g clone --single-branch https://github.com/astropy/astropy-benchmarks.git
Cloning into 'astropy-benchmarks'...
remote: Counting objects: 480, done.
remote: Compressing objects: 100% (9/9), done.
remote: Total 480 (delta 2), reused 3 (delta 0), pack-reused 471
Receiving objects: 100% (480/480), 36.45 MiB | 4.29 MiB/s, done.
Resolving deltas: 100% (232/232), done.

~/lib 10s
(base) ❯ du astropy-benchmarks
37M     astropy-benchmarks/.git
108M    astropy-benchmarks/benchmarks
144M    astropy-benchmarks

~/lib 
(base) ❯ rm -rf astropy-benchmarks

~/lib 
(base) ❯ g clone https://github.com/astropy/astropy-benchmarks.git
Cloning into 'astropy-benchmarks'...
remote: Counting objects: 96691, done.
remote: Compressing objects: 100% (343/343), done.
remote: Total 96691 (delta 1005), reused 1065 (delta 861), pack-reused 95473
Receiving objects: 100% (96691/96691), 639.43 MiB | 4.05 MiB/s, done.
Resolving deltas: 100% (95502/95502), done.

~/lib  2m 51s
(base) ❯ du astropy-benchmarks
643M    astropy-benchmarks/.git
108M    astropy-benchmarks/benchmarks
750M    astropy-benchmarks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants