Fix bandcamp download #247

JKatzwinkel · 2020-10-22T14:52:47Z

Bandcamp seems to have changed the way they embed JSON metadata into track pages. Fixes #246.

JKatzwinkel · 2020-10-22T15:10:24Z

Nose doesn't complain at all, but on running tests.py directly, test_bandcamp and test_bandcamp_slashes fail && I have no idea why.

> DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. pip 21.0 will drop support for Python 2.7 in January 2021. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support

polarisfm · 2020-10-28T06:19:17Z

Just tested it. It works with Python 3.7.3 on Debian. Thanks! 👍

Also I can confirm that the issue was real (and wouldn't output errors).

MammouthSaman · 2020-10-28T09:09:52Z

for me it still doesnt work with python 3.8 on ubuntu budgie 20.04

JKatzwinkel · 2020-10-28T18:56:50Z

Can you be more specific because it works fine for me with python 3.8.5 on a fresh Ubuntu 20.04 installation using this Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  config.vm.provision "shell", inline: <<-EOF
    sudo apt-get update && sudo apt-get install -y git python3 python3-pip
    git clone -b fix_246_bandcamp https://github.com/JKatzwinkel/SoundScrape.git && cd SoundScrape
    pip3 install --user -r requirements.txt
    nosetests
    python3 soundscrape/soundscrape.py https://anaalnathrakh.bandcamp.com/track/the-age-of-starlight-ends
    [ $(ls -1 *.mp3 | wc -l) -eq 1 ] && echo "success!!" && ls *.mp3
  EOF
end

MammouthSaman · 2020-10-29T09:07:56Z

Can you be more specific because it works fine for me with python 3.8.5 on a fresh Ubuntu 20.04 installation using this Vagrantfile:

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"
  config.vm.provision "shell", inline: <<-EOF
    sudo apt-get update && sudo apt-get install -y git python3 python3-pip
    git clone -b fix_246_bandcamp https://github.com/JKatzwinkel/SoundScrape.git && cd SoundScrape
    pip3 install --user -r requirements.txt
    nosetests
    python3 soundscrape/soundscrape.py https://anaalnathrakh.bandcamp.com/track/the-age-of-starlight-ends
    [ $(ls -1 *.mp3 | wc -l) -eq 1 ] && echo "success!!" && ls *.mp3
  EOF
end

sorry it should come from me cause i 'm not a really into programing.
i was thinking that the bugfix was already integrated in soundscrape so I had just made a pip3 install soundscrape --upgrade` and try again but stiil have the blinking cursor, nothing hapenning and no error message.

how do i proced to use this vagrant files
I try this in my shell

saman@BearPad:~$ git clone -b fix_246_bandcamp https://github.com/JKatzwinkel/SoundScrape.git && cd SoundScrape
Clonage dans 'SoundScrape'...
remote: Enumerating objects: 50, done.
remote: Counting objects: 100% (50/50), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 968 (delta 24), reused 36 (delta 11), pack-reused 918
Réception d'objets: 100% (968/968), 268.22 Kio | 778.00 Kio/s, fait.
Résolution des deltas: 100% (616/616), fait.
saman@BearPad:~/SoundScrape$     pip3 install --user -r requirements.txt
Requirement already satisfied: args>=0.1.0 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 1)) (0.1.0)
Requirement already satisfied: clint>=0.3.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 2)) (0.5.1)
Requirement already satisfied: demjson>=2.2.2 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (2.2.4)
Requirement already satisfied: fudge>=1.0.3 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 4)) (1.1.1)
Requirement already satisfied: nose>=1.3.7 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 5)) (1.3.7)
Requirement already satisfied: requests[security]>=2.9.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2.22.0)
Requirement already satisfied: setuptools>=18.0.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 7)) (45.2.0)
Requirement already satisfied: simplejson>=3.3.1 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 8)) (3.16.0)
Requirement already satisfied: soundcloud>=0.4.1 in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 9)) (0.5.0)
Requirement already satisfied: wheel>=0.24.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 10)) (0.34.2)
Requirement already satisfied: mutagen>=1.31.0 in /usr/lib/python3/dist-packages (from -r requirements.txt (line 11)) (1.44.0)
Requirement already satisfied: cryptography>=1.3.4 in /usr/lib/python3/dist-packages (from requests[security]>=2.9.0->-r requirements.txt (line 6)) (2.8)
Requirement already satisfied: idna>=2.0.0 in /usr/lib/python3/dist-packages (from requests[security]>=2.9.0->-r requirements.txt (line 6)) (2.8)
Requirement already satisfied: pyOpenSSL>=0.14 in /usr/local/lib/python3.8/dist-packages (from requests[security]>=2.9.0->-r requirements.txt (line 6)) (19.1.0)
Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from pyOpenSSL>=0.14->requests[security]>=2.9.0->-r requirements.txt (line 6)) (1.14.0)
saman@BearPad:~/SoundScrape$     nosetests
[################################] 2739/2739 - 00:00:00
[################################] 5856/5856 - 00:00:00
...FFF.
======================================================================
FAIL: test_soundcloud (test.TestSoundscrape)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/saman/SoundScrape/tests/test.py", line 41, in test_soundcloud
    self.assertTrue(new_mp3_count > mp3_count)
AssertionError: False is not true
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /resolve?url=https%3A%2F%2Fsoundcloud.com%2Ffzpz%2Frevised&limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 302 137
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /tracks/168344119?client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 200 816
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /tracks/168344119/stream?limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 429 176
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_soundcloud_hard (test.TestSoundscrape)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/saman/SoundScrape/tests/test.py", line 54, in test_soundcloud_hard
    self.assertTrue(new_mp3_count > mp3_count)
AssertionError: False is not true
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /resolve?url=https%3A%2F%2Fsoundcloud.com%2Fpuptheband&limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 302 136
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /users/41870548?client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 200 469
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /users/41870548/tracks?limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 200 1989
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /tracks/105554591/stream?limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 429 176
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: test_soundcloud_hard_2 (test.TestSoundscrape)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/saman/SoundScrape/tests/test.py", line 68, in test_soundcloud_hard_2
    self.assertTrue(new_mp3_count > mp3_count)
AssertionError: False is not true
-------------------- >> begin captured logging << --------------------
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /resolve?url=https%3A%2F%2Fsoundcloud.com%2Flostdogz%2Fsnuggles-chapstick&limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 302 137
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /tracks/337016204?client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 200 1512
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1): api.soundcloud.com:443
urllib3.connectionpool: DEBUG: https://api.soundcloud.com:443 "GET /tracks/337016204/stream?limit=200&client_id=a3dd183a357fcff9a6943c0d65664087 HTTP/1.1" 429 176
--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 8 tests in 12.726s

FAILED (failures=3)
saman@BearPad:~/SoundScrape$     python3 soundscrape/soundscrape.py https://anaalnathrakh.bandcamp.com/track/the-age-of-starlight-ends
Downloading: The Age of Starlight Ends
[################################] 4281/4281 - 00:00:00
saman@BearPad:~/SoundScrape$     [ $(ls -1 *.mp3 | wc -l) -eq 1 ] && echo "success!!" && ls *.mp3^C

beside the 3 failures on the 8 tests the track the download of the track "the age of starlight ends" worked (pretty epic song by the way).
I understood that this bugfix work pretty well, and start to think i may be at the wrong place to ask it,
but i'am still wandering what i have to do . to expect this result wwith the commandesoundscrape -b <any bandcamp link>

JKatzwinkel · 2020-10-29T10:11:36Z

Merci beaucoup pour cette helpful information! Yes, as you already noticed, these changes haven't been merged into the release branch yet, so you will not receive them via an update from the cheese shop. Until @Miserlou gets around to review and hopefully accept them, you could replace your soundscrape installation with one from my branch like this I guess (within the ~/SoundScrape folder you already created):

pip3 uninstall soundscrape
python3 setup.py install --user

Don't worry about vagrant, I just needed it in order to test the bandcamp download on ubuntu because I use a different distro. I posted my Vagrantfile to make clear what I mean by "works fine for me", and so that you could reproduce the result, which you did!

Thanks for providing the output of your test run! We have been discussing the issue of soundcloud API keys in #248, but there is not much one can do about the soundcloud tests failing at the moment, because they currently do not provide new API keys.

Again thanks for your help!

JKatzwinkel · 2020-10-29T10:25:54Z

~~@Miserlou Travis is happy.~~

MammouthSaman · 2020-10-29T10:26:57Z

works perfectly thanks a lot !!!

dseptem · 2020-10-31T22:21:43Z

Can confirm this works perfectly on windows 10 x64 using python 3.7.3.
Hopefully it gets merged soon!

xsteadfastx · 2020-11-19T10:19:40Z

tried it...works perfectly!

Miserlou · 2020-11-22T09:37:29Z

Merged and published 😎

JKatzwinkel added 2 commits October 22, 2020 16:24

new bandcamp JSON extraction function

1ca6077

change docstring to google style

e8335b3

JKatzwinkel added 4 commits October 22, 2020 17:34

fix travis setup

d5a2097

make backwards compatible

8451105

rm debug messages

9464ba0

JKatzwinkel mentioned this pull request Oct 22, 2020

Soundcloud API rate limit #248

Open

JKatzwinkel marked this pull request as draft October 28, 2020 23:01

fix html entities in track metadata & file names

807a65e

JKatzwinkel added 2 commits November 15, 2020 17:38

bring back Python 3.3 support

3ae5685

test track data containing html entity

a73969e

JKatzwinkel marked this pull request as ready for review November 15, 2020 16:39

forgot that travis doesn't support python 3.3

5889779

Miserlou merged commit 074c8b2 into Miserlou:master Nov 22, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bandcamp download #247

Fix bandcamp download #247

JKatzwinkel commented Oct 22, 2020

JKatzwinkel commented Oct 22, 2020

polarisfm commented Oct 28, 2020 •

edited

Loading

MammouthSaman commented Oct 28, 2020

JKatzwinkel commented Oct 28, 2020 •

edited

Loading

MammouthSaman commented Oct 29, 2020

JKatzwinkel commented Oct 29, 2020

JKatzwinkel commented Oct 29, 2020 •

edited

Loading

MammouthSaman commented Oct 29, 2020 •

edited

Loading

dseptem commented Oct 31, 2020

xsteadfastx commented Nov 19, 2020

Miserlou commented Nov 22, 2020

Fix bandcamp download #247

Fix bandcamp download #247

Conversation

JKatzwinkel commented Oct 22, 2020

JKatzwinkel commented Oct 22, 2020

polarisfm commented Oct 28, 2020 • edited Loading

MammouthSaman commented Oct 28, 2020

JKatzwinkel commented Oct 28, 2020 • edited Loading

MammouthSaman commented Oct 29, 2020

JKatzwinkel commented Oct 29, 2020

JKatzwinkel commented Oct 29, 2020 • edited Loading

MammouthSaman commented Oct 29, 2020 • edited Loading

dseptem commented Oct 31, 2020

xsteadfastx commented Nov 19, 2020

Miserlou commented Nov 22, 2020

polarisfm commented Oct 28, 2020 •

edited

Loading

JKatzwinkel commented Oct 28, 2020 •

edited

Loading

JKatzwinkel commented Oct 29, 2020 •

edited

Loading

MammouthSaman commented Oct 29, 2020 •

edited

Loading