Download images using CURL on a password protected website

Forums › IT Pro and developers › Download images using CURL on a password protected website

NPCtom

430 posts

Ultimate Geek

#250610 18-May-2019 14:27

I am using curl to download thousands of avatars, but unfortunately the website's images are protected by login credentials (If you go to an image URL and are not logged in you get a default avatar). I have the ability to login but cannot find a method to load those credentials into curl. Would there be a way to use my cookies file to bypass this protection?

Curl Command:

curl 'https://stratus.pnbhs.school.nz/pluginfile.php/[42000-45000]/user/icon/pnbhs/f2' -o '#1'.jpg

SirHumphreyAppleby

2844 posts

Uber Geek

#2240553 18-May-2019 14:37

I've not done this with cURL (I used Wget in the past for similar tasks), but it appears you can use either -u username:password or the usual URL syntax, https://username:password@host:port/path. If the site isn't using standard HTTP credentials, then you will probably need to add headers. The exact requirements will be site dependant.

NPCtom

430 posts

Ultimate Geek

#2240554 18-May-2019 14:39

SirHumphreyAppleby:

I've not done this with cURL (I used Wget in the past for similar tasks), but it appears you can use either -u username:password or the usual URL syntax, https://username:password@host:port/path.

I did try using wget using -u but they seem to use Microsoft account integration to login and am unable to get that to work, thats why I thought loading the cookies file into the curl command would bypass that.

SirHumphreyAppleby

2844 posts

Uber Geek

#2240555 18-May-2019 14:41

NPCtom:

I did try using wget using -u but they seem to use Microsoft account integration to login and am unable to get that to work, thats why I thought loading the cookies file into the curl command would bypass that.

Sorry, I missed that. I'm not aware of any integration. In the past I've just done a copy and paste and reused browser cookies.

NPCtom

430 posts

Ultimate Geek

#2240590 18-May-2019 16:35

Found it!

I used my browser's cookies as a token to download the images. All I had to do was download my browser's cookies for that site and execute the command

curl --cookie cookies.txt 'https://stratus.pnbhs.school.nz/pluginfile.php/[42000-45000]/user/icon/pnbhs/f2' -o '#1'.jpg

Thanks for the help Appleby.