Powershell - anthonyblackham/GIS-Wiki GitHub Wiki

Downloading files from the web using a list of URLS

Since I'm handicapped by not having linux in some environments I'm working on equivalent workflows to automate/script stuff. This brings me to powershell.

I have a list of URLS and I want to download them incrementally.

In powershell to download one file you can do from stackexchange:

(new-object System.Net.WebClient).DownloadFile('http://www.xyz.net/file.‌​txt','C:\tmp\file.tx‌​t')

But that's just one. lets say you want you want to batch download a bunch of URLS from a list, you'll have to write a powershell script.

You can use

Get-Content urls.txt

to print the list of your urls and then you'll need to pass that through a download loop

So a complete version would be scripts.ps1:

$down = New-Object "System.Net.WebClient"
$i = 1
Get-Content "F:\downloads\urls.txt" | Foreach-Object {
    $down.DownloadFile($_, [string]$i++ +'.txt')
}

Supposedly if you have a newer version (>3.0) of powershell this is simpler, you can check your version with $PSVersionTable:

Get-Content urls.txt | ForEach-Object {Invoke-WebRequest $_ -OutFile $(Split-Path $_ -Leaf)}

If running a powerscript yields:

File F:\script.ps1 cannot be loaded because the execution of scripts is disabled on this system.

Solution:

powershell -ExecutionPolicy ByPass -File script.ps1

Merging Text files from the command line:

for %f /f in (*.txt) do type "%f" >> output.txt