💾 Archived View for gmi.runtimeterror.dev › powershell-download-web-folder-contents › index.gmi captured on 2024-08-18 at 17:38:25. Gemini links have been rewritten to link to archived content

-=-=-=-=-=-=-

2022-04-19

Download Web Folder Contents with Powershell (`wget -r` replacement)

We've been working lately to use HashiCorp Packer [1] to standardize and automate our VM template builds, and we found a need to pull in all of the contents of a specific directory on an internal web server. This would be pretty simple for Linux systems using `wget -r`, but we needed to find another solution for our Windows builds.

[1] HashiCorp Packer

A coworker and I cobbled together a quick PowerShell solution which will download the files within a specified web URL to a designated directory (without recreating the nested folder structure):

$outputdir = 'C:\Scripts\Download\'
$url = 'https://win01.lab.bowdre.net/stuff/files/'
# enable TLS 1.2 and TLS 1.1 protocols
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12, [Net.SecurityProtocolType]::Tls11
$WebResponse = Invoke-WebRequest -Uri $url
# get the list of links, skip the first one ("[To Parent Directory]") and download the files
$WebResponse.Links | Select-Object -ExpandProperty href -Skip 1 | ForEach-Object {
    $fileName = $_.ToString().Split('/')[-1]                        # 'filename.ext'
    $filePath = Join-Path -Path $outputdir -ChildPath $fileName     # 'C:\Scripts\Download\filename.ext'
    $baseUrl = $url.split('/')                                      # ['https', '', 'win01.lab.bowdre.net', 'stuff', 'files']
    $baseUrl = $baseUrl[0,2] -join '//'                             # 'https://win01.lab.bowdre.net'
    $fileUrl  = '{0}{1}' -f $baseUrl.TrimEnd('/'), $_               # 'https://win01.lab.bowdre.net/stuff/files/filename.ext'
    Invoke-WebRequest -Uri $fileUrl -OutFile $filePath
}

The latest version of this script will be found on GitHub [2].

[2] GitHub

---

📧 Reply by email

PSA: Microsoft's KB5022842 breaks Windows Server 2022 VMs with Secure Boot

Using PowerCLI to list Linux VMs and Datacenter Locations

Run scripts in guest OS with vRA ABX Actions

---

Home

This page on the big web

Download Web Folder Contents with Powershell (`wget -r` replacement)

Related articles