Open Directory Downloader in GoLang with multi-threading

Hello all, So recently I was browsing through reddit and came across subreddit called OpenDirectory, which shows open directory available of server.

And I wanted to download one Open Director with all of its nested directory, so I was looking through some solution to download all files along with nested folder. I found one solution with CURL but the thing was that it’s only available to Linux(and I have windows as my main driver). I tried curl(Alias for Invoke-WebRequest) in PowerShell but it’s very difficult ;(.

As you might know, I recently started working with GO, So I thought of creating my own custom solution to download files rather then downloading some other software(which I tried but did not found any solution for it).

Source code is available on github.

I started with very simple, i.e. one file download at a time to make things very simple and easy to reason about.

Here is what I have come up with, very simple solution to download all file, one at a time.

• Read file from local file system(contains file to be downloaded in each line)
• Extract folder and file name so that nested we have exact folder structure as OD(Open Directory).
• Pass File name to downloader which downloads the file.

Here is another variant for downloader with multiple goroutines(for best performance)

What it does basically is,

• Its accepts parameter for filename(-f), number of concurrent(-c) download and destination folder(-d)
• It reads file from file system(which included list for downloaded files)
• creates multiple goroutines to download multiple files at a time. It’s better to use multithreading for faster download if you have resources available (if server gives that much throughput).
• Uses channel for syncing between main goroutines and sub-routines

With single threading I was able to download file at average of 15% of my total bandwidth but with multithreading I was able to get download of at least 80%.

I was very satisfied with the result with goroutines, so I am now using it for my downloads 🙂

Thanks for reading…