Downloading files with variable filenames based on current date

[cwhommes]

I would appreciate someone pointing me in the right direction. I need to download seven files off a webserver daily. The names of these files contain the date on which they were generated (for example 20250114-customer.csv). The files are generated daily.
How would I go about setting up a workflow and pipeline where the name of the file is dependend on the current date (TODAY())? So for example, I start retrieving yesterday's file and subsequently the files of the six days before?

[bamaer]

you didn't specify any details about what this web server would look like, but VFS supports a set of file system types that let you download files directly. VFS is supported in most workflow actions and pipeline transforms, e.g the "copy files" action to copy file locally, or the "text file input" transform to read file contents directly, without the need to download.
You can include your date in the correct format as a variable or workflow or pipeline parameter.

[cwhommes]

Hi Bart, the files I need to access are stored on a webserver that I can only access through the internet ('get http://.../file.csv'). Right now I use a HTTP action to download a file to local storage. From what I understand from the Hop documentation the VFS is specifically for Google, Dropbox, s3 etc. Or am I missing something?
I will experiment with the variables and see if I can make it work and report back. Thanks for your help.

[bamaer]

VFS works with the file systems in the list in the doc page I shared. S3, Google Storage, Dropbox etc are Apache Hop additions, but VFS (part of the Apache Commons project) supports a large number of file systems out of the box.