Automatic Directory Restructuring After Modifying "file_directory_format"??

[Nick05491]

I'll start by saying that I have very limited experience with Python. I'm an EE with basic knowledge of VBscript, C++, Matlab, etc. - not a master programmer by any means.

I modified the 'file_directory_format' within config.json to be "{site_name}/{model_username}/{api_type}/{value}". Default was "{site_name}/{username}/{api_type}/{value}/{media_type}". I wanted to simplify the directory structure and don't mind images and videos being in the same folder. I prefer to use FInder's 'Arrange By: Kind' feature.

When doing this I thought I was going to have to manually move any image or video files I had previously downloaded from their respective folders ('Images', 'Videos') to the higher-level '{value}' folder (i.e., 'Free', 'Paid'). Otherwise I assumed the code wouldn't recognize the previously downloaded files and redownload them all under the appropriate '{value}' folder. Except I forgot to manually restructure the directory for one provider I was scraping and it looks like all of the previous downloads were automatically relocated (based on the created date shown in Finder not being today's date... seems to be when I had initially downloaded these files). Some of these files hadn't even been originally downloaded with this Python scrapper (used to use a browser extension before I found this beautiful tool), yet were still relocated. I used to store media from Archived posts in the same folder as regular Posts - even these were relocated to an 'Archived' folder.

Questions are as follows:

• Can anyone confirm the code is intentionally handling this automatic directory restructuring? If I know that this was purposely written into the code and not just random luck, I'd feel better about not manually restructuring directories.
• Is there a short answer on how existing media is detected? Since files downloaded w/o this tool were relocated, it doesn't seem like the tool is saving a record of previously downloaded files and their locations. Is the code just searching for existing file names anywhere within the directory for that provider? Or within the entire ".sites" directory? This tool seems to download Profile media with file names other than "avatar.jpg" and "header.jpg". Scanning the directory for existing files under the same name would explain why the tool doesn't relocate the pre-existing generically named equivalents of these files.

Thanks for the help.

[Nick05491]

I believe I may have answered my own question. It looks like this is handled by the 'ofsorter.py' script within the extras directory. Assuming the 'user_directory' argument is the directory for a given provider, it appears the tool searches within the provider directory for matching file names like I described in my question above. Can anyone confirm?

[Nick05491]

Bumping this to see if anyone can confirm my understanding of the above. Would feel better about not manually structuring existing directories not originally created by this tool if I knew the code was designed to handle this.

[DIGITALCRIMINAL]

Yes when you change the filename structure, it automatically restructures the directory.

Media is detected by reading the metadata files. It then scans the username directory for all the filenames that is stored in the metadata files.

[Nick05491]

Appreciate the confirmation. Just to make sure I understand correctly, say I had an existing dirctory (ex. "belledelphine") that had both pics and vids from the various APIs (posts, messages, stories, etc.) that I had manually down all mixed together in the main directory (not organized into folders). The first time I scrape this account, and file names that exist in the metadata will be relocated to the appropriate folder. Is this correct?

What if the same file (w/ same name... I have seen this) had been both posted and sent in a message? Does it copy the existing file to both directories? Is the check for existing files based solely on the file names (if it also looked at the file metadata like Date Modified I suppose it may be able to match the existing file to the one in a particular API).

How does the code handle files in the directory that don't exist in the Metadata? Are they all thrown into 'Posts' by default?

Thanks for the help man and for maintaining this script... really appreciated.

[DIGITALCRIMINAL]

During it's renaming phase, the script will scan for all content located in the model folder and put it in a list. It will then try to find the content in the metadata db by looking up it's filename or media_id. If found, the content will be relocated to it's correct folder.

I remember handling wall content that was sent through messages. The script handles this by setting "linked" to True in the database, and both linked files will have "linked" pre-appended to it.

If the content isn't in the Metadata, it won't get touched.

Thanks :)