Automatically create sub directories

[tbazadaykin]

In the case when there are many files can take a long time to get their list. This problem is especially relevant when accessing via FTP.

To solve this problem, you can create subdirectories to give their names from the first three characters of the file name.

For example we have 3 files (alrady hashed):

3ef8710a2481110fce9e962d998044e6523cffaa.mpg
d3486ae9136e7856bc42212385ea797094475802.txt
d4a8bcd1a9c75d14922825185544184ef820e8f4.png

For this example we build this directory tree:

MEDIA_ROOT
|
|--+model_field_upload_to_value
    |
    |--+3ef
    |   |
    |   |---3ef8710a2481110fce9e962d998044e6523cffaa.mpg
    |
    |--+d34
    |   |
    |   |---d3486ae9136e7856bc42212385ea797094475802.txt
    |
    |--+d4a
        |
        |---d4a8bcd1a9c75d14922825185544184ef820e8f4.png

I have a project containing nearly 30,000 images (more than 4 GB), and it uses this approach. But even in this case, if you access via FTP, requires a few seconds to receive a list of directories.

From the box Django provide strftime() formating for upload_to parameter. But if you use this approach then we will upload the same files on different days. What if we extend these opportunities? Like this:

class MyModel(models.Model)
    file = models.FileField(upload_to='video/%0,3hash')

For file 3ef8710a2481110fce9e962d998044e6523cffaa.mpg this syntax says: put file into MDEIA_ROOT/video/3ef/3ef8710a2481110fce9e962d998044e6523cffaa.mpg

More compex solution:

class MyModel(models.Model)
    file = models.FileField(upload_to='video/%Y/%0,3hash/%4,6hash')

For file 3ef8710a2481110fce9e962d998044e6523cffaa.mpg this syntax says: put file into MDEIA_ROOT/video/2014/3ef/871/3ef8710a2481110fce9e962d998044e6523cffaa.mpg

[matthiask]

The os.makedirs logic in Django's FileSystemStorage is surprisingly complex:
https://github.com/django/django/blob/f8b98f0edc501ca1ca2d04d1d5938cbb68d11db5/django/core/files/storage.py#L225

[budlight]

This project might be dead, but this is definitely a useful suggestion. Its actually pretty crazy this isn't the default in django, I assume everyone wanting to use S3 has made issues like this go ignored.

Anyone putting effort into this should prob write the documentation for https://github.com/adamzap/django-storages/blob/master/storages/backends/hashpath.py and contribute there, it is a much larger project that is likely to actually see some contributions

[jordiecometrica]

@budlight Do you think we can abandon this project in favour of django-storages?

We still maintain django-hashedfilenamestorage, but mostly only for issues that affect us.

[budlight]

@jordiecometrica they do not document that storage class I believe even though it is included, also nobody has worked on it in quite some time, however I think if you were to contribute code it might be just as good to contribute it there

notice no changes since the file was committed https://github.com/adamzap/django-storages/commits/master/storages/backends/hashpath.py

[BoPeng]

I store my files on S3 and everything is straight key/object and there is no directory structure, so I would rather have the directory removed. I think these options should be provided as options at settings or Meta class level.