Since ckanext-qa (and several other extensions) are now relying on ck…

…anext-archiver to do the downloading work, need to store more info about download failures. Tightened up error handling. So this is saved in the task_status table with key "status". Also get rid of filtered list of formats to download - download all we can. Tests fixed, but they are no longer thorough.

README.rst

@@ -73,15 +73,13 @@ Configuration
 
     * ARCHIVE_DIR: path to the directory that archived files will be saved to
     * MAX_CONTENT_LENGTH: the maximum size (in bytes) of files to archive
-    * DATA_FORMATS: the data formats that are archived
 
    Alternatively, if you are running CKAN with this patch: 
    https://github.com/datagovuk/ckan/commit/83dcaf3d875d622ee0cd7f3c1f65ec27a970cd10
    then you can instead add the settings to the CKAN config file as normal:
 
     * ckanext-archiver.archive_dir
     * ckanext-archiver.max_content_length
-    * ckanext-archiver.data_formats  (space separated)
 
 
 Using Archiver
@@ -137,4 +135,4 @@ Tests should be run from the CKAN root directory (not the extension root).
 
 ::
 
-    (pyenv)~/pyenv/src/ckan$ nosetests --ckan ../ckanext-archiver/tests/
+    (pyenv)~/pyenv/src/ckan$ nosetests --ckan ../ckanext-archiver/tests/ --with-pylons=../ckanext-archiver/test.ini

ckanext/archiver/default_settings.py

@@ -7,31 +7,3 @@
 
 # Max content-length of archived files, larger files will be ignored
 MAX_CONTENT_LENGTH = int(config.get('ckanext-archiver.max_content_length', 50000000))
-
-# Only files with these mime-types or extensions will be archived.
-# To archive all files, set DATA_FORMATS = 'all'
-DEFAULT_DATA_FORMATS = [ 
-    'csv',
-    'text/csv',
-    'txt',
-    'text/plain',
-    'text/html',
-    'html',
-    'rdf',
-    'text/rdf',
-    'xml',
-    'xls',
-    'application/ms-excel',
-    'application/vnd.ms-excel',    
-    'application/xls',
-    'text/xml',
-    'tar',
-    'application/x-tar',
-    'zip',
-    'application/zip'
-    'gz',
-    'application/gzip',
-    'application/x-gzip',
-    'application/octet-stream'
-]
-DATA_FORMATS = config.get['ckanext-archiver.data_formats'].split() if 'ckan-archiver.data_formats' in config else DEFAULT_DATA_FORMATS