Skip to content

Commit

Permalink
Added evaluation of revisited Oxford and Paris ('roxford5k' and 'rpar…
Browse files Browse the repository at this point in the history 
…is6k');

Refreshed README.md and train/test comments.
  • Loading branch information
@filipradenovic
filipradenovic committed Oct 4, 2018
1 parent fc03550 commit 99b2e31
Show file tree
Hide file tree
Showing 6 changed files with 262 additions and 120 deletions.
56 changes: 39 additions & 17 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# CNNImageRetrieval: Training and evaluating CNNs for Image Retrieval
# CNN Image Retrieval in MatConvNet: Training and evaluating CNNs for Image Retrieval in MatConvNet

**CNNImageRetrieval** is a MATLAB toolbox that implements the training and testing of the approach described in our papers:
This is a MATLAB toolbox that implements the training and testing of the approach described in our papers:

> *Fine-tuning CNN Image Retrieval with No Human Annotation*,
> Radenović F., Tolias G., Chum O.,
> arXiv 2017 [[arXiv](https://arxiv.org/abs/1711.02512)]
**Fine-tuning CNN Image Retrieval with No Human Annotation**,
Radenović F., Tolias G., Chum O.,
TPAMI 2018 [[arXiv](https://arxiv.org/abs/1711.02512)]

> *CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples*,
> Radenović F., Tolias G., Chum O.,
> ECCV 2016 [[arXiv](http://arxiv.org/abs/1604.02426)]
**CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples**,
Radenović F., Tolias G., Chum O.,
ECCV 2016 [[arXiv](http://arxiv.org/abs/1604.02426)]

<img src="http://cmp.felk.cvut.cz/cnnimageretrieval/cnnimageretrieval_teaser.png" width=\textwidth/>
<img src="http://cmp.felk.cvut.cz/cnnimageretrieval/img/cnnimageretrieval_network_medium.png" width=\textwidth/>

## What is it?

Expand All @@ -28,7 +28,7 @@ In order to run this toolbox you will need:
1. MatConvNet MATLAB toolbox version [1.0-beta25](http://www.vlfeat.org/matconvnet/download/matconvnet-1.0-beta25.tar.gz)
1. All the rest (data + networks) is automatically downloaded with our scripts

## Execution
## Execution (training and testing)

Run the following script in MATLAB:

Expand All @@ -38,16 +38,28 @@ Run the following script in MATLAB:
>> train_cnnimageretrieval;
>> test_cnnimageretrieval;
```
See ```[CNNIMAGERETRIEVAL_ROOT]/examples/train_cnnimageretrieval``` and ```[CNNIMAGERETRIEVAL_ROOT]/examples/test_cnnimageretrieval``` for additional details.

## Citation
We provide the pretrained networks trained using the same parameters as in our ECCV 2016 and TPAMI 2018 papers. Performance comparison with the networks trained with our [CNN Image Retrieval in PyTorch](https://github.com/filipradenovic/cnnimageretrieval-pytorch), on the original and the revisited Oxford and Paris benchmarks:

Related publications:
| Model | Oxford | Paris | ROxf (M) | RPar (M) | ROxf (H) | RPar (H) |
|:------|:------:|:------:|:------:|:------:|:------:|:------:|
| VGG16-GeM (MatConvNet) | 87.9 | 87.7 | 61.9 | 69.3 | 33.7 | 44.3 |
| VGG16-GeM (PyTorch) | 87.2 | 87.8 | 60.5 | 69.3 | 32.4 | 44.3 |
| ResNet101-GeM (MatConvNet) | 87.8 | 92.7 | 64.7 | 77.2 | 38.5 | 56.3 |
| ResNet101-GeM (PyTorch) | 88.2 | 92.5 | 65.3 | 76.6 | 40.0 | 55.2 |

**Note**: Data and networks used for training and testing are automatically downloaded when using the example scripts.

## Related publications

### Training (fine-tuning) convolutional neural networks
```
@inproceedings{Radenovic-arXiv17a,
title={Fine-tuning {CNN} Image Retrieval with No Human Annotation},
author={Radenovi{\'c}, Filip and Tolias, Giorgos and Chum, Ond{\v{r}}ej},
booktitle = {arXiv:1711.02512},
year={2017}
@article{RTC18,
title = {Fine-tuning {CNN} Image Retrieval with No Human Annotation},
author = {Radenovi{\'c}, F. and Tolias, G. and Chum, O.}
journal = {TPAMI},
year = {2018}
}
```
```
Expand All @@ -57,4 +69,14 @@ Related publications:
booktitle = {ECCV},
year = {2016}
}
```

### Revisited benchmarks for Oxford and Paris ('roxford5k' and 'rparis6k')
```
@inproceedings{RITAC18,
author = {Radenovi{\'c}, F. and Iscen, A. and Tolias, G. and Avrithis, Y. and Chum, O.},
title = {Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking},
booktitle = {CVPR},
year = {2018}
}
```
46 changes: 32 additions & 14 deletions examples/test_cnnimageretrieval.m
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
% TEST_CNNIMAGERETRIEVAL Code to evaluate (not train) the methods presented in the papers:
% F. Radenovic, G. Tolias, O. Chum, Fine-tuning CNN Image Retrieval with No Human Annotation, arXiv 2017
% F. Radenovic, G. Tolias, O. Chum, Fine-tuning CNN Image Retrieval with No Human Annotation, TPAMI 2018
% F. Radenovic, G. Tolias, O. Chum, CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples, ECCV 2016
%
% Authors: F. Radenovic, G. Tolias, O. Chum. 2017.
Expand All @@ -18,17 +18,17 @@
download_test(data_root);

% Set test options
test_datasets = {'oxford5k', 'paris6k'}; % list of datasets to evaluate on
test_datasets = {'oxford5k', 'paris6k', 'roxford5k', 'rparis6k'}; % list of datasets to evaluate on
test_imdim = 1024; % choose test image dimensionality
use_ms = 1; % use multi-scale representation, otherwise use single-scale
use_rvec = 0; % use regional representation (R-MAC, R-GeM), otherwise use global (MAC, GeM)
use_gpu = [1]; % use GPUs (array of GPUIDs), if empty use CPU
use_gpu = [1,1,1,2,2,2,3,3,3,6,6,6,7,7,7,8,8,8]; % use GPUs (array of GPUIDs), if empty use CPU

% Choose ECCV16 fine-tuned CNN network
% network_file = fullfile(data_root, 'networks', 'retrieval-SfM-30k', 'retrievalSfM30k-siamac-alex.mat');
% network_file = fullfile(data_root, 'networks', 'retrieval-SfM-30k', 'retrievalSfM30k-siamac-vgg.mat');

% Choose arXiv17 fine-tuned CNN network
% Choose TPAMI18 fine-tuned CNN network
% network_file = fullfile(data_root, 'networks', 'retrieval-SfM-30k', 'retrievalSfM30k-gem-alex.mat');
% network_file = fullfile(data_root, 'networks', 'retrieval-SfM-120k', 'retrievalSfM120k-gem-vgg.mat');
network_file = fullfile(data_root, 'networks', 'retrieval-SfM-120k', 'retrievalSfM120k-gem-resnet101.mat');
Expand Down Expand Up @@ -177,14 +177,32 @@
qvecsLw = whitenapply(qvecs, Lw.m, Lw.P); % apply whitening on query descriptors

fprintf('>> %s: Retrieval...\n', test_datasets{d});
% raw descriptors
sim = vecs'*qvecs;
[sim, ranks] = sort(sim, 'descend');
map = compute_map (ranks, cfg.gnd);
fprintf('>> %s: mAP = %.4f, without whiten\n', test_datasets{d}, map);
% with learned whitening
sim = vecsLw'*qvecsLw;
[sim, ranks] = sort(sim, 'descend');
map = compute_map (ranks, cfg.gnd);
fprintf('>> %s: mAP = %.4f, with whiten\n', test_datasets{d}, map);
if strcmp(test_datasets{d}, 'oxford5k') || strcmp(test_datasets{d}, 'paris6k')
% % raw descriptors
% sim = vecs'*qvecs;
% [sim, ranks] = sort(sim, 'descend');
% map = compute_map (ranks, cfg.gnd);
% fprintf('>> %s: mAP = %.4f, without whiten\n', test_datasets{d}, map);
% with learned whitening
sim = vecsLw'*qvecsLw;
[sim, ranks] = sort(sim, 'descend');
map = compute_map (ranks, cfg.gnd);
fprintf('>> %s: mAP = %.4f\n', test_datasets{d}, map);
elseif strcmp(test_datasets{d}, 'roxford5k') || strcmp(test_datasets{d}, 'rparis6k')
sim = vecsLw'*qvecsLw;
[sim, ranks] = sort(sim, 'descend');
% evaluate ranks
ks = [1, 5, 10];
% search for easy (E setup)
for i = 1:numel(cfg.gnd), gnd(i).ok = [cfg.gnd(i).easy]; gnd(i).junk = [cfg.gnd(i).junk, cfg.gnd(i).hard]; end
[mapE, apsE, mprE, prsE] = compute_map (ranks, gnd, ks);
% search for easy & hard (M setup)
for i = 1:numel(cfg.gnd), gnd(i).ok = [cfg.gnd(i).easy, cfg.gnd(i).hard]; gnd(i).junk = cfg.gnd(i).junk; end
[mapM, apsM, mprM, prsM] = compute_map (ranks, gnd, ks);
% search for hard (H setup)
for i = 1:numel(cfg.gnd), gnd(i).ok = [cfg.gnd(i).hard]; gnd(i).junk = [cfg.gnd(i).junk, cfg.gnd(i).easy]; end
[mapH, apsH, mprH, prsH] = compute_map (ranks, gnd, ks);
fprintf('>> %s: mAP E: %.2f, M: %.2f, H: %.2f\n', test_datasets{d}, 100*mapE, 100*mapM, 100*mapH);
fprintf('>> %s: mP@k[%d %d %d] E: [%.2f %.2f %.2f], M: [%.2f %.2f %.2f], H: [%.2f %.2f %.2f]\n', test_datasets{d}, ks(1), ks(2), ks(3), 100*mprE, 100*mprM, 100*mprH);
end
end
8 changes: 4 additions & 4 deletions examples/train_cnnimageretrieval.m
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
% TRAIN_CNNIMAGERETRIEVAL Code to train the methods presented in the papers:
% F. Radenovic, G. Tolias, O. Chum, Fine-tuning CNN Image Retrieval with No Human Annotation, arXiv 2017
% F. Radenovic, G. Tolias, O. Chum, Fine-tuning CNN Image Retrieval with No Human Annotation, TPAMI 2018
% F. Radenovic, G. Tolias, O. Chum, CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples, ECCV 2016
%
% Note: The method has been re-coded since our ECCV 2016 paper and minor differences in performance might appear.
Expand All @@ -20,7 +20,7 @@


%-------------------------------------------------------------------------------
% Reproduce training from arXiv17 paper: Fine-tuning CNN Image Retrieval ...
% Reproduce training from TPAMI18 paper: Fine-tuning CNN Image Retrieval ...
%-------------------------------------------------------------------------------

%% RESNET101 -------------------------------------------------------------------
Expand All @@ -35,7 +35,7 @@

% Set train parameters
% We provide 2 pools of training images comprising 30k and 120k images
% The latter is used in our arXiv17 paper
% The latter is used in our TPAMI18 paper
opts.train.dbPath = fullfile(data_root, 'train', 'dbs', 'retrieval-SfM-120k.mat');
opts.train.batchSize = 5;
opts.train.numEpochs = 30;
Expand Down Expand Up @@ -77,7 +77,7 @@

% Set train parameters
% We provide 2 pools of training images comprising 30k and 120k images
% The latter is used in our arXiv17 paper
% The latter is used in our TPAMI18 paper
opts.train.dbPath = fullfile(data_root, 'train', 'dbs', 'retrieval-SfM-120k.mat');
opts.train.batchSize = 5;
opts.train.numEpochs = 30;
Expand Down
158 changes: 102 additions & 56 deletions utils/compute_map.m
View file
Original file line number Diff line number Diff line change
@@ -1,67 +1,113 @@
function [map, aps] = compute_map (ranks, gnd, verbose)
% COMPUTE_MAP computes the mAP for a given set of returned results.
function [map, aps, pr, prs] = compute_map (ranks, gnd, kappas)
% COMPUTE_MAP This function computes the mAP for a given set of returned results.
%
% mAP = compute_map (RANKS, GND);
% Usage:
% map = compute_map (ranks, gnd)
% computes mean average precsion (map) only
%
% RANKS starts from 1, size(ranks) = db_size X #queries.
% Junk results (e.g., the query itself) should be declared in the gnd stuct array
% [map, aps, pr, prs] = compute_map (ranks, gnd, kappas)
% computes mean average precision (map), average precision (aps) for each query
% computes mean precision at kappas (pr), precision at kappas (prs) for each query
%
% Authors: G. Tolias, Y. Avrithis, H. Jegou. 2013.
% Notes:
% 1) ranks starts from 1, size(ranks) = db_size X #queries
% 2) The junk results (e.g., the query itself) should be declared in the gnd stuct array
% 3) If there are no positive images for some query, that query is excluded from the evaluation

if nargin < 3
verbose = false;
end
if ~exist('kappas'), kappas = 0; end

map = 0;
nq = numel (gnd); % number of queries
aps = zeros (nq, 1);
nq = numel (gnd); % number of queries
% init map and pr
map = 0;
aps = zeros (nq, 1);
pr = zeros(1, numel(kappas));
prs = zeros (nq, numel(kappas));
nempty = 0;

for i = 1:nq
qgnd = gnd(i).ok;

if isempty(qgnd) % no positive at all, skip from the average
aps (i) = nan;
prs (i, :) = nan;
nempty = nempty + 1;
continue;
end

if isfield (gnd(i), 'junk')
qgndj = gnd(i).junk;
else
qgndj = [];
end

% positions of positive and junk images
[~, pos] = intersect (ranks (:,i), qgnd);
[~, junk] = intersect (ranks (:,i), qgndj);

pos = sort(pos);
junk = sort(junk);

k = 0;
ij = 1;

if length (junk)
% decrease positions of positives based on the number of junk images appearing before them
ip = 1;
while ip <= numel (pos)

while ( ij <= length (junk) & pos (ip) > junk (ij) )
k = k + 1;
ij = ij + 1;
end

pos (ip) = pos (ip) - k;
ip = ip + 1;
end
end

% compute ap
ap = score_ap_from_ranks1 (pos, length (qgnd));
map = map + ap;
aps (i) = ap;

% compute precision@k
for j = 1:numel(kappas)
kq = min(max(pos), kappas(j));
prs(i, j) = numel(find(pos <= kq)) ./ kq;
end
pr = pr + prs(i, :);

for i = 1:nq
qgnd = gnd(i).ok;
if isfield (gnd(i), 'junk')
qgndj = gnd(i).junk;
else
qgndj = [];
end

% positions of positive and junk images
[~, pos] = intersect (ranks (:,i), qgnd);
[~, junk] = intersect (ranks (:,i), qgndj);

pos = sort(pos);
junk = sort(junk);

k = 0;
ij = 1;

if length (junk)
% decrease positions of positives based on the number of junk images appearing before them
ip = 1;
while ip <= numel (pos)

while ( ij <= length (junk) & pos (ip) > junk (ij) )
k = k + 1;
ij = ij + 1;
end

pos (ip) = pos (ip) - k;
ip = ip + 1;
end
end

ap = compute_ap (pos, length (qgnd));

if verbose
fprintf ('query no %d -> gnd = ', i);
fprintf ('%d ', qgnd);
fprintf ('\n tp ranks = ');
fprintf ('%d ', pos);
fprintf (' -> ap=%.3f\n', ap);
end
map = map + ap;
aps (i) = ap;

map = map / (nq-nempty);
pr = pr / (nq-nempty);

end
map = map / nq;


% This function computes the AP for a query
function ap = score_ap_from_ranks1 (ranks, nres)

% number of images ranked by the system
nimgranks = length (ranks);
ranks = ranks - 1;

% accumulate trapezoids in PR-plot
ap = 0;

recall_step = 1 / nres;

for j = 1:nimgranks
rank = ranks(j);

if rank == 0
precision_0 = 1.0;
else
precision_0 = (j - 1) / rank;
end

precision_1 = j / (rank + 1);
ap = ap + (precision_0 + precision_1) * recall_step / 2;
end

end
Loading

0 comments on commit 99b2e31

Please sign in to comment.