Open-Set Support Vector Machines (OSSVM) — Results

In this repository, we provide the raw results obtained from the experiments accomplished in the paper entitled "Open-Set Support Vector Machines". We also provide the scripts we have employed for performing the statistical tests and generating diagrams present in the paper. For generating those diagrams, it is required to have Python 3 and GNU make installed.

Raw data

Raw results are available in CSV_files directory following the CSV format. Each evaluation measure has its corresponding CSV file. Those CSV files contain the following fields:

Field	Description
lenacs	Number of available classes for training
dataset	Name of the dataset
classifier	Name of the classifier
experiment	A number in the range 1–10 specifying one of the 10 experiments per combination of number of available classes, dataset, and classifier
result	Specifying the result for the respective measure indicated in the filename

The evaluation measures employed are the following:

Repr.	Description	Acronym
NA	Normalized Accuracy
HNA	Harmonic Normalized Accuracy
OSFMmacro	Macro-Averaging Open-Set F-Measure
OSFMmicro	Micro-Averaging Open-Set F-Measure
FMmacro	Multiclass Macro-Averaging F-Measure
FMmicro	Multiclass Micro-Averaging F-Measure
AKS	Accuracy on Known Samples
AUS	Accuracy on Unknown Samples

Evaluated methods include:

Repr.	Description	Acronym	Reference
svm	Support Vector Machines		Chih-Chung Chang and Chih-Jen Lin. LIBSVM: A library for Support Vector Machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27:1–27:27, April 2011.
ocsvm	One-class SVM		Dimitrios A. Pritsos and Efstathios Stamatatos. Open-set classification for automated genre identification. In European Conference on Information Retrieval, volume 7814 of Lecture Notes in Computer Science, pages 207–217, Moscow, Russia, March 2013. Springer, Berlin, Heidelberg.
svdd	Support Vector Data Description		David M. J. Tax and Robert P. W. Duin. Support vector data description. Springer Machine Learning, 54(1):45–66, January 2004.
dbc	SVM Decision Boundary Carving		Filipe de Oliveira Costa, Ewerton Silva, Michael Eckmann, Walter J. Scheirer, and Anderson Rocha. Open set source camera attribution and device linking. Elsevier Pattern Recognition Letters, 39:92–101, April 2014.
onevset	1-vs-Set Machine		Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Towards open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7):1757–1772, July 2013.
wsvm	Weibull-calibrated SVM		Walter J. Scheirer, Lalit P. Jain, and Terrance E. Boult. Probability models for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(11):2317–2324, November 2014.
pisvm	SVM with Probability of Inclusion		Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. Multi-class open set recognition using probability of inclusion. In European Conference on Computer Vision, volume 8691, part III of Lecture Notes in Computer Science, pages 393–409, September 2014.
evm	Extreme Value Machine		Ethan M. Rudd, Lalit P. Jain, Walter J. Scheirer, and Terrance E. Boult. The Extreme Value Machine. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):762–768, March 2018.
ssvm	Open-Set SVM		Proposed method

The suffix O or C in the representation name of each of those classifiers indicate if it was trained by performing open- or closed-set grid search approach, respectively.

Note: As stated in our manuscript, in this work we have discussed only the experiments with open-set grid search, as previous work have shown it to be superior to closed-set grid search. For reproducibility of the results presented in the manuscript, one should employ only the data with suffix O although below we also present the Critical Difference’s diagrams for all methods implemented with closed-set grid search as well.

Generating Critical Difference’s diagrams

For generating the Critical Difference (CD) diagrams, Python 3 is required. The required packages are specified in requirements.txt. For installation of the required packages, run:

> pip3 install -r requirements.txt

For generating the CD diagrams, run:

> make plot_CD

It will generate the CD diagrams inside the CD_diagrams/ directory.

The main script responsible for generating the diagrams is plot_CD.py.

Critical Difference’s diagrams

Note: In our paper, we have included only the CD diagrams referring to the column “Open-set grid search” herein. This supplementary material contains CD diagrams also for AKS and AUS measures, which are not presented in the paper.

Open-set grid search	Closed-set grid search
Normalized Accuracy —
Harmonic Normalized Accuracy —
Macro-averaging Open-Set F-Measure —
Micro-averaging Open-Set F-Measure —
Multiclass Macro-averaging F-Measure —
Multiclass Micro-averaging F-Measure —
Normalized Accuracy —
Normalized Accuracy —