VG Clustering

[KennethEnevoldsen]

No description provided.

[KennethEnevoldsen]

@Muennighoff and @x-tabdeveloping the VG_summarization has multiple classes pr article:

dataset[1:10]["classes"]
#['nyheter,utenriks', 'sport,fotball', 'nyheter,innenriks', 'nyheter,utenriks', 'sport,ski,langrenn', 'sport', 'sport,fotball', 'nyheter,utenriks', 'sport,norges-idrettsforbund']

Most of them are finer grained versions. E.g. 'nyheter,innenriks' (news, domestic) vs 'nyheter,utenriks' (news, international).

In total, there are 18763 documents in the test set. 1147 unique labels, 15 main categories (the first one), 220 if we include the second, and 1131 if we include three.

Unless you guys have anything to add I will go for the case with only 15 categories.

[x-tabdeveloping]

Sounds very reasonable, I think we should roll with that.

[KennethEnevoldsen]

Clustering added in #96

[KennethEnevoldsen]

Move VG STS to a new issues