Libretro Database

The github repository for databases used by RetroArch. The repository contains several different kinds of files:

• Game information database files.
  • .dat files in the clrmamepro DAT format, from many sources and across many categories of metadata. The system of dats is multifaceted: alternative or additional sources can be easily added and maintained in a self-contained constituent, some dats may overlap in the games they cover (see precedence), and some dats cover an exclusive niche of games or attributes.
  • .rdb files used by RetroArch, compiled and amalgamated from the .dat files. RetroArch Database format (no relation to Redis .RDB files) accommodates RetroArch's wide range of hardware/OS platforms.
• Cheat code .cht files. These are game-specific, remain in plain text, and are used as-is by RetroArch if manually selected by the user (see Cheat Code Documentation). The repository contains one folder for each system (unlike dats), and multiple different cheat files may exist for the same game.
• Admin/management scripts and files.

RetroArch's Usage of the Database

Libretro databases allow RetroArch to provide several automated cataloging functions:

• Validation. Reject or accept files when using the Import Scanner / Playlist Generator based on whether the ROM checksum matches the checksum of a known verified completely intact (aka "properly dumped") file.
• Game Naming. Assign a definitive and uniform display name for each game in a playlist regardless of filename.
• Thumbnail Images. Download and display thumbnail images for games based on the uniform name assigned by the database, regardless of filename. (Thumbnails are not directly assigned by the database or by checksum association, but as a secondary effect of databased game name assignment if a matching thumbnail is available on the server. Also see: Flexible Name Matching Algorithm.)
• Category Search ("Explore"). Allows the user to find/view games that match selected criteria, e.g. by Developer, Release Year, Genre, and other attributes/metadata.
• Per-Game Information View. Provide an in-app viewable informational screen for each game (Game > Information > Database Entry).

Key Field

The key field for matching varies by console typical file size (i.e. original media type).

• CRC checksum for systems with smaller file sizes, e.g. games before the advent of disc-based consoles.
• Serial Number for larger files like disc-based games, to avoid computing checksums on large files. Found within the ROM file. The serial is not metadata but encoded within the game's binary data, which is scanned (in applicable cases) as a byte array by RetroArch.

CRC and serial also serve as RetroArch's primary index.

Current build script code can be viewed as a reference for which type of key field RetroArch uses for each console system.

Precedence

Databases earlier in the list have precedence over items later in the list. E.g. definitions in /dat will over-ride /metadat in the final .rdb compile if any info conflicts for the same game (i.e. for the same key field).

Fields Specified in Game Information Databases

Database entries for games at minimum specify 1) a game's name, i.e. the display name that RetroArch will assign in playlists and 2) key field data for matching/indexing and for identifying a file. Further optional metadata may appear. For reasons of informational completeness, future-proofing, and compatibility outside RetroArch, databases contain checksum and cryptographic hashes regardless of the key used for matching.

Example of database entry within metadat/no-intro/Atari - 2600.dat for the European region version of Asteroids:

game (
	name "Asteroids (Europe)"
	description "Asteroids (Europe)"
	region "Europe"
	rom ( name "Asteroids (Europe).a26" size 8192 crc 0A2F8288 md5 8CF0D333BBE85B9549B1E6B1E2390B8D sha1 1CB8F057ACAD6DC65FEF07D3202088FF4AE355CD )
)

If other Atari - 2600.dat files exist in the repository and contain further metadata for the same crc, the data would be compiled together in the .rdb. For example, metadat/developer/Atari - 2600.dat would confer developer "Atari" to the above data.

Header Guidelines for DATs

The description " " and comment " " fields within a libretro dat's clrmamepro ( ) header should be used to clarify the origin, source, and/or purpose of the data and file. The description and comment header fields are intended for documentation purposes, are ignored by RetroArch, and can be freely changed without issue. For example, if a .dat includes 3rd party upstream data processed through a github author's build/scrape script(s), the comment and description (or other appropriate header fields) should contain information about both those aspects of the dat's origin. If the .dat file is meant to cover a particular niche of data, the description field should explain it.

The name field (and filename) of a .dat file header should match the database field that is specified in the .info file for the cores that use it (often but not always Manufacturer - Systemname or similar). The description field should be descriptive and informative about the .dat file's origin and purpose.

Repository Contents

The non-exhaustive list below serves as a guide to various folders/files in the repository.

• cht Cheat codes to various games, collected from any available source on the web including by manual contributions from users who have used RetroArch's built-in memory address/value search feature to construct new cheat codes.
• cursors Methods to query playlists.
• dat Customized DAT files maintained by the libretro team, including:
  • Subset data coverage for games or variants that do/did not have contemporary documentation by upstream database groups, e.g. Virtual Console variants of SNES games, fan translations of NEC PC-98 games, and a superceded squib for PSP Minis.
  • Game data for monolithic non-generalized cores, e.g. Cave Story, Doom, Quake, etc.
  • Data adapted from upstream sources that cover a relatively small number of systems and can therefore can be housed together in a single repository folder without conflict, e.g. DOS, ScummVM, and GameTDB coverage of GameCube and Wii data. (Though many dats from upstream groups reside in metadat.)
• metadat Several principal third-party DATs (e.g. No-Intro, Redump, MAME, TOSEC) that each cover a large number of systems and therefore require their own folders in the repository, plus various collections of metadata (some of which may be deprecated). Examples:
  • bbfc British Board of Film Classification's ratings for age-appropriateness.
  • elspa Age-appropriateness/content ratings from the Entertainment and Leisure Software Publishers Association aka the Association for UK Interactive Entertainment ("Ukie").
  • fbneo-split Includes an XML database (sourced from Logiqx's DTD ROM Management) for special use in arcade ROM scanning: it must be manually selected by the user when running a Manual Scan, it defines the component files within each ROM archive, and is not part of the .rdb compile. Also contains a typical .dat.
  • mame Similar to fbneo-split above.
  • hacks Data for modified (or "hacked") versions of commercially released games. Many of these data are set by direct manual commits on the Libretro Github.
  • homebrew Data for non-officially-published games created by independent creators/programmers.
  • libretro-dats Ad hoc databases for items that were/are not covered by upstream database groups. Currently includes fan translations of SNES games, and an additional FDS dat that may be redundant with other sources.
  • no-intro Bulk import from upstream No-Intro databases. Generally non-disc-based systems.
  • redump Bulk import from upstream Redump databases. Generally disc-based systems.
  • tosec Bulk import from upstream TOSEC databases. TOSEC data overlaps with and goes beyond other data sets (No-Intro, Redump), but has lower precedence in libretro and so generally serves as a secondary stopgap.
  • And more
• rdb The compiled RetroArch database files
• scripts Various scripts that are used to maintain the database files

Pre-emptive Databases

Some databases are maintained even if RetroArch currently has no core for the games/system, e.g. GP32, Vita, Original Xbox, and PS3.

Sources

Many source databases are in use as listed below. The table focusses on the 3rd party sources that predominantly cover each specific console library, but other/multiple sources including manual github contributions are maintained and all are compiled together in the final .rdb files (see Repository Contents and each dat's github History for details). ">" signs below indicate the precedence order when multiple sources overlap for the same subset of games/data.

System	Source	Repository
Amstrad - CPC	Clean CPC DB > TOSEC	libretro-dats
Amstrad - GX4000	TOSEC	libretro-dats
Arduboy Inc - Arduboy	No-Intro	libretro-dats
Atari - 2600	No-Intro	libretro-dats
Atari - 5200	No-Intro	libretro-dats
Atari - 7800	No-Intro	libretro-dats
Atari - 8-bit	No-Intro > TOSEC	libretro-dats
Atari - Jaguar	No-Intro	libretro-dats
Atari - Lynx	No-Intro	libretro-dats
Atari - ST	No-Intro	libretro-dats
Atomiswave
Bandai - WonderSwan	No-Intro	libretro-dats
Bandai - WonderSwan Color	No-Intro	libretro-dats
Cannonball
Casio - Loopy	No-Intro	libretro-dats
Casio - PV-1000	No-Intro	libretro-dats
Cave Story	CaveStory.org (English or Japanese)
ChaiLove
CHIP-8
Coleco - ColecoVision	No-Intro	libretro-dats
Commodore - 64	No-Intro (Note: cartridges, tapes and Preservation Project disk images)	libretro-dats
Commodore - Amiga	WHDLoad > No-Intro > TOSEC	libretro-dats
Commodore - CD32	Redump > TOSEC	libretro-dats
Commodore - CDTV	Redump > TOSEC	libretro-dats
Commodore - PET	TOSEC	libretro-dats
Commodore - Plus-4	No-Intro (Note: cartridges only)	libretro-dats
Commodore - VIC-20	No-Intro (Note: cartridges only)	libretro-dats
Dinothawr
DOS	Total DOS Collection	libretro-database-dos
Emerson - Arcadia 2001	No-Intro	libretro-dats
Entex - Adventure Vision	No-Intro	libretro-dats
Epoch - Super Cassette Vision	No-Intro	libretro-dats
Fairchild - Channel F	No-Intro	libretro-dats
FinalBurn Neo - Arcade Games 1.0.0.03	FBNeo/dats
Flashback
Funtech - Super Acan	No-Intro	libretro-dats
GamePark - GP32	No-Intro	libretro-dats
GCE - Vectrex	No-Intro	libretro-dats
Handheld Electronic Game
Hartung - Game Master	No-Intro	libretro-dats
Infocom - Z-Machine	TOSEC	libretro-dats
Jump 'n Bump
LeapFrog - Leapster Learning Game System	No-Intro	libretro-dats
LowRes NX
Id Software - Doom	Unknown
Id Software - Quake	Unknown
Id Software - Quake II	Unknown
Id Software - Quake III	Unknown
Lutro	Unknown
MAME	MAME 0.37b5, 0.78, 0.139, 0.159
Magnavox - Odyssey2	No-Intro	libretro-dats
Mattel - Intellivision	No-Intro > TOSEC	libretro-dats
Microsoft - MSX	No-Intro	libretro-dats
Microsoft - MSX 2	No-Intro	libretro-dats
Microsoft - Xbox	Redump (Note: there is no libretro emulator)	libretro-dats
MicroW8
MrBoom
NEC - PC-8001 - PC-8801.rdb	TOSEC	libretro-dats
NEC - PC-98	Redump > TOSEC	libretro-dats
NEC - PC Engine - TurboGrafx 16	No-Intro	libretro-dats
NEC - PC Engine SuperGrafx	No-Intro	libretro-dats
NEC - PC Engine CD - TurboGrafx-CD	Redump > TOSEC	libretro-dats
NEC - PC-FX	Redump > TOSEC	libretro-dats
Nintendo - e-Reader	No-Intro	libretro-dats
Nintendo - Family Computer Disk System	No-Intro	libretro-dats
Nintendo - Game Boy	No-Intro	libretro-dats
Nintendo - Game Boy Advance	No-Intro	libretro-dats
Nintendo - Game Boy Color	No-Intro	libretro-dats
Nintendo - GameCube	GameTDB > Redump > TOSEC	libretro-database-gametdb libretro-dats
Nintendo - Nintendo 3DS	No-Intro	libretro-dats
Nintendo - Nintendo 64	No-Intro	libretro-dats
Nintendo - Nintendo 64 DD	No-Intro	libretro-dats
Nintendo - Nintendo DS	No-Intro	libretro-dats
Nintendo - Nintendo DSi	No-Intro	libretro-dats
Nintendo - Nintendo Entertainment System	No-Intro (iNES 1.0 headered, NES 2.0 headered, headerless)	libretro-dats
Nintendo - Pokemon Mini	No-Intro	libretro-dats
Nintendo - Satellaview	No-Intro	libretro-dats
Nintendo - Sufami Turbo	No-Intro	libretro-dats
Nintendo - Super Nintendo Entertainment System	No-Intro	libretro-dats
Nintendo - Virtual Boy	No-Intro	libretro-dats
Nintendo - Wii	GameTDB > Redump	libretro-database-gametdb libretro-dats
Nintendo - Wii (Digital)	No-Intro	libretro-dats
Philips - CD-i	Redump	libretro-dats
Phillips - Videopac+	No-Intro	libretro-dats
PuzzleScript
RCA - Studio II	No-Intro	libretro-dats
Rick Dangerous
RPG Maker
ScummVM	Gruby's ScummVM Adventure Pack	libretro-database-scummvm
Sega - 32X	No-Intro	libretro-dats
Sega - Dreamcast	Redump > TOSEC	libretro-dats
Sega - Game Gear	No-Intro	libretro-dats
Sega - Master System - Mark III	No-Intro	libretro-dats
Sega - Mega CD - Sega CD	Redump > TOSEC	libretro-dats
Sega - Mega Drive - Genesis	No-Intro	libretro-dats
Sega - Naomi	Redump	libretro-dats
Sega - Naomi 2	Redump	libretro-dats
Sega - PICO	No-Intro	libretro-dats
Sega - Saturn	Redump > TOSEC	libretro-dats
Sega - SG-1000	No-Intro	libretro-dats
Sharp - X1	No-Intro > TOSEC	libretro-dats
Sharp - X68000	No-Intro > TOSEC	libretro-dats
Sinclair - ZX 81	TOSEC
Sinclair - ZX Spectrum	World of Spectrum > TOSEC	libretro-dats
Sinclair - ZX Spectrum +3	No-Intro	libretro-dats
SNK - Neo Geo CD	Redump	libretro-dats
SNK - Neo Geo Pocket	No-Intro	libretro-dats
SNK - Neo Geo Pocket Color	No-Intro	libretro-dats
Sony - PlayStation	Redump > TOSEC	libretro-dats
Sony - PlayStation 2	Redump	libretro-dats
Sony - PlayStation 3	GameTDB > Redump (Note: there is no libretro emulator)	libretro-database-gametdb libretro-dats
Sony - PlayStation 3 (PSN)	No-Intro (Note: there is no libretro emulator)	libretro-dats
Sony - PlayStation Portable	No-Intro	libretro-dats
Sony - PlayStation Portable (PSN)	No-Intro > Redump	libretro-dats
Sony - PlayStation Vita	No-Intro (Note: there is no libretro emulator)	libretro-dats
Spectravideo - SVI-318 - SVI-328	TOSEC	libretro-dats
The 3DO Company - 3DO	Redump > TOSEC	libretro-dats
Thomson - MO/TO	TOSEC
TIC-80	tic80.com	libretro-database-tic80
Tiger - Game.com	No-Intro	libretro-dats
Tomb Raider
Uzebox	Unknown
Vircon32
VTech - CreatiVision	No-Intro	libretro-dats
VTech - V.Smile	No-Intro	libretro-dats
WASM-4		libretro-database-wasm4
Watara - Supervision	No-Intro	libretro-dats
Wolfenstein 3D

Maintenance / Technical Usage

Building

To build a complete set of RDB files for RetroArch or to generate a single RDB file, see RetroArch/libretro-db/README.md.

Alternatively, you can run the following command to rebuild all the RDBs locally:

make build

Testing

Make sure filenames are Windows file system compatible, and are not too long (eg. ecryptfs limits filenames to 143 characters)...

find -exec basename '{}' ';' | egrep '^.{144,}$'

Adding A New Database

• Create new metadata/nameofnewdatabasefolder and appropriately named system .dat file(s) e.g. Sony - PlayStation.dat with new data
• Add the new entry to libretro-build-database.sh
• Run make build to build the RDB files
• New types for RetroArch's Explore tab require updates to RetroArch code.

Contributions

Small-Scale Corrections

A vast majority of the database's game information originates from routine imports from upstream data groups (No-Intro, Redump, TOSEC, GameTDB, etc). In cases where the .dat for the entry at issue originates from an upstream group, best practice is for a contributor to go through the channels/process of that group. Upstream changes made by the database groups will eventually be imported to the Libretro databases. A seemingly helpful "fix" to Libretro's copy of the database would be overwritten and lost by the next import from upstream.

In cases where the .dat in question is created and maintained by Libretro or does not receive bulk over-writes, github contributions are accepted. Refer to the repository contents list above and to github Histories for information about which libretro databases are applicable for github contributions.

Folder Structure Revisions

The build script specifies exact .dat files and folders in the repository, therefore organizational housekeeping revisions to the file/folder structure (e.g. combining two metadata fragments into one unified folder and file) require corresponding revisions in the build script.

Large-Scale Additions

See Adding New Database.

Databases and RetroArch Thumbnails

Currently there is no automatic process for updating libretro thumbnail repository image filenames based on game name updates in databases. RetroArch uses databases to assign a game name based on a game file's checksum (or other key), but thumbnails are only assigned if the thumbnail server image filename matches the game name or the ROM filename (with some flexibility). To help fix a thumbnail, for example in a case where the database game name has been definitively/correctly updated in a way that no longer matches the repository thumbnail name, follow the Thumbnail Repository Readme and How to Contribute to Thumbnails Guide.

Integrations

There are a few tools out there that allow integration with libretro's database.

• RetroArch CleanNaming: Python script that will rename your roms