Consolidate all results

[jyaacoub]

• Model Architecture diagrams (drawio)
• Paper Figure outlines (drawio)

code

import logging
from matplotlib import pyplot as plt

from src.analysis.figures import prepare_df, fig_combined, custom_fig

dft = prepare_df('./results/model_media/model_stats.csv')
dfv = prepare_df('./results/model_media/model_stats_val.csv')

models = {
    'DG': ('nomsa', 'binary', 'original', 'binary'),
    'esm': ('ESM', 'binary', 'original', 'binary'), # esm model
    'aflow': ('nomsa', 'aflow', 'original', 'binary'),
    # 'gvpP': ('gvp', 'binary', 'original', 'binary'),
    'gvpL': ('nomsa', 'binary', 'gvp', 'binary'),
    # 'aflow_ring3': ('nomsa', 'aflow_ring3', 'original', 'binary'),
    'gvpL_aflow': ('nomsa', 'aflow', 'gvp', 'binary'),
    # 'gvpl_esm':('ESM', 'binary', 'gvp', 'binary'),
    # 'gvpL_aflow_rng3': ('nomsa', 'aflow_ring3', 'gvp', 'binary'),
    #GVPL_ESMM_davis3D_nomsaF_aflowE_48B_0.00010636872718329864LR_0.23282479481785903D_2000E_gvpLF_binaryLE
    # 'gvpl_esm_aflow': ('ESM', 'aflow', 'gvp', 'binary'),
}

fig, axes = fig_combined(dft, datasets=['davis', 'kiba', 'PDBbind'], fig_callable=custom_fig,
            models=models, metrics=['cindex', 'mse'],
            fig_scale=(10,5), add_stats=True, title_postfix=" test set performance", box=True)
plt.xticks(rotation=45)

# fig, axes = fig_combined(dfv, datasets=['davis'], fig_callable=custom_fig,
#             models=models, metrics=['cindex', 'mse'],
#             fig_scale=(10,5), add_stats=True, title_postfix=" validation set performance", box=True, fold_labels=True)
# plt.xticks(rotation=45)

Final models - these are the ones we will show in the paper.

    models = {
        'DG': ('nomsa', 'binary', 'original', 'binary'),
        'esm': ('ESM', 'binary', 'original', 'binary'), # esm model
        'aflow': ('nomsa', 'aflow', 'original', 'binary'),
        # 'gvpP': ('gvp', 'binary', 'original', 'binary'),
        'gvpL': ('nomsa', 'binary', 'gvp', 'binary'),
        # 'aflow_ring3': ('nomsa', 'aflow_ring3', 'original', 'binary'),
        'gvpL_aflow': ('nomsa', 'aflow', 'gvp', 'binary'),
        # 'gvpl_esm':('ESM', 'binary', 'gvp', 'binary'),
        # 'gvpL_aflow_rng3': ('nomsa', 'aflow_ring3', 'gvp', 'binary'),
        #GVPL_ESMM_davis3D_nomsaF_aflowE_48B_0.00010636872718329864LR_0.23282479481785903D_2000E_gvpLF_binaryLE
        # 'gvpl_esm_aflow': ('ESM', 'aflow', 'gvp', 'binary'),
    }

[jyaacoub]

FIG 1

DATASET INFO

TABLE COUNTS

FULL TABLE COUNTS:

 | Dataset   |   Protein |   Compound |  Total Binding Entities |
 |-----------|-----------|------------|-------------------------|
 | davis     |       442 |         68 |                   30056 |
 | kiba      |       229 |       2111 |                  118254 |
 | pdbbind   |      3889 |      12639 |                   19443 |

USED TABLE COUNTS:

Due to memory limitations a couple records were excluded from our runs this is the full count that were actually used.

   Dataset  Protein  Compound  Total Binding Entities
0    davis      439        68                   29852
1     kiba      226      2111                  117590
2  pdbbind     3785     10950                   16265

SEQUENCE LENGTH DISTRIBUTION

non-overlayed or normalized plot

normalized and overlayed plot

MODEL RESULTS

All (except pocket) results - 2x3 - MSE and Cindex

Stratified with pocket results

• see below additional figures

[jyaacoub]

FIG 3 - Platinum Dataset

DATASET INFO

TABLE COUNTS

	Unique protein sequence counts: 860
	            Unique protein IDs: 361
	          Unique ligand counts: 197
	                 Total records: 1962

Distribution for the number of mutations per protein

pkd distributions (stratified by # of mutations)

Model results

Stratified figures below are better at showing the model results #147 (comment)

Raw predictive performance

This plot shows the ability for the model to just predict the pkd given the protein sequence and ligand SMILES

• Davis is terrible - no better than random
• Kiba is better
• PDBbind is best with cindex > 0.70

Delta predictive performance

Instead of looking at absolute predictive performance this plot show how well the model is able to predict the delta between a mutated and unmutated sequence.

• Here all 3 are terrible - no better than random.

[jyaacoub]

Additional figures - stratified performance results

MODEL TEST DATASETS

pocket vs full protein representation

These are the only ones we trained with the pocket representation:

NOTE: I dont think its worth training the remaining 11 model configurations (total of 55 models to train - 1 for each of the 5 folds - this would take at least a couple months at my current pace)

Platinum:

RAW model predictive performance (all vs only mutated vs only wildtype)

In pocket vs out of pocket mutation differences

RAW predictive performance:

DELTA predictive performance:

single vs multiple mutations

RAW predictive performance:

DELTA predictive performance: