How to figure out cached function memory by each function?

[bakerwy]

I am using Vetorization approach for the speed.

But it consumes huge memory as you know.

So I would like to check out memory each function holding. or memory information when the model executed.

Is there any way to do that? I tried memory profiler but I don't know how to apply this to modelx model.

Thanks in advance.

[fumitoh]

If the values of your model are mostly numpy array or pandas DataFrame or Series, then the code below should be fine.

import sys
import numpy as np
import pandas as pd


def get_sizes(space):

    result = {}
    
    for name, cells in space.cells.items():
        
        size = 0
        
        for val in cells.values():            
        
            if isinstance(val, np.ndarray):
                # For NumPy arrays
                size += val.nbytes
            elif isinstance(val, pd.DataFrame):
                # For pandas DataFrames
                size += val.memory_usage(deep=True).sum()
            elif isinstance(val, pd.Series):
                # For pandas Series
                size += val.memory_usage(deep=True)
            else:
                # Handle other types if necessary
                size += sys.getsizeof(val)
        
        if size:
            result[name] = size
    
    return dict(sorted(result.items(), key=lambda item: item[1], reverse=True))

I tested the above with BasicTerm_S in lifelib. The result shows the sizes of the values in bytes contained in the cells of a given space from the largest to the smallest.

>>> import modelx as mx

>>> m = mx.read_model("BasicTerm_S")

>>> m.Projection.result_pv()
              Premiums       Claims    Expenses  Commissions  Net Cashflow
PV         8252.085856  5501.194898  755.366026  1084.604270    910.920661
% Premium     1.000000     0.666643    0.091536     0.131434      0.110387

>>> get_size(m.Projection)
{'age': 3872,
 'claim_pp': 3872,
 'claims': 3872,
 'mort_rate': 3872,
 'mort_rate_mth': 3872,
 'pols_death': 3872,
 'premiums': 3872,
 'pols_if': 3868,
 'expenses': 3864,
 'pols_lapse': 3840,
 'commissions': 3436,
 'duration': 3388,
 'pols_maturity': 3364,
 'inflation_factor': 2904,
 'lapse_rate': 2880,
 'disc_factors': 968,
 'disc_rate_mth': 968,
 'model_point': 724,
 'result_pv': 205,
 'age_at_entry': 32,
 'net_premium_pp': 32,
 'policy_term': 32,
 'premium_pp': 32,
 'proj_len': 32,
 'pv_claims': 32,
 'pv_commissions': 32,
 'pv_expenses': 32,
 'pv_net_cf': 32,
 'pv_pols_if': 32,
 'pv_premiums': 32,
 'sum_assured': 32,
 'expense_acq': 28,
 'expense_maint': 28,
 'pols_if_init': 28,
 'inflation_rate': 24,
 'loading_prem': 24}

I would also consider changing the model so that it contains less values, using uncached Cells or user module.

[bakerwy]

def get_sizes(space):

    result = {}
    
    for name, refs in space.refs.items():
        
        size = 0
        
        for val in refs.values():            
        
            if isinstance(val, np.ndarray):
                # For NumPy arrays
                size += val.nbytes
            elif isinstance(val, pd.DataFrame):
                # For pandas DataFrames
                size += val.memory_usage(deep=True).sum()
            elif isinstance(val, pd.Series):
                # For pandas Series
                size += val.memory_usage(deep=True)
            else:
                # Handle other types if necessary
                size += sys.getsizeof(val)
        
        if size:
            result[name] = size
    
    return dict(sorted(result.items(), key=lambda item: item[1], reverse=True))

AttributeError: 'ItemsView' object has no attribute 'values'

I tried the function above, it said error below. maybe refs requires other attribute?

[fumitoh]

How about:

def get_sizes(space):

    result = {}
    
    for name, val in space.refs.items():            
        size = 0
        if isinstance(val, np.ndarray):
            # For NumPy arrays
            size = val.nbytes
        elif isinstance(val, pd.DataFrame):
            # For pandas DataFrames
            size = val.memory_usage(deep=True).sum()
        elif isinstance(val, pd.Series):
            # For pandas Series
            size = val.memory_usage(deep=True)
        else:
            # Handle other types if necessary
            size = sys.getsizeof(val)
        
        if size:
            result[name] = size
    
    return dict(sorted(result.items(), key=lambda item: item[1], reverse=True))

[bakerwy]

Hi, fumitoh. Thanks alot!

I have one more question...

I tested with the library tracemalloc for memory checking.
And result said the most of memory is used by numpy as attached below.
(Tested only 100 model points since tracemalloc took so long time for complex model)

Memory_Info_attach.txt

When model starts, the memory goes up very quickly.
After run finished, the memory goes down gradually to the amount of memory each function holds.

Do you have any advise to this?

As the functions are cached, I think numpy can use memory less and doesn't need to use that much memory...

[fumitoh]

It's up to Python when to delete unused objects, so that's why the memory usage goes down gradually. If you executte gc.collect() right after the run, then you should see the usage going down sharply.

https://docs.python.org/3/library/gc.html#gc.collect

Have you tried to export the model and see if the memory usage is still large?

[bakerwy]

Thanks. Yes I also tried exported model, it consumed also large memory.
I have an idea, and I hope you can take look at it.

Thanks