SLiM Exception

[raywray]

Hi all -- I am getting the following warnings and errors:

/anaconda3/envs/coalminer_examples_env/lib/python3.12/site-packages/stdpopsim/slim_engine.py:1523: SLiMScalingFactorWarning: You're using a scaling factor (10). This should give similar results for many situations, but is not equivalent, especially in the presence of selection. When using rescaling, you should be careful---do checks and compare results across different values of the scaling factor.
  warnings.warn(
Traceback (most recent call last):
  File "/home/raya/Documents/Projects/CoalMinerExamples/simulation_scripts/python/generate_data_stdpopsim.py", line 95, in <module>
    generate_stdpopsim_data()
  File "/home/raya/Documents/Projects/CoalMinerExamples/simulation_scripts/python/generate_data_stdpopsim.py", line 91, in generate_stdpopsim_data
    get_dro_mel_data()
  File "/home/raya/Documents/Projects/CoalMinerExamples/simulation_scripts/python/generate_data_stdpopsim.py", line 75, in get_dro_mel_data
    generate_data(species, model, contig, samples)
  File "/home/raya/Documents/Projects/CoalMinerExamples/simulation_scripts/python/generate_data_stdpopsim.py", line 22, in generate_data
    ts = engine.simulate(model, contig, samples, slim_scaling_factor=10)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/anaconda3/envs/coalminer_examples_env/lib/python3.12/site-packages/stdpopsim/slim_engine.py", line 1586, in simulate
    self._run_slim(
  File "/anaconda3/envs/coalminer_examples_env/lib/python3.12/site-packages/stdpopsim/slim_engine.py", line 1673, in _run_slim
    raise SLiMException(
stdpopsim.slim_engine.SLiMException: slim exited with code -9.

I'm not so worried about the warnings, but the error is confusing to me. I feel like this is something that is maybe within the stdpopsim package?

To recreate:

def generate_stdpopsim_data():
    get_dro_mel_data()
    
def get_dro_mel_data():
    print("**********************DRO MEL*************************")
    species = stdpopsim.get_species("DroMel")
    model = species.get_demographic_model("African3Epoch_1S16")
    contig = species.get_contig("chr2L", mutation_rate=model.mutation_rate)
    samples = {"AFR": 10}

    generate_data(species, model, contig, samples)

def generate_data(species, model, contig, samples):
    # define output libs
    stdpop_output_dir = "stdpop_output"
    vcf_dir = os.path.join(stdpop_output_dir, "vcf")
    slim_dir = os.path.join(stdpop_output_dir, "slim_scripts")

    utilities.create_directory(stdpop_output_dir)
    utilities.create_directory(vcf_dir)
    utilities.create_directory(slim_dir)
    
    # step 5: select engine
    engine = stdpopsim.get_engine("slim")
    # step 6: simulate
    ts = engine.simulate(model, contig, samples, slim_scaling_factor=10)
    
    vcf_file_path = os.path.join(vcf_dir, f"{species.id}.vcf")
    slim_file_path = os.path.join(slim_dir, f"{species.id}.slim")
    
    # write slim script to file
    with open(slim_file_path, "w") as f:
        with redirect_stdout(f):
            ts = engine.simulate(
                model,
                contig,
                samples,
                slim_script=True,
                verbosity=2,
                slim_scaling_factor=10,
            )
    
    with open(vcf_file_path, "w") as vcf_file:
        ts.write_vcf(vcf_file, contig_id="0")

generate_stdpopsim_data()

Please let me know if I'm missing something obvious. I've tested it using different chromosomes for DroMel and still get the same error. Thanks!

[petrelharp]

Hello! Well, the -9 indicates something went wrong with SLiM; I forget what (maybe out of memory?). Anyhow, here's how I'd figure it out. First, here's a more-minimal script:

import sys
import stdpopsim
species = stdpopsim.get_species("DroMel")
model = species.get_demographic_model("African3Epoch_1S16")
contig = species.get_contig("chr2L", mutation_rate=model.mutation_rate)
samples = {"AFR": 10}

engine = stdpopsim.get_engine("slim")

# write slim script to file
with open("test.slim", "w") as f:
    sys.stdout = f
    ts = engine.simulate(
        model,
        contig,
        samples,
        slim_script=True,
        verbosity=2,
        slim_scaling_factor=1000,
    )

ts = engine.simulate(model, contig, samples, slim_scaling_factor=1000)

Note that I've cranked up the scaling factor, so it doesn't take forever (however, it's still not quick!). Questions:

1. Does this run? If so - maybe the problem is indeed out of memory.
2. Whether it runs or not, the next thing to do is probably to run slim test.slim and see what goes wrong. You can even open it in SLiMgui if the problem is not apparent from the command line.
3. My first guess is that it's "out of memory", since the Drosophila sims can be Very Big, so monitor memory (and time) usage.

But, perhaps @bhaller knows what a return code of -9 means?

[raywray]

I forgot to mention that I get this error when running on both a linux-based local machine and remote cluster.

[petrelharp]

Ah - I know why it's using up so much memory with a relatively small simulation. This is because of rescaling - I'm running with a scaling factor of 1,000; so those 20 generations are supposed to be like 20,000 generations, and hence recombination rate is 1,000 times larger than normal. So, the tree sequence is necessarily big to record all those recombinations, and simplification is taking a while.

Phew - I thought something was wrong there.

I've still not been able to figure out why the return code is -9. My guess is still that it's something related to resources, which would explain why SLiM is not printing an error message. But, I haven't got the actual error message either - @raywray, how long did you have to run it for to get the error?

[bhaller]

TSK_ERR_NO_MEMORY is -2, so it isn't that. If you can get it in the debugger, setting a breakpoint on exit() might be all that is needed; you could maybe see who is calling that, and why.

[petrelharp]

There is no -9 error code from tskit either; but if it was a tskit error then SLiM would print out the error message anyhow.

[raywray]

@petrelharp I just tested it again and it ran for 1201.82 seconds before I got the -9 error.

utureWarning: Calling int on a single element Series is deprecated and will raise a TypeError in the future. Use int(ser.iloc[0]) instead
  time=int(extended_GF.time.head(1) - 1), rate=0
/anaconda3/envs/coalminer_sims_env/lib/python3.11/site-packages/stdpopsim/catalog/HomSap/demographic_models.py:161: FutureWarning: Calling int on a single element Series is deprecated and will raise a TypeError in the future. Use int(ser.iloc[0]) instead
  time=int(extended_GF.time.tail(1) + 1), rate=0
/anaconda3/envs/coalminer_sims_env/lib/python3.11/site-packages/stdpopsim/slim_engine.py:1523: SLiMScalingFactorWarning: You're using a scaling factor (10). This should give similar results for many situations, but is not equivalent, especially in the presence of selection. When using rescaling, you should be careful---do checks and compare results across different values of the scaling factor.
  warnings.warn(
An error occurred: slim exited with code -9.

Elapsed time: 1201.82 seconds