MergeVCFs step does not generate index in some cases

[pbilling]

Issue seems to due to VM running out of disk space. From a MergeVCF log:

[Thu Jun 25 08:23:53 UTC 2020] picard.vcf.MergeVcfs done. Elapsed time: 89.52 minutes.
Runtime.totalMemory()=2027290624
To get help, see http://broadinstitute.github.io/picard/index.html#GettingHelp
Exception in thread "main" htsjdk.samtools.util.RuntimeIOException: Write error; BinaryCodec in writemode; file: /cromwell_root/SHIP5536850.g.vcf.gz
	at htsjdk.samtools.util.BinaryCodec.writeBytes(BinaryCodec.java:222)
	at htsjdk.samtools.util.BlockCompressedOutputStream.writeGzipBlock(BlockCompressedOutputStream.java:389)
	at htsjdk.samtools.util.BlockCompressedOutputStream.deflateBlock(BlockCompressedOutputStream.java:352)
	at htsjdk.samtools.util.BlockCompressedOutputStream.write(BlockCompressedOutputStream.java:251)tee: /cromwell_root/stderr: No space left on device

	at htsjdk.samtools.util.BlockCompressedOutputStream.write(BlockCompressedOutputStream.java:228)
	at htsjdk.variant.variantcontext.writer.VCFWriter.writeAndResetBuffer(VCFWriter.java:128)
	at htsjdk.variant.variantcontext.writer.VCFWriter.add(VCFWriter.java:227)
	at picard.vcf.MergeVcfs.doWork(MergeVcfs.java:154)
	at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:268)
	at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:98)
	at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:108)
Caused by: java.io.IOException: No space left on device

Comparison of GVCF sizes indicates that failed steps are handling larger GVCFs than successful ones.

[pbilling]

Looks like MergeVCFs task is currently using 30 GB disk, recommend increasing disk size.

# Combine multiple VCFs or GVCFs from scattered HaplotypeCaller runs
task MergeVCFs {
  Array[File] input_vcfs
  Array[File] input_vcfs_indexes
  String output_vcf_name
  Int preemptible_tries

  # Using MergeVcfs instead of GatherVcfs so we can create indices
  # See https://github.com/broadinstitute/picard/issues/789 for relevant GatherVcfs ticket
  command {
    java -Xms2000m -jar /usr/gitc/picard.jar \
      MergeVcfs \
      INPUT=${sep=' INPUT=' input_vcfs} \
      OUTPUT=${output_vcf_name}
  }
  runtime {
    docker: "us.gcr.io/broad-gotc-prod/genomes-in-the-cloud:2.3.2-1510681135"
    preemptible: preemptible_tries
    memory: "3 GB"
    disks: "local-disk 30 HDD"
  }
  output {
    File output_vcf = "${output_vcf_name}"
    File output_vcf_index = "${output_vcf_name}.tbi"
  }
}