Skip to content

Commit

Permalink
Rebuild of the billing process. Updates to the TNRunner run, qc, and
Browse files Browse the repository at this point in the history
sample tracking apps.
  • Loading branch information
u0028003 committed Apr 8, 2024
1 parent 5a9aef7 commit 38e99de
Show file tree
Hide file tree
Showing 32 changed files with 2,063 additions and 225 deletions.
15 changes: 15 additions & 0 deletions .settings/org.eclipse.jdt.core.prefs
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
eclipse.preferences.version=1
org.eclipse.jdt.core.compiler.codegen.inlineJsrBytecode=enabled
org.eclipse.jdt.core.compiler.codegen.methodParameters=do not generate
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8
org.eclipse.jdt.core.compiler.codegen.unusedLocal=preserve
org.eclipse.jdt.core.compiler.compliance=1.8
org.eclipse.jdt.core.compiler.debug.lineNumber=generate
org.eclipse.jdt.core.compiler.debug.localVariable=generate
org.eclipse.jdt.core.compiler.debug.sourceFile=generate
org.eclipse.jdt.core.compiler.problem.assertIdentifier=error
org.eclipse.jdt.core.compiler.problem.enablePreviewFeatures=disabled
org.eclipse.jdt.core.compiler.problem.enumIdentifier=error
org.eclipse.jdt.core.compiler.problem.reportPreviewFeatures=warning
org.eclipse.jdt.core.compiler.release=disabled
org.eclipse.jdt.core.compiler.source=1.8
24 changes: 13 additions & 11 deletions Documentation/USeqDocumentation/cmdLnMenus.html
Original file line number Diff line number Diff line change
Expand Up @@ -517,7 +517,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="AggregateQCStats2"><pre>**************************************************************************************
** Aggregate QC Stats2: April 2022 **
** Aggregate QC Stats2: March 2024 **
**************************************************************************************
Parses and aggregates alignment quality statistics from log and json files produced by
the TNRunner2 DnaAlignQC and SampleConcordance workflows.
Expand All @@ -527,7 +527,7 @@ <H1>Command Line Menus</H1> <p>
-s Directory for saving the AggQC results.

Options:
-a Alignment log file match, defaults to '.+_AlignHg38.log'
-a Alignment log file match, defaults to '.+_Align.log'
-d Mark duplicates log file match, defaults to '.+_Markdup.log'
-r Read coverage json file match, defaults to '.+_UniObRC.json.gz'
-c Sample concordance json file match, defaults to '.+_SampleConcordance.json.gz'
Expand Down Expand Up @@ -870,7 +870,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="AvatarDataWrangler"><pre>**************************************************************************************
** Avatar Data Wrangler : March 2023 **
** Avatar Data Wrangler : Feb 2024 **
**************************************************************************************
Tool for assembling directories for TNRunner based on files provided by M2Gen via
download from DNAnexus. Handles patient datasets from different exome capture
Expand All @@ -893,9 +893,10 @@ <H1>Command Line Menus</H1> <p>
-t Directory to place temp files with PHI for Subject ID matching
-s Directory containing the SubjectMatchMaker 'currentRegistry_' file
-d Path to save a bash script for downloading the sequence read data.
-a Path to where the Aster datasets will be downloaded.

Example: java -jar ~/USeqApps/AvatarDataWrangler -r Resources/ -j AJobs -t SMM_PHI
-s ~/PHI/SmmRegistry/ -d dxDownloadCmds.sh
-s ~/PHI/SmmRegistry/ -d dxDownloadCmds.sh -a AsterDownloads

**************************************************************************************
</pre><br><p>
Expand Down Expand Up @@ -1465,7 +1466,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="CarisDataWrangler"><pre>**************************************************************************************
** Caris Data Wrangler : Aug 2022 **
** Caris Data Wrangler : Mar 2024 **
**************************************************************************************
The Caris Data Wrangler downloads complete patient datasets from an AWS bucket, parses
the xml test file for patient info, fetches/makes coreIds using the SubjectMatchMaker
Expand Down Expand Up @@ -1540,7 +1541,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="CBiBilling"><pre>**************************************************************************************
** CBI Billing: Sept 2023 **
** CBI Billing: Jan 2024 **
**************************************************************************************
Generates billing reports for the Cancer Bioinformatics Shared Resource.

Expand All @@ -1549,8 +1550,7 @@ <H1>Command Line Menus</H1> <p>
-m Path to the masterAccountInfo.xlsx spreadsheet updated from the prior month.
-a Path to the awsAccounts.xlsx spreadsheet.
-w Path to a dir containing the hourly and cloud 'WAF Tracking Schedule' xlsx files.
-c Path to a dir containing the cloud AWS Carahsoft xlsx expense reports. May be empty
if are none available.
-c If available, path to a dir with the cloud AWS Carahsoft xlsx expense reports.
-o Path to write the results.

Example: java -Xmx256M -jar pathTo/USeq/Apps/CBiBilling -j jiraTime.cvs -m
Expand Down Expand Up @@ -2792,7 +2792,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="JobCleaner"><pre>**************************************************************************************
** Job Cleaner : May 2022 **
** Job Cleaner : Jan 2024 **
**************************************************************************************
Zip archives particular folders, deletes particular files. Use to clean up analysis
result directories prior to cloud upload.
Expand All @@ -2801,10 +2801,12 @@ <H1>Command Line Menus</H1> <p>
-r Root directory to recursively look for files and folders.
-e File extensions and file names to delete, comma delimited, no spaces.
-n Directory names to zip archive and then delete, comma delimited, no spaces.
-m Create a merged zip archive for directories defined in -n that exist in the same
parent directory, e.g. LogsRunScripts.zip instead of Logs.zip and RunScripts.zip
-d Dry run, just list the files and directories.

Example: java -jar pathToUSeq/Apps/JobCleaner -d -n 'Logs,RunScripts' -r CJobs/ -e
'.tbi,.crai,.bai,COMPLETE'
'.tbi,.crai,.bai,COMPLETE' -m

**************************************************************************************
</pre><br><p>
Expand Down Expand Up @@ -4718,7 +4720,7 @@ <H1>Command Line Menus</H1> <p>
</pre><br><p>

<a name="ScanSeqs"><pre>**************************************************************************************
** Scan Seqs: July 2015 **
** Scan Seqs: Jan 2024 **
**************************************************************************************
Takes unshifted stranded chromosome specific PointData and uses a sliding window to
calculate several smoothed window statistics. These include a binomial p-value, a
Expand Down
Binary file modified Misc/JavaxComOrgInfoNetClasses.zip
Binary file not shown.
Binary file not shown.
28 changes: 28 additions & 0 deletions Source/edu/utah/billing/AwsAccountExpense.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
package edu.utah.billing;

import java.util.ArrayList;

public class AwsAccountExpense {

private String awsAccountNumber = null;
private float totalExpense = 0f;

public AwsAccountExpense (String number, float total) {
awsAccountNumber = number;
totalExpense = total;
}

public String getAwsAccountNumber() {
return awsAccountNumber;
}

public float getTotalExpense() {
return totalExpense;
}

public static float fetchTotalExpense(ArrayList<AwsAccountExpense> accounts) {
float total = 0f;
for (AwsAccountExpense aae: accounts) total+= aae.getTotalExpense();
return total;
}
}
20 changes: 14 additions & 6 deletions Source/edu/utah/billing/AwsXlsxAccountParser.java
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,30 @@
public class AwsXlsxAccountParser {

private TreeMap<String, String> awsAccountGroupName = new TreeMap<String, String>();
private ArrayList<String> missingAliases = new ArrayList<String>();

public AwsXlsxAccountParser(File xlsx, boolean debug) {
parseIt(xlsx);
public AwsXlsxAccountParser(File xlsx, boolean debug, HashMap<String, HashSet<String>> aliases) {
parseIt(xlsx, aliases);

if (debug) {
for (String s: awsAccountGroupName.keySet()) {
IO.pl(s+"\t"+awsAccountGroupName.get(s));
}
}

if (missingAliases.size()!=0) {
for (String a: missingAliases)IO.el("\tMissing entry in masterAccountInfo for "+a+" from "+xlsx.getName());
IO.el("\t\tCorrect and restart.\n");
System.exit(1);
}
}

public static void main(String[] args) {
AwsXlsxAccountParser p = new AwsXlsxAccountParser(new File ("/Users/u0028003/HCI/CoreAdmin/Billing/AllBillingReports/2023/6_BSR_June_2023/awsAccounts.xlsx"), true);
//AwsXlsxAccountParser p = new AwsXlsxAccountParser(new File ("/Users/u0028003/HCI/CoreAdmin/Billing/AllBillingReports/2023/6_BSR_June_2023/awsAccounts.xlsx"), true);

}

private void parseIt(File inputFile) {
private void parseIt(File inputFile, HashMap<String, HashSet<String>> aliases) {
try {

//Open up xlsx file
Expand All @@ -50,7 +57,7 @@ private void parseIt(File inputFile) {
int numRows = sheet.getPhysicalNumberOfRows();
for (int r = 0; r< numRows; r++) {
Row row = sheet.getRow(r);
if (row != null) addAccount(row);
if (row != null) addAccount(row, aliases);
}
} catch (Exception e) {
System.out.println("Aws Accounts xlsx file is not in the correct format, exiting");
Expand All @@ -59,7 +66,7 @@ private void parseIt(File inputFile) {
}
}

private void addAccount(Row row) {
private void addAccount(Row row, HashMap<String, HashSet<String>> aliases) {
int numCells = row.getLastCellNum()+1;
if (numCells < 2) return;

Expand All @@ -74,6 +81,7 @@ private void addAccount(Row row) {
String accountNumber = cell.toString().trim();
if (accountNumber.length()!=0) {
awsAccountGroupName.put(accountNumber, groupName);
if (aliases.containsKey(groupName) == false) missingAliases.add(groupName);
}
}
}
Expand Down
79 changes: 79 additions & 0 deletions Source/edu/utah/billing/AwsXlsxAccountParser2.java
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
package edu.utah.billing;

import java.io.File;
import java.io.IOException;
import java.util.TreeMap;
import org.apache.poi.ss.usermodel.Cell;
import org.apache.poi.ss.usermodel.Row;
import org.apache.poi.ss.usermodel.Sheet;
import org.apache.poi.ss.usermodel.Workbook;
import org.apache.poi.ss.usermodel.WorkbookFactory;
import util.gen.IO;


public class AwsXlsxAccountParser2 {

private TreeMap<String, String> awsAccountGroupName = new TreeMap<String, String>();

public AwsXlsxAccountParser2(File xlsx, boolean debug) {
parseIt(xlsx);

if (debug) {
for (String s: awsAccountGroupName.keySet()) {
IO.pl(s+"\t"+awsAccountGroupName.get(s));
}
}
}

public static void main(String[] args) {
//AwsXlsxAccountParser p = new AwsXlsxAccountParser(new File ("/Users/u0028003/HCI/CoreAdmin/Billing/AllBillingReports/2023/6_BSR_June_2023/awsAccounts.xlsx"), true);

}

private void parseIt(File inputFile) {
try {

//Open up xlsx file
Workbook wb = WorkbookFactory.create(inputFile);

//Find appropriate sheet
Sheet sheet = wb.getSheetAt(0);
if (sheet == null) throw new IOException("Could not find a sheet in "+inputFile+" ?");

//Iterate through rows
int numRows = sheet.getPhysicalNumberOfRows();
for (int r = 0; r< numRows; r++) {
Row row = sheet.getRow(r);
if (row != null) addAccount(row);
}
} catch (Exception e) {
System.out.println("Aws Accounts xlsx file is not in the correct format, exiting");
e.printStackTrace();
System.exit(1);
}
}

private void addAccount(Row row) {
int numCells = row.getLastCellNum()+1;
if (numCells < 2) return;

Cell groupNameCell = row.getCell(0);
if (groupNameCell == null) return;
String groupName = groupNameCell.toString().trim();
if (groupName.startsWith("INVESTIGATOR") || groupName.length()==0) return;

for (int c=1;c < numCells; c++) {
Cell cell = row.getCell(c);
if (cell != null) {
String accountNumber = cell.toString().trim();
if (accountNumber.length()!=0) awsAccountGroupName.put(accountNumber, groupName);
}
}
}

public TreeMap<String, String> getAwsAccountGroupName() {
return awsAccountGroupName;
}


}
Loading

0 comments on commit 38e99de

Please sign in to comment.