In-Network file size and data redundancy concerns

[vijayanandramamoorthy]

We started generating some sample files based on the published schema and we are concerned about the size of the final file.
Based on the numbers we are seeing now, we expect our final In-Network file for a group or individual plan to be over 500 GB in size!
The enormous file size is primarily due to 2 schema restrictions

1. Service_Code is not an array. We end up repeating the fee for some of the billing codes, for over 40 places of service, because the fee is the same irrespective of the place of service and the service can technically be performed in any of the locations.

2. TIN is required. Even though the NPI is an array, even if multiple providers share the same fee, we are forced to list the providers separately unless they all pay to the same TIN.

Does anyone else have these concerns about the file size and data redundancy?

thank you

[BobSyracuse]

There is great concern we are experiencing on the schema and the data redundancy in many areas of the schema's layout. Here is another post concerning it Reformat Root Schema To Support Multiple Plan/HIOS/EIN and Less Individual Files #93

[rgouldkc]

Regarding service code/ facility type, as pointed out previously, in our system these have no impact on the rate. Having to include each one individually will create a significant amount of redundancy in the file just to account for all of them. An array seems to me to be a good approach to help reduce the amount of redundant data. As to whether each service code would apply to every value in the npi array, the rate is driven by the provider and is established regardless of what the service_code is.

[keithwoodnc]

@shaselton Has there been any movement on this idea? There is significant concern about file size.

[shaselton-usds]

@keithwoodnc There have been updates! You can find that here in this commit: ad66fa1

[keithwoodnc]

@shaselton-usds Yes, saw this one, but I don't think it addresses the concern that TIN is still a string on it's own.

[vijayanandramamoorthy]

@shaselton-usds Both issues raised in this discussion have been addressed now. thank you!. As I mentioned above, we expect these 2 changes to make our file 100 times smaller!!!

[BobSyracuse]

Have you been able to compress the files and what file size did it produce based on your sample file? We are allowed to compress our files once created can the files be zipped? #59 . I know this does not help with the redundancy and the processing we must perform that is unneeded.

[vijayanandramamoorthy]

Yes, zipping the file gives about 88% compression. So a 500 GB file reduces to 60 GB, which still is unnecessarily huge in our opinion.