Support for measure normalization

[akariv]

I've come to realize that measure normalization is a must if we want to enable any kind of data-set comparison in the future.

The reason is that datasets come with many different structures. To give a few examples, these datasets are ones I worked with in the past couple of weeks:

• The Moldovan Budget, with measures for approved, adjusted and executed budgets
• The Israeli budget: started with measures for expenditure, commitment_limit, personnel_cap and others, repeated for (approved, adjusted and partly for executed) - that is, the data contains a column called commitment_limit_adjusted. At some point, the format changed and now it's just expenditure, commitment_limit, personnel_caps, with separate lines for approved, adjusted and executed values.
• The US budget: They have different data-sets for different 'phases' (authority & outlays), each dataset has 40+ measures - one per year since 1976 until 2021.

It is clear that each of these datasets can be described by the current FDP spec. But the spec does not provide any means to describe the 'implicit dimension' that's created by the multiple measures - e.g. for the Moldovan budget it's the phase dimension, while for the US budget it's the year.

To be able to properly compare these kind of datasets, the spec needs to provide a way to describe the difference between the measures in a systematic way.

To elaborate with an example, when describing the old-style Israeli budget, I'd like to be able to say that it contains two implicit dimensions - 'phase' and 'type(?)'. For each column containing a measure, the FDP would describe its meaning by providing values to these dimensions (e.g. the 'commitment_adjusted' column has 'commitment_limit' for 'type' and 'adjusted' for 'phase').

Once we have this kind of description, we'll be able to normalize these datasets, which has many benefits in comparing datasets, generic tools and analytics etc.

[pwalsh]

+1

Not the same, but intrinsically related to #91

[danfowler]

+1 @akariv

Looking forward to seeing the type system you've developed.

[rufuspollock]

@akariv yes - this is important and something long thought about so very excited that we will be addressing it. In general we need some kind of "numeraire" concept that we use either explicitly in the spec or implicitly in the OS processing system.

BTW see this older issue on OS main issue tracker: openspending/openspending#284

Comments:

The US budget: They have different data-sets for different 'phases' (authority & outlays), each dataset has 40+ measures - one per year since 1976 until 2021.

Do you mean a measure per year - we should not allow that i think ... (there's an issue on this but can't find it right now)

[akariv]

Alright, so here's a suggestion - let me know what you think so we can proceed.

I propose we add a new kind of dimension attribute, which could be used to normalise the fiscal data into a 'single measure' dataset.

This new attribute should be very similar to a constant attribute, with the only difference being that instead a single value it would define a value per measure defined in the model.

"dimensions": {
  "phase": {
     "attributes": {
        "phase": {      
            "byMeasure": {  (This comes instead of `source` or `constant`)
               "approved": "Approved",
               "adjusted":  "Adjusted",
               "executed": "Executed"
             }
...

When loading the data for the datapackage, implementers might use this dimension's definition in order to convert data looking like this:

cofog,approved,executed
r&d,1000000,800000

To

cofog,phase,amount
r&d,Approved,1000000
r&d,Executed,800000

[pwalsh]

+1

[pwalsh]

Moving to frictionlessdata/datapackage-fiscal#3