Handle individuals using different period structures for the same variable.

[guillett]

Hi there!

I really enjoy OpenFisca, but I recently encountered an issue.

Here is what I did:

I sent the following payload to the preview API:

{
  "familles": {
    "famille_1": {
      "enfants": [
      ],
      "parents": [
        "Bill",
        "Bob"
      ]
    }
  },
  "foyers_fiscaux": {
    "foyer_fiscal_1": {
      "declarants": [
        "Bill",
        "Bob"
      ],
      "personnes_a_charge": [
      ]
    }
  },
  "individus": {
    "Bill": {
      "salaire_de_base": {
        "2017": 20000
      },
      "tns_auto_entrepreneur_chiffre_affaires": {
        "2016": 0,
        "2016-08": 0,
        "2016-09": 0,
        "2016-10": 0,
        "2016-11": 0,
        "2016-12": 0,
        "2017-01": 0,
        "2017-02": 0,
        "2017-03": 0,
        "2017-04": 0,
        "2017-05": 0,
        "2017-06": 0,
        "2017-07": 35000,
        "2017-08": 35000,
        "2015-01": 2916.6666666666665,
        "2015-02": 2916.6666666666665,
        "2015-03": 2916.6666666666665,
        "2015-04": 2916.6666666666665,
        "2015-05": 2916.6666666666665,
        "2015-06": 2916.6666666666665,
        "2015-07": 2916.6666666666665,
        "2015-08": 2916.6666666666665,
        "2015-09": 2916.6666666666665,
        "2015-10": 2916.6666666666665,
        "2015-11": 2916.6666666666665,
        "2015-12": 2916.6666666666665
      }
    },
    "Bob": {
      "tns_auto_entrepreneur_chiffre_affaires": {
        "2015": 6000,
        "2016": 6000,
        "2016-08": 0,
        "2016-09": 0,
        "2016-10": 0,
        "2016-11": 0,
        "2016-12": 0,
        "2017-01": 0,
        "2017-02": 0,
        "2017-03": 0,
        "2017-04": 0,
        "2017-05": 0,
        "2017-06": 0,
        "2017-07": 0,
        "2017-08": 0
      }
    }
  },
  "menages": {
    "menage_1": {
      "conjoint": [
        "Bob"
      ],
      "enfants": [
      ],
      "impots_directs": {
        "2017": null
      },
      "personne_de_reference": [
        "Bill"
      ],
      "revenu_disponible": {
        "2017": null
      }
    }
  }
}

Here is what I expected to happen:

To return without error.

Here is what actually happened:

I get:

{
  "error": "Internal server error: Inconsistent input: variable tns_auto_entrepreneur_chiffre_affaires has already been set for all months contained in period 2015, and value [    0.  6000.] provided for 2015 doesn't match the total ([ 34999.99609375      0.        ]). This error may also be thrown if you try to call set_input twice for the same variable and period."
}

tns_auto_entrepreneur_chiffre_affaires legislation.openfisca.fr/code relies on set_input_divide_by_period.

For Bob a normalization is required as data is provided annually on 2015.
For Bill a normalisation is not required as data is provided monthly on 2015.

That discrepancy seems to be the origin of the problem.

Context

I identify more as a:

• Developer (I create tools that use the existing OpenFisca code).

(remove this line as well as all items in the list that don't fit your situation)

[Anna-Livia]

Thank you for this bug report, we are going to work on fixing this bug.
It is caused by the use of different time frames for Bill's and Bob's input in the same variable.

In the mean time, if you use the same input for both individuals, the API should return a valid output. For instance, for the example you mentioned, this should work:

TLDR: Bill's 2015 tns_auto_entrepreneur_chiffre_affairesis now declared annually, just like for Bob's.

{
  "familles": {
    "famille_1": {
      "enfants": [
      ],
      "parents": [
        "Bill",
        "Bob"
      ]
    }
  },
  "foyers_fiscaux": {
    "foyer_fiscal_1": {
      "declarants": [
        "Bill",
        "Bob"
      ],
      "personnes_a_charge": [
      ]
    }
  },
  "individus": {
    "Bill": {
      "salaire_de_base": {
        "2017": 20000
      },
      "tns_auto_entrepreneur_chiffre_affaires": {
        "2016": 0,
        "2016-08": 0,
        "2016-09": 0,
        "2016-10": 0,
        "2016-11": 0,
        "2016-12": 0,
        "2017-01": 0,
        "2017-02": 0,
        "2017-03": 0,
        "2017-04": 0,
        "2017-05": 0,
        "2017-06": 0,
        "2017-07": 35000,
        "2017-08": 35000,
        "2015": 35000
      }
    },
    "Bob": {
      "tns_auto_entrepreneur_chiffre_affaires": {
        "2015": 6000,
        "2016": 6000,
        "2016-08": 0,
        "2016-09": 0,
        "2016-10": 0,
        "2016-11": 0,
        "2016-12": 0,
        "2017-01": 0,
        "2017-02": 0,
        "2017-03": 0,
        "2017-04": 0,
        "2017-05": 0,
        "2017-06": 0,
        "2017-07": 0,
        "2017-08": 0
      }
    }
  },
  "menages": {
    "menage_1": {
      "conjoint": [
        "Bob"
      ],
      "enfants": [
      ],
      "impots_directs": {
        "2017": null
      },
      "personne_de_reference": [
        "Bill"
      ],
      "revenu_disponible": {
        "2017": null
      }
    }
  }
}

[benjello]

The computation are vectorized so every variable is treated the same way independently of the individual.
The input should thus be homogeneous across individuals.
I don't think there is a workaround within the core.

[fpagnoux]

There is none right now, but the web API could preprocess the input before vectorialization.

[guillett]

The use case I submitted previously is tricky

 "tns_auto_entrepreneur_chiffre_affaires": {
        "2016": 6000,
        "2016-08": 0,
        "2016-09": 0,
        "2016-10": 0,
        "2016-11": 0,
        "2016-12": 0
}

However, to me, it is reasonable to expect the following payload to be handled without trouble

{
    "individus": {
        "b": {
            "salaire_imposable": {
                "2016-11": 2556,
                "2016-12": 2556,
                "2016-07": 2556,
                "2016-06": 2556,
                "2016-05": 2556,
                "2016-04": 2556,
                "2016-03": 2556,
                "2016-02": 2556,
                "2016-01": 2556,
                "2016-09": 2556,
                "2016-08": 2556,
                "2016-10": 2556
            }
        },
        "a": {
            "salaire_imposable": {
                "2016": 23411.64
            }
        }
    },
    "foyers_fiscaux": {
        "_": {
            "declarants": [
                "a",
                "b"
            ],
            "irpp": {
                "2018": null
            }
        }
    },
    "menages": {
        "_": {
            "personne_de_reference": [
                "a"
            ],
            "conjoint": [
                "b"
            ]
        }
    },
    "familles": {
        "_": {
            "parents": [
                "a",
                "b"
            ]
        }
    }
}

curl -X POST https://fr.openfisca.org/api/v24/calculate -d @period-issue.json -H 'Content-Type: application/json' gives
{"error":"Internal server error: Inconsistent input: variable salaire_imposable has already been set for all months contained in period 2016, and value [23411.64 0. ] provided for 2016 doesn't match the total ([ 0. 30672.]). This error may also be thrown if you try to call set_input twice for the same variable and period."}.

Is that a fair expectation?
Is there any ETA to properly handle such a payload ?

Many thanks!

[Morendil]

It seems possible to hack it in, we'd need to add your case to test_entities.py as a new unit test.

I'm not entirely sure it's desirable as a long-term solution; set_input_divide_by_period seems to introduce confusion. The way set_input_divide_by_period works, as suggested by the unit test for it in test_entities.py, has less to do with dividing than with allowing both month and year scale inputs for the same variable (the same as in the case you described as "tricky"). Your use case seems legitimate but it comes in conflict with that first, officially sanctioned, use case.

It seems to me that the solution suggested above by @fpagnoux (preprocess input representations, for instance to transform a value specified for the year into 12 values for each month) would be a better way to go. It's a cleaner separation of concerns and we could (I think we should) simplify Holders by getting rid of set_input_divide_by_period.

[fpagnoux]

Is that a fair expectation?
Is there any ETA to properly handle such a payload ?

It is indeed. I'm moving this issue forward in our pipeline, and I've good hope we should be able to deliver by the end of the month.

I don't see any potential blocking issue on the implementation side. In the piece of code that builds a vectorial simulation from the JSON-like object, we just need to merge the inputs for different persons/households before building the vector, so that the default value is only applied to persons for which no input was specified.

[Morendil]

This is proving harder to fix than anticipated, see #875. Closing until we fix #887, to be revisited after.