DHIS-16705 Program Indicator Expression transformer

[luciano-fiandesio]

Summary

Work in progress

This PR addresses an issue with Enrollments and Events queries that contain a PI expression with internal subqueries as where condition.
These type of queries do not run in Doris, because Doris does not support correlation with outer layers of the parent query.

Example of a non supported query:

SELECT   enrollment,
		 sub(1 + 1)
FROM     analytics_enrollment_ur1edk5oe2n   AS subax
WHERE    ((
		  date_part('year',age(cast(
				 (
				 SELECT   scheduleddate
				 FROM     analytics_event_ur1edk5oe2n
				 WHERE    analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
				 AND      scheduleddate IS NOT NULL
				 ORDER BY occurreddate DESC
				 LIMIT    1 ) AS date), cast(coalesce(completeddate,
		  	   	 (
				   SELECT   created
				   FROM     analytics_event_ur1edk5oe2n
				   WHERE    analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
				   AND      created IS NOT NULL
				   ORDER BY occurreddate DESC
				   LIMIT    1 )) AS date)))) * 12 + date_part('month',age(cast(
						(
						SELECT   scheduleddate
						FROM     analytics_event_ur1edk5oe2n
						WHERE    analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
						AND      scheduleddate IS NOT NULL
						ORDER BY occurreddate DESC
						LIMIT    1 ) AS date), cast(coalesce(completeddate,
							(
							SELECT   created
							FROM     analytics_event_ur1edk5oe2n
							WHERE    analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
							AND      created IS NOT NULL
							ORDER BY occurreddate DESC
							LIMIT    1 )) AS date)))) > 1

This PR introduces a new component named CteOptimizationPipeline that has the following responsibilities:

• Extract PI-generated subqueries (the subqueries are mostly generated by org.hisp.dhis.parser.expression.statement.DefaultStatementBuilder so they are easy to identify).
• Transform the subqueries into CTE (Common Table Expressions)
• Rebuild the original PI CTE and add the new CTEs and modify the original where condition so that the CTEs are referenced with join statements and the where condition is preserved (including the function chain).

Transformation examples

Source

with pi_hgtnuhsqbml
as (
select
	enrollment,
	sum(1+1) as value
from
	analytics_enrollment_ur1edk5oe2n as subax
where
	((
	date_part('year', age(cast(
	(
	select
		scheduleddate
	from
		analytics_event_ur1edk5oe2n
	where
		analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
		and scheduleddate is not null
	order by
		occurreddate desc
	limit 1 ) as date),
	cast(coalesce(completeddate,
	(
	select
		created
	from
		analytics_event_ur1edk5oe2n
	where
		analytics_event_ur1edk5oe2n.enrollment = subax.enrollment
		and created is not null
	order by
		occurreddate desc
	limit 1 )) as date)))) * 12 ) > 1
	and "lw1SqmMlnfh" is not null
group by
	enrollment )
-- end of CTE
select
	ax.enrollment
from
	analytics_enrollment_ur1edk5oe2n as ax
where
	(((
	lastupdated >= '2018-01-01' and lastupdated < '2018-04-28')))
	and ( ax."uidlevel1" = 'ImspTQPwCqd' )

Target

with last_sched as (
select
	enrollment,
	scheduleddate
from
	(
	select
		enrollment,
		scheduleddate,
		row_number() over (partition by enrollment
	order by
		occurreddate desc) as rn
	from
		analytics_event_ur1edk5oe2n
	where
		scheduleddate is not null) t
where
	rn = 1),
last_created as (
select
	enrollment,
	created
from
	(
	select
		enrollment,
		created,
		row_number() over (partition by enrollment
	order by
		occurreddate desc) as rn
	from
		analytics_event_ur1edk5oe2n
	where
		created is not null) t
where
	rn = 1),
pi_hgtnuhsqbml as (
select
	subax.enrollment,
	sum(1 + 1) as value
from
	analytics_enrollment_ur1edk5oe2n as subax
left join last_sched as ls on
	subax.enrollment = ls.enrollment
left join last_created as lc on
	subax.enrollment = lc.enrollment
where
	((date_part('year',
	age(cast(ls.scheduleddate as date),
	cast(coalesce(completeddate,
	lc.created) as date)))) * 12) > 1
		and "lw1SqmMlnfh" is not null
	group by
		subax.enrollment)
select
	ax.enrollment
from
	analytics_enrollment_ur1edk5oe2n as ax
where
	(((lastupdated >= '2018-01-01'
		and lastupdated < '2018-04-28')))
	and (ax."uidlevel1" = 'ImspTQPwCqd')

What is not working

• Support for multiple PI CTEs
• Further query optimization, creating a Base CTE to avoid child CTE to scan the entire table
• Some complex expressions don't seem to work
• Increase test coverage

[luciano-fiandesio]

mmm, the class is actually used...

To fix the problem, we need to replace the deprecated setGroupByExpressions method with its recommended alternative. According to the JSqlParser library documentation, the setGroupByExpressions method has been replaced by the addGroupByExpression method, which allows adding expressions individually to the GroupByElement.

We will modify the code to use the addGroupByExpression method instead of setGroupByExpressions. This change will be made in the dataElementCountCte method of the SubqueryTransformer class.

To fix the problem, we need to replace the usage of the deprecated getGroupByExpressions() method with its recommended alternative. According to the JSqlParser library documentation, the getGroupByExpressions() method has been replaced by getGroupByExpressionList(). We will update the code to use this new method.

• Replace the call to groupBy.getGroupByExpressions() with groupBy.getGroupByExpressionList().
• Update the variable name from groupByExpressions to groupByExpressionList to reflect the new method.

To fix the problem, we need to replace the usage of the deprecated setGroupByExpressions method with its recommended alternative. The GroupByElement class provides a method addGroupByExpression which can be used to add expressions to the group by clause individually. We will iterate over the newGroupByExpressions list and add each expression using the addGroupByExpression method.

[maikelarabori]

It looks good, just minor comments/suggestions.
Cheers

[maikelarabori]

Suggested change

	E7149("Could not pre-pend CTEs to Program Indicator CTE, unexpected error: `{0}`"),
	E7149("Could not prepend CTEs to Program Indicator CTE, unexpected error: `{0}`"),

[maikelarabori]

Forgotten debug line

[maikelarabori]

Suggested change

	if (StringUtils.isEmpty(sql)) {
	if (StringUtils.isBlank(sql)) {

[maikelarabori]

I believe it's old code, but maybe select.getWithItemsList() could return an empty list, instead of a null?

[maikelarabori]

Suggested change

	WHERE rn = 1
	where rn = 1

[maikelarabori]

Forgotten comment

[maikelarabori]

Should it be getSingleExpression?

[maikelarabori]

Suggested change

	if (items == null || items.size() != 1) {
	if (CollectionUtils.size(items) != 1) {

[maikelarabori]

Suggested change

	if (selectItems == null || selectItems.size() != 1) {
	if (CollectionUtils.size(selectItems) != 1) {

[maikelarabori]

Do we need to add null to the list?
I have the impression that it will not affect consumers: if the expression is null, there is nothing to deal with, so should we even set null values?

[sonarqubecloud]

Quality Gate failed

Failed conditions
11 New issues
11 New Code Smells (required ≤ 0)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE