Ha-Flow: Missing s42 rule(s) on the shared endpoint (swT has more than one ha-flow, one of sub-flow is y-point)

[yuliiamir]

The issue occurs on the switch triplet that has some ha-flow and additional one is created.
Note, that additional flow should have different ISls than the 1st ha-flow.
The initial ha-flow has no intersection in its sub-flows paths, but after system modification (make some ISLs in maintenance mode), the following ha-flow creation causes discrepancy on the switch(shared) such as one missing rule.
"missing": [ { "cookie": -9182839640208441000, "table_id": 0, "priority": 24574, "flags": [], "match": { "ETH_SRC": { "value": 2681726922752 }, "ETH_TYPE": { "value": 2048 }, "IN_PORT": { "value": 13 }, "IP_PROTO": { "value": 17 }, "UDP_SRC": { "value": 5010 } }, "instructions": { "go_to_table": 1, "write_metadata": { "value": 537526272, "mask": 4026466304 }, "apply_actions": [ { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_SRC" }, { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_DST" } ], "write_actions": [] }, "cookie_hex": "809000000000000A" } ]

There are two missing rules when the 1st ha-flow has path when one sub-flow is y-point, and the following ha-flow sub-flows without intersection is created.
{ "as_expected": false, "rules": { "asExpected": false, "missing": [ { "cookie": -9182839635913474000, "table_id": 0, "priority": 24574, "flags": [], "match": { "ETH_DST": { "value": 7 }, "ETH_SRC": { "value": 2681726922752 }, "ETH_TYPE": { "value": 2048 }, "IN_PORT": { "value": 13 }, "IP_PROTO": { "value": 17 }, "UDP_SRC": { "value": 5010 } }, "instructions": { "go_to_table": 1, "write_metadata": { "value": 537526272, "mask": 4294901760 }, "apply_actions": [ { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_SRC" }, { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_DST" } ], "write_actions": [] }, "cookie_hex": "809000010000000A" }, { "cookie": -9182839631618507000, "table_id": 0, "priority": 24574, "flags": [], "match": { "ETH_DST": { "value": 1 }, "ETH_SRC": { "value": 2681726922752 }, "ETH_TYPE": { "value": 2048 }, "IN_PORT": { "value": 13 }, "IP_PROTO": { "value": 17 }, "UDP_SRC": { "value": 5010 } }, "instructions": { "go_to_table": 1, "write_metadata": { "value": 805961728, "mask": 4294901760 }, "apply_actions": [ { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_SRC" }, { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_DST" } ], "write_actions": [] }, "cookie_hex": "809000020000000A" } ], "misconfigured": [], "excess": [], "proper": [

STR:

1. Create ha-flow.

{
  "ha_flow_id" : "my_test_flow1",
  "shared_endpoint" : {
    "switch_id" : "00:00:00:00:00:00:00:03",
    "port_number" : 10,
    "vlan_id" : 2281,
    "inner_vlan_id" : 0
  },
  "maximum_bandwidth" : 1000,
  "path_computation_strategy" : "COST",
  "encapsulation_type" : "transit_vlan",
  "max_latency" : null,
  "max_latency_tier2" : null,
  "ignore_bandwidth" : false,
  "periodic_pings" : false,
  "pinned" : false,
  "priority" : null,
  "strict_bandwidth" : false,
  "description" : "autotest flow: It is the east, and Juliet is the sun.",
  "allocate_protected_path" : false,
  "diverse_flow_id" : null,
  "sub_flows" : [ {
    "endpoint" : {
      "switch_id" : "00:00:00:00:00:00:00:04",
      "port_number" : 10,
      "vlan_id" : 390,
      "inner_vlan_id" : 0
    },
    "description" : null
  }, {
    "endpoint" : {
      "switch_id" : "00:00:00:00:00:00:00:02",
      "port_number" : 10,
      "vlan_id" : 1158,
      "inner_vlan_id" : 0
    },
    "description" : null
  } ]
}
 

Path: No intersection : different ISLs

   {
  "shared_path": {},
  "sub_flow_paths": [
    {
      "flow_id": "my_test_flow1-b",
      "forward": [
        {
          "switch_id": "00:00:00:00:00:00:00:03",
          "input_port": 10,
          "output_port": 2
        },
        {
          "switch_id": "00:00:00:00:00:00:00:02",
          "input_port": 1,
          "output_port": 10
        }
      ]
    },
    {
      "flow_id": "my_test_flow1-a",
      "forward": [
        {
          "switch_id": "00:00:00:00:00:00:00:03",
          "input_port": 10,
          "output_port": 4
        },
        {
          "switch_id": "00:00:00:00:00:00:00:04",
          "input_port": 4,
          "output_port": 10
        }
      ],...
    }
  ]
}

3. All switches are without any discrepancy
4. Set some Isls into maintenance mode to create another ha-Flow when one sub-flow is y-point (for this case it is the 1st ha-flow Isls).
5. Create additional ha-flow, verify that one subflow is y-point

{
 "shared_path": {
   "forward": [
     {
       "switch_id": "00:00:00:00:00:00:00:03",
       "input_port": 10,
       "output_port": 1
     },
     {
       "switch_id": "00:00:00:00:00:00:00:02",
       "input_port": 3
     }
   ],
   ]
 },
 "sub_flow_paths": [
   {
     "flow_id": "my_test_flow2-b",
     "forward": [
       {
         "switch_id": "00:00:00:00:00:00:00:03",
         "input_port": 10,
         "output_port": 1
       },
       {
         "switch_id": "00:00:00:00:00:00:00:02",
         "input_port": 3,
         "output_port": 10
       }
     ],
   
   },
   {
     "flow_id": "my_test_flow2-a",
     "forward": [
       {
         "switch_id": "00:00:00:00:00:00:00:03",
         "input_port": 10,
         "output_port": 1
       },
       {
         "switch_id": "00:00:00:00:00:00:00:02",
         "input_port": 3,
         "output_port": 2
       },
       {
         "switch_id": "00:00:00:00:00:00:00:04",
         "input_port": 1,
         "output_port": 10
       }
     ]
}

Actual result:
Missing rule on shared endpoint of the HA-Flow:

{ "as_expected": false, "rules": { "asExpected": false, "missing": [ { "cookie": -9182839640208441000, "table_id": 0, "priority": 24574, "flags": [], "match": { "ETH_SRC": { "value": 2681726922752 }, "ETH_TYPE": { "value": 2048 }, "IN_PORT": { "value": 13 }, "IP_PROTO": { "value": 17 }, "UDP_SRC": { "value": 5010 } }, "instructions": { "go_to_table": 1, "write_metadata": { "value": 537526272, "mask": 4026466304 }, "apply_actions": [ { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_SRC" }, { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_DST" } ], "write_actions": [] }, "cookie_hex": "809000000000000A" } ], "misconfigured": [], "excess": [], "proper": [

The similar steps for another case, but we should change the system before the 1st flow creation to have one sub-flow as y-point, then rollback to default configuration and create another flow with no intersection between sub-flows.
After sync switch (2 missing rules), and deleting the 1st flow (with intersection) another discrepancy occurs such as excess rules
{ "as_expected": false, "rules": { "asExpected": false, "missing": [], "misconfigured": [], "excess": [ { "cookie": -9182839640208441000, "table_id": 0, "priority": 24574, "flags": [], "match": { "ETH_SRC": { "value": 2681726922752 }, "ETH_TYPE": { "value": 2048 }, "IN_PORT": { "value": 13 }, "IP_PROTO": { "value": 17 }, "UDP_SRC": { "value": 5010 } }, "instructions": { "go_to_table": 1, "write_metadata": { "value": 537526272, "mask": 4026466304 }, "apply_actions": [ { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_SRC" }, { "action_type": "SET_FIELD", "value": 4700, "field": "UDP_DST" } ], "write_actions": [] }, "cookie_hex": "809000000000000A" } ], "proper": [

{
 "shared_path": {},
 "sub_flow_paths": [
   {
     "flow_id": "my_test_flow3-b",
     "forward": [
       {
         "switch_id": "00:00:00:00:00:00:00:03",
         "input_port": 10,
         "output_port": 2
       },
       {
         "switch_id": "00:00:00:00:00:00:00:02",
         "input_port": 1,
         "output_port": 10
       }
     ],
    ...
   },
   {
     "flow_id": "my_test_flow3-a",
     "forward": [
       {
         "switch_id": "00:00:00:00:00:00:00:03",
         "input_port": 10,
         "output_port": 4
       },
       {
         "switch_id": "00:00:00:00:00:00:00:04",
         "input_port": 4,
         "output_port": 10
       }
     ],
   ...
 ]
}

Another rules issue is to have excess rules after ha-flow reroute from the path when both sub-flows have no intersection to new path when one of the sub-flow is y-point.

[yuliiamir]

Another way to catch the issue is to trigger a ha-flow swap (manually switch from main to protected path)(flaky).
The issue has been caught for `HaFlowPathSwapSpec.Able to swap main and protected paths manually'.
In this case, a ha-flow main path has a shared path, but after swapping there were no shared path. As a result, there is an excess rule(s42) on the shared switch.
before swapping there were shared path(sw3-sw2):

• sub-flow1_path=sw3-sw2, protected sw3-sw2(different isls)
• sub_flow2_path=sw3-sw2-sw1, protected_path =sw3-sw8-sw7-sw1,

after swapping there is no shared path and s42 rule is excess

Details:

"ha_flow_id" : "11Dec033442_074_springonions3381_ha-flow",
  "status" : "Up",
  "shared_endpoint" : {
    "switch_id" : "00:00:00:00:00:00:00:03",
 ... }...
  "sub_flows" : [ {
    "flow_id" : "11Dec033442_074_springonions3381_ha-flow-b",
    "endpoint" : {
      "switch_id" : "00:00:00:00:00:00:00:01",
  
  }, {
    "flow_id" : "11Dec033442_074_springonions3381_ha-flow-a",
    "endpoint" : {
      "switch_id" : "00:00:00:00:00:00:00:02",
  ..}]
  }

curl -X GET \
http://localhost:8080/api/v2/ha-flows/11Dec033442_074_springonions3381_ha-flow/paths \
Response body: {
  "shared_path" : {
    "forward" : [ {
      "switch_id" : "00:00:00:00:00:00:00:03",
      "input_port" : 11,
      "output_port" : 2
    }, {
      "switch_id" : "00:00:00:00:00:00:00:02",
      "input_port" : 1
    } ],
    "reverse" : [ ...],
    "protected_path" : { }
  },
  "sub_flow_paths" : [ {
    "flow_id" : "11Dec033442_074_springonions3381_ha-flow-b",
    "forward" : [ {
      "switch_id" : "00:00:00:00:00:00:00:03",
      "input_port" : 11,
      "output_port" : 2
    }, {
      "switch_id" : "00:00:00:00:00:00:00:02",
      "input_port" : 1,
      "output_port" : 4
    }, {
      "switch_id" : "00:00:00:00:00:00:00:01",
      "input_port" : 1,
      "output_port" : 10
    } ],
    "reverse" : [...],
    "protected_path" : {
      "forward" : [ {
        "switch_id" : "00:00:00:00:00:00:00:03",
        "input_port" : 11,
        "output_port" : 6
      }, {
        "switch_id" : "00:00:00:00:00:00:00:08",
        "input_port" : 16,
        "output_port" : 7
      }, {
        "switch_id" : "00:00:00:00:00:00:00:07",
        "input_port" : 50,
        "output_port" : 15
      }, {
        "switch_id" : "00:00:00:00:00:00:00:01",
        "input_port" : 2,
        "output_port" : 10
      } ],
      "reverse" : [ ... ]
    }
  }, {
    "flow_id" : "11Dec033442_074_springonions3381_ha-flow-a",
    "forward" : [ {
      "switch_id" : "00:00:00:00:00:00:00:03",
      "input_port" : 11,
      "output_port" : 2
    }, {
      "switch_id" : "00:00:00:00:00:00:00:02",
      "input_port" : 1,
      "output_port" : 10
    } ],
    "reverse" : [ ...],
    "protected_path" : {
      "forward" : [ {
        "switch_id" : "00:00:00:00:00:00:00:03",
        "input_port" : 11,
        "output_port" : 1
      }, {
        "switch_id" : "00:00:00:00:00:00:00:02",
        "input_port" : 3,
        "output_port" : 10
      } ],
      "reverse" : [ ...]
    }
  } ]
}

[yuliiamir]

seems like it is related to #5647