[TEST] test_delete_volume_on_deleted_vm failing on 1.2-head

[khushboo-rancher]

What's the test to develop? Please describe

def test_delete_volume_on_deleted_vm(self, api_client, ubuntu_image, ubuntu_vm, polling_for):
"""
1. Create a VM with volume
2. Delete volume should reply 422
3. Delete VM but not volume
4. Delete volume should reply 200
Ref. #652
"""
vol_name = (ubuntu_vm["spec"]["template"]["spec"]["volumes"][0]
['persistentVolumeClaim']['claimName'])

    code, data = api_client.volumes.delete(vol_name)
    assert 422 == code, f"Should fail to delete volume\n{code}, {data}"

    self.delete_vm(api_client, ubuntu_vm, polling_for)

    code, data = api_client.volumes.delete(vol_name)
    assert code in (200, 204), f"Fail to delete volume\n{code}, {data}"

    # Check Volume
    code, data = api_client.volumes.get(vol_name)

  mdata, annotations = data['metadata'], data['metadata']['annotations']

E KeyError: 'metadata'

api_client = HarvesterAPI('https://172.19.108.0', 'Bearer token-gcp2b:rbqnq89ttfgvrtzk6gsbz98fb2j7qcnjn72bs2rclckgrdpfp2grfm')
code       = 404
data       = {'code': 'NotFound',
 'links': {},
 'message': 'persistentvolumeclaims '
            '"vm-19h04m01s512888-04-15-vm-19h04m01s512888-04-15" not found',
 'status': 404,
 'type': 'error'}
polling_for = <function polling_for.<locals>._polling_for at 0x7f7872e4a830>
self       = <harvester_e2e_tests.integrations.test_1_volumes.TestVolumeWithVM object at 0x7f7873613b20>
ubuntu_image = {'display_name': 'img-19h04m01s512888-04-15',
 'id': 'default/img-19h04m01s512888-04-15',
 'ssh_user': 'ubuntu'}
ubuntu_vm  = {'actions': {'addVolume': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=addVolume',
             'backup': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=backup',
             'clone': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=clone',
             'createTemplate': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=createTemplate',
             'findMigratableNodes': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=findMigratableNodes',
             'migrate': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=migrate',
             'pause': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=pause',
             'removeVolume': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=removeVolume',
             'restart': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=restart',
             'softreboot': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=softreboot',
             'stop': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15?action=stop'},
 'apiVersion': 'kubevirt.io/v1',
 'id': 'default/vm-19h04m01s512888-04-15',
 'kind': 'VirtualMachine',
 'links': {'remove': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15',
           'self': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15',
           'update': 'https://172.19.108.0/v1/harvester/kubevirt.io.virtualmachines/default/vm-19h04m01s512888-04-15',
           'view': 'https://172.19.108.0/apis/kubevirt.io/v1/namespaces/default/virtualmachines/vm-19h04m01s512888-04-15'},
 'metadata': {'annotations': {'field.cattle.io/description': '',
                              'harvesterhci.io/vmRunStrategy': 'RerunOnFailure',
                              'harvesterhci.io/volumeClaimTemplates': '[{"type": '
                                                                      '"persistentvolumeclaim", '
                                                                      '"metadata": '
                                                                      '{"namespace": '
                                                                      '"default", '
                                                                      '"name": '
                                                                      '"vm-19h04m01s512888-04-15-vm-19h04m01s512888-04-15", '
                                                                      '"annotations": '
                                                                      '{"harvesterhci.io/imageId": '
                                                                      '"default/img-19h04m01s512888-04-15"}}, '
                                                                      '"spec": '
                                                                      '{"accessModes": '
                                                                      '["ReadWriteMany"], '
                                                                      '"resources": '
                                                                      '{"requests": '
                                                                      '{"storage": '
                                                                      '"10Gi"}}, '
                                                                      '"volumeMode": '
                                                                      '"Block", '
                                                                      '"storageClassName": '
                                                                      '"longhorn-img-19h04m01s512888-04-15"}}]',
                              'kubevirt.io/latest-observed-api-version': 'v1',
                              'kubevirt.io/storage-observed-api-version': 'v1'},
              'creationTimestamp': '2024-04-15T19:04:38Z',
              'fields': ['vm-19h04m01s512888-04-15', '39s', 'Running', 'True'],
              'finalizers': ['kubevirt.io/virtualMachineControllerFinalize',
                             'harvesterhci.io/VMController.UnsetOwnerOfPVCs'],
              'generation': 2,
              'labels': {'harvesterhci.io/creator': 'harvester',
                         'harvesterhci.io/os': ''},
              'managedFields': [{'apiVersion': 'kubevirt.io/v1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:metadata': {'f:annotations': {'f:kubevirt.io/latest-observed-api-version': {},
                                                                               'f:kubevirt.io/storage-observed-api-version': {}},
                                                             'f:finalizers': {'.': {},
                                                                              'v:"kubevirt.io/virtualMachineControllerFinalize"': {}}}},
                                 'manager': 'Go-http-client',
                                 'operation': 'Update',
                                 'time': '2024-04-15T19:04:38Z'},
                                {'apiVersion': 'kubevirt.io/v1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:status': {'.': {},
                                                           'f:conditions': {},
                                                           'f:created': {},
                                                           'f:desiredGeneration': {},
                                                           'f:observedGeneration': {},
                                                           'f:printableStatus': {},
                                                           'f:ready': {},
                                                           'f:volumeSnapshotStatuses': {}}},
                                 'manager': 'Go-http-client',
                                 'operation': 'Update',
                                 'subresource': 'status',
                                 'time': '2024-04-15T19:05:15Z'},
                                {'apiVersion': 'kubevirt.io/v1',
                                 'fieldsType': 'FieldsV1',
                                 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {},
                                                                               'f:field.cattle.io/description': {},
                                                                               'f:harvesterhci.io/vmRunStrategy': {},
                                                                               'f:harvesterhci.io/volumeClaimTemplates': {}},
                                                             'f:finalizers': {'v:"harvesterhci.io/VMController.UnsetOwnerOfPVCs"': {}},
                                                             'f:labels': {'.': {},
                                                                          'f:harvesterhci.io/creator': {},
                                                                          'f:harvesterhci.io/os': {}}},
                                              'f:spec': {'.': {},
                                                         'f:runStrategy': {},
                                                         'f:template': {'.': {},
                                                                        'f:metadata': {'.': {},
                                                                                       'f:labels': {'.': {},
                                                                                                    'f:harvesterhci.io/vmName': {}}},
                                                                        'f:spec': {'.': {},
                                                                                   'f:domain': {'.': {},
                                                                                                'f:cpu': {'.': {},
                                                                                                          'f:cores': {},
                                                                                                          'f:sockets': {}},
                                                                                                'f:devices': {'.': {},
                                                                                                              'f:disks': {},
                                                                                                              'f:inputs': {},
                                                                                                              'f:interfaces': {}},
                                                                                                'f:features': {'.': {},
                                                                                                               'f:acpi': {'.': {},
                                                                                                                          'f:enabled': {}}},
                                                                                                'f:firmware': {},
                                                                                                'f:machine': {'.': {},
                                                                                                              'f:type': {}},
                                                                                                'f:resources': {'.': {},
                                                                                                                'f:limits': {'.': {},
                                                                                                                             'f:cpu': {},
                                                                                                                             'f:memory': {}}}},
                                                                                   'f:evictionStrategy': {},
                                                                                   'f:hostname': {},
                                                                                   'f:networks': {},
                                                                                   'f:volumes': {}}}}},
                                 'manager': 'harvester',
                                 'operation': 'Update',
                                 'time': '2024-04-15T19:05:15Z'}],
              'name': 'vm-19h04m01s512888-04-15',
              'namespace': 'default',
              'relationships': [{'message': 'Resource is current',
                                 'rel': 'owner',
                                 'state': 'active',
                                 'toId': 'default/revision-start-vm-60fbb7e8-760a-4cee-b38b-9ca407168c84-1',
                                 'toType': 'apps.controllerrevision'},
                                {'rel': 'owner',
                                 'state': 'running',
                                 'toId': 'default/vm-19h04m01s512888-04-15',
                                 'toType': 'kubevirt.io.virtualmachineinstance'}],
              'resourceVersion': '37331',
              'state': {'error': False,
                        'message': 'VirtualMachine generation is 2, but latest '
                                   'observed generation is 1',
                        'name': 'in-progress',
                        'transitioning': True},
              'uid': '60fbb7e8-760a-4cee-b38b-9ca407168c84'},
 'spec': {'runStrategy': 'RerunOnFailure',
          'template': {'metadata': {'creationTimestamp': None,
                                    'labels': {'harvesterhci.io/vmName': 'vm-19h04m01s512888-04-15'}},
                       'spec': {'affinity': {},
                                'architecture': 'amd64',
                                'domain': {'cpu': {'cores': 1, 'sockets': 1},
                                           'devices': {'disks': [{'bootOrder': 1,
                                                                  'disk': {'bus': 'virtio'},
                                                                  'name': 'vm-19h04m01s512888-04-15'},
                                                                 {'disk': {'bus': 'virtio'},
                                                                  'name': 'cloudinitdisk'}],
                                                       'inputs': [{'bus': 'usb',
                                                                   'name': 'tablet',
                                                                   'type': 'tablet'}],
                                                       'interfaces': [{'macAddress': '52:54:00:de:3b:18',
                                                                       'masquerade': {},
                                                                       'model': 'virtio',
                                                                       'name': 'default'}]},
                                           'features': {'acpi': {'enabled': True}},
                                           'firmware': {},
                                           'machine': {'type': 'q35'},
                                           'memory': {'guest': '1948Mi'},
                                           'resources': {'limits': {'cpu': '1',
                                                                    'memory': '2Gi'},
                                                         'requests': {'cpu': '62m',
                                                                      'memory': '1365Mi'}}},
                                'evictionStrategy': 'LiveMigrate',
                                'hostname': 'vm-19h04m01s512888-04-15',
                                'networks': [{'name': 'default', 'pod': {}}],
                                'terminationGracePeriodSeconds': 120,
                                'volumes': [{'name': 'vm-19h04m01s512888-04-15',
                                             'persistentVolumeClaim': {'claimName': 'vm-19h04m01s512888-04-15-vm-19h04m01s512888-04-15'}},
                                            {'cloudInitNoCloud': {'userData': '#cloud-config\n'
                                                                              'package_update: '
                                                                              'true\n'
                                                                              'packages:\n'
                                                                              '- '
                                                                              'qemu-guest-agent\n'
                                                                              'runcmd:\n'
                                                                              '- '
                                                                              '- '
                                                                              'systemctl\n'
                                                                              '  '
                                                                              '- '
                                                                              'enable\n'
                                                                              '  '
                                                                              '- '
                                                                              '--now\n'
                                                                              '  '
                                                                              '- '
                                                                              'qemu-guest-agent.service\n'},
                                             'name': 'cloudinitdisk'}]}}},
 'status': {'conditions': [{'error': False,
                            'lastProbeTime': None,
                            'lastTransitionTime': '2024-04-15T19:05:12Z',
                            'lastUpdateTime': '2024-04-15T19:05:12Z',
                            'status': 'True',
                            'transitioning': False,
                            'type': 'Ready'},
                           {'error': False,
                            'lastProbeTime': None,
                            'lastTransitionTime': None,
                            'lastUpdateTime': '',
                            'status': 'True',
                            'transitioning': False,
                            'type': 'LiveMigratable'}],
            'created': True,
            'desiredGeneration': 2,
            'observedGeneration': 1,
            'printableStatus': 'Running',
            'ready': True,
            'volumeSnapshotStatuses': [{'enabled': False,
                                        'name': 'vm-19h04m01s512888-04-15',
                                        'reason': '2 matching '
                                                  'VolumeSnapshotClasses for '
                                                  'longhorn-img-19h04m01s512888-04-15'},
                                       {'enabled': False,
                                        'name': 'cloudinitdisk',
                                        'reason': 'Snapshot is not supported '
                                                  'for this volumeSource type '
                                                  '[cloudinitdisk]'}]},
 'type': 'kubevirt.io.virtualmachine'}
vol_name   = 'vm-19h04m01s512888-04-15-vm-19h04m01s512888-04-15'

harvester_e2e_tests/integrations/test_1_volumes.py:199: KeyError

[lanfon72]

@albinsun, I feel that it is falsely alarm, and according to the origin issue harvester/harvester#2677, volume and VM should be deleted simultaneously, but the implementation didn't hit it.

The failed part was added in #1186, but I don't think that after you check the volume been deleted succesfully, the volume will still available.

This test case would need to be re-implement if it is about harvester/harvester#2677 (but I feel it might not necessary because we already covered it in test case test_create_vm_do_not_start)

cc @khushboo-rancher

[khushboo-rancher]

I coudn't find the test test_create_vm_do_not_start definition.
I'm trying to find the difference between the tests test_create_vm_do_not_start and test_delete_volume_on_deleted_vm.

[lanfon72]

It's legacy test case and been replaced by integration/test_vm_functions.py::test_create_stopped_vm.
The origin issue is the side effect when testing stopped vm, so we can still implement this as a standalone test case, but current test step is not hit the origin issue.

ref: #786 (comment)

[khushboo-rancher]

Thanks @lanfon72. I see there is an extra step of attempting deletion of volume while the volume is attached to the VM in the test test_delete_volume_on_deleted_vm.
Let's correct the test test_delete_volume_on_deleted_vm and make it work as expected on our CI.

[albinsun]

Hi @khushboo-rancher and @lanfon72,

It's due to improper status check after deleting volume, should be 404 instead of 200.
Fix in PR /pull/1232.

As discussion above, this TC tests the non-simultaneous deletion case and we can treat this as new TC if test_create_vm_do_not_start already covers simultaneous case.

Please comment if I have any misunderstanding.

Thank you.

[lanfon72]

The test_create_vm_do_not_start is legacy one and the origin issue for #652 is the side effect when executing test_create_vm_do_not_start multiple times. the test_create_vm_do_not_start was using api.vms.get (same API endpoint) to check VM been deleted before delete related VM, and the newer one (rework for test_create_vm_do_not_start) is using api.vms.get_status to make sure the VMI already gracefully removed.

so, newer one is not cover the case of #652, consider the test_delete_volume_on_deleted_vm is marked to coverd #652, I will add comments to #1232 to explain what the test steps would be.

[albinsun]

Close as fixed with /pull/1269