Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delete subscriptions in bulk using python script #4626

Merged
merged 3 commits into from
Oct 28, 2024
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions scripts/utils/delete_subscriptions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
#!/usr/bin/env python3
# Copyright 2024 Telefonica Investigacion y Desarrollo, S.A.U
#
# This file is part of Orion Context Broker.
#
# Orion Context Broker is free software: you can redistribute it and/or
# modify it under the terms of the GNU Affero General Public License as
# published by the Free Software Foundation, either version 3 of the
# License, or (at your option) any later version.
#
# Orion Context Broker is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero
# General Public License for more details.
#
# You should have received a copy of the GNU Affero General Public License
# along with Orion Context Broker. If not, see http://www.gnu.org/licenses/.
#
# For those usages not covered by this license please contact with
# iot_support at tid dot es

import requests
import sys

requests.packages.urllib3.disable_warnings()

__author__ = 'Md Arqum Farooqui'

##############################################################
# BEGIN of the configuration part (don't touch above this line ;)

cb_endpoint = 'http://localhost:1026/v2/subscriptions'

headers = {
'Accept': 'application/json',
'fiware-service': 'service',
'fiware-servicepath': '/subservice'
}

page_size = 500

# WARNING! use this filter or you will remove *all* subscriptions
filter = 'offset=0'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even with offset=0, all subscription would be removed :). Note that offset/limit are not actually filters, but a mechanism to get paginated results.

Let's keep the filter varible, just in case in the future gets implemented, but I'd suggest to use something like this in this point

Suggested change
# WARNING! use this filter or you will remove *all* subscriptions
filter = 'offset=0'
# WARNING! use this filter or you will remove *all* subscriptions
# FIXME: note that filtering is not supported in the GET /v2/subscriptions. If at the it is get
# implemented (https://github.com/telefonicaid/fiware-orion/issues/1145 or additional issues),
# let's adjust this point of the script with a valid example filter
#filter = '&type=device'

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I intentionally put the offset value as the actual filter (q/type) is not working yet.
But the suggested changes gives more clarity so I'll update the PR:)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even with offset=0, all subscription would be removed :). Note that offset/limit are not actually filters, but a mechanism to get paginated results.

Let's keep the filter varible, just in case in the future gets implemented, but I'd suggest to use something like this in this point

filter variable cannot be commented as it make request bad. Please see below screenshot for reference:

filter_variable

So I can add below lines as well in the script:

# WARNING! use this filter or you will remove *all* subscriptions
# FIXME: note that filtering is not supported in the GET /v2/subscriptions. If at the it is get
# implemented (https://github.com/telefonicaid/fiware-orion/issues/1145 or additional issues),
# let's adjust this point of the script with a valid example filter
#filter = '&type=device'
# For now we can keep filter variable empty.
filter = ''

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine with it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 01f8553


# END of the configuration part (don't touch below this line ;)
##############################################################

def delete_subscriptions():
"""
delete subscriptions in current page.
:return: True if removal was ok, False otherwise
"""
try:
# Fetching subscriptions with pagination
response = requests.get(f"{cb_endpoint}?limit={page_size}&{filter}", headers=headers, verify=False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
response = requests.get(f"{cb_endpoint}?limit={page_size}&{filter}", headers=headers, verify=False)
response = requests.get(f"{cb_endpoint}?limit={page_size}{filter}", headers=headers, verify=False)

Let's assume that filter is either the empty string or a query param with the & already included (as in existing delete_entities_by_query.py)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed in 01f8553


if response.status_code == 200:
subscriptions = response.json()

# Check if subscriptions list is not empty
if not subscriptions:
print("No more subscriptions found.")
return False

# Delete each subscription
for subscription in subscriptions:
subscription_id = subscription['id']
delete_url = f"{cb_endpoint}/{subscription_id}"
delete_response = requests.delete(delete_url, headers=headers)

if delete_response.status_code != 204:
print(f"Failed to delete subscription with ID {subscription_id}. Status code: {delete_response.status_code}")
return False

else:
print(f"Failed to fetch subscriptions. Status code: {response.status_code}")
return False

except requests.exceptions.RequestException as e:
print(f"An error occurred: {e}")


def get_subscriptions_count():
"""
Return the total number of subscriptions.

:return: number of subscriptions (or -1 if there was some error)
"""

res = requests.get(cb_endpoint + '?limit=1&options=count', headers=headers, verify=False)
if res.status_code != 200:
print('Error getting subscriptions count (%d): %s' % (res.status_code, res.json()))
return -1
else:
return int(res.headers['fiware-total-count'])


def initial_statistics():
"""
Print some initial data statistics.
"""
total = get_subscriptions_count()
print(f'There are {total} subscriptions in total.')
pages = total // page_size
rest = total % page_size
print(f'This includes {pages} full pages of {page_size} subscriptions and one final page of {rest} subscriptions.')


def remove_all_subscriptions():
"""
Remove all subscriptions, page after page.
"""
i = 1
while get_subscriptions_count() > 0:
print(f'- Removing page {i}')
delete_subscriptions()
i += 1


### Main program starts here ###

# Warn user
print(f"WARNING!!!! This script will delete all the subscriptions matching the filter '{filter}'")
print("This action cannot be undone. If you are sure you want to continue type 'yes' and press Enter")

confirm = input()

if (confirm != 'yes'):
sys.exit()


# Get statistics
initial_statistics()

# Remove all subscriptions
remove_all_subscriptions()

print('All subscriptions deleted!')
Loading