Data loss after radicale server IP change & DAVx5 account conflict - any suggestions?

[floholl]

Hi,

For many years, I've been using DAVx5 on my Android phone to automatically sync a few calendars and an address book to a radicale server on my laptop. This happens automatically whenever both devices are on my home LAN, which usually means: on a daily basis.

I edit the calendars and address book exclusively on my phone rather than the laptop, so it took me a while before I realized today that the last sync to my laptop had - to my surprise - occurred many weeks ago. I then realized this was due to a recent WiFi router hardware upgrade, which had resulted in a new IP address for the laptop / radicale server, which DAVx5 was of course unaware of.

I recalled from similar previous occasions that the IP address of an existing DAVx5 account cannot be changed after the fact (#23). Normally, what I would do in such a situation (IIRC) is to first remove the existing DAVx5 account for the radicale server with the outdated IP address, and then create a new one with the updated IP address. As I recall it, the existing calendars and contacts on the phone were unaffected by this, and any new calendar events & contacts created on the phone since the last sync would then reliably sync back to the radicale server via the newly created account - just as intended.

Today, however - for whatever reason - I added the new DAVx5 account with the updated IP address without first removing the old account with the outdated IP. The result was that my calendars and address book on my Android phone suddenly looked very outdated; essentially all the edits I had made since the last sync many weeks ago were gone. Apparently, the only syncing that occurred was from the radicale server to the Android client rather than the other way around.

So now I find myself with several weeks worth of lost calendar/contacts data and have the following questions:

1. Is there any feasible path to getting my data back (through DAVx5 rather than separate backups)?
2. Is my suspicion above correct, i.e. that not removing the old DAVx5 account before creating the new one is to blame?
3. If yes, why? Does it have anything to do with the fact that calendars & address book on both accounts were named identically?

I'm mostly just trying to understand the process, so I can prevent similar disasters from happening in the future.

Thanks for any pointers!

[floholl]

An attempt to answer my own question after finding this: https://www.davx5.com/faq/synchronization/change-base-url

I now seem to recall that in that previous similar situation (i.e., IP address change of radicale server), I had actually manually changed the server's IP back to the old one, in order to perform a final sync using the old DAVx5 account, before then creating a new account with the updated IP. That's the ingredient I had missing today, and I'm guessing that's what resulted in my loss of recently created calendar events & contacts.

Can somebody confirm this? Also, note that I do still have the old DAVx5 account with the outdated IP - could that contribute towards a data rescue effort, or is it just too late, now that the radicale server has synced back to the Android client via the new account?

[devvv4ever]

Hi,

DAVx5 does not delete data if a IP is changing or not reachable anymore. Data should stay in the content provider on Android and is only removed from there if you either disable the sync for certain collections, or remove the DAVx5 account, or explicitly delete the collection via the appropriate menu option besides the collection name.

If you added new events/contacts etc to the old DAVx5 account then these data would only stay local on the device until you manually move them to the account that is syncable. It is however hard to see differences in the calendar and contacts apps, I understand that.

[floholl]

Thanks a lot!

So does this mean that, considering I still do have the old DAVX5 account, I should still be able to sync the maybe-not-lost-after-all data back to the radicale server?

I guess that would require getting the radicale server back onto the old IP address that the DAVx5 account expects it at (and the Android client onto an IP in the same subnet, of course). I'll try that.

Best,
Florian

[devvv4ever]

Yes either this, or you export calendar events via an extra app "ical Import Export" into a file and import it back to the new working DAVx5 account which will upload it to the server. May cause duplicates however, not super sure about that.

[floholl]

Thanks for the help!

I decided today that before trying to get back both devices back on their old IP addresses, I would give the "iCal Import Export" app that you mentioned a try (which on FDroid, I believe I found as "Calendar Import-Export").

Turns out that I cannot export the affected calendars via that app. Specifically, its "Export" button in the app is greyed out for these (but not for other) calendars.

It seems worth mentioning that the app also lists those affected calendars as containing "Events: 0", which I think might be the reason why "Export" is disabled.

Perhaps also of interest: Each calendar is seen only once by "Calendar Import-Export", i.e., it's not as if there are 2 "leisure" calendars, 1 "work" calendar, etc., just one for each name.

Another breakdown of the process that led me here:

1. I happily synced a DAVx5 Android client to a radicale server on a daily basis for many years.
2. A few weeks ago, I changed the IP address of the radicale server.
3. I recently realized the client had not synced new events/contacts back to the server for since the IP address update.
4. I then created a new DAVx5 account with the updated server IP (crucially: without first syncing via old IP!)
5. I then observed on the client that my Android stock calendar and address book suddenly looked like they had many weeks ago, before the IP address change.
6. I deleted the new (but not the old!) DAVx5 account on the Android client (call it a panic reaction).

Steps 5 to me suggests that step 4 triggered a sync back from the server to the client via the new DAVx5 account. Wouldn't that then permanently overwrite the local calendars/contacts on the Android client with an old, outdated version (created many weeks before with the old DAVx5 account)? After all, both accounts used exactly the same collection names. Or am I getting something wrong here, and there is in fact still a path to rescuing my data?

Step 6 explains why there are 0 events in the affected calendars now locally on my Android client, and the calendar app does indeed not show any events in any of these calendars at the moment. But perhaps deleting the new DAVx5 account was a silly thing to do? On the other hand, devvv4ever seems to suggest that as long as the old DAVx5 account remains untouched, data should not be deleted.

My 2 key questions:

1. Is my data from recent weeks gone for good?
2. If yes, which step from the ones listed above (1 to 6) triggered this accident, so I can avoid it in the future?

Many thanks for the continued support!·

[floholl]

A quick follow-up:

Since I could not use the "Calendar Import-Export" app to export calendars from my Android client (due to them containing "0 events", as described above), I instead proceeded as suggested above: I got the radicale server and the Android client back onto their original IP subnet that the old DAVx5 account expected, in order to then successfully initiate a sync between the two.

This, however, yielded the same result as originally described above for the first sync of the new DAVx5 account: My old calendar events and contacts were successfully restored from the radicale server to the Android client, but the most recent events and contacts remain lost, i.e., anything that was added on the Android client since the last sync of the old DAVx5 account to the radicale server that occurred a few weeks ago.

To me this suggests that somewhere in steps 4-6 described in my last post, I did indeed lose this most recent data on the Android device (where the only copy of that data resided), but I'm still struggling to understand which step exactly is to blame. Perhaps step 6, the deletion of the new DAVx5 account, resulted in the deletion of the local Android calendar collections, and because they were identically named as on the old DAVx5 account, they were then gone despite the old account still "sitting there"? At the same time, I'm not sure whether identically named collections associated with two different DAVx5 accounts even get into each other's hair like that - would those even access the same memory/database space on Android?

If anyone could shed some light on which step exactly might have triggered the data loss, that would certainly be appreciated as helpful towards the avoidance of similar incidents in the future.

[devvv4ever]

When you add events on Android they are all basically stored in the Android Calendar storage. It is the general database for events where all calendars apps read/write data to it. If you assign an event to the DAVx5 account, DAVx5 knows that it is responsible for syncing it from there. If you delete DAVx5 or when a resource is deactivated in the DAVx5 account (and press sync) the calendar bound to DAVx5 and thus all assigned entries get removed from this database (regardless of its sync status). It is a system process deleting all references to avoid leaving a mess. This is performed y the Android system not directly by DAVx5.
At some point this must have happened I guess since you got an older status from the server and loosing all local unsynced entries. After a few unsuccesful sync tries DAVx5 displays a warning (if the user has not disabled notifications). I/O errors are considered a soft error, since this often happens regularily during the day (think of going through a tunnel or into the basement where there is suddenly no internet connection or something like that). Soft errors are displayed not instantly to provide a smoother user experience, but after a few unsuccessful syncs.

To avoid this you can only use hostnames instead of direct IPs – then it is not relevant if the IP once changes if the server itself is staying the same.

[floholl]

Thanks, this is a very clear breakdown of what's going on behind the scenes. I think what most likely must have happened is that I deactivated the resource in DAVx5 and then pressed sync, as you describe above - ironically perhaps even in an effort to prevent further disastrous syncing to occur.

Yes, hostnames... what a fantastic idea! If only different routers and protocols were handling them in a predictable and compatible manner (looking at you, Apple and .local). Switching between manufacturers is precisely what prompted me to use IPs instead of host names in the first place, IIRC.

Anyway, now I know what (not) to do whenever I change routers, so thanks for all the time and effort to explain the nuts and bolts of it.