DavX show notification NO Internet, Planning Synch, but Internet there is.

[WillemsenGithub]

DAVx latest and some previous versions show a notification about No Internet, planning synch.
Thats not true, since there is. All other apps work fine, have Internet.

All connections are allowed, Wifi and Data.
No restrictions in app, all permissions set to allow
I use VPN
Android, LineageOS 12

Can you please stop with this rather discomforting message ?

Thanks for all the work by the way !

[ArnyminerZ]

Hello, sorry for the inconvenience. VPNs are still a bit strange in Android, and sometimes it detects that there is no Internet connection when there's one available. We are working on a solution for it, that should fix this issue that we know affects a lot of you.

[WillemsenGithub]

Thanks for quick reply.
Davx seems to reach the site to synch with.
It it helps: in the Debug Log is detects TUN0 correctly, no signs of no connection
All Permissions on Connection: Granted

[rfc2822]

To prevent misunderstanding: Prior to v4.3.5-ose it worked in the VPN-Scenario.

Then the sync framework seems to have detected connectivity (it uses JobScheduler + NETWORK_TYPE_ANY).

The problem is that we have used the analogous WorkManager check (WorkManager.CONNECTED), which then causes problems with VPNs that are "always on", even if there is no Internet connection: #336

What about just "pinging" (or sending a request, or just trying to "blindly" sync) to the desired server and examining the response for known errors?

How would we then differentiate between errors that can be ignored (because there's no connectivity) and connection errors that the user should be notified about?

There in line 418 VPNs were excluded intentionally. So i assume the method internetAvailable() at least needs some tuning or needs to generally be omitted in certain cases like VPN.

See above: there are "VPNs" that are always on (like Blokada), which cause the Internet check to always return "yes, there's a valid Internet connection", even if there was actually no Internet. So the DAVx5 work was started, but it failed and showed user notifications.

So our next try was to filter out VPNs, whether connected or not, and see whether there's a "underlying Internet connection" in the system.

You shouldn't be able to access the "unterlying internet connection" from your app in case of a systemwide VPN.

As I understand it, the "underlying Internet connection" may not be accessible, but should at least be visible in the list so that we can check against it. But maybe this is wrong…

[rfc2822]

@dr-death2 Do you have an idea why your Internet connection (taken from the list above) has INTERNET, but not VALIDATED?

☐ rmnet0
[ Transports: CELLULAR Capabilities: SUPL XCAP INTERNET NOT_RESTRICTED TRUSTED NOT_VPN NOT_ROAMING FOREGROUND NOT_CONGESTED NOT_SUSPENDED NOT_VCN_MANAGED LinkUpBandwidth>=15000Kbps LinkDnBandwidth>=14847Kbps Specifier: <TelephonyNetworkSpecifier [mSubId = 1]>]

Is there a firewall that blocks the Android system from validating this connection (they say they use captive portal detection / DNS for validation)?

[dr-death2]

Do you have an idea why your Internet connection (taken from the list above) has INTERNET, but not VALIDATED?

At least there is no firewall explicitly blocking Android. But depending on how the validation is done in detail, things might end up similar.

In our setup the mobile devices have

a) systemwide VPN
b) systemwide HTTP-Proxy inside the VPN

                     -> [Cal/CardDavServer]
                   /
[Mobile] -> [VPN] -> [Proxy] -> [Internet]

All connections from the mobile to the [Internet] were tunnelled through the VPN and the proxy. In the proxy some filter rules apply that might prevent the mobile client from accessing certain domains (i.e. many google domains). It might also be possible that a mobile client has no access to the [Internet] at all and only stays inside the private network (but this does not apply to my test case).

So eventually my mobile client >has< internet acccess, but may be not in a way the validation is able to detect. Furthermore a validated >Internet< access is not neccessary at all for the sync to work in the setup shown above.

If you give more details about how the validation works, i'll digg deeper into our logs to give you more information.

[rfc2822]

It might also be possible that a mobile client has no access to the [Internet] at all and only stays inside the private network (but this does not apply to my test case).

OK, because this would be such a case that is not yet supported.

So eventually my mobile client >has< internet acccess, but may be not in a way the validation is able to detect. Furthermore a validated >Internet< access is not neccessary at all for the sync to work in the setup shown above.

No, but then it's the "Sync without validated Internet access" special use-case, which is not yet supported (although we're working on it).

If you give more details about how the validation works, i'll digg deeper into our logs to give you more information.

The only official details I know are here:

NET_CAPABILITY_VALIDATED: indicates that the network provides actual access to the public internet when it is probed. A network behind a captive portal or a network that doesn't provide domain name resolution doesn't have this capability. This is the nearest the system can tell about a network actually providing access, although a validated network can still, in principle, be subject to IP-based filtering or suffer sudden losses of connectivity due to issues such as poor signal.

[dr-death2]

No, but then it's the "Sync without validated Internet access" special use-case, which is not yet supported (although we're working on it).

Do you have any idea when the "special use case" might be supported (days, weeks, months)?

[dr-death2]

regarding reliability of NET_CAPABILITY_VALIDATED:

"...you can't rely on NET_CAPABILITY_INTERNET && NET_CAPABILITY_VALIDATED..."
(https://stackoverflow.com/questions/36358531/internet-check-over-android-net-networkcapabilities-not-working)

regarding how NET_CAPABILITY_VALIDATED is determined:

private static final String DEFAULT_CAPTIVE_PORTAL_HTTP_URL = "http://connectivitycheck.gstatic.com/generate_204";
(https://stackoverflow.com/questions/62541728/how-does-google-validate-network-connectivity-when-using-net-capability-validate)

So this might be an explanation. I understand it the way, that the VALIDATED-check tries to access a hardcoded google-domain (*.gstatic.com). gstatic.com indeed is a domain that is on our block list.

If I'm right, a solution would be to override the predefined gstatic-domain and use the servername instead. That should give a reliable impression wether the the sync could be started or not.

BTW: By this behaviour DAVx⁵ exposes users to google tracking...

[rfc2822]

regarding reliability of NET_CAPABILITY_VALIDATED: […]

Well, WorkManager API and Android APIs which are involved in synchronization rely on it. We currently only support connections which are supported by the Android API and which don't report wrong values. We do plan to support special cases in the future (see below), but at the moment, we don't. (We have enough to do with supporting common use-cases; for instance, we had to rewrite anything from sync framework to WorkManager for so little gain as that sync is possible even when the "storage is low", which is often reported even when there are many GB free…)

So this might be an explanation. I understand it the way, that the VALIDATED-check tries to access a hardcoded google-domain (*.gstatic.com). gstatic.com indeed is a domain that is on our block list.

Then I recommend to unblock connectivitycheck.gstatic.com. You can view the source code and/or track the requests (HTTP without S if I see correctly) and see that no personal data (cookies etc) are sent when validating the connection.

If I'm right, a solution would be to override the predefined gstatic-domain and use the servername instead. That should give a reliable impression wether the the sync could be started or not.

As far as I know, there's a possibility to change the URL when you have root access: https://android.stackexchange.com/questions/186993/captive-portal-parameters At least this was possible a few years ago, when I last tried it.

However, this check is done by the system and not DAVx5.

BTW: By this behaviour DAVx⁵ exposes users to google tracking...

No, because 1. it's a system check, and Android works like it works. We have to rely on Android APIs for synchronization. We however work on support for special cases where the Android system says there's "no Internet", but these are special cases and need much extra work (for instance to keep sync with public login WiFi networks working like it is). And 2. no personal data is sent with this validation (and if you have root access, you can also change the server name).

Does it work when you unblock connectivitycheck.gstatic.com?

[dr-death2]

And 2. no personal data is sent with this validation (and if you have root access, you can also change the server name).

Not explixitly by DAVx⁵, correct. But any request to any server exposes the client IP (which may count as personal data, depending on local law), my network, my location, the client software that filed the request, the version of it, my operation system and the version of it (all the latter is PII).

Does it work when you unblock connectivitycheck.gstatic.com?

No. That does not help. The proxy-logs show, that during a DAVx⁵-sync not a single request reaches the proxy. Neither connectivitycheck.gstatic.com nor any other destination.

So i conclude, that the NET_CAPABILITY_VALIDATED-test tries to access the hard-coded test-domain >bypassing< the VPN and/or proxy, which is not possible on the client phone.

Because NET_CAPABILITY_VALIDATED seems tightly related to the captive-portal-test we tried the following:

deactivate captive-portal-test via ADB:
adb shell settings put global captive_portal_mode 0

(for reactivation use adb shell settings delete global captive_portal_mode)

And that made the sync work again! That's not a solution, but it might help narrowing down the root cause.

Another fact becomes clear to me, that seems pretty strange and unintentional:
Using NET_CAPABILITY_VALIDATED implies a network request to a pre-defined, hard-coded server x. For the usecase we discuss, it is only important if DAVx⁵ is able to reach server y. Whichever result a test returns for the accessibility of x - it tells nothing about the accessibility of y. Servers may be down, unreachable, blocked, routes may be corrupt etc. Even a VALIDATED result does not guarrantee the sync-server y to be accessible (it could be down/offline for instance). Vice versa a NON-VALIDATED result also does not imply that the sync server y is not accessible (i.e. the usecase we discuss).

Because of its non-information i think that using NET_CAPABILITY_VALIDATED is the root problem. Three possible solutions come to my mind:

1. replacing NET_CAPABILITY_VALIDATED with an own test that tests accessibility >of the desired< server
2. gernerally avoid relying on NET_CAPABILITY_VALIDATED-test results
3. make the usage of NET_CAPABILITY_VALIDATED-test results optional (i.e. by config setting)

[rfc2822]

And 2. no personal data is sent with this validation (and if you have root access, you can also change the server name).

Not explixitly by DAVx⁵, correct. But any request to any server exposes the client IP (which may count as personal data, depending on local law), my network, my location, the client software that filed the request, the version of it, my operation system and the version of it (all the latter is PII).

As far as I know, no headers like software version and operation system are sent. You can verify this by looking at the request at network level. Of course, the IP address and network-stack level OS charactistics are revealed.

However, this is how Android works. Android has network APIs and they rely on this check. Apps like DAVx5 that rely on OS network APIs get wrong information if this check is blocked.

Yes, we want to support such very special use-cases, but

1. first priority is to support more common use-cases,
2. the special use-cases have much potential to break everything (including more common use cases) and are much work for a little number of users.

Personally, I'd love a solution that doesn't even need the gstatic.com check, but it's currently not supported. We have already collected currently 10 use-cases (list is growing) where the most ones are very special and often have opposite requirements then the common use-cases. We plan to support more special use-cases in the future.

Does it work when you unblock connectivitycheck.gstatic.com?

No. That does not help. The proxy-logs show, that during a DAVx⁵-sync not a single request reaches the proxy. Neither connectivitycheck.gstatic.com nor any other destination.

When Android validates your rmnet0 (CELLULAR) connection, it will send the request to connectivitycheck.gstatic.com over the
*rmnet0 interface. If it succeeds, the VALIDATED flag will be added to rmnet0.

So according to your diagram, the request will indeed not be sent to your proxy.

So i conclude, that the NET_CAPABILITY_VALIDATED-test tries to access the hard-coded test-domain >bypassing< the VPN and/or proxy, which is not possible on the client phone.

Not bypassing – validating the rmnet0 connection (by sending the connectivitycheck.gstatic.com request) and validating the VPN connection (which is done by the VPN software, as far as I know) are two different things. And yes, if your phone blocks connectivitycheck.gstatic.com requests on the underlying Internet connection (rmnet0), rmnet0 won't be VALIDATED (as to see in your debug info). Then DAVx5 wont synchronize, because it can't rely on the VALIDATED flags of VPNs, so it filter's all VPN connections and only looks at the underlying network connections. It sees that you don't have a working non-vpn network connection and won't synchronize because this use-case is not supported at the moment.

Another fact becomes clear to me, that seems pretty strange and unintentional: Using NET_CAPABILITY_VALIDATED

Only to make it clear: we didn't invent NET_CAPABILITY_VALIDATED. We're here talking about Android design decisions, which influence the sync framework, the WorkManager etc. I'm not going to defend the Android network design here. I'm just saying that this validation is an integral part of how Android works, and it works fine with the most common use-cases like public WiFis which require a login.

implies a network request to a pre-defined, hard-coded server x. For the usecase we discuss, it is only important if DAVx⁵ is able to reach server y. Whichever result a test returns for the accessibility of x - it tells nothing about the accessibility of y. Servers may be down, unreachable, blocked, routes may be corrupt etc. Even a VALIDATED result does not guarrantee the sync-server y to be accessible (it could be down/offline for instance).

Of course, but the logic is: if the device doesn't have a working connection, it's useless to start DAVx5 to try synchronization. It would need battery without need. So apps can set flags like "start this work only when there is a working network connection", and this is what apps that require a network connection usually do (like DAVx5 at the moment). Of course you can let Android start the job without network connection, but then you will have other users which complain about increased battery usage etc.

However we're planning to improve support for these special use-cases.

[dr-death2]

Of course, but the logic is: if the device doesn't have a working connection, it's useless to start DAVx5 to try synchronization.

Understand. I wanted to point out, that the way NET_CAPABILITY_VALIDATED works, is unsuitable to rely the above logic on.

However we're planning to improve support for these special use-cases.

Do you have any clue how long it could take to support the VPN/Proxy usecase? Are we talking about some days, weeks or months?

[rfc2822]

VPN/proxy is no problem, it's currently supported ;) But "VPN without validated Internet on the underlying connection" is on the roadmap. I'd like to have it rather earlier than later, but earlier than weeks is illusory I think. So I'd guess weeks to a few months.

[dr-death2]

Does https://github.com/bitfireAT/davx5-ose/releases/tag/v4.3.5.2-ose contain any changes regarding this issue?

[rfc2822]

No; you can always find the changelogs in the release thread.

[woody106]

Hi everybody,
for two days now I had the same problem: Internet connection by wlan or mobile data; an openvpn connection to the company (tine-2.0 and davical servers) up and running and trying to sync contacts with DAVx5. But no syncing due to "there is no connection to the internet".

My solution now is: I deleted the actual app and restored davx5 with version 4.1.1-ose - now syncing of all contacts works like a charm again.

IMHO another example for "never touch a running system" ... ;)

Best regards, Woody

[rfc2822]

Can you please post the debug info connection part (of 4.1.x or 4.3.x, doesn't matter)? Normal connections with WiFi or mobile data that don't block Internet shouldn't cause any problems, OpenVPN or not.

[woody106]

Sadly I was too fast: Only 298 of 413 contacts have been synchronized. OK, better than nothing but not really good.

@rfc2822: I'll send you a log file.

Best regards, Woody

[woody106]

Am 10.08.2023 um 20:33 schrieb Ricki Hirner:

Can you please post the debug info connection part? — Reply to this email directly, view it on GitHub <#363 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ7Q4I3OUDSLQ4ZWL2KXAM3XUUSO7ANCNFSM6AAAAAA2QCIBZM>. You are receiving this because you commented.Message ID: ***@***.***>

see the attachment; all technical data are enclose, I only did a clean up of privacy data. Actual problem: Only 298 of 419 contacts are synchronized, but the older app works better than the new one, where not a single contact was transferred. Best regards

[woody106]

Last week I did an update to V4.3.5.2-ose - SSDD (same shit different day)
Again no syncing of my contacts - can anyone please help me in finding a DAVx5 Version 3.x?

Best regards,

[woody106]

Hi Ricki

Can you please post the debug info connection part? — Reply to this email directly, view it on GitHub <#363 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AZ7Q4I3OUDSLQ4ZWL2KXAM3XUUSO7ANCNFSM6AAAAAA2QCIBZM>. You are receiving this because you commented.Message ID: ***@***.***>

Did you see any reasons for DAVx5 not synchronizing CardDAV over VPN? CardDAV still not working on my shift phone. See the Discussions in #363 Best regards, woody106

[woody106]

Still an open question: Are older versions of DAVx5 Version (i.e. 3.x) available for download?
I'd like to give them a try and do some testing.

Best regards,
woody106

[dr-death2]

@rfc2822: I retested with v4.3.6.1-ose and can confirm that the sync works again for me after disabling the new settig "VPN connectivity" in the account settings. That's great. Thanks for your work and the fix.

Nevertheless i noticed some related, non-intuitive behaviour i'd like to address so that you have the opportunity to improve usability and user experience.

1. non intuitive switch:
   Switching "VPN connectivity" off in case one uses VPN connectivity is non intuitive. You should change the name of the setting or invert the boolean value to match users expectations.

2. misleading warnings
   Although the behaviour is fixed, the former warnings about "missing internet connection" and "No Internet, planning sync." are still displayed. Seems the message handling does not adapt the recently changed logic. See attachment.

3. In case the internet connection is provided through the VPN this is a global issue. So managing the new "VPN"-setting once in the global applications settings instead of multiple times in each accounts settings seems more appropriate.

[maltfield]

Can we just add a feature to the settings that tellls DavX to never "check to see if it thinks we have Internet or not".

I'm still getting the error No internet, scheduling sync in v4.3.12.1-ose from F-Droid. My internet is working fine. Why can't I just force DavX to at least try to connect?!?

Many users are going to block their OS from "calling home" so honestly it's pretty dumb to expect that the OS knows whether or not we have Internet. Why not just try the connection rather than wait forever with faulty logic?

Only to make it clear: we didn't invent NET_CAPABILITY_VALIDATED. We're here talking about Android design decisions

Obviously it's in Google's interest to collect as much data on as many Android users as possible. Just because Google tells Android developers to do something, doesn't mean you should do it. Most apps seem to know that NET_CAPABILITY_VALIDATED isn't going to be axiomatic in knowing if Interenet is working or not. I've never encountered an app refusing to use my functioning Intrenet before DavX, even if Android has a pestering message in my wifi notifications telling me that it can't call home (poor poor Google), so it seems like most devs are wisely not following everything that Google tells them to do.

A lot of users have such security protections in-place (see above), so it's not much of an edge-case. Is there any benefit at all to having the NET_CAPABILITY_VALIDATED logic in-place?

Can you please remove all of the NET_CAPABILITY_VALIDATED logic from the app?

[maltfield]

What version was this bug introduced? Which version do we need to downgrade-to to get DavX sycning again on secure Android devices?

Update: it looks like this bug was introduced in this commit on 2023-05-11

• 2ebb40b

Update 2: A workaround until this bug is fixed it to downgrade to v4.3.2

[maltfield]

See also:

• No Internet connectivity. Android will not run synchronization. #245