[YouTube] Update iOS client and add visitor data to InnerTube requests #1262

Theta-Dev · 2025-01-19T19:05:58Z

I carefully read the contribution guidelines and agree to them.
I have tested the API against NewPipe.
No API changes

I have fixed the issue with the iOS client returning invalid responses
Apparently the only issue with the iOS client is that we were not sending it visitor data cookies. Since the visitor data cookies are just protobuf objects with a random ID, timestamp and country code, they can be randomly generated and dont need to be fetched from YouTube.

Fixes #1259
Fixes TeamNewPipe/NewPipe#11934

gechoto · 2025-01-19T19:40:27Z

Does it only work with visitor data for you? What happens only with an updated IOS client but no visitor data?

Btw the visitor data could also be extracted from https://www.youtube.com/sw.js_data

Stypox · 2025-01-19T20:30:29Z

I just tested in NewPipe and it works, thank you!

Theta-Dev · 2025-01-19T20:33:50Z

Yes, it only works with the visitor data cookie. I first tried updating the client version and the issue was still there.

You can extract the visitor data cookie from YouTube, but that requires an extra request. I have been using random visitor data cookies in my own YouTube client for a few months now and they work without issues.

gechoto · 2025-01-19T22:46:26Z

Yes, it only works with the visitor data

Okay. I closed my PR, let's go with visitor data. Cool stuff, thank you.

You can extract the visitor data cookie from YouTube, but that requires an extra request.

I know I wanted to leave this info here just in case as a backup plan. If the format of the visitor data changes your solution can break. Fetching it from YT ensures it will always follow the official and up-to-date format.
(But with that the extraction might fail at some point.)

I'm not sure what is better but the random generator seems okay for now.

visitor data cookie

just out of curiosity why do you refer to it as "cookie"?
I do not see this being handled or sent as a cookie in code. Am I missing something or is this due to translation problems / lack of a better word?

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/ProtoBuilder.java

Theta-Dev · 2025-01-19T23:40:59Z

YouTube does store the visitor data as a cookie (__Secure-YEC), but probably also in local storage so it can be sent as an API parameter by the player code. It fulfills the role of an unique session ID so I call it a cookie. I dont know if there are better terms for it.

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

gechoto · 2025-01-21T14:32:29Z

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

@@ -222,28 +223,28 @@ private YoutubeParsingHelper() {
    private static final String IOS_DEVICE_MODEL = "iPhone16,2";


The device used here is an iPhone 15 Pro Max but the comments are incorrectly update to "iPhone 16".

Either change/revert the comments to iPhone 15 Pro Max or update the device model to "iPhone17,2" to match the iPhone 16 Pro Max.

The mapping can be found here: https://gist.github.com/adamawolf/3048717
or: https://theapplewiki.com/wiki/Firmware/iPhone/18.x#iPhone_16_Pro_Max

This is a nearly verbatim copy of [1], for now the code is pretty rough and I'm not even sure that this data should be passed for all requests but it does seem to unblock the iOS client for now. [1] TeamNewPipe/NewPipeExtractor#1262

litetex · 2025-01-21T20:37:47Z

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ProtoBuilder.java

+                }
+            }
+        } catch (final IOException e) {
+            throw new RuntimeException(e);


Side note: It may be better to use UncheckedIOException here since it's more specific
This might be also relevant for the rest of the file ;)

litetex

Some minor things that I would change.
But anyway nice PR!

litetex · 2025-01-21T20:40:52Z

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

@@ -1256,6 +1275,7 @@ public static JsonBuilder<JsonObject> prepareIosMobileJsonBuilder(
                        .value("platform", "MOBILE")
                        .value("osName", "iOS")
                        .value("osVersion", IOS_OS_VERSION)
+                        .value("visitorData", randomVisitorData(contentCountry))


Can this be the same as vData?

Maybe add a short comment into the code that documents this

litetex · 2025-01-21T20:41:37Z

extractor/src/main/java/org/schabi/newpipe/extractor/utils/ProtoBuilder.java

+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+
+public class ProtoBuilder {


Some documentation would be nice, e.g. what this class does

This is a nearly verbatim copy of [1], for now the code is pretty rough and I'm not even sure that this data should be passed for all requests but it does seem to unblock the iOS client for now. [1] TeamNewPipe/NewPipeExtractor#1262

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java

…youtube/YoutubeParsingHelper.java Co-authored-by: Stypox <[email protected]>

Stypox

Thank you a lot! I am going to release a new version right now, as I don't have too much time in the next few days and don't want to keep users waiting. So I'm merging this PR despite there being some finishing touches to do still.

@Theta-Dev please open a new PR to address the review comments. And @litetex @gechoto please continue the review in the other PR that is going to be opened. And thanks for reviewing :-)

jamesdrake101 · 2025-01-21T22:15:11Z

Thank you 💯 for fast-tracking this

Partial followup for #1262

Stypox · 2025-02-04T09:29:42Z

@Theta-Dev friendly reminder to open a follow-up PR to add more documentation

This is a nearly verbatim copy of [1], for now the code is pretty rough and I'm not even sure that this data should be passed for all requests but it does seem to unblock the iOS client for now. [1] TeamNewPipe/NewPipeExtractor#1262

Theta-Dev · 2025-02-04T10:02:42Z

AFAIK using random visitor data IDs with the iOS client stopped working this week, so I guess we can remove the visitor data generator again.

fix: update iOS client, add visitor data to YouTube requests

40059ed

gechoto mentioned this pull request Jan 19, 2025

[YouTube] Update IOS client #1261

Closed

3 tasks

gechoto reviewed Jan 19, 2025

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/ProtoBuilder.java Outdated Show resolved Hide resolved

nubesurrealista mentioned this pull request Jan 19, 2025

IOS Player is not valid MaintainTeam/LastPipeBender#19

Closed

7 tasks

gechoto mentioned this pull request Jan 20, 2025

Player fixes z-huang/InnerTune#1789

Open

ShareASmile added bug Issue is related to a bug ASAP Issue needs to be fixed as soon as possible youtube service, https://www.youtube.com/ labels Jan 20, 2025

zbee mentioned this pull request Jan 20, 2025

IOS player response is not valid libre-tube/LibreTube#6982

Closed

3 tasks

Thymelizabeth reviewed Jan 20, 2025

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

tuxayo mentioned this pull request Jan 20, 2025

Videos not playing anymore polymorphicshade/Tubular#192

Closed

7 tasks

jamesdrake101 reviewed Jan 20, 2025

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

move ProtoBuilder, add tests, fix incompatible rand call

65d888f

Theta-Dev force-pushed the fix/ios-client-vdata branch from 28a9bdf to 65d888f Compare January 21, 2025 02:06

ShareASmile requested changes Jan 21, 2025

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

update comment

45645b0

gechoto reviewed Jan 21, 2025

View reviewed changes

2b-zipper mentioned this pull request Jan 21, 2025

Fix playback issue erievs/FourthTube#71

Merged

litetex reviewed Jan 21, 2025

View reviewed changes

litetex mentioned this pull request Jan 21, 2025

[FIXED in 0.27.5] [YouTube] IOS player response is not valid error when loading some videos TeamNewPipe/NewPipe#11934

Closed

6 tasks

Stypox reviewed Jan 21, 2025

View reviewed changes

iPhone 15 Pro Max instead of 16 in comments

5b31ff2

Stypox reviewed Jan 21, 2025

View reviewed changes

extractor/src/main/java/org/schabi/newpipe/extractor/services/youtube/YoutubeParsingHelper.java Outdated Show resolved Hide resolved

Update extractor/src/main/java/org/schabi/newpipe/extractor/services/…

936bf2d

…youtube/YoutubeParsingHelper.java Co-authored-by: Stypox <[email protected]>

Stypox approved these changes Jan 21, 2025

View reviewed changes

Stypox merged commit 4c72032 into TeamNewPipe:dev Jan 21, 2025
3 checks passed

jdesgats mentioned this pull request Jan 22, 2025

Pass visitorData to requests kkdai/youtube#355

Open

PJansky mentioned this pull request Jan 23, 2025

"Unknown error" while playing any new music Malopieds/InnerTune#463

Open

4 tasks

AudricV changed the title ~~[YouTube] update iOS client, add visitor data to requests~~ [YouTube] Update iOS client and add visitor data to InnerTube requests Jan 24, 2025

Ales315 mentioned this pull request Jan 26, 2025

Update iOS client and add random visitor data to requests Tyrrrz/YoutubeExplode#865

Merged

Stypox added a commit that referenced this pull request Jan 26, 2025

Merge pull request #1263 from litetex/followup-1262

e0359ab

Partial followup for #1262

gechoto mentioned this pull request Jan 31, 2025

[YouTube] 2nd occurrence of "IOS player response is not valid error" when loading some videos TeamNewPipe/NewPipe#11980

Closed

6 tasks

sheepmax mentioned this pull request Feb 1, 2025

Bugfix: Extract visitor data from sw.js_data API instead of randomly… #1271

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[YouTube] Update iOS client and add visitor data to InnerTube requests #1262

[YouTube] Update iOS client and add visitor data to InnerTube requests #1262

Theta-Dev commented Jan 19, 2025 •

edited by AudricV

Loading

gechoto commented Jan 19, 2025 •

edited

Loading

Stypox commented Jan 19, 2025

Theta-Dev commented Jan 19, 2025

gechoto commented Jan 19, 2025 •

edited

Loading

Theta-Dev commented Jan 19, 2025

gechoto Jan 21, 2025 •

edited

Loading

litetex Jan 21, 2025

litetex left a comment

litetex Jan 21, 2025

litetex Jan 21, 2025

Stypox left a comment

jamesdrake101 commented Jan 21, 2025

Stypox commented Feb 4, 2025

Theta-Dev commented Feb 4, 2025 •

edited

Loading

		@@ -222,28 +223,28 @@ private YoutubeParsingHelper() {
		private static final String IOS_DEVICE_MODEL = "iPhone16,2";

[YouTube] Update iOS client and add visitor data to InnerTube requests #1262

[YouTube] Update iOS client and add visitor data to InnerTube requests #1262

Conversation

Theta-Dev commented Jan 19, 2025 • edited by AudricV Loading

gechoto commented Jan 19, 2025 • edited Loading

Stypox commented Jan 19, 2025

Theta-Dev commented Jan 19, 2025

gechoto commented Jan 19, 2025 • edited Loading

Theta-Dev commented Jan 19, 2025

gechoto Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

litetex Jan 21, 2025

Choose a reason for hiding this comment

litetex left a comment

Choose a reason for hiding this comment

litetex Jan 21, 2025

Choose a reason for hiding this comment

litetex Jan 21, 2025

Choose a reason for hiding this comment

Stypox left a comment

Choose a reason for hiding this comment

jamesdrake101 commented Jan 21, 2025

Stypox commented Feb 4, 2025

Theta-Dev commented Feb 4, 2025 • edited Loading

Theta-Dev commented Jan 19, 2025 •

edited by AudricV

Loading

gechoto commented Jan 19, 2025 •

edited

Loading

gechoto commented Jan 19, 2025 •

edited

Loading

gechoto Jan 21, 2025 •

edited

Loading

Theta-Dev commented Feb 4, 2025 •

edited

Loading