Doesn't Support extended URL with hmac Info

[Thiago-Heleno]

The provided m3u8 cant be proxied due to the way HLS-Proxy handles m3u8 files

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:5
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="skd://drmtoday?assetId=9789e0f1e78545a792860ba514daa879&variantId=avkey",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"
#EXTINF:4.004,
https://a-vrv.akamaized.net/evs3/11a41c35485da2c7d60c3b859e616a05/assets/p/9789e0f1e78545a792860ba514daa879_4830206.mp4/seg-1-v1-a1.ts?t=exp=1680116387~acl=/evs3/11a41c35485da2c7d60c3b859e616a05/assets/p/9789e0f1e78545a792860ba514daa879_4830206.mp4/*~hmac=c98f2081f7b19cdf27deca1b898c36f7c482765cb83e60555b5fd8f2ba1061d4
#EXTINF:4.004,
https://a-vrv.akamaized.net/evs3/11a41c35485da2c7d60c3b859e616a05/assets/p/9789e0f1e78545a792860ba514daa879_4830206.mp4/seg-2-v1-a1.ts?t=exp=1680116387~acl=/evs3/11a41c35485da2c7d60c3b859e616a05/assets/p/9789e0f1e78545a792860ba514daa879_4830206.mp4/*~hmac=c98f2081f7b19cdf27deca1b898c36f7c482765cb83e60555b5fd8f2ba1061d4
#EXTINF:4.004,

[warren-bank]

it works fine..
I tested your assertion by doing the following:

1. copied the text for your manifest to a file: video.m3u8
2. started a local web server (on port 80) to host this manifest at: http://127.0.0.1/video.m3u8
3. started hls-proxy (on port 8080) to proxy this manifest: http://127.0.0.1:8080/aHR0cDovLzEyNy4wLjAuMS92aWRlby5tM3U4.m3u8
4. downloaded the manifest from hls-proxy
5. inspected its content to determine whether the URLs for video segments were properly rewritten
   • they were

this is the content of this proxied manifest:

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-ALLOW-CACHE:YES
#EXT-X-PLAYLIST-TYPE:VOD
#EXT-X-VERSION:5
#EXT-X-MEDIA-SEQUENCE:1
#EXT-X-KEY:METHOD=SAMPLE-AES,URI="http://127.0.0.1:8080/aHR0cDovLzEyNy4wLjAuMS9za2Q6Ly9kcm10b2RheT9hc3NldElkPTk3ODllMGYxZTc4NTQ1YTc5Mjg2MGJhNTE0ZGFhODc5JnZhcmlhbnRJZD1hdmtleQ==.key",KEYFORMAT="com.apple.streamingkeydelivery",KEYFORMATVERSIONS="1"
#EXTINF:4.004,
http://127.0.0.1:8080/aHR0cHM6Ly9hLXZydi5ha2FtYWl6ZWQubmV0L2V2czMvMTFhNDFjMzU0ODVkYTJjN2Q2MGMzYjg1OWU2MTZhMDUvYXNzZXRzL3AvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfNDgzMDIwNi5tcDQvc2VnLTEtdjEtYTEudHM/dD1leHA9MTY4MDExNjM4N35hY2w9L2V2czMvMTFhNDFjMzU0ODVkYTJjN2Q2MGMzYjg1OWU2MTZhMDUvYXNzZXRzL3AvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfNDgzMDIwNi5tcDQvKn5obWFjPWM5OGYyMDgxZjdiMTljZGYyN2RlY2ExYjg5OGMzNmY3YzQ4Mjc2NWNiODNlNjA1NTViNWZkOGYyYmExMDYxZDQ=.ts
#EXTINF:4.004,
http://127.0.0.1:8080/aHR0cHM6Ly9hLXZydi5ha2FtYWl6ZWQubmV0L2V2czMvMTFhNDFjMzU0ODVkYTJjN2Q2MGMzYjg1OWU2MTZhMDUvYXNzZXRzL3AvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfNDgzMDIwNi5tcDQvc2VnLTItdjEtYTEudHM/dD1leHA9MTY4MDExNjM4N35hY2w9L2V2czMvMTFhNDFjMzU0ODVkYTJjN2Q2MGMzYjg1OWU2MTZhMDUvYXNzZXRzL3AvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfNDgzMDIwNi5tcDQvKn5obWFjPWM5OGYyMDgxZjdiMTljZGYyN2RlY2ExYjg5OGMzNmY3YzQ4Mjc2NWNiODNlNjA1NTViNWZkOGYyYmExMDYxZDQ=.ts

so.. please tell me precisely.. what isn't working?

[warren-bank]

oh.. wait

I see something that I don't like..

the original URI for the key uses protocol: skd://..
the proxy won't be able to download that..
so maybe (probably) before it rewrite a URL, it should check that the original protocol is either: http or https

[warren-bank]

alternatively..

https://developer.apple.com/forums/thread/50560

not sure if this is documented anywhere,
but if what this person suggests is true:

• skd:// -> http://
• skds:// -> https://

then the proxy can download these files..
it just needs to silenty rewrite the protocol before fetching

[warren-bank]

yep, here is a quick confirmation that the protocol used by the key file is a problem:

curl -I "http://127.0.0.1:8080/aHR0cDovLzEyNy4wLjAuMS9za2Q6Ly9kcm10b2RheT9hc3NldElkPTk3ODllMGYxZTc4NTQ1YTc5Mjg2MGJhNTE0ZGFhODc5JnZhcmlhbnRJZD1hdmtleQ==.key"

HTTP/1.1 500 Internal Server Error

and written to the hls-proxy console log:

proxying: http://127.0.0.1/skd://drmtoday?assetId=9789e0f1e78545a792860ba514daa879&variantId=avkey

ERROR: HTTP response status code: 404

[Thiago-Heleno]

I tried again with a still valid m3u8 file and I couldn't download the key.

Here's the main ml3u8 file that contains the resolutions:

https://pl.crunchyroll.com/evs3/9a1c55384e15c3d7a9e7ac0a41a65326/assets/9789e0f1e78545a792860ba514daa879_,4830206.mp4,4830214.mp4,4830198.mp4,4830182.mp4,4830190.mp4,.urlset/master.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cCo6Ly9wbC5jcnVuY2h5cm9sbC5jb20vZXZzMy85YTFjNTUzODRlMTVjM2Q3YTllN2FjMGE0MWE2NTMyNi9hc3NldHMvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfLDQ4MzAyMDYubXA0LDQ4MzAyMTQubXA0LDQ4MzAxOTgubXA0LDQ4MzAxODIubXA0LDQ4MzAxOTAubXA0LC51cmxzZXQvbWFzdGVyLm0zdTgiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODAxOTc2MzR9fX1dfQ__&Signature=ejPcfpdtlfyJY0wm2ETg9P-uyklOzvBOhC0wPWn75G4y6a5-K9TgijBO529xsIHkENUbQkEvOHnzVNWlqY~YxVH74pXKPq-Lpmz~gDnh1FzJBwbb-G8CTXRZIa5qodUSy2KvcgleOJF8Zibk91P~Zv9ZV07NYs3pmTgHHLGKZVKCeLdRACN1uLsYrG8oC8QjmW0ebXhxy8DvUo0JHxWUZ-wAW~BsRL59yZSmPykAlBAbhQ2O7JzkvYxSCJ2x59CgmcTOy4nfJQb~9tnBqUbnOWrbNvzIJG6t2QvhD7tbZhVfdFA6qm0WAVrreLsxn95oXcywlPdYiTNA9lWoZEy24A__&Key-Pair-Id=APKAJMWSQ5S7ZB3MF5VA

Here's one Resolution Url:

https://pl.crunchyroll.com/evs3/9a1c55384e15c3d7a9e7ac0a41a65326/assets/9789e0f1e78545a792860ba514daa879_4830206.mp4/index-v1-a1.m3u8?res=1280x720&Expires=1680198295&Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9wbC5jcnVuY2h5cm9sbC5jb20vZXZzMy85YTFjNTUzODRlMTVjM2Q3YTllN2FjMGE0MWE2NTMyNi9hc3NldHMvOTc4OWUwZjFlNzg1NDVhNzkyODYwYmE1MTRkYWE4NzlfNDgzMDIwNi5tcDQvaW5kZXgtdjEtYTEubTN1OD9yZXM9MTI4MHg3MjAiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2ODAxOTgyOTV9fX1dfQ__&Signature=Dfkx~YtGg~UMUvUtPqu8v7qpDS9Pqf7GJwGV3vmRTKoIz-5KohC6~is6Xoovzq623r78yPgp4zhpYt7unFCez715Hc7-0u7d0M6toOmmKG5~93yf777hE7TXqhTcVutrYFTRLo9x2WVU9f8U01og2jsGf8CU10h1g5GMWxZ5MzNfYfFoMphU1nt13mlrRLQeQM5jftGMswV9goYM6H-Urne0yQuO-p~NjqvEgaiqEJd3TgEtBhWzOA9985TkrthYQZ5Kiue-6p3Xecq4MSzDRX9awB~Rhg~NGkPLY-D7o5Tk4oC3AGCo22KI7XUpHhH442SeDAJJyU7FVAeb7PRQOw__&Key-Pair-Id=APKAJMWSQ5S7ZB3MF5VA

[warren-bank]

my thinking is..

1. add a new cli option, ex: --skd
2. while rewriting urls in a manifest, for each url that doesn't have a supported protocol: http, https
   • if (protocol is: skd, skds) and (---skd)
     • rewrite protocol, as suggested
   • retest: if url doesn't have a supported protocol: http, https
     • ignore, and pass the unsupported url through to the proxied manifest

[Thiago-Heleno]

That would work, We just need to find a way to process the skd request then

[warren-bank]

oh.. wait.. you're right..

what would hls-proxy do with this?

skd://drmtoday

rewriting the protocol to:

http://drmtoday

wouldn't work.. the hostname doesn't contain a top-level domain.
I have no idea how the "skd:" URI should be resolved.

I guess I'll not add any cli option --skd, but I will add a check that rewritten URLs must contain a supported protocol.

[warren-bank]

fun fact..
I just found a bug in one of the dependencies (which I also wrote).
The bug is seen here:

https://github.com/warren-bank/HLS-Proxy/blob/v3.3.0/hls-proxy/manifest_parser.js#L181
https://github.com/warren-bank/HLS-Proxy/blob/v3.3.0/hls-proxy/manifest_parser.js#L201

for example:

{
  const m3u8_url         = 'http://127.0.0.1/video.m3u8'
  const matching_url     = 'skd://drmtoday?assetId=9789e0f1e78545a792860ba514daa879&variantId=avkey'
  let resolved_match_url

  {
    const {URL} = require('@warren-bank/url')
    resolved_match_url = (new URL(matching_url, m3u8_url)).href
    console.log('mine:', resolved_match_url)
  }

  {
    const {URL} = require('url')
    resolved_match_url = (new URL(matching_url, m3u8_url)).href
    console.log('node:', resolved_match_url)
  }
}

produces the output:

mine: http://127.0.0.1/skd://drmtoday?assetId=9789e0f1e78545a792860ba514daa879&variantId=avkey
node: skd://drmtoday?assetId=9789e0f1e78545a792860ba514daa879&variantId=avkey

once I fix this..
I'll be able to take the list of URLs extracted from the manifest (as is already the case)..
and pass it though a filter before using it..
to remove any extracted URL that is unsupported (ex: skd).

[warren-bank]

my guess.. before looking at the code.. is that the URL parser uses a regex that requires a TLD.
but.. the obvious follow-up question would be.. how does it handles "localhost"?
TBD..

[warren-bank]

nope.. that wasn't the issue.
the problem is that my parser requires a path.. even if only a "/".

error: skd://drmtoday?assetId=
works: skd://drmtoday/?assetId=

hmm.. I guess I should look into that.
though, strictly speaking.. I'm pretty sure a path is requires by the spec.. and that my parser is correct.

[warren-bank]

ok.. both issues are fixed

1. the one you probably don't care about.. that my URL class now only requires that the url string contain: //host.. all other parts are optional
2. hls-proxy v3.3.1
   • uses the improved URL class
   • validates URLs embedded in manifests (after runing the redirect hook).. and only rewrites them if supported
   • tested and working..

[Thiago-Heleno]

Nice, I'll see if I can find something about the skd and reverse engineer the website im scraping

[Thiago-Heleno]

I've just found something - Apparently the .m3u8 file with this weird key is a drm_adaptive_hls, Idk what that means but I've found one that has the normal https key, It works on my machine.