CloudFront Invalidator

[TheMattSchiller]

This project changes the game for munki! Thank you for making it.

When I implemented this in our testing environment it worked fantastically, however, subsequent changes to the munki repo in S3 were not pushed to CloudFront. The fix for this is to implement a CloudFront invalidator. It is a very small amount of code that can be run as a lambda function, with S3 puts to the munki repo as its trigger, which will invalidate the associated CloudFront objects for each file added to S3.

link to lambda function

If you would like, I can write instructions on how to include this bit which you could add to the readme. I feel like it is a necessary addition as it is much too easy for CloudFront to serve stale content. For example, this issue was revealed to us when we pushed out a pkginfo with an error, and our subsequent changes to remedy the pkginfo file would make it to S3 but not to CloudFront or the clients.

Thank you again!

[clburlison]

You should not invalidate the cache! That costs money and is very heavy handed for the CloudFront CDN. You should instead look into adding time to live values (referred to in some docs as Cache-Control headers). For example /manifests & /catalogs live for 2mins in my cloudfront instances, while /pkgs, /pkginfo, etc have 24 hr time to live.

This is very easy to achieve in the CloudFront web interface. I think it is under the “behaviors” tab.

[DanLockcuff]

We have it scripted in bash to push to s3 and invalidate the whole repo in one shot using a wildcard. At least 3 times daily and have never seen a charge to date for our munki repo. I would be happy to share with you if you want.

On Wed, Oct 4, 2017 at 5:38 PM Clayton Burlison ***@***.***> wrote: You should not invalidate the cache! They costs money and is very heavy handed for the CloudFront CDN. You should instead look into adding time to live values. For example /manifests & /catalogs live for 2mins in my cloudfront instances, while /pkgs, /pkginfo, etc have 24 hr time to live. This is very easy to achieve in the CloudFront web interface. I think it is under the “behaviors” tab. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFBdmJ9qp0ynewAsyTrl03bkQdBlSanRks5spAj7gaJpZM4PuShJ> .

-- [image: --] Dan Lockcuff [image: https://]about.me/danlockcuff <https://about.me/danlockcuff?promo=email_sig&utm_source=email_sig&utm_medium=email_sig&utm_campaign=external_links>

[clburlison]

So you likely aren’t hitting the 1,000 invalidations a month limit after which you pay. With that said I highly recommend both of y’all reading the following. https://www.cloudvps.com/helpcenter/knowledgebase/content-delivery-network-cdn/cdn-cache-control-and-invalidations

Yes setting invalidations will work. No you shouldn’t be using that to “refresh” content. That is not how CDNs are designed. Munki already versions big files. Let’s take an office update for example. That is a ~1.6 GB file, if using the skuless installer, when you invalidate you will remove this from the edge servers. Then the next update will require a pull from s3 to cache to the CloudFront edge servers. Doing this multiple times a day is very inefficient.

Also, CloudFront makes it very easy to set the Cache-Control headers for entire directory paths using the web interface. You can also script this with something like aws sync if that is more you fancy (could even do both).

[erikng]

+1 with what Clayton is saying. I do the same thing by having TTL’s on the manifests and catalogs. Thanks, Erik Gomez

…

________________________________ From: Clayton Burlison <[email protected]> Sent: Wednesday, October 4, 2017 6:33:14 PM To: AaronBurchfield/CloudFront-Middleware Cc: Subscribed Subject: Re: [AaronBurchfield/CloudFront-Middleware] CloudFront Invalidator (#3) So you likely aren’t hitting the 1,000 invalidations a month limit after which you pay. With that said I highly recommend both of y’all reading the following. https://www.cloudvps.com/helpcenter/knowledgebase/content-delivery-network-cdn/cdn-cache-control-and-invalidations Yes setting invalidations will work. No you shouldn’t be using that to “refresh” content. That not how CDNs are designed. Munki already versions big files. Let’s take an office update for example. That is a ~1.6 GB file, if using the skuless installer, when you invalidate you will remove this from the edge servers. Then the next update will require a pull from s3 to cache to the CloudFront edge servers. Doing this multiple times a day is very inefficient. Also, CloudFront makes it very easy to set the Cache-Control headers for entire directory paths using the web interface. You can also script this with something like aws sync if that is more you fancy (could even do both). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#3 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFa_GKXXOQ_Pfet8_No_B76OewzbUUHPks5spDHagaJpZM4PuShJ>.

[AaronBurchfield]

Thanks @TheMattSchiller for bringing this up. This behavior may not be obvious to those new to CloudFront, I'll probably update the readme to include a note about this for future readers.

Personally I use this script to wrap the aws command line tools and specify a desired cache-control for each munki repo subdirectory while syncing to s3.

Thanks @clburlison for providing a concise explanation for using cache control over invalidations.

I'll leave this open as a reminder to myself to update the documentation.

[erikng]

You can create TTL rules outside of the CLI tools as well. No need to use cache-control. You just setup your rules one time and upload to s3 normally. Thanks, Erik Gomez

…

________________________________ From: Aaron Burchfield <[email protected]> Sent: Wednesday, October 4, 2017 9:18:20 PM To: AaronBurchfield/CloudFront-Middleware Cc: Erik Gomez; Comment Subject: Re: [AaronBurchfield/CloudFront-Middleware] CloudFront Invalidator (#3) Thanks @TheMattSchiller<https://github.com/themattschiller> for bringing this up. This behavior may not be obvious to those new to CloudFront, I'll probably update the readme to include a note about this for future readers. Personally I use this script<https://github.com/AaronBurchfield/push_repo/blob/master/push_repo.py> to wrap the aws command line tools and specify a desired cache-control for each munki repo subdirectory while syncing to s3. Thanks @clburlison<https://github.com/clburlison> for providing a concise explanation for using cache control over invalidations. I'll leave this open as a reminder to myself to update the documentation. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#3 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFa_GL2Gz_s8ksDY8sLR8WoM6wYKFrpqks5spFiMgaJpZM4PuShJ>.