feat: FDR Latest pulling at the edge (S3 upload/download)

[RohinBhargava]

Short description of the changes made

• This change introduces pulling FDR Latest specs directly from S3, using already existing S3 asset buckets. The reason we can do this is because we no longer have shape transformations for FDR docs shapes, and instead have one shape that is registered and read.

• This PR chooses to use FDR to upload S3 specs, however, this can easily be moved into the CLI (before this, there should be a discussion of security concerns of distributing S3 keys to clients/this is not an issue in Next since the api keys are used in the backend part of next). For spec pulling, this is done directly at the site that needs the data (some FDR apis or edge functions, because we need to index algolia records).

*The idea is that this should improve performance, since we see that FDR is a current bottleneck, and that there is one less hop to deliver the data to the end user.

THIS NEEDS TO BE THOROUGHLY TESTED

• Coupling this with the OpenApi Parser v2 rollout is beneficial, because we can properly flight/test this new approach.

What was the motivation & context behind this PR?

• Performance issues
• Future vision to parse data at the edge
• Integration with dynamic snippets

How has this PR been tested?

• To be tested in FDR dev.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Updated (UTC)
rbac.ferndocs.com	❌ Failed (Inspect)		Jan 12, 2025 11:16pm

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
fern-shell	⬜️ Ignored (Inspect)		Jan 12, 2025 11:16pm

[github-actions]

🌿 Preview your docs: https://fern-preview-c8c7e24c-ad09-449e-88c7-97d80b1d5675.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-4f6ba162-2087-4f89-adf0-c97f478c3531.docs.buildwithfern.com

[github-actions]

📦 Next.js Bundle Analysis for fern-platform-monorepo

This analysis was generated by the Next.js Bundle Analysis action. 🤖

This PR introduced no changes to the JavaScript bundle! 🙌

[github-actions]

PR Preview

• angellist.docs.withchima.com
• api-docs.codecombat.com
• api.babelstreet.com
• api.labelstud.io
• api.navipartner.com
• api.qdrant.tech
• apidocs.polytomic.com
• column.docs.withchima.com
• dev.documentation.sayari.com
• dev.hume.ai
• developer.artlist.io
• developer.balkan.id
• developer.devrev.ai
• developer.ideogram.ai
• developer.nibtravelinsurance.com
• developer.projectmanager.com
• developer.yocobeta.co.za
• developers.ada.cx
• developers.awork.com
• developers.beehiiv.com
• developers.duve.com
• developers.gappify.com
• developers.upwardli.com
• docs-beta.projectmanager.com
• docs-next.scoutos.com
• docs.aiaplatform.com.au
• docs.airtop.ai
• docs.amniscient.com
• docs.anterior.com
• docs.argolabs.ai
• docs.athenaintel.com
• docs.benchify.com
• docs.boundaryml.com
• docs.brightmove.com
• docs.bytebot.ai
• docs.cable.tech
• docs.cartesia.ai
• docs.coactive.ai
• docs.cohere.com
• docs.conductorquantum.com
• docs.credal.ai
• docs.darwinbio.io
• docs.darwincloud.xyz
• docs.data.aviato.co
• docs.dataglade.com
• docs.deeptune.com
• docs.edp.codeclimate.com
• docs.edp.dev
• docs.eyelevel.ai
• docs.fileforge.com
• docs.flagright.com
• docs.fluidstack.io
• docs.frankieone.com
• docs.getanalog.ai
• docs.getkard.com
• docs.getscorecard.ai
• docs.givechariot.com
• docs.goldenbasis.com
• docs.hellodata.ai
• docs.impiloplatform.com
• docs.instabase.com
• docs.joincandidhealth.com
• docs.kota.io
• docs.letta.com
• docs.mavenagi.com
• docs.mercoa.com
• docs.monite.com
• docs.multion.ai
• docs.nominal.io
• docs.ntropy.com
• docs.openledger.com
• docs.paradex.trade
• docs.payos.money
• docs.pier-finance.com
• docs.predictionguard.com
• docs.privategpt.dev
• docs.propexo.com
• docs.qlty.ai
• docs.qlty.sh
• docs.qlty.us
• docs.reka.ai
• docs.revert.dev
• docs.rightbrain.ai
• docs.schematichq.com
• docs.scorecard.io
• docs.scoutos.com
• docs.scrapybara.com
• docs.sphinxbio.com
• docs.stack-auth.com
• docs.staging.paradex.trade
• docs.subconscious.ai
• docs.superagent.sh
• docs.syndicate.io
• docs.termina.ai
• docs.trykeet.com
• docs.trypinnacle.app
• docs.upshift.dev
• docs.usebridge.com
• docs.vellum.ai
• docs.withabound.com
• docs.withchima.com
• documentation.babelstreet.com
• documentation.sayari.com
• fern.assemblyai.com
• fern.ssoready.com
• gusto.docs.withchima.com
• handbook.codeclimate.com
• help.getzep.com
• plantstore.dev
• reference.flatfile.com
• rippling.docs.withchima.com
• secure.docs.propexo.com
• www.intern.mavenagi.com
• www.plantstore.dev

[github-actions]

Playwright test results

 96 passed
 3 flaky
 2 skipped

Details

 101 tests across 9 suites
 1 minute, 37 seconds
 faa4a4d

Flaky tests

chromium › forward-proxy/express.spec.ts › home page 404
chromium › skew-protection/index.test.ts › should contain ?dpl= or x-deployment-id header on all scripts and prefetch requests
chromium › smoke/favicon.spec.ts › Check if favicon exists and URL does not return 404 for developers.gappify.com

Skipped tests

chromium › posthog.spec.ts › Posthog loads successfully
chromium › websockets/websocket.spec.ts › websocket test

[github-actions]

🌿 Preview your docs: https://fern-preview-63fccc1f-e901-4e5d-a736-1903f9f2d0e6.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-63c12ca5-83b7-4130-abd2-3943bf01444c.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-e128101d-3eff-4842-9bf6-d38be5388d6a.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-3c991263-19b0-45a5-9018-c6ffd12a7522.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-983525e0-6040-4929-bb68-60292249d171.docs.buildwithfern.com

[dsinghvi]

It's a large PR, but this makes sense to me