wip: transition fern-bot to use step functions

[armandobelardo]

Fixes: FER-1831

Trying this out, but will otherwise make a lambda that triggers a bunch of other instances of another lambda

The general idea: take advantage of aws lambda step functions
First lambda, runs on schedule:

• Run through the Github app installations and create a new "dataset" in s3 that has the necessary data to run the subsequent actions
  Second lambda, runs as a follow on step to the first lambda, triggered automatically:
• Type: Map, meaning it can take a dataset in s3 and parallelize the execution of a lambda on each individual record of the dataset
• Runs the existing update action, but instead of looping over the repos, expects repo data to be passed in as a payload (e.g. from the s3 dataset)

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
fern-dashboard-dev	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 21, 2024 3:36pm
fern-dev	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 21, 2024 3:36pm
fern-prod	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jun 21, 2024 3:36pm

2 Ignored Deployments

Name	Status	Preview	Comments	Updated (UTC)
fontawesome-cdn	⬜️ Ignored (Inspect)	Visit Preview		Jun 21, 2024 3:36pm
private-docs	⬜️ Ignored (Inspect)			Jun 21, 2024 3:36pm

[github-actions]

PR Preview

• docs.stack-auth.com
• gemini2.ferndocs.com
• docs.darwincloud.xyz
• docs.kota.io
• mongodb.ferndocs.com
• gemini.ferndocs.com
• api.labelstud.io
• docs.fluidstack.io
• docs.fileforge.com
• docs.instabase.com
• docs.reka.ai
• docs.aiaplatform.com.au
• fern.ssoready.com
• docs.anterior.com
• developers.beehiiv.com
• docs.scorecard.io
• fern.humanloop.com
• fern.octo.ai
• docs.goldenbasis.com
• developers.mavenagi.com
• docs.subconscious.ai
• docs.mavenagi.com
• docs.sphinxbio.com
• help.getzep.com
• api.qdrant.tech
• developers.upwardli.com
• docs.darwinbio.io
• dev.documentation.sayari.com
• fern.assemblyai.com
• docs.usebridge.com
• fern.uploadcare.com
• docs.impiloplatform.com
• docs.multion.ai
• developer.projectmanager.com
• dev.hume.ai
• docs.predictionguard.com
• docs-beta.projectmanager.com
• angellist.docs.withchima.com
• apidocs.polytomic.com
• docs.athenaintelligence.ai
• docs.flagright.com
• developer.devrev.ai
• docs.hellodata.ai
• docs.bytebot.ai
• documentation.sayari.com
• docs.pier-finance.com
• docs.coactive.ai

[linear]

FER-1831 split up the update actions to shard out the run across multiple jobs to limit execution time and memory usage

[dsinghvi]

Couple questions:

• What repos do we store in s3 (all or just ones with fern folders?)
• Would be great to have an end-to-end test that runs to make sure PRs get created (we can setup a demo repo to do this as well)

[armandobelardo]

@dsinghvi

1. all the ones fern bot is installed on, the thought there is that this first step can be used to multi-plex to whatever jobs we want to add without having to reinvent the wheel
2. sure will add