fix: hide hidden and noindex pages from sitemap

[abvthecity]

Fixes FER-2522

Before this PR:

• hidden and noindex pages are included in the sitemap

After this PR:

• hidden and noindex pages are excluded from the sitemap
• nodes inherit "hidden" from parents. i.e. if a section is hidden, all of its children become hidden as well.

See here for why noindex pages should also be excluded from sitemap.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
fern-dashboard-dev	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 23, 2024 0:22am
fern-dev	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 23, 2024 0:22am
fern-prod	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jul 23, 2024 0:22am

3 Skipped Deployments

Name	Status	Preview	Comments	Updated (UTC)
fern-dashboard	⬜️ Ignored (Inspect)			Jul 23, 2024 0:22am
fontawesome-cdn	⬜️ Ignored (Inspect)	Visit Preview		Jul 23, 2024 0:22am
private-docs	⬜️ Ignored (Inspect)			Jul 23, 2024 0:22am

[github-actions]

PR Preview

• docs.schematichq.com
• plantstore.dev
• www.plantstore.dev
• docs.boundaryml.com
• docs.qlty.ai
• docs.rightbrain.ai
• docs.getkard.com
• docs.syndicate.io
• docs.qlty.us
• hume.ferndocs.com
• docs.stack-auth.com
• gemini2.ferndocs.com
• docs.darwincloud.xyz
• docs.kota.io
• mongodb.ferndocs.com
• gemini.ferndocs.com
• api.labelstud.io
• docs.fluidstack.io
• docs.fileforge.com
• docs.instabase.com
• docs.reka.ai
• docs.aiaplatform.com.au
• fern.ssoready.com
• docs.anterior.com
• developers.beehiiv.com
• docs.scorecard.io
• fern.humanloop.com
• docs.goldenbasis.com
• developers.mavenagi.com
• docs.subconscious.ai
• docs.mavenagi.com
• docs.sphinxbio.com
• help.getzep.com
• api.qdrant.tech
• developers.upwardli.com
• docs.darwinbio.io
• dev.documentation.sayari.com
• fern.assemblyai.com
• docs.usebridge.com
• fern.uploadcare.com
• docs.impiloplatform.com
• docs.multion.ai
• developer.projectmanager.com
• dev.hume.ai
• docs.predictionguard.com
• docs-beta.projectmanager.com
• angellist.docs.withchima.com
• apidocs.polytomic.com

[github-actions]

🌿 Preview your docs: https://fern-preview-2df91bdb-edb3-4c0d-80b4-b52926746fe8.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-41f240f3-d308-45c4-b7e5-8bc7349df9fa.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-50197c22-0cf1-4b68-910d-e3718ccde1b1.docs.buildwithfern.com

[linear]

FER-2522 [Cohere] Exclude hidden pages from the sitemap XML

Currently, Fern's sitemap XML includes locations for hidden pages (like docs/setting-up). Cohere's existing sitemap does not. We should exclude hidden pages from the sitemap.

[github-actions]

🌿 Preview your docs: https://fern-preview-7edf64f3-f68f-4da2-a891-f647c77d11ea.docs.buildwithfern.com

[github-actions]

🌿 Preview your docs: https://fern-preview-d2a836c0-484f-4b55-bdf6-593202deef10.docs.buildwithfern.com