feat: add workflow and script to check edit links on docs #3557
Conversation
anshgoyalevil
requested review from
derberg,
akshatnema,
magicmatatjahu,
Mayaleeeee and
asyncapi-bot-eve
as code owners
WalkthroughThis pull request introduces a comprehensive enhancement to the documentation link checking and editing workflow. A new GitHub Actions workflow is added to automatically check for broken links weekly. The changes include modifications to the Changes
Assessment against linked issues
Poem
Finishing Touches
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
✅ Deploy Preview for asyncapi-website ready!Built without sensitive environment variables
To edit notification comments on pull requests, go to your Netlify site configuration. |
|
⚡️ Lighthouse report for the changes in this PR:
Lighthouse ran on https://deploy-preview-3557--asyncapi-website.netlify.app/ |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
.github/workflows/check-edit-links.yml (2)
32-37: Improve shell script robustness.The error extraction step could be more robust by following shell scripting best practices.
- name: Extract 404 URLs from output id: extract-404 run: | - ERRORS=$(sed -n '/URLs returning 404:/,$p' output.log) - echo "errors<<EOF" >> $GITHUB_OUTPUT - echo "$ERRORS" >> $GITHUB_OUTPUT - echo "EOF" >> $GITHUB_OUTPUT + { + ERRORS="$(sed -n '/URLs returning 404:/,$p' output.log)" + echo "errors<<EOF" + echo "${ERRORS}" + echo "EOF" + } >> "$GITHUB_OUTPUT"🧰 Tools
🪛 actionlint (1.7.4)
32-32: shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:2:23: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:3:19: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:4:15: Double quote to prevent globbing and word splitting
(shellcheck)
38-49: Consider adding error categorization in Slack notification.The Slack notification could be more informative by categorizing errors.
SLACK_MESSAGE: | 🚨 The following URLs returned 404 during the link check:${{ steps.extract-404.outputs.errors }}
+ Note: These errors might be due to: + - Missing documentation files + - Incorrect edit link paths + - Repository structure changes
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
.github/workflows/check-edit-links.yml(1 hunks)components/layout/DocsLayout.tsx(1 hunks)config/edit-page-config.json(2 hunks)package.json(1 hunks)scripts/markdown/check-editlinks.js(1 hunks)
🧰 Additional context used
🪛 eslint
scripts/markdown/check-editlinks.js
[error] 58-58: Unexpected await inside a loop.
(no-await-in-loop)
[error] 59-59: Unexpected await inside a loop.
(no-await-in-loop)
🪛 actionlint (1.7.4)
.github/workflows/check-edit-links.yml
32-32: shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:2:23: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:3:19: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:4:15: Double quote to prevent globbing and word splitting
(shellcheck)
⏰ Context from checks skipped due to timeout of 180000ms (1)
- GitHub Check: Lighthouse CI
🔇 Additional comments (4)
components/layout/DocsLayout.tsx (1)
37-39: Ensure correct conversion of.mdxto.mdin edit linksBy replacing
.mdxwith.mdin edit links, please verify that all.mdxfiles correspond to.mdfiles in the repository. Ensure that this change does not introduce broken edit links for any files that should remain as.mdx.config/edit-page-config.json (1)
4-4: Verify the updatedhrefpath for the AsyncAPI Generator docsThe
hreffor/tools/generatorhas been updated to a new path. Please confirm thathttps://github.com/asyncapi/generator/tree/master/apps/generator/docsis the correct location for the generator's documentation and that the link is accessible..github/workflows/check-edit-links.yml (1)
3-7: LGTM! Well-configured workflow triggers.The workflow is appropriately scheduled to run weekly and includes manual trigger capability.
package.json (1)
27-27: Verify the edit links checker script and its dependencies.The script addition looks good, but let's ensure the required files and dependencies are in place.
✅ Verification successful
✓ Edit links checker implementation is complete and properly configured
All required dependencies and configuration files are in place:
- node-fetch-2 is available in package.json
- edit-page-config.json exists with valid mappings
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Description: Verify the existence of the edit links checker script and related files # Check if the script file exists if ! [ -f "scripts/markdown/check-editlinks.js" ]; then echo "Error: check-editlinks.js script is missing" exit 1 fi # Check for required dependencies in package.json rg "scripts/markdown/check-editlinks.js" -lLength of output: 105
Script:
#!/bin/bash # Check script content and dependencies cat scripts/markdown/check-editlinks.js # Also check package.json for any related dependencies cat package.json | jq '.dependencies, .devDependencies'Length of output: 9008
Script:
#!/bin/bash # Check if the config file exists if [ -f "config/edit-page-config.json" ]; then echo "Config file exists" cat config/edit-page-config.json else echo "Config file missing" fiLength of output: 705
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #3557 +/- ##
=========================================
Coverage 100.00% 100.00%
=========================================
Files 19 20 +1
Lines 668 732 +64
=========================================
+ Hits 668 732 +64 ☔ View full report in Codecov by Sentry. |
jest.config.js
Outdated
| 'scripts/compose.js', | ||
| 'scripts/tools/categorylist.js', | ||
| 'scripts/tools/tags-color.js', | ||
| 'scripts/markdown/check-editlinks.js' |
There was a problem hiding this comment.
I have added the new script to ignore Jest coverage to pass CI. Will create a new good first issue for other contributors to add test
There was a problem hiding this comment.
I think we should have a practice to add test along with the code, so don't add the file here. Instead add the relevant tests for the file.
There was a problem hiding this comment.
uhm okay. will add it
There was a problem hiding this comment.
Actionable comments posted: 1
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
jest.config.js(1 hunks)
⏰ Context from checks skipped due to timeout of 180000ms (3)
- GitHub Check: Redirect rules - asyncapi-website
- GitHub Check: Header rules - asyncapi-website
- GitHub Check: Pages changed - asyncapi-website
🔇 Additional comments (1)
jest.config.js (1)
14-14: LGTM!The testMatch pattern correctly excludes Netlify edge function tests while including other test files.
|
@asyncapi/bounty_team |
scripts/markdown/check-editlinks.js
Outdated
|
|
||
| if (invalidUrls.length === 0) { | ||
| console.log('All URLs are valid.'); | ||
| process.exit(0); |
There was a problem hiding this comment.
We shouldn't use process.exit functions like this. Make the conditional rendering more appropriate. Like, make this if block for invalidUrls only.
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (3)
tests/fixtures/markdown/check-editlinks-data.js (1)
3-5: Ensure consistency in URL path formatsThe URL paths in the test data are inconsistent - some start with '/' while others don't. This inconsistency could lead to issues in path matching and URL generation.
Apply this diff to maintain consistency:
- urlPath: 'docs/concepts/application', + urlPath: '/docs/concepts/application', - urlPath: 'concepts/application', + urlPath: '/concepts/application', - urlPath: '/tools/cli', + urlPath: 'tools/cli',Also applies to: 8-10, 13-15
tests/markdown/check-editlinks.test.js (1)
90-93: Add timeout handling test caseThe network error test case only covers generic errors. Add a specific test case for request timeouts.
Add this test case:
it('should handle request timeouts', async () => { fetch.mockImplementation(() => new Promise((_, reject) => { setTimeout(() => reject(new Error('Request timeout')), 5000); })); await expect(processBatch(testBatch)).rejects.toThrow('Request timeout'); });scripts/markdown/check-editlinks.js (1)
103-136: Implement pagination for large directoriesThe
generatePathsfunction loads all files into memory at once, which could cause issues with large directories.Consider implementing pagination to process files in chunks:
async function* generatePathsIterator(folderPath, editOptions, batchSize = 100) { const stack = [{ path: folderPath, relativePath: '' }]; let batch = []; while (stack.length > 0) { const { path: currentPath, relativePath } = stack.pop(); const files = await fs.readdir(currentPath); for (const file of files) { const filePath = path.join(currentPath, file); const relativeFilePath = path.join(relativePath, file); const stats = await fs.stat(filePath); if (stats.isDirectory()) { stack.push({ path: filePath, relativePath: relativeFilePath }); } else if (stats.isFile() && file.endsWith('.md') && file !== '_section.md') { const urlPath = relativeFilePath.split(path.sep).join('/').replace('.md', ''); batch.push({ filePath, urlPath, editLink: determineEditLink(urlPath, filePath, editOptions) }); if (batch.length >= batchSize) { yield batch; batch = []; } } } } if (batch.length > 0) { yield batch; } }
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (6)
jest.config.js(1 hunks)package.json(1 hunks)scripts/markdown/check-editlinks.js(1 hunks)tests/fixtures/markdown/check-editlinks-data.js(1 hunks)tests/fixtures/markdown/edit-page-config.json(1 hunks)tests/markdown/check-editlinks.test.js(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- package.json
- jest.config.js
🧰 Additional context used
📓 Learnings (1)
scripts/markdown/check-editlinks.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
🔇 Additional comments (2)
tests/fixtures/markdown/edit-page-config.json (1)
15-17: Verify empty value fallback handlingThe empty value entry is used as a fallback, but this could lead to incorrect URL generation if not handled carefully. Ensure that the
determineEditLinkfunction properly validates the target before using it.✅ Verification successful
Empty value fallback is properly implemented and tested
The empty value entry in the configuration is intentionally used as a fallback, with proper handling in the
determineEditLinkfunction. The implementation adds necessary path components and has comprehensive test coverage verifying this behavior.🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Check if determineEditLink handles null/undefined targets ast-grep --pattern 'function determineEditLink($_, $_, $_) { $$$ if (!target) { $$$ } $$$ }'Length of output: 106
Script:
#!/bin/bash # Search for determineEditLink function with broader context rg -l "determineEditLink" # Search for files containing both "edit" and "link" rg -l "edit.*link|link.*edit" --type js --type ts # Look for test files related to edit links fd "edit.*test|test.*edit" -e js -e ts -e jsx -e tsxLength of output: 402
Script:
#!/bin/bash # Check implementation cat scripts/markdown/check-editlinks.js # Check test cases cat tests/markdown/check-editlinks.test.js # Check test data cat tests/fixtures/markdown/check-editlinks-data.jsLength of output: 13950
scripts/markdown/check-editlinks.js (1)
147-153: 🛠️ Refactor suggestionImprove error handling in main function
The main function's error handling could be more informative and should set appropriate exit codes.
Apply this diff to improve error handling:
if (invalidUrls.length > 0) { console.log('\nURLs returning 404:\n'); invalidUrls.forEach((url) => console.log(`- ${url.editLink} generated from ${url.filePath}\n`)); console.log(`\nTotal invalid URLs found: ${invalidUrls.length}`); + process.exitCode = 1; } else { console.log('All URLs are valid.'); + process.exitCode = 0; }Likely invalid or redundant comment.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
scripts/markdown/check-editlinks.js (4)
3-3: Consider using native fetch or updating to latest node-fetchThe code uses
node-fetch-2which is an older version. Consider using the nativefetchAPI (available since Node.js 18) or updating to the latest version ofnode-fetch.-const fetch = require('node-fetch-2'); +const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
6-10: Move ignore list to configuration fileConsider moving the
ignoreFilesarray to the configuration file (edit-page-config.json) to make it more maintainable and configurable without code changes.
142-142: Enhance error message detailThe error message could be more informative by including the original error details.
- throw new Error(`Error processing directory ${folderPath}:`, err); + throw new Error(`Error processing directory ${folderPath}: ${err.message}`);
163-163: Enhance error message detailSimilar to the previous suggestion, include the original error message for better debugging.
- throw new Error('Failed to check edit links:', error); + throw new Error(`Failed to check edit links: ${error.message}`);
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/markdown/check-editlinks.js(1 hunks)
🧰 Additional context used
📓 Learnings (1)
scripts/markdown/check-editlinks.js (2)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:80-93
Timestamp: 2025-01-14T09:23:32.728Z
Learning: In the AsyncAPI website's edit link generation system, the `editOptions` array in `edit-page-config.json` includes a fallback entry with an empty string value (`''`) that matches any URL path, ensuring that `determineEditLink()` function always finds a target and never returns null.
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
🔇 Additional comments (4)
scripts/markdown/check-editlinks.js (4)
12-20: LGTM! Well-documented helper functionThe pause function is well-implemented with clear JSDoc documentation.
58-79: Align batch processing pattern with codebase standardsThe current implementation uses concurrent processing with
Promise.all, but based on the codebase patterns (as seen in the Dashboard script), sequential processing is preferred.Would you like to maintain consistency with the codebase's sequential pattern or keep the current concurrent implementation? Please confirm your preference.
88-101: LGTM! Robust edit link determinationThe implementation correctly handles the fallback case and special path prefixes. The code aligns with the established pattern where an empty string value in
editOptionsserves as a catch-all fallback.
172-172: LGTM! Clean exportsThe module exports are well-organized and include all necessary functions.
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
.github/workflows/check-edit-links.yml (3)
4-6: Add detailed cron syntax documentation.While the cron syntax is correct, consider adding a more detailed comment explaining each field for better maintainability:
schedule: - - cron: '0 0 * * 0' # Runs every week at midnight on Sunday + - cron: '0 0 * * 0' # minute(0) hour(0) day(*) month(*) weekday(0=Sunday) - Runs every Sunday at midnight UTC
38-40: Consider moving the if condition.As suggested in the past review, consider moving the if condition above the step name for better readability and to follow the standard pattern in GitHub Actions:
- - name: Notify Slack - if: ${{ steps.extract-404.outputs.errors != '' }} + - if: ${{ steps.extract-404.outputs.errors != '' }} + name: Notify Slack
49-49: Document the MSG_MINIMAL parameter.The
MSG_MINIMALparameter's purpose isn't immediately clear. Add a comment explaining its effect on the Slack notification:- MSG_MINIMAL: true + MSG_MINIMAL: true # Sends a compact message without additional metadata like GitHub workflow details
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
.github/workflows/check-edit-links.yml(1 hunks)package.json(1 hunks)scripts/dashboard/build-dashboard.js(1 hunks)scripts/markdown/check-edit-links.js(1 hunks)tests/markdown/check-edit-links.test.js(1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
- package.json
🧰 Additional context used
📓 Learnings (2)
scripts/markdown/check-edit-links.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
.github/workflows/check-edit-links.yml (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: .github/workflows/check-edit-links.yml:25-29
Timestamp: 2025-01-08T15:16:27.655Z
Learning: In GitHub workflows running scripts with process.exit statements for error handling (like check-editlinks.js in asyncapi/website), avoid adding error suppression (|| true) at the workflow level as it would mask the intended error reporting mechanism.
🪛 eslint
scripts/dashboard/build-dashboard.js
[error] 184-184: Replace ·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause· with ⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎
(prettier/prettier)
🪛 actionlint (1.7.4)
.github/workflows/check-edit-links.yml
32-32: shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:2:23: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:3:19: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:4:15: Double quote to prevent globbing and word splitting
(shellcheck)
⏰ Context from checks skipped due to timeout of 180000ms (1)
- GitHub Check: Test NodeJS PR - macos-13
🔇 Additional comments (6)
scripts/markdown/check-edit-links.js (2)
83-93:⚠️ Potential issueHandle undefined 'target' to prevent runtime errors in 'determineEditLink'.
In the
determineEditLinkfunction, if no matching edit option is found,targetcould beundefined, leading to aTypeErrorwhen accessingtarget.value. Please add a check to handle the case whentargetisundefinedto prevent runtime errors.Apply this diff to fix the issue:
function determineEditLink(urlPath, filePath, editOptions) { // Remove leading 'docs/' if present for matching const pathForMatching = urlPath.startsWith('docs/') ? urlPath.slice(5) : urlPath; const target = editOptions.find((edit) => pathForMatching.includes(edit.value)); + if (!target) { + // Handle the case where no matching edit option is found + return null; // Or handle appropriately based on your application's needs + } // Handle the empty value case (fallback) if (target.value === '') { return `${target.href}/docs/${urlPath}.md`; } // For other cases with specific targets return `${target.href}/${path.basename(filePath)}`; }Ensure that downstream code can handle a
nullreturn value from this function, or adjust the handling accordingly.
3-3:⚠️ Potential issueCorrect the import statement for 'node-fetch'.
It appears that
'node-fetch-2'is not a valid package. To import version 2 ofnode-fetch, you should use'node-fetch'. Please update the import statement accordingly.Apply this diff to fix the import statement:
-const fetch = require('node-fetch-2'); +const fetch = require('node-fetch');Ensure that the
node-fetchpackage is installed in yourpackage.jsondependencies.scripts/dashboard/build-dashboard.js (1)
184-184: Reformat 'module.exports' for better readability as per Prettier guidelines.The static analysis tool indicates that the
module.exportsstatement should be reformatted for consistency and readability.Apply this diff to format
module.exports:-module.exports = { getLabel, monthsSince, mapGoodFirstIssues, getHotDiscussions, getDiscussionByID, getDiscussions, writeToFile, start, processHotDiscussions, pause }; +module.exports = { + getLabel, + monthsSince, + mapGoodFirstIssues, + getHotDiscussions, + getDiscussionByID, + getDiscussions, + writeToFile, + start, + processHotDiscussions, + pause +};Ensure that the updated formatting aligns with the project's coding standards and passes the linter checks.
🧰 Tools
🪛 eslint
[error] 184-184: Replace
·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause·with⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎(prettier/prettier)
tests/markdown/check-edit-links.test.js (1)
2-2:⚠️ Potential issueCorrect import and mocking of 'node-fetch'.
It appears that
'node-fetch-2'is not a valid package name. Replace'node-fetch-2'with'node-fetch'in both the import statement and the Jest mock.Apply these diffs:
At line 2:
-const fetch = require('node-fetch-2'); +const fetch = require('node-fetch');At line 13:
-jest.mock('node-fetch-2', () => jest.fn()); +jest.mock('node-fetch', () => jest.fn());Ensure that the
node-fetchpackage is correctly installed and updated in yourpackage.jsondependencies.Also applies to: 13-13
.github/workflows/check-edit-links.yml (2)
9-11: LGTM! Well-structured job configuration.The job configuration follows best practices with clear naming and appropriate runner selection.
25-29: Approved: Proper error handling implementation.The link checker step correctly allows the script's built-in error handling to work without suppression, as noted in the previous learning. This ensures proper error reporting and Slack notifications.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
scripts/markdown/check-edit-links.js (3)
3-3: Consider using native fetch or updating to latest node-fetch.The script uses
node-fetch-2which is an older version. Consider:
- Using the native
fetchAPI (available since Node.js 18.0.0)- Updating to the latest
node-fetchversion-const fetch = require('node-fetch-2'); +// If using Node.js >= 18.0.0 +const fetch = (...args) => import('node-fetch').then(({default: fetch}) => fetch(...args));
19-19: Extract timeout duration as a named constant.The timeout duration is a magic number. Consider extracting it to a named constant at the module level for better maintainability.
+const URL_CHECK_TIMEOUT_MS = 5000; + async function processBatch(batch) { - const TIMEOUT_MS = 5000; + const TIMEOUT_MS = URL_CHECK_TIMEOUT_MS;
51-51: Extract batch size as a named constant.The batch size is a magic number. Consider extracting it to a named constant at the module level for better configuration.
+const URL_CHECK_BATCH_SIZE = 5; + async function checkUrls(paths) { const result = []; - const batchSize = 5; + const batchSize = URL_CHECK_BATCH_SIZE;
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/markdown/check-edit-links.js(1 hunks)
🧰 Additional context used
📓 Learnings (1)
scripts/markdown/check-edit-links.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
🔇 Additional comments (2)
scripts/markdown/check-edit-links.js (2)
102-135: Well-implemented recursive file processing.The function is well-structured with:
- Proper error handling
- Concurrent file processing
- Clear documentation
- Edge case handling (e.g., _section.md files)
137-163: Clean main function implementation with proper exports.The implementation is solid with:
- Comprehensive error handling
- Clear console output
- Well-organized exports
- Proper module execution check
scripts/markdown/check-edit-links.js
Outdated
| * @returns {Promise<string[]>} Array of URLs that returned 404 | ||
| */ | ||
| async function processBatch(batch) { | ||
| const TIMEOUT_MS = 5000; |
There was a problem hiding this comment.
Make it configurable with env variable as well.
scripts/markdown/check-edit-links.js
Outdated
| */ | ||
| async function checkUrls(paths) { | ||
| const result = []; | ||
| const batchSize = 5; |
There was a problem hiding this comment.
Kindly make it configurable using env variable.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
scripts/markdown/check-edit-links.js (3)
7-11: Consider moving ignore list to configuration file.The
ignoreFileslist could be moved to the configuration file (config/edit-page-config.json) to make it more maintainable and configurable without code changes.
102-135: Consider adding additional file validation.While the implementation is solid, consider adding validation for:
- Maximum file size to prevent processing extremely large files
- File name patterns to exclude temporary or backup files (e.g.,
.md~,.md.bak)
137-163: Consider enhancing error reporting.While the error handling is good, consider adding:
- Total number of URLs checked in the success message
- Structured error output (e.g., JSON format) for better parsing
- Error categorization (timeout vs 404 vs other errors)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
scripts/markdown/check-edit-links.js(1 hunks)
🧰 Additional context used
📓 Learnings (1)
scripts/markdown/check-edit-links.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
🔇 Additional comments (3)
scripts/markdown/check-edit-links.js (3)
18-42: Well-implemented batch processing with timeout handling!The implementation includes:
- Configurable timeout via environment variable
- Proper request cancellation using AbortController
- Comprehensive error handling
- Test coverage exclusions where appropriate
49-70: Efficient implementation of concurrent batch processing!The implementation includes:
- Configurable batch size via environment variable
- Concurrent processing with Promise.all
- Rate limiting prevention with pauses between batches
79-92: Clean and efficient edit link determination logic!The implementation correctly handles both specific targets and fallback cases, with proper path manipulation.
|
@coderabbitai resolve |
✅ Actions performedComments resolved. |
|
@coderabbitai full review |
✅ Actions performedFull review triggered. |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (6)
scripts/markdown/check-edit-links.js (2)
19-19: Parse environment variableTIMEOUT_MSto a number
process.env.TIMEOUT_MSreturns a string. To ensureTIMEOUT_MSis used correctly in numerical operations, parse it usingNumber(). This prevents potential issues if the environment variable is set.Apply this diff to parse
TIMEOUT_MS:- const TIMEOUT_MS = process.env.TIMEOUT_MS || 5000; + const TIMEOUT_MS = Number(process.env.TIMEOUT_MS) || 5000;
51-51: Parse environment variableBATCH_SIZEto a numberSimilarly,
process.env.BATCH_SIZEshould be parsed to a number to ensure correct batch size handling.Apply this diff to parse
BATCH_SIZE:- const batchSize = process.env.BATCH_SIZE || 5; + const batchSize = Number(process.env.BATCH_SIZE) || 5;tests/markdown/check-edit-links.test.js (1)
90-93: Add test case for request timeouts inprocessBatchConsider adding a test case to simulate a timeout in the
processBatchfunction to ensure the timeout logic works as expected.Here's how you might add the test case:
it('should handle request timeouts', async () => { fetch.mockImplementation(() => new Promise((resolve) => setTimeout(resolve, 10000))); await expect(processBatch(testBatch)).rejects.toThrow('The user aborted a request.'); });This test simulates a fetch request that exceeds the specified timeout, triggering the
AbortController.scripts/dashboard/build-dashboard.js (3)
Line range hint
10-15: Consider making delay durations configurable.The
pausefunction is well-implemented and documented. However, the delay durations (500ms for GraphQL queries and 1000ms for batch processing) are hardcoded throughout the code. Consider extracting these values into configuration constants for better maintainability and flexibility.+const RATE_LIMIT_DELAY_MS = 500; +const BATCH_PROCESSING_DELAY_MS = 1000; async function pause(ms) { return new Promise((res) => { setTimeout(res, ms); }); }🧰 Tools
🪛 eslint
[error] 184-184: Replace
·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause·with⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎(prettier/prettier)
184-184: Improve module exports formatting for better readability.The module exports line is quite long. Consider formatting it across multiple lines for better readability.
-module.exports = { getLabel, monthsSince, mapGoodFirstIssues, getHotDiscussions, getDiscussionByID, getDiscussions, writeToFile, start, processHotDiscussions, pause }; +module.exports = { + getLabel, + monthsSince, + mapGoodFirstIssues, + getHotDiscussions, + getDiscussionByID, + getDiscussions, + writeToFile, + start, + processHotDiscussions, + pause +};🧰 Tools
🪛 eslint
[error] 184-184: Replace
·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause·with⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎(prettier/prettier)
Line range hint
17-42: Enhance error handling and rate limiting.Consider the following improvements:
- Create custom error types for different failure scenarios
- Add retry logic when approaching rate limits
- Consider using environment variable validation
Example implementation:
class GitHubAPIError extends Error { constructor(message, query, response) { super(message); this.name = 'GitHubAPIError'; this.query = query; this.response = response; } } // Validate environment variables early function validateEnv() { if (!process.env.GITHUB_TOKEN) { throw new Error('GITHUB_TOKEN environment variable is required'); } } // Add retry logic for rate limits async function executeWithRetry(queryFn, maxRetries = 3) { for (let attempt = 1; attempt <= maxRetries; attempt++) { try { const result = await queryFn(); if (result.rateLimit.remaining <= 100) { const resetTime = new Date(result.rateLimit.resetAt); const waitTime = resetTime - new Date(); if (waitTime > 0) { await pause(waitTime); } } return result; } catch (error) { if (attempt === maxRetries) throw error; await pause(1000 * attempt); } } }🧰 Tools
🪛 eslint
[error] 184-184: Replace
·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause·with⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎(prettier/prettier)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
.github/workflows/check-edit-links.yml(1 hunks)components/layout/DocsLayout.tsx(1 hunks)config/edit-page-config.json(2 hunks)jest.config.js(1 hunks)package.json(1 hunks)scripts/dashboard/build-dashboard.js(1 hunks)scripts/markdown/check-edit-links.js(1 hunks)tests/fixtures/markdown/check-editlinks-data.js(1 hunks)tests/markdown/check-edit-links.test.js(1 hunks)
🧰 Additional context used
📓 Learnings (3)
tests/markdown/check-edit-links.test.js (2)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: tests/markdown/check-edit-links.test.js:20-46
Timestamp: 2025-01-14T14:58:38.076Z
Learning: In the AsyncAPI website codebase, the edit-page-config.json contains a fallback match entry that ensures determineEditLink function always has a valid target.value, making null checks unnecessary.
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:80-93
Timestamp: 2025-01-14T09:23:32.728Z
Learning: In the AsyncAPI website's edit link generation system, the `editOptions` array in `edit-page-config.json` includes a fallback entry with an empty string value (`''`) that matches any URL path, ensuring that `determineEditLink()` function always finds a target and never returns null.
scripts/markdown/check-edit-links.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
.github/workflows/check-edit-links.yml (2)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: .github/workflows/check-edit-links.yml:30-37
Timestamp: 2025-01-14T14:57:08.491Z
Learning: In GitHub Actions workflows, avoid using {...} blocks for output redirection. Instead, use individual redirects with proper quoting as recommended by GitHub Docs. For example:
```yaml
echo "key<<EOF" >> "${GITHUB_OUTPUT}"
echo "${value}" >> "${GITHUB_OUTPUT}"
echo "EOF" >> "${GITHUB_OUTPUT}"
Learnt from: anshgoyalevil
PR: #3557
File: .github/workflows/check-edit-links.yml:25-29
Timestamp: 2025-01-08T15:16:27.655Z
Learning: In GitHub workflows running scripts with process.exit statements for error handling (like check-editlinks.js in asyncapi/website), avoid adding error suppression (|| true) at the workflow level as it would mask the intended error reporting mechanism.
</details>
</details><details>
<summary>🪛 eslint</summary>
<details>
<summary>scripts/dashboard/build-dashboard.js</summary>
[error] 184-184: Replace `·getLabel,·monthsSince,·mapGoodFirstIssues,·getHotDiscussions,·getDiscussionByID,·getDiscussions,·writeToFile,·start,·processHotDiscussions,·pause·` with `⏎··getLabel,⏎··monthsSince,⏎··mapGoodFirstIssues,⏎··getHotDiscussions,⏎··getDiscussionByID,⏎··getDiscussions,⏎··writeToFile,⏎··start,⏎··processHotDiscussions,⏎··pause⏎`
(prettier/prettier)
</details>
</details>
<details>
<summary>🪛 actionlint (1.7.4)</summary>
<details>
<summary>.github/workflows/check-edit-links.yml</summary>
32-32: shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects
(shellcheck)
---
32-32: shellcheck reported issue in this script: SC2086:info:2:23: Double quote to prevent globbing and word splitting
(shellcheck)
---
32-32: shellcheck reported issue in this script: SC2086:info:3:19: Double quote to prevent globbing and word splitting
(shellcheck)
---
32-32: shellcheck reported issue in this script: SC2086:info:4:15: Double quote to prevent globbing and word splitting
(shellcheck)
</details>
</details>
</details>
<details>
<summary>🔇 Additional comments (11)</summary><blockquote>
<details>
<summary>components/layout/DocsLayout.tsx (1)</summary>
`37-39`: **Handle '.mdx' file extensions correctly**
Great job handling posts with '.mdx' extensions by replacing them with '.md' in the `generateEditLink` function. This ensures edit links point to the correct files.
</details>
<details>
<summary>jest.config.js (1)</summary>
`9-10`: **LGTM!**
The formatting change looks good.
</details>
<details>
<summary>config/edit-page-config.json (1)</summary>
`4-4`: **Verify the new generator docs path.**
The path has been updated to reflect the repository restructure. Let's verify that the new path exists and is accessible.
<details>
<summary>✅ Verification successful</summary>
<!-- [monologue]
[task verify_path_1/1]
Verify if the new generator docs path exists and is accessible.
[observations]
- Executed curl command to check the URL status
- Received HTTP/2 200 response
- The path "https://github.com/asyncapi/generator/tree/master/apps/generator/docs" is accessible
- GitHub successfully serves the content, confirming the path exists
[analysis]
The verification shows:
- The new path structure is valid and accessible
- The repository restructure was correctly reflected in the config
- No further verification needed as we got a successful response
[actions]
Generate a final response confirming the path is valid and accessible
-->
**Path verification successful**
The new generator docs path is valid and accessible.
</details>
<details>
<summary>🏁 Scripts executed</summary>
The following scripts were executed for the analysis:
Script:
```shell
#!/bin/bash
# Description: Verify that the new generator docs path exists
# Expected: 200 OK response indicating the path exists
curl -I "https://github.com/asyncapi/generator/tree/master/apps/generator/docs"
Length of output: 5010
.github/workflows/check-edit-links.yml (4)
4-6: LGTM! Weekly schedule is appropriate.
The cron schedule 0 0 * * 0 correctly runs the workflow at midnight every Sunday.
17-21: LGTM! Node.js setup follows best practices.
Using node-version-file with .nvmrc ensures consistent Node.js versions across different environments.
38-49: LGTM! Slack notification is well configured.
The Slack notification:
- Triggers only when 404 errors are found
- Includes a clear title and formatted error message
- Uses MSG_MINIMAL to reduce noise
30-37: 🛠️ Refactor suggestion
Add proper quoting to shell variables.
The shell script needs proper quoting to prevent word splitting and globbing issues.
- name: Extract 404 URLs from output
id: extract-404
run: |
- ERRORS=$(sed -n '/URLs returning 404:/,$p' output.log)
- echo "errors<<EOF" >> $GITHUB_OUTPUT
- echo "$ERRORS" >> $GITHUB_OUTPUT
- echo "EOF" >> $GITHUB_OUTPUT
+ ERRORS="$(sed -n '/URLs returning 404:/,$p' output.log)"
+ echo "errors<<EOF" >> "${GITHUB_OUTPUT}"
+ echo "${ERRORS}" >> "${GITHUB_OUTPUT}"
+ echo "EOF" >> "${GITHUB_OUTPUT}"Likely invalid or redundant comment.
🧰 Tools
🪛 actionlint (1.7.4)
32-32: shellcheck reported issue in this script: SC2129:style:2:1: Consider using { cmd1; cmd2; } >> file instead of individual redirects
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:2:23: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:3:19: Double quote to prevent globbing and word splitting
(shellcheck)
32-32: shellcheck reported issue in this script: SC2086:info:4:15: Double quote to prevent globbing and word splitting
(shellcheck)
tests/fixtures/markdown/check-editlinks-data.js (3)
1-17: LGTM! Test data structure is well-organized.
The determineEditLinkData array provides good coverage for different documentation scenarios:
- Root docs paths
- Concept docs
- Tool docs
19-30: LGTM! Tutorial paths are correctly mapped.
The processBatchData array correctly maps tutorial documentation paths to their edit links.
32-83: Verify the existence of all test paths.
Let's ensure all the test paths exist in the repository.
✅ Verification successful
All test paths verified successfully.
All files in the test data exist in the repository under the markdown/docs/ directory. The paths in the test data correctly represent the actual file structure, just without the leading /markdown/docs/ prefix.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that all test paths exist in the repository
# Expected: All paths should be found
# Extract and verify markdown file paths
fd -t f -e md "^(guides|migration|reference|tools|tutorials)" markdown/docs/
Length of output: 78
Script:
#!/bin/bash
# First, let's understand the directory structure
echo "=== Directory Structure ==="
ls -la
echo -e "\n=== Search for markdown directory ==="
fd -t d "markdown"
echo -e "\n=== Search for any markdown files ==="
fd -t f -e md
echo -e "\n=== Try to find specific files ==="
# Search for some specific files from the test paths
fd "index.md|message-validation.md|validate.md" -t f
Length of output: 15761
package.json (1)
27-27: LGTM! Script follows naming conventions.
The new test:editlinks script is:
- Properly named following the test script pattern
- Correctly positioned with other test scripts
- Well integrated with the new workflow
scripts/markdown/check-edit-links.js
Outdated
| * @returns {Promise<string[]>} Array of URLs that returned 404 | ||
| */ | ||
| async function processBatch(batch) { | ||
| const TIMEOUT_MS = process.env.TIMEOUT_MS || 5000; |
There was a problem hiding this comment.
| const TIMEOUT_MS = process.env.TIMEOUT_MS || 5000; | |
| const TIMEOUT_MS = Number(process.env.DOCS_LINK_CHECK_TIMEOUT) || 5000; |
Make this more specific
scripts/markdown/check-edit-links.js
Outdated
| */ | ||
| async function checkUrls(paths) { | ||
| const result = []; | ||
| const batchSize = process.env.BATCH_SIZE || 5; |
There was a problem hiding this comment.
| const batchSize = process.env.BATCH_SIZE || 5; | |
| const batchSize = Number(process.env.DOCS_LINK_CHECK_BATCH_SIZE) || 5; |
There was a problem hiding this comment.
Add test case for request timeouts in processBatch
it('should handle request timeouts', async () => {
fetch.mockImplementation(() => new Promise((resolve) => setTimeout(resolve, 10000)));
await expect(processBatch(testBatch)).rejects.toThrow('The user aborted a request.');
});
There was a problem hiding this comment.
Kindly correct the file name for this test file.
|
Also, add links of the workflow runs for this new GitHub action from your forked repo. |
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (3)
scripts/markdown/check-edit-links.js (3)
19-19: Improve environment variable naming.Consider using a more descriptive name for the timeout environment variable to better indicate its purpose.
-const TIMEOUT_MS = Number(process.env.DOCS_LINK_CHECK_TIMEOUT) || 5000; +const EDIT_LINK_REQUEST_TIMEOUT_MS = Number(process.env.DOCS_EDIT_LINK_REQUEST_TIMEOUT_MS) || 5000;
101-134: Consider optimizing directory traversal.The current implementation reads the entire directory tree synchronously. For large documentation repositories, this could be optimized.
Consider using a streaming directory reader like
readdir-enhancedor implementing a generator-based solution:async function* walkDirectory(dir) { const files = await fs.readdir(dir); for (const file of files) { const filePath = path.join(dir, file); const stats = await fs.stat(filePath); if (stats.isDirectory()) { yield* walkDirectory(filePath); } else { yield filePath; } } }
146-148: Enhance error reporting format.Consider structuring the error output in a more machine-readable format for better integration with CI/CD pipelines.
- invalidUrls.forEach((url) => console.log(`- ${url.editLink} generated from ${url.filePath}\n`)); + console.log(JSON.stringify({ + total: invalidUrls.length, + errors: invalidUrls.map(url => ({ + editLink: url.editLink, + sourcePath: url.filePath + })) + }, null, 2));
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
scripts/markdown/check-edit-links.js(1 hunks)tests/fixtures/markdown/check-edit-links-data.js(1 hunks)tests/markdown/check-edit-links.test.js(1 hunks)
🧰 Additional context used
📓 Learnings (2)
scripts/markdown/check-edit-links.js (1)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:58-59
Timestamp: 2025-01-08T15:15:00.759Z
Learning: In the AsyncAPI codebase, batch processing operations (like in the Dashboard script and check-editlinks.js) follow a sequential pattern using await in loops, which is the preferred approach for maintaining consistency across the codebase.
tests/markdown/check-edit-links.test.js (3)
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: tests/markdown/check-edit-links.test.js:20-46
Timestamp: 2025-01-14T14:58:38.076Z
Learning: In the AsyncAPI website codebase, the edit-page-config.json contains a fallback match entry that ensures determineEditLink function always has a valid target.value, making null checks unnecessary.
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-editlinks.js:80-93
Timestamp: 2025-01-14T09:23:32.728Z
Learning: In the AsyncAPI website's edit link generation system, the `editOptions` array in `edit-page-config.json` includes a fallback entry with an empty string value (`''`) that matches any URL path, ensuring that `determineEditLink()` function always finds a target and never returns null.
Learnt from: anshgoyalevil
PR: asyncapi/website#3557
File: scripts/markdown/check-edit-links.js:83-88
Timestamp: 2025-01-14T15:12:29.840Z
Learning: In the AsyncAPI website's edit link generation logic, the `editUrls` configuration (config/edit-page-config.json) includes a fallback entry with an empty value that acts as a catch-all matcher. Therefore, `editOptions.find()` will always return a match, making null checks on the result unnecessary and untestable.
🔇 Additional comments (3)
tests/markdown/check-edit-links.test.js (1)
1-167: Well-structured and comprehensive test suite!The test implementation is thorough and covers:
- All exported functions
- Edge cases (404s, timeouts, network errors)
- Proper mocking of external dependencies
- Clear test descriptions
scripts/markdown/check-edit-links.js (2)
25-31: LGTM: Proper request timeout implementation.The implementation correctly:
- Uses AbortController for request cancellation
- Cleans up the timeout using clearTimeout
- Sends HEAD requests to minimize data transfer
58-65: LGTM: Efficient concurrent batch processing.The implementation correctly uses Promise.all for concurrent processing while maintaining a delay between batches to prevent rate limiting.
|
Another workflow run with the latest commit in this PR. (I made some intentional changes to config so that it finds out those 404 urls and dump the urls to slack.) Slack message: https://asyncapi.slack.com/archives/C0884HHQZS5/p1737262577722289 (The following run was triggered by cron expression on master branch. Since an error wasn't planted on master branch, this wf passed) https://github.com/anshgoyalevil/website/actions/runs/12848952843 |
|
/rtm |
Description
Related issue(s)
fixes #3221
Summary by CodeRabbit
New Features
Bug Fixes
.mdxextensionChores
Documentation