Skip to content

Commit

Permalink
feat: update from table
Browse files Browse the repository at this point in the history
  • Loading branch information
Do1e committed Dec 15, 2024
1 parent 6aa2393 commit ebc0011
Show file tree
Hide file tree
Showing 6 changed files with 166 additions and 140 deletions.
26 changes: 22 additions & 4 deletions .github/workflows/deploy.yml
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
name: Deploy
on:
push:
branches:
- main
schedule:
- cron: "0 4,16 * * *"
workflow_dispatch:
jobs:
build:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -40,9 +38,29 @@ jobs:
path: workers-site/node_modules
key: ${{ runner.os }}-${{ env.cache-name }}-${{ hashFiles('workers-site/package-lock.json') }}
- run: npm i # 执行 Blogroll 的依赖安装
- run: npm run gen # 相当于 node index.js,生成 opml.xml,opml.json 和 data.json
- run: npm run update # 相当于 node update_files.js,从 Seatable 更新数据到 README.md
env:
SEATABLE_API_TOKEN: ${{ secrets.SEATABLE_API_TOKEN }}
- name: Commit and push if README.md changed
env:
DEPLOY_REPO: [email protected]:nju-lug/blogroll.git
DEPLOY_BRANCH: main
SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
run: |
if [ -n "$(git status --porcelain README.md)" ]; then
git config --local user.email "github-actions[bot]@users.noreply.github.com"
git config --local user.name "github-actions[bot]"
git add README.md
git commit -m "update README.md from github actions"
mkdir -p ~/.ssh
echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa
ssh-keyscan github.com >> ~/.ssh/known_hosts
git push $DEPLOY_REPO HEAD:$DEPLOY_BRANCH
else
echo "No changes detected"
fi
- run: npm run gen # 相当于 node index.js,生成 opml.xml,opml.json 和 data.json
- run: npm run build
- name: Deploy to Cloudflare Workers
uses: cloudflare/wrangler-action@v3
Expand Down
30 changes: 5 additions & 25 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@

欢迎在线浏览:https://blogroll.njulug.org/

聚合页面使用 Vue 框架编写,在每次 Push 之后,与每天定时 0 点和 12 点的时候,均会通过 GitHub Action 自动集成和部署到 Cloudflare 上。
聚合页面使用 Vue 框架编写,每天定时 0 点和 12 点,会通过 GitHub Action 自动集成和部署到 Cloudflare 上。

聚合页面由 [@OrangeX4](https://github.com/OrangeX4) 维护,如发现页面上有任何 Bug,欢迎在本 Repo 中提出 Issues。

Expand All @@ -36,38 +36,18 @@ https://t.me/NJULUG_Blogroll
提 Pull Request 将其删除,同时我们也会通过 Github Action 的自动更新中的 Log 来判断是否失效。


## 添加方式
## 添加/编辑方式

先 Fork 这个项目,编辑这个 `README.md` 文件的内容,在 **表格最下面一行** 添加(也即按时间顺序),最后提交 Pull Request 进行更改。
填写表单:[https://table.nju.edu.cn/dtable/forms/b7e232c1-b52b-43ad-8058-3400594cba5a/](https://table.nju.edu.cn/dtable/forms/b7e232c1-b52b-43ad-8058-3400594cba5a/)

如果无 RSS 源,可以使用 `---` 代替,聚合页面将不会抓取。

Pull Request 规范:标题为自己的名字,内容可以是对自己和博客的介绍。

> 南大协同表格支持建设中...
编辑表单:[https://table.nju.edu.cn/dtable/collection-tables/36161685-5d74-4d48-928f-b6b40174da28](https://table.nju.edu.cn/dtable/collection-tables/36161685-5d74-4d48-928f-b6b40174da28)。如果是之前在README中填写的,可重新[填写表单](https://table.nju.edu.cn/dtable/forms/b7e232c1-b52b-43ad-8058-3400594cba5a/),保证`Name`字段一致即可。

如果无 RSS 源,可以使用 `---` 代替,聚合页面将不会抓取,仅展示HTML链接。

## Lists

| Name | RSS | HTML |
| -- | -- | -- |
| OrangeX4's Blog | https://blog.orangex4.workers.dev/atom.xml | https://blog.orangex4.workers.dev/ |
| Idealclover's Blog | https://idealclover.top/feed | https://idealclover.top/ |
| Cmj's Blog | https://blog.caomingjun.com/atom.xml | https://blog.caomingjun.com/ |
| Mexii's Blog | https://blog.mexii.dev/atom.xml | https://blog.mexii.dev/ |
| LadderOperator's Blog | https://ladderoperator.top/index.xml | https://ladderoperator.top |
| Antares's Blog | https://chr.fan/feed | https://chr.fan |
| lyc8503's Blog | https://blog.lyc8503.net/atom.xml | https://blog.lyc8503.net/ |
| YeungYeah 的乱写地 | https://scottyeung.top/atom.xml | https://scottyeung.top/ |
| yaoge123's Blog | https://www.yaoge123.com/blog/feed | https://www.yaoge123.com/ |
| 南雍随笔 | https://ydjsir.com.cn/atom.xml | https://ydjsir.com.cn/ |
| Kevinpro's Blog | --- | https://www.yuque.com/kevinpro |
| Domon | https://www.domon.cn/rss/ | https://www.domon.cn |
| 极东魔术昼寝结社 | https://blog.jaoushingan.com/atom.xml | https://blog.jaoushingan.com |
| Chivalric Gong | --- | https://gmy-acoustics.github.io/ |
| Persvadisto's Blog | https://persvadisto.github.io/atom.xml | https://persvadisto.github.io/ |
| Yukino's Blog | https://02hyc.github.io/Blog/atom.xml | https://02hyc.github.io/Blog/ |
| Do1e | https://www.do1e.cn/feed | https://www.do1e.cn |

## OPML

Expand Down
186 changes: 78 additions & 108 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@ const Parser = require('rss-parser');
const parser = new Parser();
// 引入 RSS 生成器
const RSS = require('rss');
// 引入 SeaTableAPI
const { Base } = require('seatable-api');

// 相关配置
const opmlXmlContentTitle = 'NJU-LUG Blogroll';
Expand All @@ -18,12 +16,12 @@ const opmlXmlPath = './web/public/opml.xml';
const rssXmlPath = './web/public/rss.xml';
const opmlXmlContentOp = '<opml version="2.0">\n <head>\n <title>' + opmlXmlContentTitle + '</title>\n </head>\n <body>\n\n';
const opmlXmlContentEd = '\n </body>\n</opml>';
const seatableToken = process.env.SEATABLE_API_TOKEN;

// 解析 README 中的表格,转为 JSON
const pattern = /\| *([^\|]*) *\| *(http[^\|]*) *\| *(http[^\|]*) *\|/g;
const pattern = /\| *([^\|]*) *\| *(http[^\|]*|---) *\| *(http[^\|]*|---) *\|/g;
const readmeMdContent = fs.readFileSync(readmeMdPath, { encoding: 'utf-8' });
// 生成 opml.json
let opmlJson = [];
const opmlJson = [];
let resultArray;
while ((resultArray = pattern.exec(readmeMdContent)) !== null) {
opmlJson.push({
Expand All @@ -32,118 +30,90 @@ while ((resultArray = pattern.exec(readmeMdContent)) !== null) {
htmlUrl: resultArray[3].trim()
});
}

// 解析 SeaTable 中的表格,转为 JSON
async function parseSeaTableToJson(opmlJson) {
const seatableBase = new Base({
server: "https://table.nju.edu.cn",
APIToken: seatableToken
});
await seatableBase.auth();
const tables = await seatableBase.getTables();
const rows = await seatableBase.listRows(tables[0]['name']);
rows.forEach(row => {
if (row['Name'] && (!opmlJson.some(item => item.title === row['Name'])) && row['RSS'].startsWith('http') && row['HTML'].startsWith('http')) {
opmlJson.push({
title: row['Name'],
xmlUrl: row['RSS'],
htmlUrl: row['HTML']
});
}
});
return opmlJson;
}
// 保存 opml.json 和 opml.xml
fs.writeFileSync(opmlJsonPath, JSON.stringify(opmlJson, null, 2), { encoding: 'utf-8' });
const opmlXmlContent = opmlXmlContentOp
+ opmlJson.map((lineJson) => ` <outline title="${lineJson.title}" xmlUrl="${lineJson.xmlUrl}" htmlUrl="${lineJson.htmlUrl}" />\n`).join('')
+ opmlXmlContentEd;
fs.writeFileSync(opmlXmlPath, opmlXmlContent, { encoding: 'utf-8' });

// 异步处理
(async () => {
// 如果定义 SeaTable Token,则将 SeaTable 中的数据合并到 opmlJson 中
if (seatableToken !== undefined && seatableToken !== '' && seatableToken !== null) {
try {
opmlJson = await parseSeaTableToJson(opmlJson);
} catch (err) {
console.log(err);
}
}

// 保存 opml.json 和 opml.xml
fs.writeFileSync(opmlJsonPath, JSON.stringify(opmlJson, null, 2), { encoding: 'utf-8' });
const opmlXmlContent = opmlXmlContentOp
+ opmlJson.map((lineJson) => ` <outline title="${lineJson.title}" xmlUrl="${lineJson.xmlUrl}" htmlUrl="${lineJson.htmlUrl}" />\n`).join('')
+ opmlXmlContentEd;
fs.writeFileSync(opmlXmlPath, opmlXmlContent, { encoding: 'utf-8' });
// 用于存储各项数据
const dataJson = [];

// 异步处理
(async () => {
for (const lineJson of opmlJson) {

// 用于存储各项数据
const dataJson = [];

for (const lineJson of opmlJson) {

try {

// 读取 RSS 的具体内容
const feed = await parser.parseURL(lineJson.xmlUrl);

// 数组合并
dataJson.push.apply(dataJson, feed.items.filter((item) => item.title && item.content && item.pubDate).map((item) => {
const pubDate = new Date(item.pubDate);
return {
name: lineJson.title,
xmlUrl: lineJson.xmlUrl,
htmlUrl: lineJson.htmlUrl,
title: item.title,
link: item.link,
summary: item.summary ? item.summary : item.content,
pubDate: pubDate,
pubDateYYMMDD: pubDate.toISOString().split('T')[0]
}
}));

} catch (err) {

// 网络超时,进行 Log 报告
console.log(err);
console.log("-------------------------");
console.log("xmlUrl: " + lineJson.xmlUrl);
console.log("-------------------------");
try {

// 读取 RSS 的具体内容
if (!lineJson.xmlUrl.startsWith('http')) {
continue;
}
const feed = await parser.parseURL(lineJson.xmlUrl);

// 数组合并
dataJson.push.apply(dataJson, feed.items.filter((item) => item.title && item.content && item.pubDate).map((item) => {
const pubDate = new Date(item.pubDate);
return {
name: lineJson.title,
xmlUrl: lineJson.xmlUrl,
htmlUrl: lineJson.htmlUrl,
title: item.title,
link: item.link,
summary: item.summary ? item.summary : item.content,
pubDate: pubDate,
pubDateYYMMDD: pubDate.toISOString().split('T')[0]
}
}));

} catch (err) {

// 网络超时,进行 Log 报告
console.log(err);
console.log("-------------------------");
console.log("xmlUrl: " + lineJson.xmlUrl);
console.log("-------------------------");

}
}

// 按时间顺序排序
dataJson.sort((itemA, itemB) => itemA.pubDate < itemB.pubDate ? 1 : -1);
// 默认为保存前 n 项的数据, 并保证不超过当前时间
const curDate = new Date();
const dataJsonSliced = dataJson.filter((item) => item.pubDate <= curDate).slice(0, Math.min(maxDataJsonItemsNumber, dataJson.length));
fs.writeFileSync(dataJsonPath, JSON.stringify(dataJsonSliced, null, 2), { encoding: 'utf-8' });

// 生成 RSS 文件
var feed = new RSS({
title: 'NJU-LUG Blogroll',
description: '南京大学 Linux User Group 收集同学和校友们的 Blog',
feed_url: 'https://blogroll.njulug.org/rss.xml',
site_url: 'https://blogroll.njulug.org/',
image_url: 'https://blogroll.njulug.org/assets/logo.56c0d74c.png',
docs: 'https://blogroll.njulug.org/',
managingEditor: 'NJU-LUG',
webMaster: 'NJU-LUG',
copyright: '2022 NJU-LUG',
language: 'cn',
pubDate: dataJson[0].pubDate,
ttl: '60',
});

// 按时间顺序排序
dataJson.sort((itemA, itemB) => itemA.pubDate < itemB.pubDate ? 1 : -1);
// 默认为保存前 n 项的数据, 并保证不超过当前时间
const curDate = new Date();
const dataJsonSliced = dataJson.filter((item) => item.pubDate <= curDate).slice(0, Math.min(maxDataJsonItemsNumber, dataJson.length));
fs.writeFileSync(dataJsonPath, JSON.stringify(dataJsonSliced, null, 2), { encoding: 'utf-8' });

// 生成 RSS 文件
var feed = new RSS({
title: 'NJU-LUG Blogroll',
description: '南京大学 Linux User Group 收集同学和校友们的 Blog',
feed_url: 'https://blogroll.njulug.org/rss.xml',
site_url: 'https://blogroll.njulug.org/',
image_url: 'https://blogroll.njulug.org/assets/logo.56c0d74c.png',
docs: 'https://blogroll.njulug.org/',
managingEditor: 'NJU-LUG',
webMaster: 'NJU-LUG',
copyright: '2022 NJU-LUG',
language: 'cn',
pubDate: dataJson[0].pubDate,
ttl: '60',
for (let item of dataJsonSliced) {
feed.item({
title: item.title,
description: item.summary,
url: item.link, // link to the item
author: item.name, // optional - defaults to feed author property
date: item.pubDate.toISOString(), // any format that js Date can parse.
});
}

for (let item of dataJsonSliced) {
feed.item({
title: item.title,
description: item.summary,
url: item.link, // link to the item
author: item.name, // optional - defaults to feed author property
date: item.pubDate.toISOString(), // any format that js Date can parse.
});
}
// 保存 rss.xml 文件
const rssXmlContent = feed.xml();
fs.writeFileSync(rssXmlPath, rssXmlContent, { encoding: 'utf-8' });

// 保存 rss.xml 文件
const rssXmlContent = feed.xml();
fs.writeFileSync(rssXmlPath, rssXmlContent, { encoding: 'utf-8' });
})();
})();
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 2 additions & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
{
"name": "blogroll",
"version": "1.0.0",
"version": "2.0.0",
"scripts": {
"dev": "cd web && vite",
"update": "node update_files.js",
"gen": "node index.js",
"build": "cd web && vite build",
"preview": "cd web && vite preview --port 5050"
Expand Down
57 changes: 57 additions & 0 deletions update_files.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
// 文件读取包
const fs = require('fs');
// 引入 SeaTableAPI
const { Base } = require('seatable-api');
// exit函数
const { exit } = require('process');

// 相关配置
const seatableToken = process.env.SEATABLE_API_TOKEN;
const readmeMdPath = './README.md';
// 读取 README.md
const readmeMdContent = fs.readFileSync(readmeMdPath, { encoding: 'utf-8' });

// 解析 SeaTable 中的表格,转为 JSON
async function parseSeaTableToJson() {
const seatableBase = new Base({
server: "https://table.nju.edu.cn",
APIToken: seatableToken
});
try {
await seatableBase.auth();
} catch (err) {
console.log('Seatable API Token 无效,请检查环境变量 SEATABLE_API_TOKEN 是否正确设置。');
exit(1);
}
const tables = await seatableBase.getTables();
const rows = await seatableBase.listRows(tables[0]['name']);
const rows_reverse = rows.reverse();
var opmlJson = [];
rows_reverse.forEach(row => {
// 根据Name去重并保留最后一个
if (row['Name'] && !(opmlJson.find((item) => item.title === row['Name']))) {
if (opmlJson.find((item) => item.htmlUrl === row['HTML'])) {
return;
}
opmlJson.push({
title: row['Name'],
xmlUrl: row['RSS'],
htmlUrl: row['HTML']
});
}
});
return opmlJson.reverse();
}


(async () => {
// 从 SeaTable 中读取数据
const opmlJson = await parseSeaTableToJson();

// 更新 README.md 中的表格内容
const tableStart = readmeMdContent.indexOf('| -- | -- | -- |') + 22;
const tableEnd = readmeMdContent.indexOf('## OPML') - 2;
const tableContent = opmlJson.map((lineJson) => `| ${lineJson.title} | ${lineJson.xmlUrl} | ${lineJson.htmlUrl} |`).join('\n') + '\n';
const newReadmeMdContent = readmeMdContent.slice(0, tableStart) + tableContent + readmeMdContent.slice(tableEnd);
fs.writeFileSync(readmeMdPath, newReadmeMdContent, { encoding: 'utf-8' });
})();

0 comments on commit ebc0011

Please sign in to comment.