Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

用egg 在容器中启动后，在几乎没人使用的情况下，内存在一天内逐步上升到1.6g。建议能有配置项能够优化内存分配 #123

Open
kaihekaihe opened this issue Jun 24, 2019 · 17 comments
Open

用egg 在容器中启动后，在几乎没人使用的情况下，内存在一天内逐步上升到1.6g。建议能有配置项能够优化内存分配 #123

kaihekaihe opened this issue Jun 24, 2019 · 17 comments
Labels
help wanted

Comments

@kaihekaihe
Copy link

详情见这个issue：
eggjs/egg#3777
@hubcarl
Copy link
Collaborator

hubcarl commented Jun 27, 2019 •
edited
Loading

@kaihekaihe 你怎么启动应用的？ 不会是启动的开发模式(npm run dev)吧？ 另外确认一下，你的 Egg 容器启动了多少 worker ， Egg 默认是系统的核数

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 27, 2019

@kaihekaihe 部署的话，除了先进行项目构建外，部署就和普通的 node 应用一样部署就可以. 针对这个问题，先确认两点：

  • 确认一下启动方式： 是开发模式(npm run dev) 还是发布模式 (npm run build && npm start)
  • 确认一下 Egg 启动的进程数，发布模式默认是系统核数。 本地我启动了一下，一个 worker 大概 30M 内存， 差不多占用 500MB. 指定 Egg workers 数，请看：eggjs/egg-scripts
    image
  • 在没人使用的情况下， 内存一直涨， 那就感觉不是 SSR 渲染导致的内存问题。如果alinode 用不了，自己用这种方式排查一下： yuque.com/hubcarl/blog/dxrrhq

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 27, 2019

  • 这种情况是要排查一下， 刚我又在本地压侧了一下，内存回收正常

ab -c 50 -n 1000 http://127.0.0.1:7001
ab -c 50 -n 1000 http://127.0.0.1:7001/admin/article/list

  • 我克隆了egg vue的模板，没有加任何其他大的开发功能进去

建议先排查一下自己添加的功能，同时后用工具分析一下内存情况 @kaihekaihe， 也可以提供可复现的 Demo 我这边排查一下

@kaihekaihe
Copy link
Author

@kaihekaihe 你怎么启动应用的？ 不会是启动的开发模式(npm run dev)吧？ 另外确认一下，你的 Egg 容器启动了多少 worker ， Egg 默认是系统的核数
npm i
npm run build
npm run start

用ps aux,有：

node /opt/entry-frame/node_modules/.bin/egg-scripts start
node --require xxx/node_modules/egg-scripts/node_modules/source-map-support/register.js ...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...
/usr/local/node/bin/node --require .../source-map-support/register.js /...

@kaihekaihe
Copy link
Author

我用了type orm有没有影响？

@kaihekaihe
Copy link
Author

一般是这样，每多一天，内存会增加100m左右，这个egg vue启动后占用内存在1.2g左右，随后，会每天增加100m，直到接近1.7g

@kaihekaihe
Copy link
Author

不知道你那边是不是一直启动这个应用，每天的内存增量是多少

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 29, 2019

@kaihekaihe 可以直接部署这个模板应用观察一下内情情况。 另外 设置一下 EGG_WORKERS 环境变量，限制一下 worker数，比如设置为 4 个

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 29, 2019

我用了type orm有没有影响？

因为不知道代码怎么使用的，建议可以直接部署这个模板应用观察一下内情情况。

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 29, 2019

不知道你那边是不是一直启动这个应用，每天的内存增量是多少

我们这边线上使用的，内存不会一直增加的，即使增加后，也会回收掉的

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 29, 2019 •
edited
Loading

@kaihekaihe 简单点，可以先本地对比试试 （npm run build , npm start)

  • 直接使用该模板，然后不停的访问，用Mac 活动监视器看一下 node 内存情况, 是否一直增加，不回收
  • 用你修改的代码，然后不停的访问，用Mac 活动监视器看一下 node 内存情况, 是否一直增加，不回收

@hubcarl
Copy link
Collaborator

hubcarl commented Jun 29, 2019

如果还不行，可以加 QQ：715493250 直接聊，请备注一下

@kaihekaihe
Copy link
Author

@kaihekaihe 可以直接部署这个模板应用观察一下内情情况。 另外 设置一下 EGG_WORKERS 环境变量，限制一下 worker数，比如设置为 4 个

根据我上面的ps aux。有多少个worker，为什么要限制worker数呢

@kaihekaihe
Copy link
Author

不知道你那边是不是一直启动这个应用，每天的内存增量是多少

我们这边线上使用的，内存不会一直增加的，即使增加后，也会回收掉的

这个模板 占用的内存，一直维持在多少

@hubcarl
Copy link
Collaborator

hubcarl commented Jul 2, 2019 •
edited
Loading

@kaihekaihe 具体多少，要看你启动的 worker 数量。 上面截图显示，1个 worker 也就 30M - 40M

@kaihekaihe
Copy link
Author

kaihekaihe commented Jul 29, 2019 •
edited
Loading

memory: 2019-7-29 14:30:57: rss:110.08MB, heapTotal:55.27MB, heapUsed:48.64MB
memory: 2019-7-29 14:31:01: rss:109.15MB, heapTotal:55.27MB, heapUsed:48.75MB
memory: 2019-7-29 14:31:03: rss:109.44MB, heapTotal:56.27MB, heapUsed:48.72MB
memory: 2019-7-29 14:31:05: rss:110.07MB, heapTotal:55.77MB, heapUsed:48.77MB
memory: 2019-7-29 14:31:06: rss:109.64MB, heapTotal:55.77MB, heapUsed:48.70MB
memory: 2019-7-29 14:31:07: rss:110.06MB, heapTotal:55.27MB, heapUsed:48.65MB
memory: 2019-7-29 14:31:09: rss:110.05MB, heapTotal:55.27MB, heapUsed:48.64MB
memory: 2019-7-29 14:31:10: rss:109.79MB, heapTotal:55.77MB, heapUsed:48.64MB
memory: 2019-7-29 14:31:12: rss:108.97MB, heapTotal:56.27MB, heapUsed:48.70MB
memory: 2019-7-29 14:31:18: rss:88.71MB, heapTotal:55.77MB, heapUsed:49.40MB
memory: 2019-7-29 14:31:19: rss:109.45MB, heapTotal:56.27MB, heapUsed:48.69MB
memory: 2019-7-29 14:31:19: rss:88.44MB, heapTotal:55.27MB, heapUsed:49.34MB
memory: 2019-7-29 14:31:19: rss:109.60MB, heapTotal:55.27MB, heapUsed:48.63MB
memory: 2019-7-29 14:31:19: rss:109.98MB, heapTotal:55.27MB, heapUsed:48.60MB
memory: 2019-7-29 14:31:19: rss:110.02MB, heapTotal:55.27MB, heapUsed:48.63MB

[2019-07-29 14:31:21.215] [cfork:master:81] worker:1221 disconnect (exitedAfterDisconnect: false, state: disconnected, isDead: false, worker.disableRefork: false)
[2019-07-29 14:31:21.222] [cfork:master:81] new worker:1242 fork (state: none)
2019-07-29 14:31:21,222 INFO 81 [master] app_worker#113:1221 disconnect, suicide: false, state: disconnected, current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","112","113","114","115"]
2019-07-29 14:31:21,223 INFO 81 [master] app_worker#115:1242 start, state: none, current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","112","113","114","115"]
[2019-07-29 14:31:21.223] [cfork:master:81] worker:1221 exit (code: null, exitedAfterDisconnect: false, state: dead, isDead: true, isExpected: true, worker.disableRefork: false)
2019-07-29 14:31:21,223 ERROR 81 nodejs.AppWorkerDiedError: [master] app_worker#113:1221 died (code: null, signal: SIGKILL, suicide: false, state: dead), current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","112","114","115"]
at Master.onAppExit (/opt/entry-frame/node_modules/egg-cluster/lib/master.js:510:21)
at emitOne (events.js:116:13)
at Master.emit (events.js:211:7)
at Messenger.sendToMaster (/opt/entry-frame/node_modules/egg-cluster/lib/utils/messenger.js:137:17)
at Messenger.send (/opt/entry-frame/node_modules/egg-cluster/lib/utils/messenger.js:102:12)
at EventEmitter.cluster.on (/opt/entry-frame/node_modules/egg-cluster/lib/master.js:353:22)
at emitThree (events.js:141:20)
at EventEmitter.emit (events.js:217:7)
at ChildProcess.worker.process.once (internal/cluster/master.js:185:13)
at Object.onceWrapper (events.js:317:30)
name: "AppWorkerDiedError"
pid: 81
hostname: entry-frame-65555cc2c9c84b2-4lp4z

[2019-07-29 14:31:21.812] [cfork:master:81] worker:1207 disconnect (exitedAfterDisconnect: false, state: disconnected, isDead: false, worker.disableRefork: false)
[2019-07-29 14:31:21.816] [cfork:master:81] new worker:1249 fork (state: none)
2019-07-29 14:31:21,816 INFO 81 [master] app_worker#112:1207 disconnect, suicide: false, state: disconnected, current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","112","114","115","116"]
2019-07-29 14:31:21,816 INFO 81 [master] app_worker#116:1249 start, state: none, current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","112","114","115","116"]
[2019-07-29 14:31:21.817] [cfork:master:81] worker:1207 exit (code: null, exitedAfterDisconnect: false, state: dead, isDead: true, isExpected: true, worker.disableRefork: false)
2019-07-29 14:31:21,817 ERROR 81 nodejs.AppWorkerDiedError: [master] app_worker#112:1207 died (code: null, signal: SIGKILL, suicide: false, state: dead), current workers: ["29","31","53","60","62","68","74","84","93","94","101","102","114","115","116"]
at Master.onAppExit (/opt/entry-frame/node_modules/egg-cluster/lib/master.js:510:21)
at emitOne (events.js:116:13)
at Master.emit (events.js:211:7)
at Messenger.sendToMaster (/opt/entry-frame/node_modules/egg-cluster/lib/utils/messenger.js:137:17)
at Messenger.send (/opt/entry-frame/node_modules/egg-cluster/lib/utils/messenger.js:102:12)
at EventEmitter.cluster.on (/opt/entry-frame/node_modules/egg-cluster/lib/master.js:353:22)
at emitThree (events.js:141:20)
at EventEmitter.emit (events.js:217:7)
at ChildProcess.worker.process.once (internal/cluster/master.js:185:13)
at Object.onceWrapper (events.js:317:30)
name: "AppWorkerDiedError"
pid: 81

这是日志打印。每次打印完memory之后，其中有一个worker 自动关了。看日志文件没有找到报错，不知道是不是打印memory的代码有问题。
worker数量一致维持在15个

我在代码里extend/application.js加了这段heapdump打印代码：https://blog.csdn.net/zdhsoft/article/details/56671395
今天是第二天，容器总占用内存是1349m，日志里所有worker rss 加总1603m，heaptotal求和834m，heapused：731m

@hubcarl
Copy link
Collaborator

hubcarl commented Jul 30, 2019

@kaihekaihe 排查问题时，可以设置一下 process.env.EGG_WORKERS 环境变量，比如启动 1 个worker

https://github.com/eggjs/egg-scripts

workers - numbers of app workers, default to process.env.EGG_WORKERS, if unset, egg will use os.cpus().length as default.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hubcarl @kaihekaihe