Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move server 3 to use new ethernet card and have external IP #95

Closed
mitar opened this issue Nov 4, 2016 · 28 comments
Closed

Move server 3 to use new ethernet card and have external IP #95

mitar opened this issue Nov 4, 2016 · 28 comments

Comments

@mitar
Copy link
Member

mitar commented Nov 4, 2016 •
edited
Loading

Prerequisite for: #83

At the same time it would be useful to do: #82
@ck2qsuZT
Copy link

Did we order a new network card? Does the one you showed me last time we were in the network closet work?

@mitar
Copy link
Member Author

mitar commented Nov 15, 2016

I upgraded the server to new kernel and after that I have not noticed issues with the one I bought before (4 ports one) anymore. So I didn't know any new one.

@mitar
Copy link
Member Author

mitar commented Nov 15, 2016

(I worry now more that server 3 might not have enough CPU power for nodewatcher. I had to move online council app to campus server because server 2 was too slow. Those servers are really meant for file sharing only.)

@ck2qsuZT
Copy link

Would be pretty easy and not to expensive to upgrade, especially if we get
a used server process, those things are a dime a dozen. Would we need more
ram also?

On Nov 15, 2016 01:35, "Mitar" [email protected] wrote:

(I worry now more that server 3 might not have enough CPU power for
nodewatcher. I had to move online council app to campus server because
server 2 was too slow. Those servers are really meant for file sharing
only.)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#95 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHwZSvYobm1VFVavQ_Sct43DdtFHEMwyks5q-XzXgaJpZM4KqFly
.

@ck2qsuZT
Copy link

Especially if its for all house Internet all houses can chip in a bit?

On Nov 15, 2016 07:33, "James Martins" [email protected] wrote:

Would be pretty easy and not to expensive to upgrade, especially if we get
a used server process, those things are a dime a dozen. Would we need more
ram also?

On Nov 15, 2016 01:35, "Mitar" [email protected] wrote:

(I worry now more that server 3 might not have enough CPU power for
nodewatcher. I had to move online council app to campus server because
server 2 was too slow. Those servers are really meant for file sharing
only.)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#95 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHwZSvYobm1VFVavQ_Sct43DdtFHEMwyks5q-XzXgaJpZM4KqFly
.

@mitar
Copy link
Member Author

mitar commented Nov 15, 2016

So yes, server 1 and server 2 should be probably replaced with a proper server, it does not need too much disk (2 TB), but memory is good to have (32 GB?).

But the question is where to get funding from for this. So for now we will be running on server 3, and later on we can migrate this to a new server.

@ck2qsuZT
Copy link

ck2qsuZT commented Nov 16, 2016 •
edited
Loading

Why not convert the makerspace computer into a server? it has a pretty nice server CPU (i7-5820k), decent server motherboard (ASRock Fatal1ty X99X Killer, it supports ECC if we get a XEON E5 processor even though that's probably unecessary) and honestly we could make a Windows virtual machine with PCI passthrough of a graphics card and then get an intel nuc ($100-300 Maybe BSCIT? ) into the makerspace as thin client. NX protocol is extremely responsive and if we wanted to go crazy we could have a dedicated cable and ethernet port from the server to the thin client. We would need more RAM if we were to do this though but it prevents the nice expensive makerspace computer from getting stolen. The only thing we would need then is:

  • 2 2TB disks in RAID1 ($100*2)
  • maybe a system SSD since they're pretty cheap ($100 for 256GB) (we can probably get this through BSCIT)
  • The makerspace motherboard is DDR4 so RAM costs ($6.5/GB for really good server ram, $6/GB for desktop ram) I'd say just populate a slot or two with maximum size (16GB/slot with our current CPU, 32GB/slot with a XEON E5 CPU) then we can upgrade later if needed when RAM gets cheaper (keep in mind though that server RAM doesn't change in price much.)
  • a graphics card if we want to do the pci passthrough option ($100-$300) (we can probably get this through BSCIT)
  • a rack-mount case runs from ($100-$600)
  • we can sell our current CPU for a slightly better but mostly similar XEON (Intel Xeon E5-1650 v3) if we really need 32GB RAM/slot, ECC, etc. for about an extra $400
  • we can also probably sell the current case for about $100 which will help a bit.

so in all:

  • $300(16GB RAM Desktop RAM, cheapest case, no GPU, no SSD)
    to
  • $2000 (64GB Server RAM, SSD, nice case, nice GPU, upgraded CPU, nice new intel nuc (up to $700from BSCIT)

@mitar
Copy link
Member Author

mitar commented Nov 16, 2016 •
edited
Loading

I think we do not need a different CPU and stuff. To my knowledge we got money to buy replacement hardware for stolen hardware, so maybe the only thing which we need is a rack-mount case and move whatever is there there.

I would not go with thin client. And I would not go with changing what we have. Whatever we have is better than those file-server servers.

@clohard, how is makerspace computer fixing going?

@ck2qsuZT
Copy link

ck2qsuZT commented Nov 16, 2016 •
edited
Loading

Did we get money or hardware from BSCIT?

@mitar
Copy link
Member Author

mitar commented Nov 16, 2016

Hm, we got money, but maybe then central office bought the equipment for us. This is something new they are trying this semester.

@ck2qsuZT
Copy link

Yeah, this semester central has been buying hardware for NM then a manager needs to go pick it up. If that be the case and we don't want to do a thin client then we would have a worthless GPU

@mitar
Copy link
Member Author

mitar commented Nov 16, 2016

We can do deep-learning on the GPU. :-)

@ck2qsuZT
Copy link

or folding at home ^_^

On Wed, Nov 16, 2016 at 4:22 AM, Mitar [email protected] wrote:

We can do deep-learning on the GPU. :-)

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#95 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AHwZShmkoyf0JqpHyOvhVSqSx7KxQXATks5q-oUGgaJpZM4KqFly
.

@mitar
Copy link
Member Author

mitar commented Nov 16, 2016

Or we can sell the GPU. Motherboard probably have internal GPU no?

@ck2qsuZT
Copy link

Or we can use it as a thin client host =p. There is usually a relatively powerful iGPU in modern desktop CPUs but not server CPUs. The GPU built into the motherboard is pretty much text only.

@mitar
Copy link
Member Author

mitar commented Nov 16, 2016

The GPU built into the motherboard is pretty much text only.

Good enough. :-)

@ck2qsuZT
Copy link

I feel like the PCI issues may be related to our 9 year old Motherboard:
www.supermicro.com/products/motherboard/Xeon3000/3000/PDSML-LN2_.cfm

@mitar
Copy link
Member Author

mitar commented Jan 27, 2017

I think that since I upgraded kernel to new version there was no ethernet issues anymore. So we should just proceed and start using the new network card with public IP.

@ck2qsuZT
Copy link

ck2qsuZT commented Mar 6, 2017

Why use an external network card anyways, is the advantage more ports (not saying this is a negligible advantage) or is there something else going on with the motherboard's Ethernet ports?

@mitar
Copy link
Member Author

mitar commented Mar 6, 2017

We need extra ports: internal network, antenna network, external network.

@mitar
Copy link
Member Author

mitar commented Mar 6, 2017

Also, one can do in the future batching over all ports and achieve 4 Gbit/s throughput.

@ck2qsuZT
Copy link

ck2qsuZT commented Mar 6, 2017

This and #104 can probably both be fixed at the same time. This should be done via salt, correct?

@mitar
Copy link
Member Author

mitar commented Mar 6, 2017

Yes.

@mitar
Copy link
Member Author

mitar commented Apr 15, 2017 •
edited
Loading

Let's start work on this. We have 6 physical interfaces now in the server, one card with 4 ports, and two on the mother board:

$ ip link show
2: p1p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:05 brd ff:ff:ff:ff:ff:ff
3: p1p2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:04 brd ff:ff:ff:ff:ff:ff
4: p1p3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:07 brd ff:ff:ff:ff:ff:ff
5: p1p4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:06 brd ff:ff:ff:ff:ff:ff
6: p5p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:30:48:8c:bf:78 brd ff:ff:ff:ff:ff:ff
7: p6p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:30:48:8c:bf:79 brd ff:ff:ff:ff:ff:ff

The plan is to connect:

  • one interface to our external IP for the server, p1p1
  • one interface to local network, p1p2
  • one interface to debug network for antenna uplink, p1p3

mitar added a commit to cloyne/servers that referenced this issue Apr 15, 2017
@mitar
Configured new interfaces.
926077c 
See: cloyne/network#95
@mitar
Copy link
Member Author

mitar commented Apr 16, 2017

This now work, but during boot it was for 60 seconds for some reason because of some network configuration issues. Not sure what. /var/log/boot.log contains many:

 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device security                                                                            [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Starting configure network device                                                                                     [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]
 * Stopping configure network device                                                                                     [ OK ]
 * Stopping configure network device security                                                                            [ OK ]

@mitar
Copy link
Member Author

mitar commented Apr 16, 2017

So now the output of ip link is:

2: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:05 brd ff:ff:ff:ff:ff:ff
3: p1p2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:04 brd ff:ff:ff:ff:ff:ff
4: p1p3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:07 brd ff:ff:ff:ff:ff:ff
5: p1p4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:26:55:d9:cf:06 brd ff:ff:ff:ff:ff:ff
6: p5p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:30:48:8c:bf:78 brd ff:ff:ff:ff:ff:ff
7: p6p1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:30:48:8c:bf:79 brd ff:ff:ff:ff:ff:ff

@mitar
Copy link
Member Author

mitar commented Apr 16, 2017

The issue with network configuration was that I configured a gateway both for external IP interface and lan interface. Removing it from the lan interface solved the problem.

mitar added a commit to cloyne/servers that referenced this issue Apr 16, 2017
@mitar
Remove the gateway from the lan interface.
df5ac9e 
See: cloyne/network#95
@mitar
Copy link
Member Author

mitar commented Apr 16, 2017

I also upgraded the server to Ubuntu Xenial 16.04.

@mitar mitar closed this as completed Apr 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mitar @ck2qsuZT