Help me understand Ethernet motion control

[Marius Liebenberg]

I am wondering about how Ethernet motion control works together with
linuxcnc. How is the synchronized motion achieved over the link? Or what
kind of information is passed over the link to the controller?

[sam sokolik]

I will take a stab...

The short answer is linuxcnc Ethernet motion is no different than any of
the other interface card solution..

The longer (as I understand it) answer is...

Remember that linuxcnc doesn't use motion devices like smoothstepper or
galil. (these are buffering devices that move motion off of the
computer.) These have advantages that the computer that is sending the
data doesn't have to be realtime (or the link between the computer and
the device doesn't need to be realtime). From my perspective - they
have too many disadvantages. (but I am biased..)

In linuxcnc - most interfaces are 'just' hardware step generators and or
pwm generators and or i/o and or encoder counters and or other things I
can't think of at the moment. (ie mesa, pico, vital...) This moves the
heavy lifting off of the computer. So on a normal system running one of
these interfaces the computer accesses the card every servo period.
(maybe 1 to 10khz). This reads encoder positions, updates stepgens,
pwm, i/o and so on. The link between the interface card and the
computer is realtime be it pci, parallel port, ethernet.... So
initially the realtime Ethernet interface was rt-net on xenomai. This
worked but was a pain that a) you needed rtnet and b) There was only a
few network interfaces supported... Now the Ethernet interface uses
rt-preempt which is a lot less work to setup - expecially now that jeff
has added support to master. (on wheezy you can use the rt_preempt
kernel from synaptic)

Overall - pretty exciting stuff!

sam

Post by Marius Liebenberg
I am wondering about how Ethernet motion control works together with
linuxcnc. How is the synchronized motion achieved over the link? Or what
kind of information is passed over the link to the controller?

[Jeff Epler]

Mesa's 7i80 cards have a HostMot2 firmware that communicates via an
ethernet interface using standard UDP packets. The LinuxCNC HAL driver,
hm2_eth, simply uses these UDP packets as a communications bus, just
like it uses the PCI bus for a PCI Mesa card.

So for instance, to write all 72 GPIOs on the 7i80hd, you send a packet
with the contents (expressed in hex)
83C2 (write 3 words in sequential addresses in memory space 0)
0010 (starting at address 0100, the GPIO module)
AAAAAAAA (4 bytes for first header's GPIO outputs)
BBBBBBBB (4 bytes for second header's GPIO outputs)
CCCCCCCC (4 bytes for third header's GPIO outputs)
(If you were also setting stepgen frequencies or PWM values, they would
be additional commands within the same packet). This is the same thing
that happens with a PCI Mesa card:
memory_mapped_io[0x100/4] = 0xAAAAAAAA;
memory_mapped_io[0x104/4] = 0xBBBBBBBB;
memory_mapped_io[0x108/4] = 0xCCCCCCCC;
or with an EPP Mesa card:
ADDR 0010
DATA AAAAAAAA
DATA BBBBBBBB
DATA CCCCCCCC
just transporting the data in a different way.

I don't understand the technical details of ethernet, but based on some
wikipedia browsing it looks like a full ethernet packet with the 16 byte
UDP payload would weigh 672 bits and take under 10microseconds to
transmit, so there's enough bandwidth and the packet size alone doesn't
pose any problems for latency.

A dedicated, point-to-point ethernet connection is used, so there are no
ethernet switches or other devices connected that can interfere to add
latency or the potential to packets lost due to collision. On the
remote side, a dedicated part of the FPGA chip handles packets in real
time as well. On the Linux side, the PREEMPT-RT kernel has apprently
removed most latency even from the regular kernel ethernet drivers.
(so no need for special "rtnet" drivers)

Using a 2kHz servo period (.5ms interval), Peter Wallace of Mesa has run
one of these boards with an earlier iteration of the driver for over 30
billion packets (real-world months) without interruption, and for me the
driver has worked properly for as long as I cared to let it run --
though that's only been for minutes at a time, nothing like Peter's
extended test.

I suspect that we'll discover that certain NICs are better than others
at giving realtime performance, just like with the PCs themselves. For
instance, Sam has seen on one system that the onboard RTL-chipset NIC
worked reliably, and a PCI NIC with Intel chipset did not. I've seen
good performance with the only NIC I tried, a PCI-E NIC with an Intel
chipset.

Jeff

[Charles Steinkuehler]

Post by Jeff Epler
A dedicated, point-to-point ethernet connection is used, so there are no
ethernet switches or other devices connected that can interfere to add
latency or the potential to packets lost due to collision. On the
remote side, a dedicated part of the FPGA chip handles packets in real
time as well. On the Linux side, the PREEMPT-RT kernel has apprently
removed most latency even from the regular kernel ethernet drivers.
(so no need for special "rtnet" drivers)

It's also possible to communicate via raw packets using the low-latency
hooks provided in most Linux Ethernet drivers (as used for things like
libpcap, the high-speed stock trading folks, and the HPC world). This
avoids most of the IP stack latency overhead you get with the standard
Linux kernel but still allows use of stock Linux Ethernet drivers, so
you get broad hardware support and other folks will maintain the driver
code for you. :)

[Marius Liebenberg]

Thanks a lot guys. A lot of information as usual. I am very proud to be
part of this community of very clever and helpful people.
I will no doubt have lot to ask at a later stage about this topic. I
have a lot of code to go scan and reading up to do.
Any suggestions as to where to start. What is the simplest driver to
look at for now?

Post by Marius Liebenberg
I am wondering about how Ethernet motion control works together with
linuxcnc. How is the synchronized motion achieved over the link? Or what
kind of information is passed over the link to the controller?

[Jeff Epler]

Post by Marius Liebenberg
Thanks a lot guys. A lot of information as usual. I am very proud to be
part of this community of very clever and helpful people.
I will no doubt have lot to ask at a later stage about this topic. I
have a lot of code to go scan and reading up to do.
Any suggestions as to where to start. What is the simplest driver to
look at for now?

The only driver in LinuxCNC that uses ethernet is the one for Mesa's
7i80 card. It's in src/hal/drivers/mesa-hostmot2; the code that's
concerned with ethernet cards is in the file hm2_7i80.c (631 lines),
though it will require some understanding of the common parts of the
hostmot2 driver, e.g., hostmot2.c (1643 lines), tram.c (212 lines)
organizing the data to be read or written, and ioport.c (595 lines)
managing simple digital I/O pins might be a minimum to read to
understand the full story of how digital I/O pins work on hostmot2.
Then it will be easier to understand other parts of hostmot2 like pwm,
counter, and step generator (various other files), the things that
really make hostmot2 interesting...

Jeff