Why the ip header checksum field of the IP(ICMP) packet sent by F-stack is '0'

[TrevorTaoARM]

When I start f-stack with /usr/local/nginx_fstack/sbin/nginx on my arm64 server, I 'ping' the fstack(dpdk) port from another host,
and the ping can not get the desired result.
I used tcpdump to capture the ping(ICMP) packet, and found that the value of IP header checksum field in the ICMP reply packet sent by the fstack is 0, which is not the desired result by the host.
The capture of the packet is showed by Wireshark here:

Can you tell me why the IP header cksum field is 0?

And the f-stack.conf is set as default:
...

TCP segment offload, default: disabled.

tso=0
...

[whl739]

It seems that f-stack enabled tx checksum offloading, but the hardware didn't fill a checksum.
You can paste the startup info, and check if f-stack enables tx checksum offloading.

[whl739]

Nginx should configure daemon off, and then some logs will be printed in stdout.

[TrevorTaoARM]

@whl739 Yes, I do see the f-stack enabled the tx checksum offloading, and I just surround it by comment
pconf->hw_features.tx_csum_ip = 1;
and Now the 'ping' can work now.
But it seems nginx still cannot work now, I can not wget/curl from the nginx server.

[TrevorTaoARM]

@whl739 Now the nginx can work now, I just turned off the following 3 checksum offload temporarily:
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
If you had better solution or any suggestion, please tell me too.
Thank you.

[whl739]

Generally, performance will be better when enable hardware offloads.
But in you case, it seems that there are some problems with tx checksum offloads.
I didn't do any tests on an arm server, no other suggestions.

[sunrash]

hi @whl739 , I have a similar problem.
Hardware:

2x servers :
82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb (2x port) 
I350 Gigabit Network Connection 152 onboard (4x port)
Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz 
128GB RAM

Software

Centos 7.6.1810
BOOT_IMAGE=/vmlinuz-3.10.0-957.1.3.el7.x86_64 root=UUID=ef9cbceb-bfea-4e5d-9e62-f5dac37a8755 ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8 default_hugepagesz=1GB hugepagesz=1GB hugepages=16

Network devices using DPDK-compatible driver
0000:06:00.0 '82599ES 10-Gigabit SFI/SFP+ Network Connection 10fb' drv=igb_uio unused=ixgbe

Other network devices using kernel driver

f-stack config:

[dpdk]
lcore_mask=1
channel=4
promiscuous=1
numa_on=1
tso=0
vlan_strip=0
port_list=0
[port0]
addr=192.168.1.2
netmask=255.255.255.0
broadcast=192.168.1.255
gateway=192.168.1.1
pcap=/data/nginx/a.pcap
[freebsd.boot]
hz=100
fd_reserve=1024
kern.ipc.maxsockets=262144
net.inet.tcp.syncache.hashsize=4096
net.inet.tcp.syncache.bucketlimit=100
net.inet.tcp.tcbhashsize=65536
kern.ncallout=262144
[freebsd.sysctl]
kern.ipc.somaxconn=32768
kern.ipc.maxsockbuf=16777216
net.link.ether.inet.maxhold=5
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.sendspace=16384
net.inet.tcp.recvspace=8192
net.inet.tcp.nolocaltimewait=1
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sack.enable=1
net.inet.tcp.blackhole=1
net.inet.tcp.msl=2000
net.inet.tcp.delayed_ack=0
net.inet.udp.blackhole=1
net.inet.ip.redirect=0

in nginx.conf :

daemon off;
kernel_network_stack on;

When I start f-stack with /data/f-stack/nginx/sbin/nginx on my first server, I 'ping' the fstack(dpdk) port from another host, and the ping can not get the desired result.
TCPDUMP out on second server:

192.168.1.66.38206 > 192.168.1.2.http: Flags [S], **cksum 0x83c3 (incorrect -> 0x126b)**, seq 4238118135, win 29200, options [mss 1460 v6  ifup-plusb  ifup-routes  ifup-wireless      network-functions-ipv6
,sackOK,TS val 1730066862 ecr 0,nop,wscale 7], length 0                                                                                ifcfg-eno2  ifcfg-enp6s0f0  ifdown       ifdown-bnep      ifdown-ipv6  ifdown-ppp     ifdown-tunnel  ifup-TeamPort  ifup-eth   ifup-is
12:06:17.151890 IP (tos 0x0, ttl 64, id 13658, offset 0, flags [DF], proto TCP (6), length 60)                                         dn  ifup-post   ifup-sit     init.ipv6-global
    192.168.1.66.38206 > 192.168.1.2.http: Flags [S], **cksum 0x83c3 (incorrect -> 0x0e81)**, seq 4238118135, win 29200, options [mss 1460 ifcfg-eno3  ifcfg-enp6s0f1  ifdown-Team  ifdown-eth       ifdown-isdn  ifdown-routes  ifup           ifup-aliases   ifup-ippp  ifup-pl
,sackOK,TS val 1730067864 ecr 0,nop,wscale 7], length 0

Nginx STDOUT in first server:

[dpdk]: lcore_mask=1
[dpdk]: channel=4
[dpdk]: promiscuous=1
[dpdk]: numa_on=1
[dpdk]: tso=0
[dpdk]: vlan_strip=0
[dpdk]: port_list=0
[port0]: addr=192.168.1.2
[port0]: netmask=255.255.255.0
[port0]: broadcast=192.168.1.255
[port0]: gateway=192.168.1.1
[port0]: lcore_list=0
[port0]: pcap=/data/nginx/a.pcap
[freebsd.boot]: fd_reserve=1024
[freebsd.boot]: kern.ipc.maxsockets=262144
[freebsd.boot]: net.inet.tcp.syncache.hashsize=4096
[freebsd.boot]: net.inet.tcp.syncache.bucketlimit=100
[freebsd.boot]: net.inet.tcp.tcbhashsize=65536
[freebsd.boot]: kern.ncallout=262144
[freebsd.sysctl]: kern.ipc.somaxconn=32768
[freebsd.sysctl]: kern.ipc.maxsockbuf=16777216
[freebsd.sysctl]: net.link.ether.inet.maxhold=5
[freebsd.sysctl]: net.inet.tcp.fast_finwait2_recycle=1
[freebsd.sysctl]: net.inet.tcp.sendspace=16384
[freebsd.sysctl]: net.inet.tcp.recvspace=8192
[freebsd.sysctl]: net.inet.tcp.nolocaltimewait=1
[freebsd.sysctl]: net.inet.tcp.cc.algorithm=cubic
[freebsd.sysctl]: net.inet.tcp.sendbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.recvbuf_max=16777216
[freebsd.sysctl]: net.inet.tcp.sendbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.recvbuf_auto=1
[freebsd.sysctl]: net.inet.tcp.sendbuf_inc=16384
[freebsd.sysctl]: net.inet.tcp.recvbuf_inc=524288
[freebsd.sysctl]: net.inet.tcp.sack.enable=1
[freebsd.sysctl]: net.inet.tcp.blackhole=1
[freebsd.sysctl]: net.inet.tcp.msl=2000
[freebsd.sysctl]: net.inet.tcp.delayed_ack=0
[freebsd.sysctl]: net.inet.udp.blackhole=1
[freebsd.sysctl]: net.inet.ip.redirect=0
f-stack -c1 -n4 --proc-type=primary EAL: Probing VFIO support...
EAL: PCI device 0000:02:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:02:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:02:00.2 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:02:00.3 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:10fb net_ixgbe
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL:   probe driver: 8086:10fb net_ixgbe
lcore: 0, port: 0, queue: 0
create mbuf pool on socket 0
create ring:dispatch_ring_p0_q0 success, 2047 ring entries are now free!
Port 0 MAC: 90 e2 ba dd 4d d8
Port 0 modified RSS hash function based on hardware support,requested:0x3ffffc configured:0x38d34
RX checksum offload supported
TX ip checksum offload supported
TX TCP&UDP checksum offload supported
TSO is disabled
port[0]: rss table size: 128
set port 0 to promiscuous mode ok
Checking link status.done
Port 0 Link Up - speed 10000 Mbps - full-duplex
link_elf_lookup_symbol: missing symbol hash table
link_elf_lookup_symbol: missing symbol hash table
Timecounters tick every 10.000 msec
Timecounter "ff_clock" frequency 100 Hz quality 1
f-stack-0: Ethernet address: 90:e2:ba:dd:4d:d8

If I change the tso in f-stack.conf to tso=1 , ping and http(nginx) not works, but if I change the following parameters:
pconf->hw_features.rx_csum = 0;
pconf->hw_features.tx_csum_ip = 0;
pconf->hw_features.tx_csum_l4 = 0;
pconf->hw_features.tx_tso = 0
in the file lib/ff_dpdk_if.c and make the assembly, and PING is works, but HTTP(nginx) not works.

TCPDUMP:

13:18:37.626474 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.1.66 is-at 90:e2:ba:dd:4d:dc, length 28
3:18:37.626600 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.1.2 > 192.168.1.66: ICMP echo reply, id 51337, seq 1, length 64
13:18:38.625912 IP (tos 0x0, ttl 64, id 64173, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.1.66 > 192.168.1.2: ICMP echo request, id 51337, seq 2, length 64

13:20:49.161356 IP (tos 0x0, ttl 64, id 55738, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.66.38218 > 192.168.1.2.http: Flags [S], cksum 0x83c3 **(incorrect -> 0x1f5a)**, seq 398926811, win 29200, options [mss 1460,sackOK,TS val 1734539873 ecr 0,nop,wscale 7], length 0
13:20:50.163873 IP (tos 0x0, ttl 64, id 55739, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.66.38218 > 192.168.1.2.http: Flags [S], cksum 0x83c3 **(incorrect -> 0x1b6f)**, seq 398926811, win 29200, options [mss 1460,sackOK,TS val 1734540876 ecr 0,nop,wscale 7], length 0
13:20:52.167915 IP (tos 0x0, ttl 64, id 55740, offset 0, flags [DF], proto TCP (6), length 60)
192.168.1.66.38218 > 192.168.1.2.http: Flags [S], cksum 0x83c3 **(incorrect -> 0x139b)**, seq 398926811, win 29200, options [mss 1460,sackOK,TS val 1734542880 ecr 0,nop,wscale 7], length 0

[Jianlin-lv]

@whl739 We encountered the same problem and made the software work by rolling back dpdk related commits.
“git reset --hard 9da1cd9 ”

[whl739]

It seemed that there were csum offloading related problems when interacting with the newer version of dpdk.
We will do some tests and fix it.

[sunrash]

@whl739 I did the test using the testpmd app and got the following results (Maybe this will help.):
./testpmd -c 0xFF00FF00 -n 4 -- --nb-cores=2 --enable-rx-cksum --disable-rss --rxd=1024 --txd=1024 -i --rxfreet=16 --portmask=0x3 --tx-offloads=0xf

testpmd> set fwd csum
Set csum packet forwarding mode
testpmd> start
csum packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 9 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  csum packet forwarding packets/burst=32
  nb forwarding cores=2 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0xe Tx offloads=0xf
    RX queue: 0
      RX desc=1024 - RX free threshold=16
      RX threshold registers: pthresh=8 hthresh=8  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 7              RX-dropped: 0             RX-total: 7
  Bad-ipcsum: 0              Bad-l4csum: 0             Bad-outer-l4csum: 0
  TX-packets: 7              TX-dropped: 0             TX-total: 7
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 7              RX-dropped: 0             RX-total: 7
  TX-packets: 7              TX-dropped: 0             TX-total: 7
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

./testpmd -c 0xFF00FF00 -n 4 -- --nb-cores=2 --disable-rss --rxd=1024 --txd=1024 -i --rxfreet=16 --portmask=0x3 --tx-offloads=0x0

set fwd csum
Set csum packet forwarding mode
testpmd> start
csum packet forwarding - ports=1 - cores=1 - streams=1 - NUMA support enabled, MP allocation mode: native
Logical Core 9 (socket 1) forwards packets on 1 streams:
  RX P=0/Q=0 (socket 0) -> TX P=0/Q=0 (socket 0) peer=02:00:00:00:00:00

  csum packet forwarding packets/burst=32
  nb forwarding cores=2 - nb forwarding ports=1
  port 0: RX queue number: 1 Tx queue number: 1
    Rx offloads=0x0 Tx offloads=0x0
    RX queue: 0
      RX desc=1024 - RX free threshold=16
      RX threshold registers: pthresh=8 hthresh=8  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=1024 - TX free threshold=32
      TX threshold registers: pthresh=32 hthresh=0  wthresh=0
      TX offloads=0x0 - TX RS bit threshold=32
testpmd> stop
Telling cores to stop...
Waiting for lcores to finish...

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 6              RX-dropped: 0             RX-total: 6
  Bad-ipcsum: 0              Bad-l4csum: 0             Bad-outer-l4csum: 0
  TX-packets: 6              TX-dropped: 0             TX-total: 6
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 6              RX-dropped: 0             RX-total: 6
  TX-packets: 6              TX-dropped: 0             TX-total: 6
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.
testpmd> stop

[TrevorTaoARM]

After some tests and trials, it seems I had found the cause and solution of this issue. I would submit a PR to resolve it ASAP.

[jfb8856606]

After F-Stack upgrate DPDK to 18.11 LTS, found some issues, include this csum error.

And the master branch has reverted the commit 8850115 "DPDK:upgrade to 18.11.0 LTS".

I have created a dev branch to fix this issue by commit
d9665c9. But there is some other issue in this branch, suach as the "kni" feature.

[sunrash]

@TrevorTaoARM @jfb8856606 @whl739
I tested the patch, it works for my configuration.