|
1 year ago | |
---|---|---|
apps | 1 year ago | |
config | 1 year ago | |
docs | 4 years ago | |
dpdk @ a5dce55556 | 1 year ago | |
dpdk-iface-kmod | 2 years ago | |
io_engine | 3 years ago | |
mtcp | 1 year ago | |
util | 2 years ago | |
.buildtest_dpdk.sh | 2 years ago | |
.buildtest_netmap.sh | 2 years ago | |
.buildtest_psio.sh | 2 years ago | |
.check_hyperthreading.sh | 2 years ago | |
.gitignore | 1 year ago | |
.gitmodules | 1 year ago | |
.travis.yml | 2 years ago | |
CHANGELOG | 5 years ago | |
LICENSE | 5 years ago | |
Makefile.am | 4 years ago | |
Makefile.in | 1 year ago | |
README | 1 year ago | |
README.config | 4 years ago | |
README.devel | 4 years ago | |
README.md | 1 year ago | |
README.netmap | 2 years ago | |
TODOS | 4 years ago | |
aclocal.m4 | 1 year ago | |
compile | 1 year ago | |
config.guess | 4 years ago | |
config.h.in | 2 years ago | |
config.sub | 4 years ago | |
configure | 1 year ago | |
configure.ac | 1 year ago | |
install-sh | 1 year ago | |
missing | 1 year ago | |
setup_linux_env.sh | 2 years ago | |
setup_mtcp_dpdk_env.sh | 2 years ago | |
setup_mtcp_onvm_env.sh | 2 years ago |
mTCP is a highly scalable user-level TCP stack for multicore systems. mTCP source code is distributed under the Modified BSD License. For more detail, please refer to the LICENSE. The license term of io_engine driver and ported applications may differ from the mTCP’s.
We require the following libraries to run mTCP.
libdpdk
(Intel’s DPDK package*) or libps
(PacketShader I/O engine library) or netmap
driverlibnuma
libpthread
librt
libgmp
(for DPDK/ONVM driver)Compling PSIO/DPDK/NETMAP/ONVM driver requires kernel headers.
apt-get install linux-headers-$(uname -r)
We have modified the dpdk package to export net_device stat data
(for Intel-based Ethernet adapters only) to the OS. To achieve this, we have
created a new LKM dpdk-iface-kmow. We also modified
mk/rte.app.mk
file to ease the compilation
process of mTCP applications. We recommend using our package for DPDK
installation.
You can optionally use CCP’s congestion control implementation rather than mTCP’s. You’ll have wider selection of congestion control algorithms with CCP. (Currently this feature is experimental and under revision.)
Using CCP for congestion control (disabled by
default), requires the CCP library. If you would like to enable CCP, simply run
configure script with --enable-ccp
option.
Install Rust. Any installation method should be fine. We recommend using rustup:
curl https://sh.rustup.rs -sSf | sh -- -y -v --default-toolchain nightly
````
2. Install the CCP command line utility:
```bash
cargo install portus --bin ccp
Build the library (comes with Reno and Cubic by default, use ccp get
to add others):
ccp makelib
You will also need to link your application against -lccp
and -lstartccp
as demonstrated in apps/example/Makefie.in
mtcp: mtcp source code directory
mtcp/src: source code
mtcp/src/include: mTCP’s internal header files
mtcp/lib: library file
mtcp/include: header files that applications will use io_engine: event-driven packet I/O engine (io_engine)
io_engine/driver - driver source code
io_engine/lib - io_engine library
io_engine/include - io_engine header files
io_engine/samples - sample io_engine applications (not mTCP’s) dpdk - Intel’s Data Plane Development Kit
dpdk/… apps: mTCP applications
apps/example - example applications (see README)
apps/lighttpd-1.4.32 - mTCP-ported lighttpd (see INSTALL)
apps/apache_benchmark - mTCP-ported apache benchmark (ab) (see README-mtcp) util: useful source code for applications config: sample mTCP configuration files (may not be necessary)
mTCP can be prepared in four ways.
bash
git submodule init
git submodule update
2. Setup DPDK.
bash
./setup_mtcp_dpdk_env.sh [<path to $RTE_SDK>]
dpdk/
submodule as our DPDK driver. FYI, you can pass a different
dpdk source directory as command line argument.bash
sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
export RTE_SDK=`echo $PWD`/dpdk
export RTE_TARGET=x86_64-native-linuxapp-gcc
4. Setup mtcp library:
bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET
make
bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case ./configure
script prints an error, run the
following command; and then re-do step-4 (configure again):
bash
autoreconf -ivf
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --disable-hwcsum
to disable checksum offloading.libmtcp.a
in mtcp/lib
mtcp/include
apps/example
apps/example
epserver.conf
for server-side configurationepwget.conf
for client-side configurationbash
./setup_linux_env.sh [<path to $RTE_SDK>]
- Press [29] to unbind the Ethernet ports
- Press [30] to remove igb_uio.ko driver
- Press [33] to remove hugepage mappings
- Press [34] to quit the tool
### PSIO VERSION
1. make in io_engine/driver:
bash
make
bash
./install.py <# cores> <# cores>
- refer to http://shader.kaist.edu/packetshader/io_engine/
- you may need to change the ip address in install.py:46
3. Setup mtcp library:
bash
./configure --with-psio-lib=<$path_to_ioengine>
# e.g. ./configure --with-psio-lib=`echo $PWD`/io_engine
make
bash
./configure --with-psio-lib=`echo $PWD`/io_engine CFLAGS="-DMAX_CPUS=8"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding).
- In case ./configure
script prints an error, run the
following command; and then re-do step-3 (configure again):
bash
autoreconf -ivf
libmtcp.a
in mtcp/lib
mtcp/include
apps/example
apps/example
epserver.conf
for server-side configurationepwget.conf
for client-side configurationRun the applications!
NEW: Now you can run mTCP applications (server + client) locally.
A local setup is useful when only 1 machine is available for the experiment.
ONVM configurations are placed as .conf
files in apps/example directory.
ONVM basics are explained in https://github.com/sdnfv/openNetVM.
Before running the applications make sure that onvm_mgr is running.
Also, no core overlap between applications and onvm_mgr is allowed.
Set up the dpdk interfaces:
./setup_mtcp_onvm_env.sh
bash
sudo ifconfig dpdk0 x.x.x.x netmask 255.255.255.0 up
Setup mtcp library
./configure --with-dpdk-lib=$<path_to_dpdk> --with-onvm-lib=$<path_to_onvm_lib>
# e.g. ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=`echo $ONVM_HOME`/onvm
make
bash
./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> CFLAGS="-DMAX_CPUS=32"
Please note that your NIC should support RSS queues equal to the MAX_CPUS value
(since mTCP expects a one-to-one RSS queue to CPU binding)../configure
script prints an error, run the
following command; and then re-do step-4 (configure again):
bash
autoreconf -ivf
- checksum offloading in the NIC is now ENABLED (by default)!!!
- this only works for dpdk at the moment
- use ./configure --with-dpdk-lib=$RTE_SDK/$RTE_TARGET --with-onvm-lib=$<path_to_onvm_lib> --disable-hwcsum
to disable checksum offloading.
- check libmtcp.a
in mtcp/lib
- check header files in mtcp/include
- check example binary files in apps/example
5. Check the configurations in apps/example
- epserver.conf
for server-side configuration
- epwget.conf
for client-side configuration
- you may write your own configuration file for your application
6. Run the applications!
7. You can revert back all your changes by running the following script.
bash
./setup_linux_env.sh
EAL: FATAL: Cannot init memory
Cannot mmap memory for rte_config at [0x7ffff7fb6000], got [0x7ffff7e74000] - please use '--base-virtaddr' option
EAL: Cannot mmap device resource file /sys/bus/pci/devices/0000:06:00.0/resource3 to address: 0x7ffff7ff1000
To prevent this, use the base virtual address parameter to run the ONVM manager (core list arg 0xf8
isn’t actually used by mtcp NFs but is required), e.g.:
bash
cd openNetVM/onvm
./go.sh 1,2,3 1 0xf8 -s stdout -a 0x7f000000000
### NETMAP VERSION
See README.netmap for details.
## Tested environments
mTCP runs on Linux-based operating systems (2.6.x for PSIO) with generic
x86_64 CPUs, but to help evaluation, we provide our tested environments
as follows.
Intel Xeon E5-2690 octacore CPU @ 2.90 GHz 32 GB of RAM (4 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Debian 6.0.7 (Linux 2.6.32-5-amd64)
Intel Core i7-3770 quadcore CPU @ 3.40 GHz 16 GB of RAM (2 memory channels)
10 GbE NIC with Intel 82599 chipset (specifically Intel X520-DA2)
Ubuntu 10.04 (Linux 2.6.32-47)
Event-driven PacketShader I/O engine (extended io_engine-0.2)
- PSIO is currently only compatible with Linux-2.6.
We tested the DPDK version (polling driver) with Linux-3.13.0 kernel.
## Notes
1. mTCP currently runs with fixed memory pools. That means, the size of
TCP receive and send buffers are fixed at the startup and does not
increase dynamically. This could be performance limit to the large
long-lived connections. Be sure to configure the buffer size
appropriately to your size of workload.
2. The client side of mTCP supports mtcp_init_rss() to create an
address pool that can be used to fetch available address space in
O(1). To easily congest the server side, this function should be
called at the application startup.
3. The supported socket options are limited for right now. Please refer
to the mtcp/src/api.c for more detail.
4. The counterpart of mTCP should enable TCP timestamp.
5. mTCP has been tested with the following Ethernet adapters:
1. Intel-82598 ixgbe (Max-queue-limit: 16)
2. Intel-82599 ixgbe (Max-queue-limit: 16)
3. Intel-I350 igb (Max-queue-limit: 08)
4. Intel-X710 i40e (Max-queue-limit: ~)
5. Intel-X722 i40e (Max-queue-limit: ~)
## Frequently asked questions
1. How can I quit the application?
- Use ^C to gracefully shutdown the application. Two consecutive
^C (separated by 1 sec) will force quit.
2. My application doesn’t use the address specified from ifconfig.
- For some Linux distros(e.g. Ubuntu), NetworkManager may re-assign
a different IP address, or delete the assigned IP address.
- Disable NetworkManager temporarily if that’s the case.
NetworkManager will be re-enabled upon reboot.
bash
sudo service network-manager stop
Yes, mTCP allows static route and arp configuration. Go to the config directory and see sample_route.conf or sample_arp.conf. Copy and adapt it to your condition and link (ln -s) the config directory to the application directory. mTCP will find config/route.conf and config/arp.conf for static configuration.
Do not remove I/O driver (ps_ixgbe/igb_uio
) while running mTCP
applications. The application will panic!
Use the ps_ixgbe/dpdk driver contained in this package, not the one from some other place (e.g., from io_engine github).
GitHub issue board is the preferred way to report bugs and ask questions about mTCP. CONTACTS FOR THE AUTHORS User mailing list EunYoung Jeong M. Asim Jamshed