Commit 5a8379cd authored by Raphael Defosseux's avatar Raphael Defosseux

doc(oran): fixed a few typos and better explanation of the CPU allocation scheme

Signed-off-by: default avatarRaphael Defosseux <raphael.defosseux@eurecom.fr>
parent 65ff4b78
...@@ -67,10 +67,36 @@ Tested libxran releases: ...@@ -67,10 +67,36 @@ Tested libxran releases:
2. We recommend you to start with a fresh installation of OS (either RHEL or Ubuntu). You have to install realtime kernel on your OS (Operating System). Based on your OS you can search how to install realtime kernel. 2. We recommend you to start with a fresh installation of OS (either RHEL or Ubuntu). You have to install realtime kernel on your OS (Operating System). Based on your OS you can search how to install realtime kernel.
3. Once the realtime kernel is installed then you have to change the boot arguments. You can use `tuned` command for this or you can do it manually via re-building the grub. 3. Once the realtime kernel is installed then you have to change the boot arguments. You can use `tuned` command for this or you can do it manually via re-building the grub.
In below example we have shown the output of `/proc/cmdline` for two different servers, each of them have different number of numa nodes. Be careful in isolating the CPUs in your environment. ### 1.1.0 CPU allocation
**This section is important to read, regardless of the operating system you are using.**
Your server could be:
* 1-socket CPU (See [the Ubuntu example](#112-ubuntu-22043-lts)): all the processors are sharing a single memory system.
* 2-socket CPU (see [the RHEL example](#111-rhel-92)): processors are grouped in 2 memory systems.
- Usually the even (ie `0,2,4,...`) CPUs are on the 1st socket
- And the odd (ie (`1,3,5,...`) CPUs are on the 2nd socket
DPDK, OAI and kernel threads require to be properly allocated to extract maximum real-time performance for your use case.
**NOTE**: The default OAI 7.2 configuration file requires isolated CPU 0,2,4 for DPDK/libXRAN, CPU 6 for `ru_thread` and CPU 8 for `L1_rx_thread`. 1. **NOTE**: Currently the default OAI 7.2 configuration file requires isolated **CPUs 0,2,4** for DPDK/libXRAN, **CPU 6** for `ru_thread` and **CPU 8** for `L1_rx_thread`. It is preferrable to have all these threads on the same socket.
2. Allocating CPUs to the OAI nr-softmodem is done using the `--thread-pool` option. Allocating 4 CPUs is the minimal configuration but we recommend to allocate at least **8** CPUs. And they can be on a different socket as the DPDK threads.
3. And to avoid kernel preempting these allocated CPUs, it is better to force the kernel to use un-allocated CPUs.
Let summarize for example on a `32-CPU` system, regardless of the number of sockets:
| Applicative Threads | Allocated CPUs |
| ------------------- | -------------- |
| XRAN DPDK usage | 0,2,4 |
| OAI `ru_thread` | 6 |
| OAI `L1_rx_thread` | 8 |
| OAI `nr-softmodem` | 1,3,5,7,9,11,13,15 |
| kernel | 16-31 |
In below example we have shown the output of `/proc/cmdline` for two different servers, each of them have different number of numa nodes. Be careful in isolating the CPUs in your environment.
Modifying the `linux` command line usually requires to edit the `/etc/default/grub`, run a `grub` command and reboot the server.
### 1.1.1 RHEL 9.2 ### 1.1.1 RHEL 9.2
...@@ -84,7 +110,7 @@ NUMA: ...@@ -84,7 +110,7 @@ NUMA:
``` ```
```bash ```bash
mitigations=off usbcore.autosuspend=-1 intel_iommu=on intel_iommu=pt selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 audit=0 skew_tick=1 isolcpus=managed_irq,domain,0,2,4,6,8,10,12,14,16 nohz_full=0,2,4,6,8,10,12,14,16 rcu_nocbs=0,2,4,6,8,10,12,14,16 rcu_nocb_poll intel_pstate=disable nosoftlockup tsc=nowatchdog cgroup_disable=memory mce=off hugepagesz=1G hugepages=40 hugepagesz=2M hugepages=0 default_hugepagesz=1G skew_tick=1 isolcpus=managed_irq,domain,0,2,4,6,8,10,12,14 intel_pstate=disable nosoftlockup tsc=reliable mitigations=off usbcore.autosuspend=-1 intel_iommu=on intel_iommu=pt selinux=0 enforcing=0 nmi_watchdog=0 softlockup_panic=0 audit=0 skew_tick=1 isolcpus=managed_irq,domain,0,2,4,6,8,10,12,14,16 nohz_full=0,2,4,6,8,10,12,14,16 rcu_nocbs=0,2,4,6,8,10,12,14,16 rcu_nocb_poll intel_pstate=disable nosoftlockup tsc=nowatchdog cgroup_disable=memory mce=off hugepagesz=1G hugepages=40 hugepagesz=2M hugepages=0 default_hugepagesz=1G skew_tick=1 isolcpus=managed_irq,domain,0,2,4,6,8,10,12,14 kthread_cpus=18-35 intel_pstate=disable nosoftlockup tsc=reliable
``` ```
### 1.1.2 Ubuntu 22.04.3 LTS ### 1.1.2 Ubuntu 22.04.3 LTS
...@@ -97,7 +123,7 @@ NUMA: ...@@ -97,7 +123,7 @@ NUMA:
``` ```
```bash ```bash
isolcpus=0-2,8-17 nohz_full=0-2,8-17 rcu_nocbs=0-2,8-17 rcu_nocb_poll nosoftlockup default_hugepagesz=1GB hugepagesz=1G hugepages=20 amd_iommu=on iommu=pt mitigations=off skew_tick=1 selinux=0 enforcing=0 tsc=nowatchdog nmi_watchdog=0 softlockup_panic=0 audit=0 vt.handoff=7 isolcpus=0-15 nohz_full=0-15 rcu_nocbs=0-15 kthread_cpus=16-31 rcu_nocb_poll nosoftlockup default_hugepagesz=1GB hugepagesz=1G hugepages=20 amd_iommu=on iommu=pt mitigations=off skew_tick=1 selinux=0 enforcing=0 tsc=nowatchdog nmi_watchdog=0 softlockup_panic=0 audit=0 vt.handoff=7
``` ```
### 1.1.3 Common ### 1.1.3 Common
...@@ -343,14 +369,14 @@ git checkout oran_e_maintenance_release_v1.0 ...@@ -343,14 +369,14 @@ git checkout oran_e_maintenance_release_v1.0
Apply patches (available in `oai_folder/cmake_targets/tools/oran_fhi_integration_patches/E`) Apply patches (available in `oai_folder/cmake_targets/tools/oran_fhi_integration_patches/E`)
```bash ```bash
cp ~/openairinterface5g/cmake_targets/tools/oran_fhi_integration_patches/bronze/*.patch . cp ~/openairinterface5g/cmake_targets/tools/oran_fhi_integration_patches/E/*.patch .
git apply oaioran_E.patch git apply oaioran_E.patch
``` ```
### 2.1.3. Build (valid for any release) ### 2.1.3. Build (valid for any release)
Set up the environment (change the path if you use different folders). We recommend you copy all variables in a file and then you source that file evertime you compile the source code. Set up the environment (change the path if you use different folders). We recommend you copy all variables in a file and then you source that file everytime you compile the source code.
```bash ```bash
export XRAN_LIB_DIR=~/phy/fhi_lib/lib/build export XRAN_LIB_DIR=~/phy/fhi_lib/lib/build
...@@ -392,7 +418,7 @@ sudo apt install -y libnuma-dev ...@@ -392,7 +418,7 @@ sudo apt install -y libnuma-dev
sudo dnf install -y numactl-devel sudo dnf install -y numactl-devel
./build_oai --gNB --ninja -t oran_fhlib_5g (Add, -I if you are building for the first time on server for installing external dependencies) ./build_oai --gNB --ninja -t oran_fhlib_5g (Add, -I if you are building for the first time on server for installing external dependencies)
#check if all the libraries are properly linked to liboai_transpro.so #check if all the libraries are properly linked to liboai_transpro.so
ldd ran_build/build/lliboran_fhlib_5g.so ldd ran_build/build/liboran_fhlib_5g.so
linux-vdso.so.1 (0x00007ffc9bdfc000) linux-vdso.so.1 (0x00007ffc9bdfc000)
librte_node.so.0.200.2 => /usr/local/lib64/librte_node.so.0.200.2 (0x00007f2da93bd000) librte_node.so.0.200.2 => /usr/local/lib64/librte_node.so.0.200.2 (0x00007f2da93bd000)
librte_graph.so.0.200.2 => /usr/local/lib64/librte_graph.so.0.200.2 (0x00007f2da93b2000) librte_graph.so.0.200.2 => /usr/local/lib64/librte_graph.so.0.200.2 (0x00007f2da93b2000)
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment