admin/TripleCross
1
0
Fork 0
mirror of https://github.com/h3xduck/TripleCross.git synced 2025-12-25 02:43:07 +08:00
Code Issues Packages Projects Releases Wiki Activity

Updated some style aspects, updated positions of tables and figures, other changes.

Browse Source
This commit is contained in:
h3xduck
2022-06-11 16:32:00 -04:00
parent e5bb65925d
commit e697dc867d
16 changed files with 1135 additions and 1099 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
docs/chapters/chapter2.tex
View File

@@ -12,13 +12,13 @@ In this section we will detail the origins of eBPF in the Linux kernel. By offer
\subsection{Introduction to the BPF system}
Nowadays eBPF is not officially considered to be an acronym anymore\cite{ebpf_io}, but it remains largely known as "extended Berkeley Packet Filters", given its roots in the Berkeley Packet Filter (BPF) technology, now known as classic BPF.
BPF was introduced in 1992 by Steven McCanne and Van Jacobson in the paper "The BSD Packet Filter: A New Architecture for User-level Packet Capture"\cite{bpf_bsd_origin}, as a new filtering technology for network packets in the BSD platform. It was first integrated in the Linux kernel on version 2.1.75\cite{ebpf_history_opensource}.
BPF was introduced in 1992 by Steven McCanne and Van Jacobson in the paper "The BSD Packet Filter: A New Architecture for User-level Packet Capture"\cite{bpf_bsd_origin}, as a new filtering technology for network packets in the BSD platform. It was first integrated in the Linux kernel on version 2.1.75 \cite{ebpf_history_opensource}.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=12cm, keepaspectratio=true]{classic_bpf.jpg}
\caption{Sketch of the functionality of classic BPF}
\caption{Functionality of classic BPF. Based on the figure at the original paper \cite{bpf_bsd_origin_bpf_page2}.}
\label{fig:classif_bpf}
\end{figure}
@@ -26,10 +26,10 @@ Figure \ref{fig:classif_bpf} shows how BPF was integrated in the existing networ
\subsection{The BPF virtual machine} \label{subsection:bpf_vm}
In a technical level, BPF comprises both the BPF filter programs developed by the user and the BPF module included in the kernel which allows for loading and running the BPF filters. This BPF module in the kernel works as a virtual machine\cite{bpf_bsd_origin_bpf_page1}, meaning that it parses and interprets the filter program by providing simulated components needed for its execution, turning into a software-based CPU. Because of this reason, it is usually referred as the BPF Virtual Machine (BPF VM). The BPF VM comprises the following components:
In a technical level, BPF comprises both the BPF filter programs developed by the user and the BPF module included in the kernel which allows for loading and running the BPF filters. This BPF module in the kernel works as a virtual machine \cite{bpf_bsd_origin_bpf_page1}, meaning that it parses and interprets the filter program by providing simulated components needed for its execution, turning into a software-based CPU. Because of this reason, it is usually referred as the BPF Virtual Machine (BPF VM). The BPF VM comprises the following components:
\begin{itemize}
\item \textbf{An accumulator register}, used to store intermediate values of operations.
\item \textbf{An index register}, used to modify operand addresses, it is usually incorporated to optimize vector operations\cite{index_register}.
\item \textbf{An index register}, used to modify operand addresses, it is usually incorporated to optimize vector operations \cite{index_register}.
\item \textbf{An scratch memory store}, a temporary storage.
\item \textbf{A program counter}, used to point to the next machine instruction to execute in a filter program.
\end{itemize}
@@ -42,13 +42,6 @@ As we mentioned in section \ref{subsection:bpf_vm}, the components of the BPF VM
\item If it returns \textit{false}, the packet is not accepted by the filter (and thus the network stack will be the next to operate it).
\end{itemize}
\begin{figure}[H]
\centering
\includegraphics[width=8cm]{cbpf_prog.jpg}
\caption{Execution of a BPF filter.}
\label{fig:cbpf_prog}
\end{figure}
Figure \ref{fig:cbpf_prog} shows an example of a BPF filter upon receiving a packet. In the figure, green lines indicate that the condition is true and red lines that it is evaluated as false. Therefore, the execution works as a control flow graph (CFG) which ends on a boolean value\cite{bpf_bsd_origin_bpf_page5}. The figure presents an example BPF program which accepts the following frames:
\begin{itemize}
\item Frames with an IP packet as a payload directed from IP address X.
@@ -57,12 +50,19 @@ Figure \ref{fig:cbpf_prog} shows an example of a BPF filter upon receiving a pac
\item Frames not from the ARP protocol directed from IP address Y to IP address X.
\end{itemize}
\begin{figure}[ht]
\centering
\includegraphics[width=8cm]{cbpf_prog.jpg}
\caption{Execution of a BPF filter.}
\label{fig:cbpf_prog}
\end{figure}
\subsection{BPF bytecode instruction format}
In order to implement the CFG to be run at the BPF VM, BPF filter programs are made up of BPF bytecode, which is defined by a new BPF instruction set. Therefore, a BPF filter program is an array of BPF bytecode instructions\cite{bpf_organicprogrammer_analysis}.
In order to implement the CFG to be run at the BPF VM, BPF filter programs are made up of BPF bytecode, which is defined by a new BPF instruction set. Therefore, a BPF filter program is an array of BPF bytecode instructions \cite{bpf_organicprogrammer_analysis}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|c|c|c|c|}
\hline
& OPCODE & JT & JF & K\\
@@ -70,11 +70,11 @@ In order to implement the CFG to be run at the BPF VM, BPF filter programs are m
BITS & 16 & 8 & 8 & 32\\
\hline
\end{tabular}
\caption{Table showing BPF instruction format. It is a fixed-length 64 bit instruction, the number of bits used by each field are indicated.}
\caption{BPF instruction format.}
\label{table:bpf_inst_format}
\end{table}
Table \ref{table:bpf_inst_format} shows the format of a BPF bytecode instruction. As it can be observed, it is a compound of:
Table \ref{table:bpf_inst_format} shows the format of a BPF bytecode instruction. As it can be observed, it is a fixed-length 64 bit instruction composed of:
\begin{itemize}
\item An \textbf{opcode}, similar to assembly opcode, it indicates the operation to be executed.
\item Field \textbf{jt} indicates the offset to the next instruction to jump in case a condition is evaluated as \textit{true}.
@@ -82,14 +82,7 @@ Table \ref{table:bpf_inst_format} shows the format of a BPF bytecode instruction
\item Field \textbf{k} is miscellaneous and its contents vary depending on the instruction opcode.
\end{itemize}
\begin{figure}[H]
\centering
\includegraphics[width=8cm]{bpf_instructions.png}
\caption{Table of supported classic BPF instructions, as shown by McCanne and Jacobson\cite{bpf_bsd_origin_bpf_page7}}
\label{fig:bpf_instructions}
\end{figure}
Figure \ref{fig:bpf_instructions} shows how BPF instructions are defined according to the BPF instruction set. As we mentioned, similarly to assembly, instructions include an opcode which indicates the operation to execute, and the multiple arguments defining the arguments of the operation. The table shows, in order by rows, the following instruction types\cite{bpf_bsd_origin_bpf_page8}:
Figure \ref{fig:bpf_instructions} shows how BPF instructions are defined according to the BPF instruction set. As we mentioned, similarly to assembly, instructions include an opcode which indicates the operation to execute, and the multiple arguments defining the arguments of the operation. The table shows, in order by rows, the following instruction types \cite{bpf_bsd_origin_bpf_page8}:
\begin{itemize}
\item Rows 1-4 are \textbf{load instructions}, copying the addressed value into the index or accumulator register.
\item Rows 4-6 are \textbf{store instructions}, copying the accumulator or index register into the scratch memory store.
@@ -98,33 +91,39 @@ Figure \ref{fig:bpf_instructions} shows how BPF instructions are defined accordi
\item Row 20 is a \textbf{return instruction}, it is positioned in the final end of the CFG, and indicate whether the filter accepts the packet (returning true) or otherwise rejects it (return false).
\end{itemize}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=8cm]{bpf_address_mode.png}
\caption{Table explaining the column address modes in Figure\ref{fig:bpf_instructions}, as shown by McCanne and Jacobson\cite{bpf_bsd_origin_bpf_page8}}
\label{fig:bpf_address_mode}
\includegraphics[width=8cm]{bpf_instructions.png}
\caption{Supported classic BPF instructions, as shown by McCanne and Jacobson \cite{bpf_bsd_origin_bpf_page7}}
\label{fig:bpf_instructions}
\end{figure}
The column \textit{addr modes} in figure \ref{fig:bpf_instructions} describes how the parameters of a BPF instruction are referenced depending on the opcode. The address modes are detailed in figure \ref{fig:bpf_address_mode}. As it can be observed, paremeters may consist of immediate values, offsets to memory positions or on the packet, the index register or combinations of the previous.
\begin{figure}[htbp]
\centering
\includegraphics[width=8cm]{bpf_address_mode.png}
\caption{BPF address modes, as shown by McCanne and Jacobson \cite{bpf_bsd_origin_bpf_page8}}
\label{fig:bpf_address_mode}
\end{figure}
\subsection{An example of BPF filter with tcpdump}
At the time, by filtering packets before they are handled by the kernel instead of using an user-level application, BPF offered a performance improvement between 10 and 150 times the state-of-the art technologies of the moment\cite{bpf_bsd_origin_bpf_page1}. Since then, multiple popular tools began to use BPF, such as the network tracing tool \textit{tcpdump}\cite{tcpdump_page}.
At the time, by filtering packets before they are handled by the kernel instead of using an user-level application, BPF offered a performance improvement between 10 and 150 times the state-of-the art technologies of the moment \cite{bpf_bsd_origin_bpf_page1}. Since then, multiple popular tools began to use BPF, such as the network tracing tool \textit{tcpdump} \cite{tcpdump_page}.
\textit{tcpdump} is a command-line tool that enables to capture and analyse the network traffic going through the system. It works by setting filters on a network interface, so that it shows the packets that are accepted by the filter. Still today, \textit{tcpdump} uses BPF for the filter implementation. We will now show an example of BPF code used by \textit{tcpdump} to implement a simple filter:
\textit{tcpdump} is a command-line tool that enables to capture and analyse the network traffic going through the system. It works by setting filters on a network interface, so that it shows the packets that are accepted by the filter. Still today, \textit{tcpdump} uses BPF for the filter implementation. Figure \ref{fig:bpf_tcpdump_example} shows an example of BPF code used by \textit{tcpdump} to implement a simple filter.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=10cm]{tcpdump_example.png}
\caption{BPF bytecode tcpdump needs to set a filter to display packets directed to port 80.}
\label{fig:bpf_tcpdump_example}
\end{figure}
Figure \ref{fig:bpf_tcpdump_example} shows how tcpdump sets a filter to display traffic directed to all interfaces (\textit{-i any}) directed to port 80. Flag \textit{-d} instructs tcpdump to display BPF bytecode.
In figure \ref{fig:bpf_tcpdump_example} we can see how tcpdump sets a filter to display traffic directed to all interfaces (\textit{-i any}) directed to port 80. Flag \textit{-d} instructs tcpdump to display BPF bytecode.
In the example, using the \textit{jf} and \textit{jt} fields, we can label the nodes of the CFG described by the BPF filter. Figure \ref{fig:tcpdump_ex_sol} describes the shortest graph path that a true comparison will need to follow to be accepted by the filter. Note how instruction 010 is checking the value 80, the one our filter is looking for in the port.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=6cm]{cBPF_prog_ex_sol.png}
\caption{Shortest path in the CFG described in the example of figure \ref{fig:bpf_tcpdump_example} that a packet needs to follow to be accepted by the BPF filter set with \textit{tcpdump}.}
@@ -136,7 +135,7 @@ This section discusses the current state of eBPF in the Linux kernel. By buildin
The addition of classic BPF in the Linux kernel set the foundations of eBPF, but nowadays it has already extended its presence to many other components other than traffic filtering. Similarly to how BPF filters were included in the networking module of the Linux kernel, we will now study the necessary changes made in the kernel to support these new program types. Table \ref{table:ebpf_history} shows the main updates that were incorporated and shaped modern eBPF of today.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|c|c|}
\hline
Description & Kernel version & Year\\
@@ -155,25 +154,28 @@ Description & Kernel version & Year\\
\hline
\end{tabular}
\caption{Table showing relevant eBPF updates. Note that only those relevant for our research objectives are shown. This is a selection of the official complete table at \cite{ebpf_funcs_by_ver}.}
\caption{Relevant eBPF updates. Note that only those relevant for our research objectives are shown. This is a selection of the official complete table at \cite{ebpf_funcs_by_ver}.}
\label{table:ebpf_history}
\end{table}
As it can be observed in the table above, the main breakthrough happened in the 3.15 version, where Alexei Starovoitov, along with Daniel Borkmann, decided to expand the capabilities of BPF by remodelling the BPF instruction set and overall architecture\cite{brendan_gregg_bpf_book}.
As it can be observed in the table above, the main breakthrough happened in the 3.15 version, where Alexei Starovoitov, along with Daniel Borkmann, decided to expand the capabilities of BPF by remodelling the BPF instruction set and overall architecture \cite{brendan_gregg_bpf_book}.
Figure \ref{fig:ebpf_architecture} offers an overview of the current eBPF architecture. During the subsequent subsections, we will proceed to explain its components in detail.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15cm]{ebpf_arch.jpg}
\caption{Figure showing overall eBPF architecture in the Linux kernel and the process of loading an eBPF program. Based on\cite{brendan_gregg_bpf_book} and \cite{ebpf_io_arch}.}
\caption{eBPF architecture in the Linux kernel and the process of loading an eBPF program. Based on \cite{brendan_gregg_bpf_book} and \cite{ebpf_io_arch}.}
\label{fig:ebpf_architecture}
\end{figure}
\subsection{eBPF instruction set} \label{subsection:ebpf_inst_set}
The eBPF update included a complete remodel of the instruction set architecture (ISA) of the BPF VM. Therefore, eBPF programs will need to follow the new architecture in order to be interpreted as valid and executed.
\begin{table}[H]
Table \ref{table:ebpf_inst_format} shows the new instruction format for eBPF programs \cite{ebpf_inst_set}. As it can be observed, it is a fixed-length 64 bit instruction. The new fields are similar to x86\_64 assembly, incorporating the typically found immediate and offset fields, and source and destination registers \cite{8664_inst_set_specs}. Similarly, the instruction set is extended to be similar to the one typically found on x86\_64 systems, the complete list can be consulted in the official documentation \cite{ebpf_inst_set}.
%Should I talk about assembly or this more in detail?
\begin{table}[htbp]
\begin{tabular}{|c|c|c|c|c|c|}
\hline
& IMM & OFF & SRC & DST & OPCODE \\
@@ -181,17 +183,13 @@ The eBPF update included a complete remodel of the instruction set architecture
BITS & 32 & 16 & 4 & 4 & 8\\
\hline
\end{tabular}
\caption{Table showing eBPF instruction format. It is a fixed-length 64 bit instruction, the number of bits used by each field are indicated.}
\caption{eBPF instruction format.}
\label{table:ebpf_inst_format}
\end{table}
Table \ref{table:ebpf_inst_format} shows the new instruction format for eBPF programs\cite{ebpf_inst_set}. The new fields are similar to x86\_64 assembly, incorporating the typically found immediate and offset fields, and source and destination registers\cite{8664_inst_set_specs}. Similarly, the instruction set is extended to be similar to the one typically found on x86\_64 systems, the complete list can be consulted in the official documentation\cite{ebpf_inst_set}.
%Should I talk about assembly or this more in detail?
With respect to the BPF VM registers, they get extended from 32 to 64 bits of length, and the number of registers is incremented to 10, instead of the original accumulator and index registers. These registers are also adapted to be similar to those in assembly, as it is shown in table \ref{table:ebpf_regs}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|c|m{21em}|}
\hline
eBPF register & x86\_64 register & Purpose\\
@@ -209,16 +207,16 @@ r9 & r15 & Callee saved register, value preserved between calls\\
r10 & rbp & Frame pointer for stack, read only\\
\hline
\end{tabular}
\caption{Table showing eBPF registers and their purpose in the BPF VM.\cite{ebpf_inst_set}\cite{ebpf_starovo_slides}.}
\caption{eBPF registers and their purpose in the BPF VM. \cite{ebpf_inst_set} \cite{ebpf_starovo_slides}.}
\label{table:ebpf_regs}
\end{table}
\subsection{JIT compilation}
We mentioned in subsection \ref{subsection:ebpf_inst_set} that eBPF registers and instructions describe an almost one-to-one correspondence to those in x86 assembly. This is in fact not a coincidence, but rather it is with the purpose of improving a functionality that was included in Linux kernel 3.0, called Just-in-Time (JIT) compilation\cite{ebpf_JIT}\cite{ebpf_JIT_demystify_page13}.
We mentioned in subsection \ref{subsection:ebpf_inst_set} that eBPF registers and instructions describe an almost one-to-one correspondence to those in x86 assembly. This is in fact not a coincidence, but rather it is with the purpose of improving a functionality that was included in Linux kernel 3.0, called Just-in-Time (JIT) compilation \cite{ebpf_JIT} \cite{ebpf_JIT_demystify_page13}.
JIT compiling is an extra step that optimizes the execution speed of eBPF programs. It consists of translating BPF bytecode into machine-specific instructions, so that they run as fast as native code in the kernel. Machine instructions are generated during runtime, written directly into executable memory and executed there\cite{ebpf_JIT_demystify_page14}.
JIT compiling is an extra step that optimizes the execution speed of eBPF programs. It consists of translating BPF bytecode into machine-specific instructions, so that they run as fast as native code in the kernel. Machine instructions are generated during runtime, written directly into executable memory and executed there \cite{ebpf_JIT_demystify_page14}.
Therefore, when using JIT compiling (a setting defined by the variable \textit{bpf\_jit\_enable}\cite{jit_enable_setting}, BPF registers are translated into machine-specific registers following their one-to-one mapping and bytecode instructions are translated into machine-specific instructions\cite{ebpf_starovo_slides_page23}. There no longer exists an interpretation step by the BPF VM, since we can execute the code directly\cite{brendan_gregg_bpf_book_bpf_vm}.
Therefore, when using JIT compiling (a setting defined by the variable \textit{bpf\_jit\_enable} \cite{jit_enable_setting}, BPF registers are translated into machine-specific registers following their one-to-one mapping and bytecode instructions are translated into machine-specific instructions \cite{ebpf_starovo_slides_page23}. There no longer exists an interpretation step by the BPF VM, since we can execute the code directly \cite{brendan_gregg_bpf_book_bpf_vm}.
The programs developed during this project will always have JIT compiling active.
@@ -228,10 +226,10 @@ We introduced in figure \ref{fig:ebpf_architecture} the presence of the so-calle
The verifier performs a series of tests which every eBPF program must pass in order to be accepted. Otherwise, user programs could leak privileged data, result in kernel memory corruption, or hang the kernel in an infinite loop, between others. Therefore, the verifier limits multiple aspects of eBPF programs so that they are restricted to the intended functionality, whilst at the same time offering a reasonable amount of freedom to the developer.
The following are the most relevant checks that the verifier performs in eBPF programs\cite{ebpf_verifier_kerneldocs}\cite{ebpf_JIT_demystify_page17-22}:
The following are the most relevant checks that the verifier performs in eBPF programs \cite{ebpf_verifier_kerneldocs} \cite{ebpf_JIT_demystify_page17-22}:
\begin{itemize}
\item Tests for ensuring overall control flow safety:
\subitem No loops allowed (bounded loops accepted since kernel version 5.3\cite{ebpf_bounded_loops}.
\subitem No loops allowed (bounded loops accepted since kernel version 5.3 \cite{ebpf_bounded_loops}.
\subitem Function call and jumps safety to known, reachable functions.
\subitem Sleep and blocking operations not allowed (to prevent hanging the kernel).
\item Tests for individual instructions:
@@ -247,13 +245,13 @@ These checks are performed by two main algorithms:
\end{itemize}
\subsection{eBPF maps} \label{subsection:ebpf_maps}
An eBPF map is a generic storage for eBPF programs used to share data between user and kernel space, to maintain persistent data between eBPF calls and to share information between multiple eBPF programs\cite{ebpf_maps_kernel}.
An eBPF map is a generic storage for eBPF programs used to share data between user and kernel space, to maintain persistent data between eBPF calls and to share information between multiple eBPF programs \cite{ebpf_maps_kernel}.
A map consists of a key + value tuple. Both fields can have an arbitrary data type, the map only needs to know the length of the key and the value field at its creation\cite{bpf_syscall}. Programs can open maps by specifying their ID, and lookup or delete elements in the map by specifying its key, also insert new ones by supplying the element value and they key to store it with.
A map consists of a key + value tuple. Both fields can have an arbitrary data type, the map only needs to know the length of the key and the value field at its creation \cite{bpf_syscall}. Programs can open maps by specifying their ID, and lookup or delete elements in the map by specifying its key, also insert new ones by supplying the element value and they key to store it with.
Therefore, creating a map requires a struct with the following fields:
Therefore, creating a map requires a struct with the fields shown in table \ref{table:ebpf_map_struct}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|c|}
\hline
FIELD & VALUE\\
@@ -264,11 +262,13 @@ value\_size & Size of the data structure to use as value field\\
max\_entries & Maximum number of elements in the map\\
\hline
\end{tabular}
\caption{Table showing common fields for creating an eBPF map.}
\caption{Common fields for creating an eBPF map.}
\label{table:ebpf_map_struct}
\end{table}
\begin{table}[H]
Table \ref{table:ebpf_map_types} describes the main types of eBPF maps that are available for use. During the development of our rootkit, we will mainly focus on hash maps (BPF\_MAP\_TYPE\_HASH), provided that they are simple to use and we do not require of any special storage for our research purposes.
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
TYPE & DESCRIPTION\\
@@ -280,12 +280,10 @@ BPF\_MAP\_TYPE\_PROG\_ARRAY & Stores descriptors of eBPF programs\\
\hline
\hline
\end{tabular}
\caption{Table showing types of eBPF maps. Only those used in our rootkit are displayed, the full list can be consulted in the man page \cite{bpf_syscall}}
\caption{Types of eBPF maps. Only those used in our rootkit are displayed, the full list can be consulted in the man page \cite{bpf_syscall}}
\label{table:ebpf_map_types}
\end{table}
Table \ref{table:ebpf_maps} describes the main types of eBPF maps that are available for use. During the development of our rootkit, we will mainly focus on hash maps (BPF\_MAP\_TYPE\_HASH), provided that they are simple to use and we do not require of any special storage for our research purposes.
\subsection{The eBPF ring buffer} \label{subsection:bpf_ring_buf}
eBPF ring buffers are a special kind of eBPF maps, providing a one-way directional communication system, going from an eBPF program in the kernel to an user space program that subscribes to its events.
@@ -294,9 +292,9 @@ eBPF ring buffers are a special kind of eBPF maps, providing a one-way direction
\subsection{The bpf() syscall} \label{subsection:bpf_syscall}
The bpf() syscall is used to issue commands from user space to kernel space in eBPF programs. This syscall is multiplexor, meaning that it can perform a great range of actions, changing its behaviour depending on the parameters.
The main operations that can be issued are described in table \ref{table:bpf_syscall}:
The main operations that can be issued are described in table \ref{table:ebpf_syscall}:
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{5cm}|>{\centering\arraybackslash}p{5cm}|}
\hline
COMMAND & ATTRIBUTES & DESCRIPTION\\
@@ -313,13 +311,13 @@ BPF\_MAP\_DELETE\_ELEM & Map ID and struct with key to search in the map & Delet
BPF\_PROG\_LOAD & Struct describing the type of eBPF program to load & Load an eBPF program in the kernel\\
\hline
\end{tabular}
\caption{Table showing types of syscall actions. Only those relevant to our research are shown the full list and attribute details can be consulted in the man page \cite{bpf_syscall}}
\caption{Types of syscall actions. Only those relevant to our research are shown the full list and attribute details can be consulted in the man page \cite{bpf_syscall}}
\label{table:ebpf_syscall}
\end{table}
With respect to the program type indicated with BPF\_PROG\_LOAD, this parameter indicates the type of eBPF program, setting the context in the kernel in which it will run, and to which modules it will have access to. The types of programs relevant for our research are described in table \ref{table:ebpf_prog_types}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{5cm}|}
\hline
PROGRAM TYPE & DESCRIPTION\\
@@ -336,20 +334,20 @@ BPF\_PROG\_TYPE\_XDP & Program to filter, redirect and monitor network events fr
BPF\_PROG\_TYPE\_SCHED\_CLS & Program to filter, redirect and monitor events using the Traffic Control classifier\\
\hline
\end{tabular}
\caption{Table showing types of eBPF programs. Only those relevant to our research are shown. The full list and attribute details can be consulted in the man page \cite{bpf_syscall}.}
\caption{Types of eBPF programs. Only those relevant to our research are shown. The full list and attribute details can be consulted in the man page \cite{bpf_syscall}.}
\label{table:ebpf_prog_types}
\end{table}
In section \ref{section:TODO}, we will proceed to analyse in detail the different program types and what capabilities` they offer.
\subsection{eBPF helpers} \label{subsection:ebpf_helpers}
Our last component to cover of the eBPF architecture are the eBPF helpers. Since eBPF programs have limited accessibility to kernel functions (which kernel modules commonly have free access to), the eBPF system offers a set of limited functions called helpers\cite{ebpf_helpers}, which are used by eBPF programs to perform certain actions and interact with the context on which they are run. The list of helpers a program can call varies between eBPF program types, since different programs run in different contexts.
Our last component to cover of the eBPF architecture are the eBPF helpers. Since eBPF programs have limited accessibility to kernel functions (which kernel modules commonly have free access to), the eBPF system offers a set of limited functions called helpers \cite{ebpf_helpers}, which are used by eBPF programs to perform certain actions and interact with the context on which they are run. The list of helpers a program can call varies between eBPF program types, since different programs run in different contexts.
It is important to highlight that, just like commands issued via the bpf() syscall can only be issued from the user space, eBPF helpers correspond to the kernel-side of eBPF program exclusively. Note that we will also find a symmetric correspondence to those functions of the bpf() syscall related to map operations (since these are accessible both from user and kernel space).
Table \ref{table:ebpf_helpers} lists the most relevant general-purpose eBPF helpers we will use during the development of our project. We will later detail those helpers exclusive to an specific eBPF program type in the sections on which they are studied.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
eBPF helper & DESCRIPTION\\
@@ -380,7 +378,7 @@ bpf\_ringbuf\_submit() & Submit data to an specific eBPF ring buffer, and notify
bpf\_tail\_call() & Jump to another eBPF program preserving the current stack\\
\hline
\end{tabular}
\caption{Table showing common eBPF helpers. Only those relevant to our research are shown. Those helpers exclusive to an specific program type are not listed. The full list and attribute details can be consulted in the man page \cite{ebpf_helpers}.}
\caption{Common eBPF helpers. Only those relevant to our research are shown. Those helpers exclusive to an specific program type are not listed. The full list and attribute details can be consulted in the man page \cite{ebpf_helpers}.}
\label{table:ebpf_helpers}
\end{table}
@@ -392,24 +390,24 @@ In the previous subsection \ref{subsection:bpf_syscall} we introduced the new ty
\subsection{XDP} \label{subsection:xdp}
eXpress Data Path (XDP) programs are a novel type of eBPF program that allows for the lowest-latency traffic filtering and monitoring in the whole Linux kernel. In order to load an XDP program, a bpf() syscall with the command BPF\_PROG\_LOAD and the program type BPF\_PROG\_TYPE\_XDP must be issued.
These programs are directly attached to the Network Interface Controller (NIC) driver, and thus they can process the packet before any other module\cite{xdp_gentle_intro}.
These programs are directly attached to the Network Interface Controller (NIC) driver, and thus they can process the packet before any other module \cite{xdp_gentle_intro}.
\begin{figure}[H]
\centering
\includegraphics[width=15cm]{xdp_diag.jpg}
% Either this caption, or change the text afterwards. I still need to know whether to put the long explanation here or on the paragraph, it gets repetitive.
\caption{Figure showing how the eBPF XDP and TC modules are integrated in the network processing in the Linux kernel.}
\label{fig:xdp_diag}
\end{figure}
Figure \ref{fig:xdp_diag} shows how XDP is integrated in the network processing of the Linux kernel. After receiving a raw packet (in the figure, \textit{xdp\_md}, which consists on the raw bytes plus some very basic metadata about the packet) from the incoming traffic, XDP program can perform the following actions\cite{xdp_manual}:
Figure \ref{fig:xdp_diag} shows how XDP is integrated in the network processing of the Linux kernel. After receiving a raw packet (in the figure, \textit{xdp\_md}, which consists on the raw bytes plus some very basic metadata about the packet) from the incoming traffic, XDP program can perform the following actions \cite{xdp_manual}:
\begin{itemize}
\item Analyse the data between the packet buffer bounds.
\item Modify the packet contents, and modify the packet length.
\item Decide between one of the actions displayed in table \ref{table:xdp_actions_av}.
\end{itemize}
\begin{table}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15cm]{xdp_diag.jpg}
% Either this caption, or change the text afterwards. I still need to know whether to put the long explanation here or on the paragraph, it gets repetitive.
\caption{XDP and TC modules integration in the network processing module of the Linux kernel.}
\label{fig:xdp_diag}
\end{figure}
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
ACTION & DESCRIPTION\\
@@ -422,12 +420,12 @@ XDP\_TX & Return the packet at the same NIC it was received from. Packet modific
XDP\_DROP & Drops the packet completely, kernel networking will not be notified.\\
\hline
\end{tabular}
\caption{Table showing XDP relevant return values.}
\caption{Relevant XDP return values.}
\label{table:xdp_actions_av}
\end{table}
Some of the XDP-exclusive eBPF helpers we will be discussing in later sections are shown in table \ref{table:xdp_helpers}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
eBPF helper & DESCRIPTION\\
@@ -438,7 +436,7 @@ bpf\_xdp\_adjust\_head() & Enlarges or reduces the extension of a packet, by mov
bpf\_xdp\_adjust\_tail() & Enlarges or reduces the extension of a packet, by moving the address of its last byte.\\
\hline
\end{tabular}
\caption{Table showing relevant XDP-exclusive eBPF helpers.}
\caption{Relevant XDP-exclusive eBPF helpers.}
\label{table:xdp_helpers}
\end{table}
@@ -446,11 +444,11 @@ bpf\_xdp\_adjust\_tail() & Enlarges or reduces the extension of a packet, by mov
\subsection{Traffic Control} \label{subsection:tc}
Traffic Control (TC) programs are also indicated for networking instrumentation. Similarly to XDP, their module is positioned before entering the overall network processing of the kernel. However, as it can be observed in figure \ref{fig:xdp_diag}, they differ in some aspects:
\begin{itemize}
\item TC programs receive a network buffer with metadata (in the figure, \textit{sk\_buff}) about the packet in it. This renders TC programs less ideal than XDP for performing large packet modifications (like new headers), but at the same time the additional metadata fields make it easier to locate and modify specific packet fields\cite{tc_differences}.
\item TC programs receive a network buffer with metadata (in the figure, \textit{sk\_buff}) about the packet in it. This renders TC programs less ideal than XDP for performing large packet modifications (like new headers), but at the same time the additional metadata fields make it easier to locate and modify specific packet fields \cite{tc_differences}.
\item TC programs can be attached to the \textit{ingress} or \textit{egress} points, meaning that an eBPF program can operate not only over incoming traffic, but also over the outgoing packets.
\end{itemize}
With respect to how TC programs operate, the Traffic Control system in Linux is greatly complex and would require a complete section by itself. In fact, it was already a complete system before the appearance of eBPF. Full documentation can be found at \cite{tc_docs_complete}. For this document, we will explain the overall process needed to load a TC program\cite{tc_direct_action}:
With respect to how TC programs operate, the Traffic Control system in Linux is greatly complex and would require a complete section by itself. In fact, it was already a complete system before the appearance of eBPF. Full documentation can be found at \cite{tc_docs_complete}. For this document, we will explain the overall process needed to load a TC program \cite{tc_direct_action}:
\begin{enumerate}
\item The TC program defines a so-called queuing discipline (qdisc), a packet scheduler that issues packets in a First-In-First-Out (FIFO) order as soon as they are received. This qdisc will be attached to an specific network interface (e.g.: wlan0).
\item Our TC eBPF program is attached to the qdisc. It will work as a filter, being run for every of the packets dispatched by the qdisc.
@@ -458,7 +456,7 @@ With respect to how TC programs operate, the Traffic Control system in Linux is
Similarly to XDP, the TC eBPF programs can decide an action to be executed on a packet by specifying a return value. These actions are almost analogous to the ones in XDP, as it can be observed in table \ref{table:tc_actions}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
ACTION & DESCRIPTION\\
@@ -471,12 +469,12 @@ TC\_ACT\_RECLASSIFY & Return the packet to the back of the qdisc scheduling queu
TC\_ACT\_SHOT & Drops the packet completely, kernel networking will not be notified.\\
\hline
\end{tabular}
\caption{Table showing TC relevant return values. Full list can be consulted at \cite{tc_ret_list_complete}.}
\caption{Relevant TC return values. Full list can be consulted at \cite{tc_ret_list_complete}.}
\label{table:tc_actions}
\end{table}
Finally, as in XDP, there exists a list of useful BPF helpers that will be relevant for the creation of our rootkit. They are shown in table \ref{table:tc_helpers}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
eBPF helper & DESCRIPTION\\
@@ -496,14 +494,14 @@ bpf\_skb\_change\_tail() & Enlarges or reduces the extension of a packet, by mov
\hline
\hline
\end{tabular}
\caption{Table showing relevant TC-exclusive eBPF helpers.}
\caption{Relevant TC-exclusive eBPF helpers.}
\label{table:tc_helpers}
\end{table}
%TODO This section might benefit from some diagrams, maybe. It was a bit to extense already, so skipping it from now
\subsection{Tracepoints} \label{subsection:tracepoints}
Tracepoints are a technology in the Linux kernel that allows to hook functions in the kernel, connecting a 'probe': a function that is executed every time the hooked function is called\cite{tp_kernel}. These tracepoints are set statically during kernel development, meaning that for a function to be hooked, it needs to have been previously marked with a tracepoint statement indicating its traceability. At the same time, this limits the number of tracepoints available.
Tracepoints are a technology in the Linux kernel that allows to hook functions in the kernel, connecting a 'probe': a function that is executed every time the hooked function is called \cite{tp_kernel}. These tracepoints are set statically during kernel development, meaning that for a function to be hooked, it needs to have been previously marked with a tracepoint statement indicating its traceability. At the same time, this limits the number of tracepoints available.
The list of tracepoint events available depends on the kernel version and can be visited under the directory \textit{/sys/kernel/debug/tracing/events}.
@@ -514,9 +512,9 @@ Also, note that the probe functions that are called when hitting a tracepoint re
In eBPF, a program can issue a bpf() syscall with the command BPF\_PROG\_LOAD and the program type BPF\_PROG\_TYPE\_TRACEPOINT, specifying which is the function with the tracepoint to attach to and an arbitrary function probe to call when it is hit. This function probe is defined by the user in the eBPF program submitted to the kernel.
\subsection{Kprobes}
Kprobes are another tracing technology of the Linux kernel whose functionality has been become available to eBPF programs. Similarly to tracepoints, kprobes enable to hook functions in the kernel, with the only difference that it is dynamically attached to any arbitrary function, rather than to a set of predefined positions\cite{kprobe_manual}. It does not require that kernel developers specifically mark a function to be probed, but rather kprobes can be attached to any instruction, with a short list of blacklisted exceptions.
Kprobes are another tracing technology of the Linux kernel whose functionality has been become available to eBPF programs. Similarly to tracepoints, kprobes enable to hook functions in the kernel, with the only difference that it is dynamically attached to any arbitrary function, rather than to a set of predefined positions \cite{kprobe_manual}. It does not require that kernel developers specifically mark a function to be probed, but rather kprobes can be attached to any instruction, with a short list of blacklisted exceptions.
As it happened with tracepoints, the probe functions have access to the parameters of the original hooked function. Also, the kernel maintains a list of kernel symbols (addresses) which are relevant for tracing and that offer us insight into which functions we can probe. It can be visited under the file \textit{/proc/kallsyms}, which exports symbols of kernel functions and loaded kernel modules\cite{kallsyms_kernel}.
As it happened with tracepoints, the probe functions have access to the parameters of the original hooked function. Also, the kernel maintains a list of kernel symbols (addresses) which are relevant for tracing and that offer us insight into which functions we can probe. It can be visited under the file \textit{/proc/kallsyms}, which exports symbols of kernel functions and loaded kernel modules \cite{kallsyms_kernel}.
Also similarly, since tracepoints could be found in their \textit{enter} and \textit{exit} variations, kprobes have their counterpart, name kretprobes, which call the hooked probe once a return instruction is reached after the hooked symbol. This means that a kretprobe hooked to a kernel function will call the probe function once it exits.
@@ -543,7 +541,7 @@ In section \ref{section:modern_ebpf}, we discussed the overall architecture of t
Nowadays, there exist multiple popular alternatives for writing and running eBPF programs. We will overview which they are and proceed to analyse in further detail the option that we will use for the development of our rootkit.
\subsection{BCC}
BPF Compiler Collection (BCC) is one of the first and well-known toolkits for eBPF programming available\cite{bcc_github}. It allows to include eBPF code into user programs. These programs are developed in python, and the eBPF code is embedded as a plain string. An example of a BCC program is included in %TODO ANNEX???
BPF Compiler Collection (BCC) is one of the first and well-known toolkits for eBPF programming available \cite{bcc_github}. It allows to include eBPF code into user programs. These programs are developed in python, and the eBPF code is embedded as a plain string. An example of a BCC program is included in %TODO ANNEX???
Although BCC offers a wide range of tools to easy the development of eBPF programs, we found it not to be the most appropriate for our large-scale eBPF project. This was in particular due to the feature of eBPF programs being stored as a python string, which leads to difficult scalability, poor development experience given that programming errors are detected at runtime (once the python program issues the compilation of the string), and simply better features from competing libraries.
@@ -560,17 +558,17 @@ bpftool is not a development framework like BCC, but one of the most relevant to
Although we will not be covering bpftool during our overview on the constructed eBPF rootkit, it was used extensively during the development and became a key tool for debugging eBPF programs, particularly to peek data at eBPF maps during runtime.
\subsection{Libbpf}
libbpf\cite{libbpf_github} is a library for loading and interacting with eBPF programs, which is currently maintained in the Linux kernel source tree\cite{libbpf_upstream}. It is one of the most popular frameworks to develop eBPF applications, both because it makes eBPF programming similar to common kernel development and because it aims at reducing kernel-version dependencies, thus increasing programs portability between systems\cite{libbpf_core}. During our research, however, we will not make use of this functionalities given that a portable program is not in our research goals.
libbpf \cite{libbpf_github} is a library for loading and interacting with eBPF programs, which is currently maintained in the Linux kernel source tree \cite{libbpf_upstream}. It is one of the most popular frameworks to develop eBPF applications, both because it makes eBPF programming similar to common kernel development and because it aims at reducing kernel-version dependencies, thus increasing programs portability between systems \cite{libbpf_core}. During our research, however, we will not make use of this functionalities given that a portable program is not in our research goals.
As we discussed in section \ref{section:modern_ebpf}, eBPF programs are composed of both the eBPF code in the kernel and a user space program that can interact with it. With libbpf, the eBPF kernel program is developed in C (a real program, not a string later compiled as with BCC), while user programs are usually developed in C, Rust or GO. For our project, we will use the C version of libbpf, so both the user and kernel side of our rootkit will be developed in this language.
% Cites in the following paragraph?
When using libbpf with the C language, both the user-side and kernel eBPF program are compiled together using the Clang/LLVM compiler, translating C instructions into eBPF bytecode. As a clarification, Clang is the front-end of the compiler, translating C instructions into an intermediate form understandable by LLVM, whilst LLVM is the back-end compiling the intermediate code into eBPF bytecode. As it can be observed in figure \ref{fig:libbpf}, the result of the compilation is a single program, comprising the user-side which will launch a user process, the eBPF bytecode to be run in the kernel, and other structures libbpf generates about eBPF maps and other meta data. This program is encapsulated as an ELF file (a common executable format).
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=12cm, keepaspectratio=true]{libbpf_prog.jpg}
\caption{Sketch of the compilation and loading process of a program developed with libbpf.}
\caption{Compilation and loading process of a program developed with libbpf.}
\label{fig:libbpf}
\end{figure}
@@ -578,7 +576,7 @@ Finally, we will overview one of the main functionalities of libbpf to simplify
Table \ref{table:libbpf_skel} describes the API offered by the BPF skeleton. Note that <name> is subtituted by the name of the program being compiled.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|>{\centering\arraybackslash}p{10cm}|}
\hline
Function name & Description\\
@@ -593,7 +591,7 @@ Function name & Description\\
<name>\_\_destroy() & Detach and unload the eBPF programs from the kernel.\\
\hline
\end{tabular}
\caption{Table showing BPF skeleton functions.}
\caption{BPF skeleton functions.}
\label{table:libbpf_skel}
\end{table}
@@ -604,7 +602,7 @@ Note that the BPF skeleton also offers further granularity at the time of dealin
\section{Security features in eBPF}
As we have shown in section \ref{section:modern_ebpf}, eBPF has been an active part of the Linux kernel from its 3.18 version. However, as with many other components of the kernel, its availability to the user depends on the parameters with which the kernel has been compiled. Specifically, eBPF is only available to kernels compiled with the flags specified in table \ref{table:ebpf_kernel_flags}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|c|c|>{\centering\arraybackslash}p{8cm}|}
\hline
Flag & Value & Description\\
@@ -637,20 +635,20 @@ CONFIG\_XDP\_SOCKETS & y & Enable XDP\\
\label{table:ebpf_kernel_flags}
\end{table}
The above table is based on BCC's documentation\ref{table:ebpf_kernel_flags}, but the full list of eBPF-related flags can be extracted in a live system via bpftool, as detailed in Annex \ref{annex:bpftool_flags_kernel}. Nowadays, all mainstream Linux distributions include kernels with full support for eBPF.
Table \ref{table:ebpf_kernel_flags} is based on BCC's documentation, but the full list of eBPF-related flags can be extracted in a live system via bpftool, as detailed in Annex \ref{annex:bpftool_flags_kernel}. Nowadays, all mainstream Linux distributions include kernels with full support for eBPF.
\subsection{Access control}
It must be noted that, similarly to kernel modules, loading an eBPF program requires privileged access in the system. In old kernel versions, this means either an user having full root permissions, or having the Linux capability\cite{ubuntu_caps} CAP\_SYS\_ADMIN. Therefore, there existed two main options:
\subsection{Access control} \label{subsection:access_control}
It must be noted that, similarly to kernel modules, loading an eBPF program requires privileged access in the system. In old kernel versions, this means either an user having full root permissions, or having the Linux capability \cite{ubuntu_caps} CAP\_SYS\_ADMIN. Therefore, there existed two main options:
%TODO some words about capabilities
\begin{itemize}
\item \textbf{Privileged users} can load any kind of eBPF program and use any functionality.
\item \textbf{Unprivileged users} can only load and attach eBPF programs of type BPF\_PROG\_TYPE\_SOCKET\_FILTER\cite{evil_ebpf_p9}, offering the very limited functionality of filtering packets received on a socket.
\item \textbf{Unprivileged users} can only load and attach eBPF programs of type BPF\_PROG\_TYPE\_SOCKET\_FILTER \cite{evil_ebpf_p9}, offering the very limited functionality of filtering packets received on a socket.
\end{itemize}
More recently, in an effort to further granulate the permissions needed for loading, attaching and running eBPF programs, CAP\_SYS\_ADMIN has been substituted by more specific capabilities\cite{ebpf_caps_intro}\cite{ebpf_caps_lwn}. The current system is therefore described in table \ref{table:ebpf_caps_current}.
More recently, in an effort to further granulate the permissions needed for loading, attaching and running eBPF programs, CAP\_SYS\_ADMIN has been substituted by more specific capabilities \cite{ebpf_caps_intro} \cite{ebpf_caps_lwn}. The current system is therefore described in table \ref{table:ebpf_caps_current}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{4cm}|>{\centering\arraybackslash}p{10cm}|}
\hline
Capabilities & eBPF functionality\\
@@ -673,11 +671,11 @@ CAP\_SYS\_ADMIN & Privileged eBPF. Includes iterating over eBPF maps, and CAP\_B
Therefore, eBPF network programs usually require both CAP\_BPF and CAP\_NET\_ADMIN, whilst tracing programs require CAP\_BPF and CAP\_PERFMON. CAP\_SYS\_ADMIN still remains as the (non-preferred) capability to assign to eBPF programs with complete access in the system.
Although for a long time there have existed efforts towards enhancing unprivileged eBPF, it remains a worrying feature\cite{unprivileged_ebpf}. The main issue is that the verifier must be prepared to detect any attempt to extract kernel memory access or user memory modification by unprivileged eBPF programs, which is a complex task. In fact, there have existed numerous security vulnerabilities which allow for privilege escalation using eBPF, that is, execution of privileged eBPF programs by exploiting vulnerabilities in unprivileged eBPF\cite{cve_unpriv_ebpf}.
Although for a long time there have existed efforts towards enhancing unprivileged eBPF, it remains a worrying feature \cite{unprivileged_ebpf}. The main issue is that the verifier must be prepared to detect any attempt to extract kernel memory access or user memory modification by unprivileged eBPF programs, which is a complex task. In fact, there have existed numerous security vulnerabilities which allow for privilege escalation using eBPF, that is, execution of privileged eBPF programs by exploiting vulnerabilities in unprivileged eBPF \cite{cve_unpriv_ebpf}.
This influx of security vulnerabilities leads to the recent inclusion of an attribute into the kernel which allows for setting whether unprivileged eBPF is allowed in the system or not. This parameter is named \textit{kernel.unprivileged\_bpf\_disabled}, its values can be seen in table \ref{table:unpriv_ebpf_values}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{4cm}|>{\centering\arraybackslash}p{10cm}|}
\hline
Value & Meaning\\
@@ -694,7 +692,7 @@ Value & Meaning\\
\label{table:unpriv_ebpf_values}
\end{table}
Nowadays, most Linux distributions have set value 1 to this parameter, therefore disallowing unprivileged eBPF completely. These include Ubuntu\cite{unpriv_ebpf_ubuntu}, Suse Linux\cite{unpriv_ebpf_suse} or Red Hat Linux\cite{unpriv_ebpf_redhat}, between others.
Nowadays, most Linux distributions have set value 1 to this parameter, therefore disallowing unprivileged eBPF completely. These include Ubuntu \cite{unpriv_ebpf_ubuntu}, Suse Linux \cite{unpriv_ebpf_suse} or Red Hat Linux \cite{unpriv_ebpf_redhat}, between others.
@@ -704,19 +702,19 @@ Multiple of the techniques incorporated in our rootkit require a deep understand
\subsection{Memory pages and faults}
Linux systems divide the available random access memory (RAM) into 'pages', subsections of an specific length, usually 4 KB. The collection of all pages is called physical memory.
Likewise, individual memory sections need to be assigned to each running process in the system, but instead of assigning a set of pages from physical memory, a new address space is defined, named virtual memory, which is divided into pages as well. These virtual memory pages are related to physical memory pages via a page table, so that each virtual memory address of a process can be translated into a real, physical memory address in RAM\cite{mem_page_arch}. Figure \ref{fig:mem_arch_pages} shows a diagram of the described architecture.
Likewise, individual memory sections need to be assigned to each running process in the system, but instead of assigning a set of pages from physical memory, a new address space is defined, named virtual memory, which is divided into pages as well. These virtual memory pages are related to physical memory pages via a page table, so that each virtual memory address of a process can be translated into a real, physical memory address in RAM \cite{mem_page_arch}. Figure \ref{fig:mem_arch_pages} shows a diagram of the described architecture.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=13cm]{mem_arch_pages.jpg}
\caption{Memory translation of virtual pages to physical pages.}
\label{fig:mem_arch_pages}
\end{figure}
As we can observe in the figure, each virtual page is related to one physical page. However, RAM needs to maintain multiple processes and data simultaneously, and therefore sometimes the operating system (OS) will remove them from physical memory when it believes they are no longer being used. This leads to the occurrence of two type of memory events\cite{page_faults}:
As we can observe in the figure, each virtual page is related to one physical page. However, RAM needs to maintain multiple processes and data simultaneously, and therefore sometimes the operating system (OS) will remove them from physical memory when it believes they are no longer being used. This leads to the occurrence of two type of memory events \cite{page_faults}:
\begin{itemize}
\item \textbf{Major page faults} occur when a process tries to access a virtual page, but the related physical page has been removed from RAM. In this case, the OS will need to request a secondary storage (such as a hard disk) for the data removed, and allocate a new physical page for the virtual page. Figure \ref{fig:mem_major_page_fault} illustrates a major page fault.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=11cm]{mem_major_page_fault.jpg}
\caption{Major page fault after a page was removed from RAM.}
@@ -724,7 +722,7 @@ As we can observe in the figure, each virtual page is related to one physical pa
\end{figure}
\item \textbf{Minor page faults} occur when a process tries to access a virtual page, and although the related physical page exists, the connection in the page table has not been completed. A common event when these fault happen is on fork() calls, since with the purpose of making the call more efficient, the page table of the parent is not always completely copied into the child, leading into multiple minor page faults once the child tries to access the data on them. Figure \ref{fig:mem_minor_page_fault} illustrates a minor page fault after a fork.
\end{itemize}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=11cm]{mem_minor_page_fault.jpg}
\caption{Minor page fault after a fork() in which the page table was not copied completely.}
@@ -734,14 +732,13 @@ As we can observe in the figure, each virtual page is related to one physical pa
\subsection{Process virtual memory}
In the previous subsection we have studied that each process disposes of a virtual address space. We will now describe how this virtual memory is organized in a Linux system.
\begin{figure}[H]
Figure \ref{fig:mem_proc_arch} describes how virtual memory is distributed within a process in the x86\_64 architecture. As we can observe, it is partitioned into multiple sections:
\begin{figure}[htbp]
\centering
\includegraphics[width=6cm]{memory.jpg}
\caption{Virtual memory architecture of a process\cite{mem_arch_proc}.}
\caption{Virtual memory architecture of a process \cite{mem_arch_proc}.}
\label{fig:mem_proc_arch}
\end{figure}
Figure \ref{fig:mem_proc_arch} describes how virtual memory is distributed within a process in the x86\_64 architecture. As we can observe, it is partitioned into multiple sections:
\begin{itemize}
\item Lower and upper memory addresses are reserved for the kernel.
\item A section where shared libraries code is stored.
@@ -758,14 +755,14 @@ Between all the sections we identified in a process virtual memory, the stack wi
Firstly, we will present how the stack is structured, and which operations can be executed on it. Figure \ref{fig:stack_pres} presents a stack during the execution of a program. Table \ref{table:systemv_abi_other} explains the purpose of the most relevant registers related to the stack and program execution:
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=14cm]{stack_pres.jpg}
\caption{Simplified stack representation showing only stack frames.}
\label{fig:stack_pres}
\end{figure}
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{2cm}|>{\centering\arraybackslash}p{10cm}|}
\hline
Register & Purpose\\
@@ -786,13 +783,13 @@ As it can be observed in figure \ref{fig:stack_pres}, the stack grows towards lo
As table \ref{table:systemv_abi_other} explains, the rbp and rsp registers are used for keeping track of the starting and final position of the current stack frame respectively. We can see in figure \ref{fig:stack_pres} that their value is a memory address pointing to their stack positions. On the other hand, the rip register does not point to the stack, but rather to the .text section (see figure \ref{fig:mem_proc_arch}), where it points to the next instruction to be executed. However, as we will now see, its value must also be stored in the stack frame when a nested function is called, since after the nested function exits we need to restore the execution in the same instruction of the original function.
As with any LIFO structure, the stack supports two main operations: \textit{push} and \textit{pop}. In the x86\_64 architecture, it operates with chunks of data of either 16, 32 or 64 bytes.
As with any LIFO structure, the stack supports two main operations: \textit{push} and \textit{pop}. In the x86\_64 architecture, it operates with chunks of data of either 16, 32 or 64 bytes. Table \ref{fig:stack_ops} shows a representation of these operations in the stack.
\begin{itemize}
\item A \textbf{push} operation writes data in the free memory pointed by register rsp. It then moves the value of rsp to point to the new end of the stack.
\item A \textbf{pop} operation moves the value of rsp by 16, 32 or 64 bytes, and reads the data previously saved in that position.
\end{itemize}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=10cm]{stack_ops.jpg}
\caption{Representation of push and pop operations in the stack.}
@@ -800,17 +797,16 @@ As with any LIFO structure, the stack supports two main operations: \textit{push
\end{figure}
As we mentioned, the stack stores function parameters, return addresses and local variables inside a stack frame. We will now study how the processor uses the stack in order to call, execute, and exit a function. To illustrate this process, we will simulate the execution of function \lstinline{func(char* a, char* b, char* c)} \lstinline{}. Figures \ref{fig:stack_before} and \ref{fig:stack} show a representation of the stack during these operations.
As we mentioned, the stack stores function parameters, return addresses and local variables inside a stack frame. We will now study how the processor uses the stack in order to call, execute, and exit a function. To illustrate this process, we will simulate the execution of function \lstinline{func(char* a, char* b, char* c)} \lstinline{}:
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=14cm]{stack_before.jpg}
\caption{Stack representation right before starting the function call process.}
\label{fig:stack_before}
\end{figure}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=14cm]{stack.jpg}
\caption{Stack representation right after the function preamble.}
@@ -818,14 +814,14 @@ As we mentioned, the stack stores function parameters, return addresses and loca
\end{figure}
\begin{enumerate}
\item The function arguments are pushed into the stack. We can see them in the stack in reverse order.
\item The function arguments are pushed into the stack. We can see them in the stack of figure \ref{fig:stack} in reverse order.
\item The function is called:
\begin{enumerate}
\item The value of register rip is pushed into the stack, so that it is saved for when the function exists. We can see it on the figure as 'ret'.
\item The value of register rip is pushed into the stack, so that it is saved for when the function exists. We can see it on figure \ref{fig:stack} as 'ret'.
\item The value of rip changes to point to the first instruction of the called function.
\item We execute what is called as the \textit{function preamble}\cite{8664_params_abi_p18}, which prepares the stack frame for the called function:
\item We execute what is called as the \textit{function preamble} \cite{8664_params_abi_p18}, which prepares the stack frame for the called function:
\begin{enumerate}
\item The value of rbp is pushed into the stack, so that we can restore the previous stack frame when the function exits. We can see it on the figure as the 'saved frame pointer'.
\item The value of rbp is pushed into the stack, so that we can restore the previous stack frame when the function exits. We can see it on figure \ref{fig:stack} as the 'saved frame pointer'.
\item The value of rsp is moved into rbp. Therefore, now rbp points to the end of the previous stack frame.
\item The value of rsp is usually decremented (since the stack needs to go to lower memory addresses) so that we allocate some space for function variables.
\end{enumerate}
@@ -849,8 +845,8 @@ In section \ref{subsection:stack}, we studied how the stack works and which is t
\item The original value of the rbp register (sfp), to restore the frame pointer of the original stack frame.
\end{itemize}
Although this process is simple enough, it opens the possibility for an attacker to easily hijack the flow of execution if it can modify the value of ret, as it is shown in figure \ref{fig:stack_ret_hij_simple}:
\begin{figure}[H]
Although this process is simple enough, it opens the possibility for an attacker to easily hijack the flow of execution if it can modify the value of ret, as it is shown in figure \ref{fig:stack_ret_hij_simple}.
\begin{figure}[htbp]
\centering
\includegraphics[width=15cm]{stack_ret_hij_simple.jpg}
\caption{Execution hijack overwriting saved rip value.}
@@ -878,7 +874,7 @@ int main(int argc, char *argv[]){
During the execution of the above program, since the char array \textit{buffer} is a buffer of length 12 stored in the stack, then if the value of \textit{bar} is larger than 12 bytes it will overflow the allocated space in the stack. This is usually the case of using unsafe functions for processing user input such as strcpy(), which does not check whether the array fits in the buffer. Figure \ref{fig:buffer_overflow} shows how the overflow happens in the stack.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15cm]{buffer_overflow.jpg}
\caption{Stack buffer overflow overwriting ret value.}
@@ -889,7 +885,7 @@ As we can observe in the figure, the new data written into the buffer has also o
Usually, an attacker exploiting a program vulnerable to stack buffer overflow is interested in running arbitrary (malicious) code. For this, the attacker follows the process shown in figure \ref{fig:buffer_overflow_shellcode}:
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15cm]{buffer_overflow_shellcode.jpg}
\caption{Executing arbitrary code exploiting a buffer overflow vulnerability.}
@@ -917,7 +913,7 @@ Return Oriented Programming (ROP) is an exploitation technique that takes advant
ROP tackles this challenge by designing a method of reconstructing malicious code from parts of already-existing code, as in a 'collage'. Assembly instructions are selected from multiple places, so that, when put together and executed sequentially, they recreate the shellcode which the attacker wants to execute. These pieces of code are called ROP gadgets, and consist of a set of arbitrary instructions followed by a final \textit{ret} instruction, which triggers the function exit and pops the value of ret. These gadgets may belong to any code in the process memory, usually selected between the code of the shared libraries (see figure \ref{fig:stack}) to which the process is linked.
Finding ROP gadgets and writing ROP-compatible payloads manually is hard, thus multiple programs exist that automatically scan the system libraries and construct provide the gadgets given the shellcode to execute\cite{rop_prog_finder}.
Finding ROP gadgets and writing ROP-compatible payloads manually is hard, thus multiple programs exist that automatically scan the system libraries and construct provide the gadgets given the shellcode to execute \cite{rop_prog_finder}.
However, we will now illustrate how ROP works with an example. Suppose that an attacker has discovered a buffer overflow vulnerability, but the stack is marked as not executable. The attacker wants to execute the assembly code shown in snippet \ref{code:rop_ex}:
@@ -928,7 +924,7 @@ mov rax, [rsp]
After finding the address of the ROP gadgets manually or using an automated tool, the attacker takes advantage of a buffer overflow (or, in our case, a direct write using eBPF's bpf\_probe\_write\_user()) to overwrite the vale of ret with the address of the first ROP gadget, and also additional data in the stack. Figure \ref{fig:rop_compund} shows how we can execute the original program using ROP:
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=16cm]{ROPcompound.jpg}
\caption{Steps for executing code sample using ROP.}
@@ -953,14 +949,14 @@ This section presents an overview on the most relevant aspects of the network sy
\subsection{An overview on the network layer}
Firstly, we will describe the data structure we will be dealing with in networking programs. This will be Ethernet frames containing TCP/IP packets. Figure \ref{fig:frame} shows the frame in its completeness:
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=14cm]{frame.jpg}
\caption{Ethernet frame with TCP/IP packet.}
\label{fig:frame}
\end{figure}
As we can observe, we can distinguish five different network layers in the frame. This division is made according to the OSI model\cite{network_layers}:
As we can observe, we can distinguish five different network layers in the frame. This division is made according to the OSI model \cite{network_layers}:
\begin{itemize}
\item Layer 1 corresponds to the physical layer, and it is processed by the NIC hardware, even before it reaches the XDP module (see figure \ref{fig:xdp_diag}). Therefore, this layer is discarded and completely invisible to the kernel. Note that it does not only include a header, but also a trailer (a Frame Check Sequence, a redundancy check included to check frame integrity).
\item Layer 2 is the data layer, it is in charge of transporting the frame via physical media, in our case an Ethernet connection. Most relevant fields are the MAC destination and source, used for performing physical addressing.
@@ -973,7 +969,7 @@ As we can observe, we can distinguish five different network layers in the frame
\subsection{Introduction to the TCP protocol} \label{subsection:tcp}
We will now focus our view on the transport layer, specifically on the TCP protocol, since it will be a major concern at the time of designing the network capabilities of our rootkit.
Firstly, since TCP aims to offer a reliable and ordered packet transmission\cite{tcp_reliable}, it includes sequence numbers (see table \ref{fig:frame}) which mark the order in which they are transmitted. However, since the physical medium may corrupt or lose packets during the transmission, TCP must incorporate mechanisms for ensuring the order and delivery of all packets:
Firstly, since TCP aims to offer a reliable and ordered packet transmission \cite{tcp_reliable}, it includes sequence numbers (see table \ref{fig:frame}) which mark the order in which they are transmitted. However, since the physical medium may corrupt or lose packets during the transmission, TCP must incorporate mechanisms for ensuring the order and delivery of all packets:
\begin{itemize}
\item Mechanism for opening and establishing a reliable connection between two parties.
\item Mechanism for ensuring that packets are retransmitted in case of an error during the connection.
@@ -981,7 +977,7 @@ Firstly, since TCP aims to offer a reliable and ordered packet transmission\cite
With respect to the establishment of a reliable connection, this is achieved via a 3-way handshake, in which certain TCP flags will be set in a series of interchanged packets (see in figure \ref{fig:frame} the field TCP flags). Most relevant TCP flags are described in table \ref{table:tcp_flags}.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{4cm}|>{\centering\arraybackslash}p{10cm}|}
\hline
Flag & Purpose\\
@@ -1000,8 +996,8 @@ RST & Abruptly terminates the connection, usually sent when a host receives an u
\label{table:tcp_flags}
\end{table}
Taking the above into account, figure \ref{fig:tcp_conn} shows a depiction of the 3-way handshake\cite{tcp_handshake}:
\begin{figure}[H]
Taking the above into account, figure \ref{fig:tcp_conn} shows a depiction of the 3-way handshake \cite{tcp_handshake}:
\begin{figure}[htbp]
\centering
\includegraphics[width=12cm]{tcp_conn.jpg}
\caption{TCP 3-way handshake.}
@@ -1017,7 +1013,7 @@ With respect to maintaining the integrity of the connection once it starts, TCP
\item The sender receives the ACK packet and stops the timer. If, for any reason, the ACK packet is not received before the timer ends, then the same packet is retransmitted.
\end{enumerate}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=12cm]{tcp_retransmission.jpg}
\caption{TCP packet retransmission on timeout.}
@@ -1025,14 +1021,14 @@ With respect to maintaining the integrity of the connection once it starts, TCP
\end{figure}
\section{ELF binaries} \label{section:elf}
This section details the Executable and Linkable Format (ELF)\cite{elf}, the format in which we find executable files (between other file types) in Linux systems. We will perform an analysis from a security standpoint, that is, mainly oriented to describe the most relevant sections and the permissions incorporated into them. We will also focus on several of these sections which will be relevant for designing our attack.
This section details the Executable and Linkable Format (ELF) \cite{elf}, the format in which we find executable files (between other file types) in Linux systems. We will perform an analysis from a security standpoint, that is, mainly oriented to describe the most relevant sections and the permissions incorporated into them. We will also focus on several of these sections which will be relevant for designing our attack.
After that, we will overview the security hardening techniques that have been historically incorporated into Linux to mitigate possible exploitation techniques when running ELF executables (such as the stack buffer overflow we explained in section \ref{subsection: buf_overflow}). During the design of our rootkit, we will attempt to bypass these techniques using multiple workarounds.
\subsection{The ELF format and Lazy Binding} \label{subsection:elf_lazy_binding}
Linux supports multiple tools that enable a deep inspection of ELF binaries and its sections. Table \ref{table:elf_tools} shows the main tools we will use during this analysis:
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{3cm}|>{\centering\arraybackslash}p{10cm}|}
\hline
Tool & Purposes\\
@@ -1055,7 +1051,7 @@ Firstly, we will analyse the main sections we can find in an ELF executable. We
The commands used and complete list of headers can be found in Annex \ref{annexsec:readelf_sec_headers}. The most relevant sections are described in table \ref{table:elf_sec_headers}:
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{1cm}|>{\centering\arraybackslash}p{9cm}|>{\centering\arraybackslash}p{2cm}|}
\hline
Tool & Purpose & Permissions\\
@@ -1084,11 +1080,11 @@ Tool & Purpose & Permissions\\
As it can be observed in table \ref{table:elf_sec_headers}, we can find that all sections have the Alloc flag, meaning they will be loaded into process memory during runtime (see table \ref{TODO}, they have not been shown in previous diagrams for simpleness).
Apart from those we have already discussed previously, we can find the GOT and PLT sections, whose purpose is to support Position Independent Code (PIC), that is, instructions whose address in virtual memory is not hardcoded by the compiler into the executable, but rather they are not known until resolved at runtime. This is usually the case of shared libraries, which can be loaded into virtual memory starting at any address\cite{plt_got_overlord}.
Apart from those we have already discussed previously, we can find the GOT and PLT sections, whose purpose is to support Position Independent Code (PIC), that is, instructions whose address in virtual memory is not hardcoded by the compiler into the executable, but rather they are not known until resolved at runtime. This is usually the case of shared libraries, which can be loaded into virtual memory starting at any address \cite{plt_got_overlord}.
Therefore, in order to call a function of a shared library, the dynamic linker follows a process called 'Lazy binding'\cite{plt_got_technovelty}:
Therefore, in order to call a function of a shared library, the dynamic linker follows a process called 'Lazy binding' \cite{plt_got_technovelty}:
\begin{enumerate}
\item From the .text section, instead of calling a direct absolute address as usual, a PLT stub (in the .plt section) is called. Snippet \ref{code:lazy_bind_1} shows a call to the function timerfd\_settime, implemented by the shared library glibc and thus using a PLT:
\item From the .text section, instead of calling a direct absolute address as usual, a PLT stub (in the .plt section) is called. Snippet \ref{code:lazy_bind_1} shows a call to the function timerfd\_settime, implemented by the shared library glibc and thus using a PLT.
\begin{lstlisting}[language=C, caption={Call to PLT stub seen from objdump.}, label={code:lazy_bind_1}]
$ objdump -d simple_timer
4014cb: b9 00 00 00 00 mov $0x0,%ecx
@@ -1099,14 +1095,14 @@ $ objdump -d simple_timer
\item In the PLT stub, the flow of execution jumps to an address which is stored in the GOT section, which is the absolute address of the function at glibc. This address must be written there by the dynamic linker but, according to lazy binding, the first time to call this function the linker has not calculated that address yet.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15.5cm]{sch_gdb_plt.png}
\caption{PLT stub for timerfd\_settime, seen from gdb-peda.}
\label{fig:lazy_bind_2}
\end{figure}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15.5cm]{sch_gdb_got_prev.png}
\caption{Inspecting address stored in GOT section before dynamic linking, seen from gdb-peda.}
@@ -1115,14 +1111,14 @@ $ objdump -d simple_timer
\item As we can see in figures \ref{fig:lazy_bind_2} and \ref{fig:lazy_bind_3}, the PLT stub calls address 0x4010a0, which leads to a dynamic linking routine, which proceeds to write the address into the GOT section and jump back to the start of the PLT stub. This time, the memory address at GOT to which the PLT jumps is already loaded with the address to the function at the shared library, as shown by figure \ref{fig:lazy_bind_4}.
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15.5cm]{sch_gdb_got_after.png}
\caption{Inspecting address stored in GOT section after dynamic linking, seen from gdb-peda.}
\label{fig:lazy_bind_4}
\end{figure}
\begin{figure}[H]
\begin{figure}[htbp]
\centering
\includegraphics[width=15.5cm]{sch_glibc_func.png}
\caption{Glibc function to which PLT jumps using address stored at GOT, seen from gdb-peda.}
@@ -1140,7 +1136,7 @@ During section \ref{section:attacks_stack}, we presented multiple of the classic
Table \ref{table:compilers} shows the compilers that we will be considering during this study. We will be exclusively looking at those security features that are included by default.
\begin{table}[H]
\begin{table}[htbp]
\begin{tabular}{|>{\centering\arraybackslash}p{5cm}|>{\centering\arraybackslash}p{8cm}|}
\hline
Compiler & Security features by default\\
@@ -1166,7 +1162,7 @@ Data Execution Prevention, also known as No Execute, is the option of marking th
The creation of advanced techniques like ROP is one reaction to this mitigation, that circumvents this protection.
\textbf{ASLR}\\
Address Space Layout Randomization is a technique that randomizes the position of memory sections in a process virtual memory, including the heap, stack and libraries, so that an attacker cannot rely on known addresses during exploitation (e.g: libraries are loaded at a different memory address each time the program is run, so ROP gadgets change their position)\cite{aslr_pie_intro}.
Address Space Layout Randomization is a technique that randomizes the position of memory sections in a process virtual memory, including the heap, stack and libraries, so that an attacker cannot rely on known addresses during exploitation (e.g: libraries are loaded at a different memory address each time the program is run, so ROP gadgets change their position) \cite{aslr_pie_intro}.
In the context of a stack buffer overflow attack, the memory position of the stack is random, and therefore even if shellcode is injected into the stack by an attacker, the address at which it resides cannot be written into the saved value of rip in order to hijack the flow of execution.
@@ -1179,7 +1175,7 @@ Relocation Read-Only is a hardening technique that mitigates the possibility of
Two settings for RELRO are the most widespread, either Partial RELRO (which only marks sections of the .got section not related to the PLT as read-only, leaving .got.plt writeable) or Full RELRO (which marks the .got section as read-only completely). Binaries with only Partial RELRO are still non-secure, as the address at which the PLT section jumps can still be overwriten (including from eBPF, as we will explain) \cite{relro_redhat}.
\textbf{Intel CET}\\
Intel Control-flow Enforcement Technology is a hardening feature fully incorporated in Windows 10 systems \cite{cet_windows} and a work in progress in Linux\cite{cet_linux}. Its purpose is to defeat ROP attacks and other derivates (e.g: Jump-oriented programming, JOP), by adding a strict kernel-supported control of the return addresses and strong restrictions over jump and call instructions.
Intel Control-flow Enforcement Technology is a hardening feature fully incorporated in Windows 10 systems \cite{cet_windows} and a work in progress in Linux \cite{cet_linux}. Its purpose is to defeat ROP attacks and other derivates (e.g: Jump-oriented programming, JOP), by adding a strict kernel-supported control of the return addresses and strong restrictions over jump and call instructions.
In Linux, the kernel will support a hidden 'shadow stack' that will save the return addresses for each call. This prevents modifying the saved value of rip in the stack, since the kernel would realise that the flow of execution has been modified. We can also find that modern compilers (such as GCC 10.3.0) already generate Intel CET-related instructions such as \textit{endbr64}, whose purpose is to be placed at the start of functions, marking that as the only address to which an indirect jump can land (otherwise, jumps will be rejected if not landing at \textit{endbr64}).