Calling Conventions

Function Calling Conventions

• Function Calling Convention: A promise to call and return a function.
• When one function calls another function, the program execution flow moves to the other function. Afterwards, when the called function returns, it returns to the original function and continues the existing execution flow.
  • Therefore, when calling a function, you must save the Caller’s Stack frame and Return Address to continue the execution flow after returning. And the caller must pass the arguments requested by the callee and receive the return value when the callee’s execution ends.
• Function calling conventions are usually applied when a developer writes code in a high-level language and the functions are compiled according to the calling convention by the compiler.

Types of Calling Conventions

x86
	- cdecl
	- stdcall
	- fastcall
	- thiscall
	
x86-64
	- Calling Convention of System V AMD64 ABI
	- MS ABI Calling Convention

x86 Calling Convention

Traditionally, parameters were stored in the stack, but with the expansion from x86 to x86-64, registers, which are faster than the stack, were mainly used.

Because the number of registers is limited, registers and stacks are usually used together in modern calling conventions.

Caller and Callee

The most important concepts when understanding function calling conventions are Caller and Callee.

• Caller: A function that calls a function
• Callee: Function called

For example, if you have the following code:

int add(int a, int b) {
    return a + b;
}

int main() {
    int result = add(1, 2);
}

The main() function is a caller because it calls the add() function, and the add() function is a callee because it was called.

When a function call occurs, Caller and Callee have different roles.

• Caller must pass arguments to Callee.
• Callee executes a function using the passed arguments.
• When the callee finishes executing the function, it passes the return value to the caller.
• The caller continues the existing execution flow after receiving the return value.

In this process, stack, register, return address, stack frame, etc. are used.

Stack Frame

Stack Frame is the stack area used by the function when it is executed.

When a function is called, a Stack Frame unique to that function is usually created, and when the function execution is completed, the Stack Frame is cleaned up.

The following information is generally stored in the Stack Frame.

• Local variables in functions
• Stack Frame information from previous function
• Return Address
• function arguments
• temporary data

Based on x86, Stack Frame is usually managed using ebp and esp registers.

• esp: Register pointing to the top of the current stack
• ebp: Register pointing to the Stack Frame reference point of the current function

A typical function prologue appears in the following form:

push ebp
mov ebp, esp
sub esp, 0x10

The above code stores the ebp value of the previous function on the stack, moves the current esp value to ebp, and secures space on the stack for local variables.

A function epilogue usually appears in the following form:

mov esp, ebp
pop ebp
ret

Alternatively, it may be abbreviated as the leave command as follows.

leave
ret

leave internally performs mov esp, ebp and pop ebp.

Return Address

Return Address is the address to which the callee function should return after execution is complete.

If the caller uses the call instruction when calling the callee, CPU stores the address of the next instruction to be executed in the stack. This value is Return Address.

For example, if you have the following code:

call add
mov [result], eax

When the call add instruction is executed, the address of mov [result], eax is stored on the stack.

Afterwards, when the add function ends and the ret command is executed, the Return Address stored in the stack is taken out and moved to that address again.

In other words, function calls and returns proceed roughly as follows.

1. Caller calls Callee with the call command.
2. CPU stores the Return Address on the stack.
3. The execution flow moves to the callee function.
4. The Callee function is executed.
5. The Callee function executes the ret command.
6. Move back to the Return Address saved in the stack.
7. The caller resumes its previous execution flow.

Function Argument Passing

One of the important things in calling convention is how to pass function arguments.

Function arguments can be passed in two main ways.

• Passing arguments on the stack
• Passing arguments in registers

In x86, function arguments are usually passed by pushing them onto the stack.

push 2
push 1
call add

The above code is an example of a function call such as add(1, 2) at the assembly level.

When pushing an argument onto the stack, the right argument is usually pushed first. So, push 2 first, and then push 1.

The reason for doing this is to make it easy to access the first argument inside the function as a fixed location such as [ebp+8] and the second argument as [ebp+12].

For example, based on x86 cdecl, arguments can be referenced inside a function as follows.

mov eax, [ebp+8]
mov edx, [ebp+12]
add eax, edx

Here, [ebp+8] means the first argument, and [ebp+12] means the second argument.

In x86-64, argument passing using registers is mainly used.

Based on System V AMD64 ABI, integer arguments are transmitted through the registers in the following order.

• rdi
• rsi
• rdx
• rcx
• r8
• r9

For example, if there is a function call like this:

add(1, 2);

Based on System V AMD64 ABI, the arguments are transmitted approximately as follows.

mov rdi, 1
mov rsi, 2
call add

The first argument 1 is passed as rdi, and the second argument 2 is passed as rsi.

Returning Values

When the function finishes executing, the return value must be passed to the caller.

The return value is also delivered through a location determined according to the calling convention.

In x86, the return value is generally stored in the eax register.

mov eax, 3
ret

The above code has a similar meaning as the function returns 3.

In x86-64, the return value is generally stored in the rax register.

mov rax, 3
ret

In other words, after calling the function, the Caller can check the eax or rax value to obtain the return value of Callee.

Stack Cleanup

One important difference in the x86 Calling Convention is who is responsible for stack cleanup.

If you push arguments onto the stack to call a function, you must clear the arguments from the stack after the function call is finished.

At this time, the calling convention varies depending on whether the stack is cleaned up by the caller or the callee.

Typically, in cdecl, the caller cleans up the stack.

push 2
push 1
call add
add esp, 8

In the code above, add esp, 8 is a code that organizes the two arguments that the caller put on the stack.

Since one argument is 4 bytes and there are 2 arguments, a total of 8 bytes are organized.

On the other hand, for stdcall, Callee cleans up the stack.

ret 8

ret 8 is a command that moves to the Return Address and additionally clears 8 bytes from the stack.

In other words, the big difference between cdecl and stdcall is who cleans up the stack.

• cdecl: Caller cleans up the stack
• stdcall: Callee cleans up the stack

cdecl

cdecl is a calling convention frequently used in x86.

The characteristics of cdecl are as follows.

• Arguments are pushed onto the stack in right-to-left order.
• The return value is stored in eax.
• The caller is responsible for cleaning up the stack.
• Variable argument functions can be used.

For example, if there is a function call like this:

add(1, 2);

In cdecl, it roughly looks like this:

push 2
push 1
call add
add esp, 8

The caller puts the argument on the stack, and after calling the function, it cleans the stack directly with add esp, 8.

For variable argument functions such as printf, it is difficult for Callee to clean up the stack because the number of arguments may change at each call time.

Therefore, caller-side stack cleanup, as used by cdecl, is suitable for variadic functions.

stdcall

stdcall is an x86 Calling Convention widely used in Windows API.

The characteristics of stdcall are as follows.

• Arguments are pushed onto the stack in right-to-left order.
• The return value is stored in eax.
• The callee is responsible for cleaning up the stack.

For example, if there is a function call like this:

push 2
push 1
call add

On the caller side, stack cleanup code such as add esp, 8 is not included after calling the function.

Instead, when the Callee function returns, it cleans up the stack as follows.

ret 8

In other words, for stdcall, Callee cleans up the stack according to the argument size.

fastcall

fastcall is a calling convention to speed up function calls by passing some arguments to registers rather than the stack.

Since there are not many registers in x86, all arguments cannot be passed as registers, and only some arguments are passed as registers.

Based on Microsoft fastcall, the first and second arguments are usually passed as ecx and edx, respectively.

• First argument: ecx
• Second argument: edx
• Remaining arguments: stack

For example, if there is a function call like this:

func(1, 2, 3);

Arguments can be passed approximately in the following form.

mov ecx, 1
mov edx, 2
push 3
call func

Using registers can improve function call performance because they can be faster than accessing stack memory.

thiscall

thiscall is a calling convention used in member function calls in C++.

In C++, member functions of a class internally use this pointer to point to the object itself.

For example, if you have the following code:

obj.func(1);

Since func is a member function, you need to know which object it is a function of.

At this time, the pointer that points to the object itself is this.

Based on x86 Microsoft thiscall, this pointer is usually passed to the ecx register.

mov ecx, obj
push 1
call func

In other words, thiscall can be seen as a calling convention that determines how to pass the this pointer in a C++ member function.

System V AMD64 ABI

System V AMD64 ABI is a calling convention mainly used in Unix x86-64 environments such as Linux and macOS.

Integer arguments are passed in the following order.

• rdi
• rsi
• rdx
• rcx
• r8
• r9

The return value is usually passed as rax.

If there are more than 6 arguments, the remaining arguments are passed through the stack.

For example, if there is a function call like this:

func(1, 2, 3, 4, 5, 6, 7);

In System V AMD64 ABI, the arguments are transmitted approximately as follows.

mov rdi, 1
mov rsi, 2
mov rdx, 3
mov rcx, 4
mov r8, 5
mov r9, 6
push 7
call func

The first to sixth arguments are passed to registers, and the seventh argument is passed to the stack.

MS x64 Calling Convention

MS x64 Calling Convention is a calling convention used in the Windows x64 environment.

Integer arguments are passed in the following order.

• rcx
• rdx
• r8
• r9

The return value is usually passed as rax.

If there are more than 4 arguments, the remaining arguments are passed through the stack.

For example, if there is a function call like this:

func(1, 2, 3, 4, 5);

In the MS x64 Calling Convention, arguments are passed roughly as follows.

mov rcx, 1
mov rdx, 2
mov r8, 3
mov r9, 4
push 5
call func

The first to fourth arguments are passed to registers, and the fifth argument is passed to the stack.

System V AMD64 ABI and MS x64 Calling Convention are both used in the x86-64 environment, but the register order for passing arguments is different.

• System V AMD64 ABI: rdi, rsi, rdx, rcx, r8, r9
• MS x64: rcx, rdx, r8, r9

Caller-saved and Callee-saved Register

Calling Convention also defines register preservation rules.

Because register values may change when calling a function, it is necessary to determine who should keep which registers.

Registers can be broadly divided into Caller-saved Register and Callee-saved Register.

• Caller-saved Register: A register that must be saved directly when a caller needs it.
• Callee-saved Register: A register that the Callee must save before use and restore before returning.

Caller-saved Register is a register whose value is not abnormal even if it changes after a function call.

Therefore, if the caller needs to use the corresponding register value even after calling the function, it must be saved directly before calling the function.

A callee-saved register is a register whose value the callee must not change.

If Callee needs to use the register, the existing value must be saved at the start of the function and restored to the original value before the function ends.

For example, if Callee needs to use ebx, it could look like this:

push ebx
mov ebx, 1
pop ebx
ret

Because of these rules, caller and callee can predict each other’s register use.

Why Calling Conventions Matter

Calling Convention not only determines the function call method, but is also very important in binary analysis and exploits.

When doing reversing, if you know the calling convention, you can obtain the following information.

• How many arguments does the function have?
• Where are the arguments stored?
• Where is the return value sent?
• How the stack is cleaned up after a function call
• Which registers should the function preserve?

For example, if you see add esp, 0x10 after calling a function in an x86 binary, it may be the cdecl method because the caller is cleaning up the stack.

push 4
push 3
push 2
push 1
call func
add esp, 0x10

Conversely, if the function ends with ret 0x10, it is possible that Callee is the way to clean up the stack.

ret 0x10

Also, if you see that values are entered into rdi, rsi, rdx, etc. right before a function call in the x86-64 binary, it may be a System V AMD64 ABI environment.

mov rdi, 1
mov rsi, 2
mov rdx, 3
call func

In Windows x64 binaries, rcx, rdx, r8, and r9 are often used right before a function call.

mov rcx, 1
mov rdx, 2
mov r8, 3
mov r9, 4
call func

In other words, if you know the calling convention, you can more accurately analyze the structure of functions and data flow in assembly code.

Summary

Calling Convention is a promise about calling and returning a function.

During the function call process, the following elements must be determined.

• Where to send the argument
• Where to send the return value
• Who will clean up the stack?
• Which registers to preserve
• How to manage Return Address and Stack Frame

In x86, arguments are mainly passed through the stack, and in x86-64, arguments are mainly passed through registers.

However, because the number of registers is limited, as the number of arguments increases, the stack is also used.

Understanding calling conventions helps you understand how C code is converted into function calls in assembly, and helps you analyze function arguments, return values, and stack structures when performing reversal or binary exploitation.