Notes on the SysV ABI calling convention

Get ready to align the stack

The SysV ABI defines the AMD64 calling convention used by both Linux and Mac. The official SysV x86_64 reference may be found here. They use “eightbyte” as a unit of memory, since it’s the size of a 64-bit register.

I’m ignoring stuff like long double and __int128.

Integers and pointers are passed in INTEGER registers. Floats and doubles are passed in SSE registers. Struct arguments are translated to a sequence of INTEGER and SSE registers following these rules:

  1. If it’s larger than eight eightbytes, it gets passed in memory. (But actually there’s an extra rule later on that implies that if it’s larger than two eightbytes, it’s passed in memory – except maybe super wide SIMD vectors.)

  2. If it’s weird (has unaligned members, or weird C++ stuff), it gets passed in memory.

  3. Else, no struct member falls across an eightbyte boundary, so split it up into eightbyte chunks.

  4. Each eightbyte chunk might still contain multiple members. If it’s all floats it’s SSE, otherwise use an INTEGER register.

Now that we can classify arguments, we need to assign them to registers.

  1. Each INTEGER eightbyte gets assigned the next available out of %rdi, %rsi, %rdx, %rcx, %r8, %r9

  2. Each SSE eightbyte gets assigned the next available out of %xmm0, …, %xmm7

  3. If at any point there’s not enough registers left, the whole argument goes in memory instead.

We also need to handle the return value. It gets classified in the same way as arguments. Then:

  1. If it would get passed in memory, the caller allocates space for it on the stack and passes a pointer to this space as %rdi, like an invisible extra zeroth argument. This same pointer must be returned in %rax.

  2. Each INTEGER eightbyte get assigned the next available out of %rax, %rdx

  3. Each SSE eightbyte gets assigned the next available out of %xmm0, %xmm1

Finally, let’s deal with the stuff that gets passed in memory. In order, push to the stack:

  1. Padding, if needed, so that the stack will be 16-byte aligned at the end, and enough space for the return value, if needed.

  2. Each argument that needs to be passed in memory, in order