CSAW CTF Quals: Reversing 400 Keygenme32

This linux executable took three inputs, and printed a smiley (good) or frowny (bad) face based on these inputs. First, I looked at the program in IDA, and found a method that determines which face prints. The disassembly for this method is below.

This function takes four inputs. Upon inspection in GDB, two of arguments passed are directly from the original input, when the file was ran in terminal. For the above disassembly, arg_8 is the first input, and arg_C is the second. These are compared at the end of the program to arg0 and arg0+4, which both come from the first input, in some sort of modification. In order to get successful output from this section

arg0 == arg8 XOR 31333337

arg0+4 = a rearrangement of arg_C. To get from arg_C to the corresponding arg0+4, the last byte stays the same, the first byte becomes the third, and the second and third byte to go to first a second.

For example 0x11223344 -> 0x22331144

However, this does nothing to tell us how to get the first two values. I spent a lot of time looking at the code without really getting anywhere. Thankfully, there are ways around this. Imp3rial and I ended up scripting GDB in order to get the immediate values. We accomplish this by writing to the .gdbinit file, and using popen to automate things. The problem was solved using the following script (some imports were used in test version, but not include in final functionality).

import os, sys, pexpect, fdpexpect, socket, struct, time
from subprocess import Popen, PIPE
so = socket.socket()
so.connect(('128.238.66.219', 14549))
temp = so.recv(1024)
print(temp)
for i in xrange(100):
 temp = so.recv(1024)
 print("new range " + temp)
 inputStuff = temp.split(' ')[-1].strip("\n")
 print('input stuff ' + inputStuff)
 f = open('/home/student/.gdbinit', 'w')
 writeString = "file ./keygenme32.elf\n"
 writeString += 'break _Z5checkiiii\n'
 writeString += 'start ' + inputStuff + ' 0x11 0x11\n'
 writeString += 'run\n'
 writeString += 'x/x $ebp+0x8\n'
 writeString += 'x/x $ebp+0xc\n'
 writeString += 'quit\n'
 f.write(writeString)
 f.close()
 a = Popen('gdb', shell=True, stdout=PIPE, stderr=PIPE)
 temp = a.communicate()[0].split(' ')[-21].split('\t')
 #print(temp) 
 input1 = temp[1][0:10]
 input2 = temp[2][0:10]

 result1 = int(input1,16) ^ 0x31333337
 result2 = input2[0:2] + input2[6:8] + input2[2:6]+ input2[8:10]
 result1 = str(hex(result1).zfill(8)).strip("L")
 print(result1)
 print(result2)
 output = str(int(result1,0)) + " " + str(int(result2,0)) + "\n"
 so.send(output)
 print("output is " + output)

 print(so.recv(1024))
 print(so.recv(1024))
 print("##########################################################")

Post competition, it appears that the code implemented some sort of vm, with its own instructions. However, looking at writeups, it appears that most people did what I did, and scripted a debugger.

key{#r3vers1ng_emul4t3d_cpuz_a1n7_h4rd}

-albinotomato