Although Apple has yet to follow through on its promise to publish the FaceTime video calling protocol, some details are starting to emerge. Arjun Roychowdhury did a little packet sniffing and reports that the calls seem to go over vanilla SIP and STUN. The mystery is how the iPhone can seamlessly transition from a GSM voice call to video. There is some encrypted stuff that goes on when the phone registers to the FaceTme servers, so my guess it that protocol includes the phone number, such that Apple can maintain a database that maps phone numbers to IP addresses.
On the other hand, the Apple support site says it uses ports 4080 and 5223 which are hardly standard for SIP.
It appears that registration is a combination of SMS and XMPP, with SIP just being used to set up the RTP session after the call us established. The audio and video streams are not encrypted.
It also appears that while Apple has said the protocol will be "open" the connection requires a client-side SSL certificate, giving Apple the ability to grant or deny access on a per-device basis.