There is some information on the FTP server properties required to enable FTPS support here - http://wiki.alfresco.com/wiki/File_Server_Subsystem#FTP_Server_Configuration
You need to follow the instructions for setting up the keystore and truststore files, via the Java6 JSSE Reference Guide link on the Wiki page. That will give you the values required for the ftp.keyStore, ftp.trustStore and ftp.passphrase values.
You can force all connections to use FTPS by setting ftp.requireSecureSession=true.
There's debug output that can be enabled by setting the 'SSL' debug flag via ftp.sessionDebug=SSL, and enabling the log4j.logger.org.alfresco.fileserver=debug log4j output.
The FTPS support runs over the same socket as normal connections, the connection is switched into SSL mode at the request of the client, this is usually done before the username/password is sent. The client can switch the socket back to plaintext mode using the 'CCC' command.