FTP Sanity. Ryan Hadley

A few years back, while working for PowerOne Media, we were writing up a report of all single points of failures in IT. It was quite obvious that the FTP situation was bad. PowerOne had recently acquired several companies, including Careersite (the company I was with). Every company had at least 1 FTP server, some had several. We had just completed “merging” all the company networks together and assumed there had to be a better way to handle all the FTP traffic. We searched all the open source projects that had anything to do with FTP and couldn’t find anything with all the functionality we wanted:

• Easy Redundancy
• One FTP Service for multiple backends
• Must run in Linux
• Ability to “clean up” after itself (more on this later)

Having found nothing adequate, we created our own: Generic-FTP.

And because I was working for two of the best bosses I had ever worked for (Dale and Bruce. And seriously, not only were they great managers, but they were great teachers, I’m sure they’d argue against that though.) I was given the OK to open source it under the GPL. Sadly, it’s gotten very little attention sitting out there, but I do know that some co-workers who had moved on to other companies have found use for it

The concept was simple. We would use a normal linux FTP daemon (ProFTPD was chosen at the time for its built in ability to use MySQL authentication, making it very easy to keep the secondary FTP server in a cold-standby situation). Next we needed some scripts to handle the uploads appropriately. The scripts would pull the full path to the file (/opt/ftp/root/careersite/example/directory) and check in the config file what it should do with a file uploaded to that directory. If it didn’t find any config for that directory it would move up a directory (/opt/ftp/root/careersite/example) and continue moving up unitl it reached the ftp root or found a config for that directory. The config file would assign actions to those directories. Actions are plugins that are dropped in the configured action directory for Generic-FTP, plus one built-in action called “hold”. It would process the files accordingly, moving them in the end to an archive directory. Files in the archive directory would be purged according to the Generic-FTP configuration of days to keep an archive for.

Since this was really just fancy scripting, I took up the task of programming it myself using PHP. Which is just as useful for scripting as Perl no matter what anyone tells you. If you rolled your eyes at the idea of “server scripting programming in PHP” then please go home.

I created some actions for general use:

• rsync: to rsync the file to a remote rsync server
• rsync-local: to rsync the file to a folder on the localhost (i.e. copy the file to a network mount: /mnt/smb/windowsboxshare)
• smbclient: to use the smbclient command to copy the file to a remote smb/windows share

So then we configured Generic-FTP to take over for all our other FTP servers. It could accept files for any backend service, windows or linux based, and properly get the file to their location. It could also accept files for pickup by customers, things like reports/exports/etc, using the “Hold” action. The “Hold” action also has a time limit for each file, so after X number of days the file will be archived one final time and then purged. And details of each transaction was properly logged, making monitoring with Nagios/BigBrother very simple.

It performed wonderfully. We had quite a bit (sorry, no hard data) of traffic going through it at one time. We had to play with the number of concurrent incoming FTP connections, the interval of the “backups” to the cold-standby secondary server, and the number of concurrent”actions” allowed to run.

It also served its purpose well, knocking off a good handful of single points of failure. There were a couple occasions when we had drives go out and whatnot that caused us to fail over to the secondary cold-standby.

Please download it here and see if it can make your life simpler like has made ours. There is still more that I’d like to do to it, cleaning up the code/documentation/install process being the top priority. Also I’d like to write up a full How-To on how to use it with ProFTPD to get a full cold-standby FTP server up and running. Please try it out and let me know your thoughts.

• 1 Comment
• Posted In Engineering, QA, IT