03/10/2007FTP Gateway to S3

Recently I’ve announced that I will be developing an FTP server that would serve as a layer over S3 storage. I went ahead and had a proof of concept server working in about a day. It supports all the features you expect it to but still lacks administrative controls. I’ve found out however that Windows can’t directly mount FTP servers as drives but there are at least two free apps that do exactly that. It works great, for example, when browsing remote storage you see icons embedded in executables and it doesn’t require downloading whole files in background (in contrast with JungleDisk). Anyway, I think WebDAV and SFTP support will come as well in future versions.

What I wanted to talk about are details of the implementation for this app. Some of it is already implemented some is not, but generally it is going to work this way (referring to Windows version for now):

  • The server itself is a Windows Service, running from the moment system boots up (many other good things come out of this)
  • Users can run a tray agent that controls the server and reports its status
  • The server is controlled via XML-RPC (you can write your own controller too)
  • The exciting thing is that the server can run on a remote machine but you would still be able to use the tray agent in the same way as if it was local
  • This also means that it can be used not just for personal use but also as a file server in your organization (or home network). It can run on EC2 as well.
  • This calls for permissions system, but guess what, the server is based on Twisted and we use their auth subsystem. It has ready to use implementations for UNIX password db files, LDAP and PAM hooks. And I think I’ll add Windows domain support as well.
  • This gives full integration with other Intranet services and security
  • You don’t have to use all these advanced features, it will work just fine as a local gateway
  • Don’t forget it reuses S3 Backup codebase and will have file revision control and renaming features

There’s a long way to go but I think a first version of this gateway will be included in one of the next betas. I’d love to get feedback on this, you can leave a comment here or email me privately. Thanks.

Update: another thing worth a mention, with splitting support uploads are truly streaming so if you have a decent Internet connection you will be able use S3 as a target for whole drive backup (for ex. via Windows built-in Backup tool or Norton Ghost)