5 weeks agoS3 Backup beta 18: New file-db

S3 Backup beta 18 rev.1518

This version introduces a rewrite of one of the most important parts of the application — backup logic. This was made possible by redesigning a file-db component which I’ll explain in a later post.

The new backup logic works by synchronizing two “virtual” filesystems created from the local one and S3-based one by filtering and reordering them based on backup rules. This affords some of long-requested features, for example, if you change the exclusion masks in some of your backup rules and the new mask excludes some files that were previously backed up, this change will not make the backup job delete the remote copy on next run. This isn’t a trivial matter — the app compares the local and remote storage and given that file mask it would not expect the filtered file on the remote storage, so the new system does well to make the correct decision in this and many other corner cases, like when a different backup jobs or different rules in the same backup job point to the same remote path.

I’ll be later writing on the systems involved (file-db, rule indexing, virtual filesystems, filesystem middleware and more) so that users who are interested in the inner workings of the app have a better grasp of what doing a backup involves.

Another very noticeable change in this version is faster backup job planning phase. We are working on even more speedups, but it’s already much faster than before.

The app now also stores all the debug logs in your user settings folder. We will introduce online reports, notifications and reminders in a later version, but this should work for now.

The file-db format had to be changed, so the first time you run the new version it will automatically be migrated. This happens momentarily and transactionally, so it cannot corrupt it. The log of the conversion is written in the logs folder in case you want to inspect it. You need to know that once it is migrated, the older version will not recognize it and during backups would believe all of the files need to be reuploaded. Basically, once you run beta 18 on a certain computer, do not try to use earlier version on it. If you have an older version installed on a different computer — it would not be influenced in any way, the file-db is local. The migration process is solid, but if you want to keep a copy of the older file-db, it’s stored in %appdata%/s3 backup/filedb.db

The new file-db structure affords much faster operation which is used to ensure its consistency on-the-fly. This matters if you alter the remote storage via different app or from another computer. For example, if you have deleted some of the backed up files from S3 using a different app, the file-db would still think they are there and the next time backup is run, it would not reupload them. But in beta 18, if you browse to the folder where some of the files were removed, the app will immediately notice that they are missing and correct file-db accordingly. These actions are logged and can be seen in the debug log (from the main menu: Tools → Backup Log).

The same happens when listing remote folders during normal backup operation, but not all folders are listed at that stage, so we’ll add a way to manually reindex remote storage, if there’s a need for that and to do it automatically at a schedule, just in case.

The deletion propagation was disabled midway through the life of beta 17 and is now enabled again.

Some more notes on this release:

  • Every time the app runs it will persistently write a log of all its operations to its appdata folder (to open it in explorer type %appdata%/s3 backup in the address bar).
  • The crash logs are also directed to that folder. Normally they would be written to the same folder as the app itself, but program files folder is restricted, so this has better chance to get the log successfully written.
  • When the new bucket is created but the DNS records had no time to update yet, the requests to S3 are met with a redirect to a location that would get the request to the correct datacenter. In this revision, this endpoint is cached, so the redirect is only encountered once. This leads to much faster backups, directory listing and operation in general for new buckets.
  • The GUI code for backup progress reporting was tweaked so it doesn’t try to keep up with every change which allows the backup, especially the planning stage, to work faster.
  • The way directory structure is stored in S3 relies on every folder to contain at least one file or subfolder. This means that empty folders must have a special invisible file in them (S3 object with the key exactly matching the directory path). If that file is not present, the folder would disappear as soon as the last file in it is deleted. To make sure this doesn’t happen, the app will automatically detect if that file is missing and create it if necessary. This is completely transparent to the user and has no performance impact but is reported in the debug logs.
  • The requirements for bucket names have changed since Amazon started offering a choice of datacenter location and some old buckets are only accessible by using the old method, and buckets that are located in locations other than the US-East datacenter must be accessed in a different way. So, to make them both accessible both methods need to be supported and picked automatically depending on the bucket name. S3 Backup was capable of this since beta 16, but beta 18 improves this further by being smarter about this choice.
  • The bucket location menu in bucket manager now has country flags matching the location of the datacenter.
  • UNC support was improved a little, but please note that it will get properly exposed in the UI in a later release.
  • Fixed migration from very old versions (backups.xml in a certain old format was preventing the updated app from starting)

If you come across any issues, please let us know and we’ll do our best to fix it ASAP. We are planning to make beta 19 a release candidate and have 1.0 version out in February, but you can order it while it’s still in beta and get a nice discount.

Download S3 Backup beta 18