Quick nav: S3 Backup News Software Sitemap

Home → News

S3 Backup beta 18 rev.1518

This version introduces a rewrite of one of the most important parts of the application -- backup logic. This was made possible by redesigning a file-db component which I'll explain in a later post.

The new backup logic works by synchronizing two "virtual" filesystems created from the local one and S3-based one by filtering and reordering them based on backup rules. This affords some of long-requested features, for example, if you change the exclusion masks in some of your backup rules and the new mask excludes some files that were previously backed up, this change will not make the backup job delete the remote copy on next run. This isn't a trivial matter -- the app compares the local and remote storage and given that file mask it would not expect the filtered file on the remote storage, so the new system does well to make the correct decision in this and many other corner cases, like when a different backup jobs or different rules in the same backup job point to the same remote path.

I'll be later writing on the systems involved (file-db, rule indexing, virtual filesystems, filesystem middleware and more) so that users who are interested in the inner workings of the app have a better grasp of what doing a backup involves.

Another very noticeable change in this version is faster backup job planning phase. We are working on even more speedups, but it's already much faster than before.

The app now also stores all the debug logs in your user settings folder. We will introduce online reports, notifications and reminders in a later version, but this should work for now.

The file-db format had to be changed, so the first time you run the new version it will automatically be migrated. This happens momentarily and transactionally, so it cannot corrupt it. The log of the conversion is written in the logs folder in case you want to inspect it. You need to know that once it is migrated, the older version will not recognize it and during backups would believe all of the files need to be reuploaded. Basically, once you run beta 18 on a certain computer, do not try to use earlier version on it. If you have an older version installed on a different computer -- it would not be influenced in any way, the file-db is local. The migration process is solid, but if you want to keep a copy of the older file-db, it's stored in %appdata%/s3 backup/filedb.db

The new file-db structure affords much faster operation which is used to ensure its consistency on-the-fly. This matters if you alter the remote storage via different app or from another computer. For example, if you have deleted some of the backed up files from S3 using a different app, the file-db would still think it's there and the next time backup is run, it would not reupload it. With beta 18, if you browse to the folder where some of the files were removed the app will immediately notice that they are missing and correct file-db accordingly. These actions are logged and can be seen in the debug log (from the main menu: Tools -> Backup Log).

The same happens when listing remote folders during normal backup operation, but not all folders are listed at that stage, so we'll add a way to manually reindex remote storage, if there's a need for that and to do it automatically at a schedule, just in case.

The deletion propagation was disabled midway through the life of beta 17 and is now enabled again.

Some more notes on this release:

• Every time the app runs it will persistently write a log of all its operations to its appdata folder (to open it in explorer type %appdata%/s3 backup in the address bar).
• The crash logs are also directed to that folder. Normally they would be written to the same folder as the app itself, but program files folder is restricted, so this has better chance to get the log successfully written.
• When the new bucket is created but the DNS records had no time to update yet, the requests to S3 are met with a redirect to a location that would get the request to the correct datacenter. In this revision, this endpoint is cached, so the redirect is only encountered once. This leads to much faster backups, directory listing and operation in general for new buckets.
• The GUI code for backup progress reporting was tweaked so it doesn't try to keep up with every change which allows the backup, especially the planning stage, to work faster.
• The way directory structure is stored in S3 relies on every folder to contain at least one file or subfolder. This means that empty must have a special invisible file in them (S3 object with the key exactly matching the directory path). If that file is not present, the folder would disappear as soon as the last file in it is deleted. To make sure this doesn't happen, the app will automatically detect if that file is missing and create it if necessary. This is completely transparent to the user and has no performance impact but is reported in the debug logs.
• The requirements for bucket names have changed since Amazon started offering a choice of datacenter location and some old buckets are only accessible by using the old methods, and buckets that are located in locations other than the US-East datacenter must to be accessed in a different way. So, to make them both accessible both methods need to be supported and picked automatically depending on the bucket name. S3 Backup was capable of this since beta 16, but beta 18 improves this further by being smarter about this choice.
• The bucket location menu in bucket manager now has country flags matching the location of the datacenter.
• UNC support was improved a little, but please note that it will get properly exposed in the UI in a later release.
• Fix migration from very old versions (backups.xml in one old format was preventing the updated app from starting)

If you come across any issues, please let us know and we'll do our best to fix it ASAP. We are planning to make beta 19 a release candidate and have 1.0 version out in February, but you can order it while it's still in beta and get a nice discount.

Download S3 Backup beta 18

S3 Backup beta 17 rev.1422

Some changes in this release:

• Reduce the passes over data to a minimum. The data needs to be hashed for future change detection, compressed, encrypted and resulting stream hashed again to ensure its integrity in transfer. All of this is done as efficiently as possible with minimal I/O by doing multiple operations during the same pass.
• Support for international bucket names: accented, Cyrillic and most other characters. This is accomplished by IDNA encoding
• New encoding / decoding layer
• Better upload reporting taking into account things like size changes due to compression and retries in estimating the time remaining.
• Improved connection pooling.
• Support for transforming proxies (specifically Transfer-Encoding: chunked)
• Support for non-alphanumeric characters in backup job names.
• There was an issue for a couple revisions that was causing buckets with capital letters in their names to be inaccessible, this was fixed soon after it was discovered
• Backup code sometimes was causing remote directories to be repeatedly recreated. This was fixed.
• New menu code (important step in porting the app to new GUI backend code, but not a visible change).

Beta 17 was done quite some time ago, but we decided against releasing it on holidays to make sure that if any issues crop up, we would be able to put any time necessary to fix it immediately.

So even while we release beta 17 only now, beta 18 well underway already. The changes coming there will be related to the core backup algorithm that should make it yet faster especially for backup sets with huge amount of small files. Currently known issues:

• If you cancel an operation during the planning phase it will pop up a debug log with a traceback. It's just a small UI glitch we'll correct in one of the nearest releases.

If you come across any other issues, please let us know and we'll do our best to fix it ASAP.

This is release is part of our runup to a 1.0 release which gives you an opportunity to order it at a discount.

S3 Backup beta 16 rev.1353

It's time for beta 16. There are quite a few more changes that will be available shortly, but beta 16 is no small improvement. Here's a list of changes that are available right now:

• Port all code to an entirely new S3 abstraction layer. Among other things this adds support for EU and US-West buckets and much better connection pooling (better performance).
• Add choice of bucket location when creating one via Bucket manager.
• Better support for proxies in areas other than S3 connectivity.
• Better application state architecture. Fixes a problem with changed settings applying only partially until the app is restarted.
• Better support for logging program output in case of possible early crash. (Might come in handy if any such issue will arise in the future)
• Faster SSL connections.
• Work around another case of corrupted last-modified dates in filesystem.
• Fix a small issue with Unicode in filesystem.
• Fix "time too skewed" reporting.
• Fix "permission denied" errors in backup set selection tree.
• On Windows 2000 make installer put msvcrt DLLs in a location that OS can handle.
• Update dependencies.
• New beta expiry date is Febrary 15, 2010. It is safe to say at this point that 1.0 will be released much earlier than that.

We push out this release because it has changed quite a bit of internally, but we are aware of a couple imperfections:

• The upload progress code temporarily works on a file level only, that is, you will not see the progress change while the file blocks are being uploaded.
• Connection pooling is partially disabled.

Both issues will be corrected very soon.

This is a transitional release leading up to 1.0 release. While the app is still in beta you can order S3 Backup at significant discount.

S3 Backup beta 15 rev.1287

Changes in this release:

• Upgraded to Python 2.6.4
• New logging subsystem
• Improved timezone handling (application is now throughly timezone-aware)
• Handle corrupted file "last-modified" values (like if filesystem reports that date to be years into the future)
• Improved installer. The app will now install on top of any older or newer versions, no need to ever uninstall previously installed versions.
• Better handle filesystem changing during the backup process itself (cases like temprorary files etc).
• More efficient encryption and compression for smaller files (without any disk access).
• A couple improvements to Reaction.

This is one of our transitional releases leading up to 1.0 release. During this period you can order S3 Backup at significant discount.

S3 Backup beta 15 rev.1255

Sorry for taking a bit longer with this release. The reason for that was that I spent some time working on new code. The process of building an executable was completely overhauled and while it brings no immediately obvious changes for the user it's an important step for some of the things we have planned and to be announced at their own time.

(Also, another beta extension for S3 Webmaster was issued. I expect that it will get some care rather soon and get an actually updated release at last.)

• Almost immediately after rev.1221 a rev.1224 was issued with a fix to Reaction bug in lock management. I apologize for having that one in the first place, but I also have to note that the bug was of a kind that would go unnoticed if Reaction wasn't written with thought given to such things -- it did a very good job detecting that something was wrong.
• The issue with Windows 7 scheduling that appeared in the last revision was resolved. We always try to test the app in a couple of different environments before releasing a new version, but as far as permissions on task scheduling operations go, systems seem to vary wildly. So if you have any messages pop up when configuring the scheduling of the backup job, let us know so we could look into it. Note however that this does not apply to the running on schedule part -- that functionality is solid. It's the configuration part that proved to be quirky.
• When the client computer time is skewed, Amazon denies access for security reasons (to fight against so called "replay attacks"). This condition was not properly reported in a few recent revisions, but this is fixed now. If your computer clock or timezone settings are way off you will get a message telling you what time Amazon expects the client machine to have.
• The app is linked against newest Microsoft CRT libraries now.
• I will not detail the multiple changes to the application building process and internal changes, as this time that is something that is quite hard to explain without getting into details on internal architecture. So I'll just mention that there were quite a few more.

Now we are back to migrating the codebase and releasing new revisions every couple of days. We were made aware of a couple temporary issues caused by mismatch in some parts of the app between the old and the new codebase, but those will be resolved shortly.

Still, please do report whenever you see any odd behavior.

Also, let me mention again that while S3 Backup is in beta you can order a license at discount.

See Archives for more