Ad
Ad
Ad
Pages: [1] 2 »   Bottom of Page
Print
Author Topic: Amazon Glacier + Lightroom ???  (Read 9454 times)
Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« on: August 23, 2012, 03:41:02 AM »
ReplyReply

Amazon just introduced a new online archival service called Glacier. But it's unlike Dropbox, Google Drive and the like, in that it is not meant for real-time access. The idea is that you can quickly upload stuff for free, but retrieval might take a few hours and it costs. Storage itself is much cheaper than anything else with only $0.01 per GB and month. So it's really meant as an archive or last-line-of-defense backup.
AFAIK there is not front end supplied yet, but only some APIs which developers can use to cook up their own applications.

So that makes me wonder if is would be a good idea to integrate Glacier into LR. Maybe as a plugin. I could imagine that during every import an archive of the new images is created and sent to the glacier. Or alternatively this could be done automatically once a month or week.
Thanks to my OCD-like editing I have less than 100GB of image in total including all Photoshop files. That would imply costs of less than $12 per year for me which even in the long run isn't more expensive than buying external disks.

So, what do you think? Any plugin programmers around who are interested?


(as a technical side note: I believe the service is based on magnetic tape. This would explain the wait times for retrieval and why it is not possible to manipulate uploaded files. Even today, when it comes to large volume, long-term storage nothing beats tape)
Logged
Alan Goldhammer
Sr. Member
****
Offline Offline

Posts: 1673


WWW
« Reply #1 on: August 23, 2012, 07:49:48 AM »
ReplyReply

This is very interesting compared to what I'm current paying for Mozy home backup.  I guess the delay in being able to access one's data is probably OK for me since I would only need it if there were an absolute crisis where both my computer HD and backup HD failed on the same day which is not all that likely.  I'm going to take a look at this and make sure that there are not any hidden gotchas here.  Thanks for the tip!!!!!

I haven't read the stuff on the site yet so I don't know how the backup time is configured.  Mozy Home automatically backs up things twice a day and you can configure what files are backed up.

Alan
Logged

PeterAit
Sr. Member
****
Offline Offline

Posts: 1946



WWW
« Reply #2 on: August 23, 2012, 08:10:25 AM »
ReplyReply

For most of us, who are using cable or DSL connections, wouldn't this be glacially slow for large backups? A cable modem's upload speed is about 700k/sec, and that's under ideal conditions (neighbors not streaming movies, etc.). A 100 GB upload would take 40 hours.
Logged

Peter
"Photographic technique is a means to an end, never the end itself."
View my photos at http://www.peteraitken.com
Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« Reply #3 on: August 23, 2012, 08:14:28 AM »
ReplyReply

The thing is, right now this is not meant as a service for end user. Amazon provides some APIs for developers to come up with front ends or to integrate glacier into existing programs as a service (where the user might not even be aware that he's using Amazon).
So, I really hope that soon some body comes up with a LR plugin or something. Actually I have little hope, that Adobe will integrate it directly in LR as they have their own cloud based service Revel which unfortunately is not very useful for me and it's far too expensive.
Logged
Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« Reply #4 on: August 23, 2012, 08:18:48 AM »
ReplyReply

Peter, that's right. But you would only have to do this once. If you then keep on updating once a month or so, it will go much faster. Also, this is meant as an long(er)-term archival service, so hopefully in a couple of years time, upload speeds will have increased.

But one idea would be to have to uploads done automatically according to some user defined schedule. Say every sunday night, starting at 11pm or so. Then it would be less distracting (that is when 11pm is not in the middle of your working hours  Wink )
Logged
Alan Goldhammer
Sr. Member
****
Offline Offline

Posts: 1673


WWW
« Reply #5 on: August 23, 2012, 10:55:55 AM »
ReplyReply

The thing is, right now this is not meant as a service for end user. Amazon provides some APIs for developers to come up with front ends or to integrate glacier into existing programs as a service (where the user might not even be aware that he's using Amazon).
I did some reading on the site and this is correct.  To use it as is, you would need to upload individual packets of data with each use and the manage them.  It's pretty cumbersome to do it this way and until someone writes an API to manage this better I'll stick with what Mozy does for me with is fully automated.  To Peter's point about upload speeds, yes the initial upload takes a long time but as noted, you only need to do it once.  I think the way the Amazon service is presently set up without an API, you are going to run into problems in this regard.  I suspect they are going after the business user who has an Internet connection much faster than those of us who are home users.
Logged

Samotano
Jr. Member
**
Offline Offline

Posts: 88



WWW
« Reply #6 on: August 28, 2012, 10:20:13 AM »
ReplyReply

You can send them a hard drive and they will upload the content for you.  In comparing this with their S3 service, I assumed Glacier was the natural end user version of it.  Given its pricing, I find it quite attractive for my photo library backup and the pain of downloading/uploading is a small price to pay for offsite backup (I always have my own onsite backup). 

A much bigger problem though is to figure out a sustainable workflow for it.  Update it once a year or once every few months? How? What about files that have been modified? I am curious to hear in the following months from people who have tried it in the context of photo backup.
Logged
stormyboy
Jr. Member
**
Offline Offline

Posts: 65


« Reply #7 on: August 28, 2012, 12:01:27 PM »
ReplyReply

I saw this link on another site.  It's a Windows client for Amazon Glacier.  I haven't tried it.
http://fastglacier.com/
Logged
Samotano
Jr. Member
**
Offline Offline

Posts: 88



WWW
« Reply #8 on: August 28, 2012, 12:14:11 PM »
ReplyReply

Nice!  Thanks for sharing this.
Does anyone know how this would work to update files?  Say I have uploaded my library.  Then I modify 100 files.  Now I want the backup to reflect these updates.  What do you do? 
Logged
Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« Reply #9 on: August 28, 2012, 12:41:20 PM »
ReplyReply

In Glacier, you can only upload and delete files. There is no such thing as modification or overwriting. It's due to the fact, that (most likely) magnetic tape is used on which you can only write data serially. You would have to delete the old files and uploaded the modified ones again.
However, that's not really an issue as there is no need to write changes into the raw file. Even when working with DNG where the XMP is embedded in the file itself. All information is stored in the catalog.
Personally would be to convert to DNG during import and write the most important metadata, like copyright information, geo data, Keywords, and such into the DNG file before uploading.

EDIT: Forgot to mention, that there's a charge of $0.03 per GB for data deleted prior to 90 days.
« Last Edit: August 28, 2012, 01:23:15 PM by Fips » Logged
Snoopy Lane
Newbie
*
Offline Offline

Posts: 6


« Reply #10 on: August 28, 2012, 06:33:45 PM »
ReplyReply

It's 2012.  Most likely not tape.
Logged
howardm
Sr. Member
****
Offline Offline

Posts: 784


« Reply #11 on: August 28, 2012, 06:40:24 PM »
ReplyReply

There is no other storage medium that has a lower cost per GB and as low power requirements.

At somewhere between 1.5TB and 3TB per tape capacity and a price of well under $50, it's pretty hard
to beat good ol' tape (ala LTO5)


It's 2012.  Most likely not tape.
Logged
cybis
Full Member
***
Offline Offline

Posts: 164



WWW
« Reply #12 on: August 28, 2012, 08:52:29 PM »
ReplyReply


Yeah, thank you!
Logged

stormyboy
Jr. Member
**
Offline Offline

Posts: 65


« Reply #13 on: August 28, 2012, 09:39:10 PM »
ReplyReply

Hi, cybis.  I haven't tried it yet, but I did find some discussion about it:
https://forums.aws.amazon.com/thread.jspa?threadID=102829
Logged
Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« Reply #14 on: August 29, 2012, 02:15:39 AM »
ReplyReply

Quote
[...] it's pretty hard to beat good ol' tape (ala LTO5)

Exactly. And don't forget the 15-30 years durability.


Regarding clients. There will be a lot of programs hacked together in short time now to make use of Glacier. But as it is meant to be really, really secure, I'd give it some time and wait for a proven-to-be-reliable solution, hopefully with good LR integration.
Logged
cybis
Full Member
***
Offline Offline

Posts: 164



WWW
« Reply #15 on: August 29, 2012, 09:01:14 AM »
ReplyReply

Hi, cybis.  I haven't tried it yet, but I did find some discussion about it:
https://forums.aws.amazon.com/thread.jspa?threadID=102829

Thanks @stormyboy, I gave it a go today and it worked just fine and is easy to use. Now I'm just waiting on Amazon to publish the tool to transfer files from S3 to Glacier...

Also FastGlacier is free only for non-commercial use. They have a pro version for $29.95 that they claim is faster.
Logged

Samotano
Jr. Member
**
Offline Offline

Posts: 88



WWW
« Reply #16 on: August 29, 2012, 10:18:08 AM »
ReplyReply

So one possible workflow is:
1. Upload data to Amazon Glacier (say only RAW files).
  Question: can the folder structure be preserved on Glacier?

2. One month later (or 6 month later - depending on user's needs) user has new RAWs files to upload.
  I seem to understand that the user cannot populate the directory structure created in 1.  What then?

My library is organized as follows:
Photo_Library>Digital>YYYY_MM>YYYY_MM_eventname>Originals>YYYYMMDD_XX_XXXX.dng

Also, one would need a way to keep track of what was backed-up and what wasn't

« Last Edit: August 29, 2012, 10:20:32 AM by Samotano » Logged
john beardsworth
Sr. Member
****
Offline Offline

Posts: 2829



WWW
« Reply #17 on: August 29, 2012, 10:20:27 AM »
ReplyReply

Exactly. And don't forget the 15-30 years durability.
Regarding clients. There will be a lot of programs hacked together in short time now to make use of Glacier. But as it is meant to be really, really secure, I'd give it some time and wait for a proven-to-be-reliable solution, hopefully with good LR integration.
If you've got a general client, I wonder if you can roll your own integration in LR using Publish services. You'd create a new service using the Hard Drive option, probably choosing DNG as the output format, and a smart collection which picks up images edited in the last x hours.
Logged

Fips
Full Member
***
Offline Offline

Posts: 195



WWW
« Reply #18 on: August 30, 2012, 11:43:05 AM »
ReplyReply

Yes, that's more or less it. But as uploading large amounts of data can take quite a while, I would split the process in two parts: One LR plugin and one external program.

The plugin one has to write would provide a new publish service and create a database which keeps track of what's already uploaded and when that happened. All the files are then prepared, i.e. the metadata is saved to those files. If they are proprietary raw file, a DNG copy is created somewhere.
Then comes the external program into play which handles the upload according to a time schedule set in the plugin dialogue. The advantage of this is that LR does not need to run during uploading.

In principle, this shouldn't be too complicated. I had a look at the LR SKD and it's all done in Lua which provides modules for HTTP and SQL (so in principle one could even do in all within a single plugin!).
Logged
cybis
Full Member
***
Offline Offline

Posts: 164



WWW
« Reply #19 on: August 30, 2012, 10:13:10 PM »
ReplyReply

So I played a little with FastGlacier today. To understand Glacierís limitations it may help to imagine that Amazonís servers are located on Neptune. A round trip to Neptune takes about 9 hours at light speed. One can have a high bandwidth connection to Neptune but latency will always be what it is - long.

Every file operation that requires two way communications will take that long. For example in dealing with duplicate files, it would be impractical to warn the user and ask what to do. So Glacier gives each file a unique id; therefore multiple files with the same name and path can coexist (yes glacier keeps paths intact). This makes incremental backups impractical.

In the same fashion, downloading files will be as fast as your bandwidth permits but will only reach you hours after a request.

The usual file browser user interface is impractical. FastGlacier user interface simply shows one long list of files with their path, i.e. no collapsing folders, no drag and dropping files between folders, etc.

As already mentioned Glacier is for long term archival, not for last month files that you might still be working with next week. Since itís at least just as important to backup yesterdayís files as it is for last yearís files, I would use S3-jungle disk or similar for recent files and Glacier for the rest.

Iím not sure yet what the best way to organize files for archiving in Glacier is. If you first group files together in zip or rar archives, it seems to me it would make the whole thing even less flexible.

Thoughts?
 
Logged

Pages: [1] 2 »   Top of Page
Print
Jump to:  

Ad
Ad
Ad