Head in the Cloud

My journey to work in cloud technology

Creating my own backup tool with AWS

November 12, 2017 · AWS

As of today, I am no longer a CrashPlan customer, as they have phased out their retail client service to focus on larger customers. I looked at other solutions, but $5 per month per machine seems to be the going rate, so I figure I'll try to create my own solution and save some money. I need an automated backup solution for my home computer and my work laptop.

Obviously, this is a problem already solved by many people (Duplicati, for one), so this is just a learning experiment for me.

My first attempt is to use command line tools to automate a copy from the local file system to an S3 bucket.

  1. Create the bucket.
    1. Pretty straight forward, except that this is my first time using encryption at rest. I chose the default AES-256 encryption, which should protect the data anyone but me accessing it. I also chose to create a lifecycle policy to rotate the files to cheaper storage in 30 days, then to Glacier after 60 days, then retain for 10 years. I figure if I haven't accessed a file in 10 years, I really don't need it anymore :-). Initially, I am putting all client backups into one bucket. I think the next iteration will be to create the bucket programmatically and pass the bucket name to create the user and policy.
    2. Create the user.
      1. I have created a programmatic access user for each device that needs to be backed up with S3 access. Once the concept is proved, I'll tighten it down so only this user can access only their specific bucket.

    3. Add the user profile to awscli.
      1. My first time configuring multiple users with the --profile option, very easy to do.

    4. Test the command.
      1. My first attempt was to use the aws s3 sync command to begin the copy. This command is being run from a Windows 10 machine with bash (ubuntu) installed and aws configured.
         aws s3 sync --profile myClient /mnt/c s3://mybucket.backup/
        This worked fairly well until it got down into some the Windows internals, then stopped. I don't think the OS liked that. So, I chose to update my home folder and backup drive instead. Having the two processes running at the same time seemed to speed things up a bit, my clumsy attempt at manually parallelization :-)

So far, so good, taking a while to push about a terabyte of data over my home internet access. The next step will be to automate the process, which requires learning how to create recurring tasks in Windows and Mac. I wish all OSes would just use crontab, sigh...

I'll do some analysis of the costs over the next 90 days and see if I can actually save money. I think the data transfer and regular S3 costs might be more than $5, but the lifecycle policy should kick in and as long as you are doing differential backups, then the data charges should be much cheaper too. Let's see how long it takes this service to pay for itself...

The next step will be to develop an app client for this process, which is a bit more than I know how to do at this point. I'll dig through github and see if anyone has built this sort of thing already.

 

NOTE: Found this after I started experimenting, official step by step.