How do you back up your work? Any solutions you’ve tried that ultimately didn't work out?
Since this applies to all digital artists, I thought it would be a good topic to share ideas on.
Let’s talk back ups.
Let’s talk back ups.
tarekith
https://tarekith.com
https://tarekith.com
-
jestermgee
- Posts: 4500
- Joined: Mon Apr 26, 2010 6:38 am
Re: Let’s talk back ups.
A topic I am passionate about and would STRESS to EVERYONE that they spend the time to invest/test solutions that will work for them BEFORE getting stuck into anything.
Being the tech savvy guy I am and having to help countless friends and family when things go wrong there is nothing worse than having to tell someone that their child's first 4 years of photos and videos are now gone forever. Having been producing music for 15+ years you can imagine how many projects I have, how many samples I have collected and also I have thousands of field and sound effects recordings some of which were captured on film sets and in restricted areas of businesses so no chance to ever get them again easily.
I offer advice on how to design backup solutions and have helped others to setup solutions that work for them. I have had some major hardware failures in my time with one accident frying every drive in my production system (including an internal backup drive0, motherboard and rom drives. I still, to this day (touch wood) have never lost more than a few MB of data thanks to backups!
Do I need to Backup my Data?
Seems like a stupid question but it really depends. Some people live on social media now days and may not even have a computer, just a smart phone. Their photos are all posted to facebook and live on there so in essence are "backed up" as you would think. But things can go wrong even in the larger world so here is what I have people answer:
- Is the data important and irreplaceable?
Photos, videos, written documents are all important and cannot be replaced if gone. If a single file is important and cannot be replaced it needs to be backed up which means it needs to exist in more than 1 location.
- Do you have more than 1 copy in different places?
All hard data storage solutions (drives, discs etc) will die. The best way to think is in terms of the next time you access the drive it will be dead so is there another location the data is stored? There should always be an original and a backup at a MINIMUM.
- How much data do you need to backup in the next 10 years?
The amount increases with each passing year so if you have 1GB of important data within a year, next you will probably have 2GB in the same space considering file sizes get larger with better qualities. Make sure the solution will cater for the future.
- Automated backup!!!
Having an external USB drive that you just plug in when you remember and copy files is not a backup solution. People become complacent and lazy and I have heard this far too many times: "Oh man, you mean my hard drive is fried. I was just about to backup that data too...". It is painful to tell someone they have just lost everything and the only backup they have was from a year ago.
Things to consider when deciding on backups
Before getting stuck into backing up things there are a few things to consider:
- Do I need to backup just selected files (photos, videos etc) or my whole system (such as a system snapshot).
Many external USB drives marketed at users ship with some basic free software that can make automatic backups of selected files on your PC. As long as you keep a copy on the PC and on the backup drive then this can be sufficient. Backing up an entire OS is a lot more involved and takes a bit more planning and space.
- Do I need incremental backups?
Incremental backups will save changes since the last backup rather than just updating a backup image. This can be useful for instance if you have a "mirror" backup system that simply creates a mirrored backup of your files deleting files you have deleted yourself. Incremental backups mean if you accidentally delete something that is reflected in the backup or need to go back to a version from some time ago you can.
- Do I want local or remote backups?
Local backup solutions would be from computer to an attached hard drive or network server within the home. The benefit to a local storage solution is you typically have access to huge amounts of cheap storage to store a lot of data but the downside is that the original data and backup will be within the same premises so wont be protected from something like a fire or flood. Remote backups to a service such as dropbox or google drive can offer secured and reliable storage in the "cloud" but require monthly access fees and the storage of files is out of your control. Having a mix of BOTH options is one of the best solutions.
I run RAID in my system so don't need extra backups
lol. RAID is not a backup solution in itself, just a step towards redundancy. It is entirely possible that a RAID array can become corrupt or fail. I have had a server with ALL drives fail due to a pinched data and +12v wire in the case not to mention that power related issues can cause havoc. If data is only stored on a RAID system and not in 2 locations, it is not secured.
Cost?
What will the ongoing cost be and can you maintain that "forever"? One thing to be cautious of with solutions such as dropbox for instance is it is a great service and may do exactly what you need it to... now...but companies know how to make money and you may find in a few years after you reach your data limits that an upgrade to a new storage plan is significant and not affordable. It can then be difficult to find a new solution so always be sure to check what the ongoing costs will be.
My Solution
There are many solutions out there and all have their pros and cons and I use a combination of 4 different backup solutions for different reasons:
Complete OS Backup
I make complete OS backups every 3-6 months using Acronis (PC). This takes a complete backup of my windows drive (with software installed) as well as a separate backup of my sample data and VST library drive. This is all stored on a removable HDD. This is a manual process and just in the event I have an OS issue, it's not to backup my important data.
Automated Backups to Server
I have a main production PC, Secondary production PC, portable laptop for DJ work and 2 additional family computers that all have data I need backed up. I built a file and media server back in 2005 and designed a completely automated system for backups using windows "robocopy" and 7zip.
On my server I run a simple batch file that runs in a constant loop. Every 5 minutes this batch file will poll the network for any of my computers and when it finds one it will then start to upload all the directories I have set as backup locations for that PC. It does a simple mirror copy to a folder on the server making a 1:1 copy of the directories I want to backup. Once this has completed the batch file then does not poll that machine again until it goes offline/online again.
Incremental Mirror Backup
On my server I have another process that creates an "incremental" image every 24 hours for my main production PC data that saves an incremental image of file changes since the last incremental image. This doubles up my backups on the same drive but allows me to go back to any point in time and get files that may have been deleted or changed. To do this I use 7zip via command to create a new zipped incremental backup.
Server Backup to Ext HDD
For every drive in my server I have a 1:1 external drive as a backup. I use a simple HDD docking station and again, using batch files and robocopy I simply insert a drive into the dock and my looping batch script detects which drive it is and does a mirror to external drives. These are stored in a case so in the event of an emergency I just grab that single case of drives and have all my data from all machines I care about. Also acts as a backup in case my server fails.
Backups to Dropbox / Gdrive / Facebook / Email
I use all of these methods also for creating backups. Google Drive is quite impressive. You get 15GB free, can drag and drop files easily and Gmail accounts are quick and easy to setup. I have 3 different ones and store different info across them all.
So my data lives in a minimum of 3 locations at one time and I consider this almost safe.
Being the tech savvy guy I am and having to help countless friends and family when things go wrong there is nothing worse than having to tell someone that their child's first 4 years of photos and videos are now gone forever. Having been producing music for 15+ years you can imagine how many projects I have, how many samples I have collected and also I have thousands of field and sound effects recordings some of which were captured on film sets and in restricted areas of businesses so no chance to ever get them again easily.
I offer advice on how to design backup solutions and have helped others to setup solutions that work for them. I have had some major hardware failures in my time with one accident frying every drive in my production system (including an internal backup drive0, motherboard and rom drives. I still, to this day (touch wood) have never lost more than a few MB of data thanks to backups!
Do I need to Backup my Data?
Seems like a stupid question but it really depends. Some people live on social media now days and may not even have a computer, just a smart phone. Their photos are all posted to facebook and live on there so in essence are "backed up" as you would think. But things can go wrong even in the larger world so here is what I have people answer:
- Is the data important and irreplaceable?
Photos, videos, written documents are all important and cannot be replaced if gone. If a single file is important and cannot be replaced it needs to be backed up which means it needs to exist in more than 1 location.
- Do you have more than 1 copy in different places?
All hard data storage solutions (drives, discs etc) will die. The best way to think is in terms of the next time you access the drive it will be dead so is there another location the data is stored? There should always be an original and a backup at a MINIMUM.
- How much data do you need to backup in the next 10 years?
The amount increases with each passing year so if you have 1GB of important data within a year, next you will probably have 2GB in the same space considering file sizes get larger with better qualities. Make sure the solution will cater for the future.
- Automated backup!!!
Having an external USB drive that you just plug in when you remember and copy files is not a backup solution. People become complacent and lazy and I have heard this far too many times: "Oh man, you mean my hard drive is fried. I was just about to backup that data too...". It is painful to tell someone they have just lost everything and the only backup they have was from a year ago.
Things to consider when deciding on backups
Before getting stuck into backing up things there are a few things to consider:
- Do I need to backup just selected files (photos, videos etc) or my whole system (such as a system snapshot).
Many external USB drives marketed at users ship with some basic free software that can make automatic backups of selected files on your PC. As long as you keep a copy on the PC and on the backup drive then this can be sufficient. Backing up an entire OS is a lot more involved and takes a bit more planning and space.
- Do I need incremental backups?
Incremental backups will save changes since the last backup rather than just updating a backup image. This can be useful for instance if you have a "mirror" backup system that simply creates a mirrored backup of your files deleting files you have deleted yourself. Incremental backups mean if you accidentally delete something that is reflected in the backup or need to go back to a version from some time ago you can.
- Do I want local or remote backups?
Local backup solutions would be from computer to an attached hard drive or network server within the home. The benefit to a local storage solution is you typically have access to huge amounts of cheap storage to store a lot of data but the downside is that the original data and backup will be within the same premises so wont be protected from something like a fire or flood. Remote backups to a service such as dropbox or google drive can offer secured and reliable storage in the "cloud" but require monthly access fees and the storage of files is out of your control. Having a mix of BOTH options is one of the best solutions.
I run RAID in my system so don't need extra backups
lol. RAID is not a backup solution in itself, just a step towards redundancy. It is entirely possible that a RAID array can become corrupt or fail. I have had a server with ALL drives fail due to a pinched data and +12v wire in the case not to mention that power related issues can cause havoc. If data is only stored on a RAID system and not in 2 locations, it is not secured.
Cost?
What will the ongoing cost be and can you maintain that "forever"? One thing to be cautious of with solutions such as dropbox for instance is it is a great service and may do exactly what you need it to... now...but companies know how to make money and you may find in a few years after you reach your data limits that an upgrade to a new storage plan is significant and not affordable. It can then be difficult to find a new solution so always be sure to check what the ongoing costs will be.
My Solution
There are many solutions out there and all have their pros and cons and I use a combination of 4 different backup solutions for different reasons:
Complete OS Backup
I make complete OS backups every 3-6 months using Acronis (PC). This takes a complete backup of my windows drive (with software installed) as well as a separate backup of my sample data and VST library drive. This is all stored on a removable HDD. This is a manual process and just in the event I have an OS issue, it's not to backup my important data.
Automated Backups to Server
I have a main production PC, Secondary production PC, portable laptop for DJ work and 2 additional family computers that all have data I need backed up. I built a file and media server back in 2005 and designed a completely automated system for backups using windows "robocopy" and 7zip.
On my server I run a simple batch file that runs in a constant loop. Every 5 minutes this batch file will poll the network for any of my computers and when it finds one it will then start to upload all the directories I have set as backup locations for that PC. It does a simple mirror copy to a folder on the server making a 1:1 copy of the directories I want to backup. Once this has completed the batch file then does not poll that machine again until it goes offline/online again.
Incremental Mirror Backup
On my server I have another process that creates an "incremental" image every 24 hours for my main production PC data that saves an incremental image of file changes since the last incremental image. This doubles up my backups on the same drive but allows me to go back to any point in time and get files that may have been deleted or changed. To do this I use 7zip via command to create a new zipped incremental backup.
Server Backup to Ext HDD
For every drive in my server I have a 1:1 external drive as a backup. I use a simple HDD docking station and again, using batch files and robocopy I simply insert a drive into the dock and my looping batch script detects which drive it is and does a mirror to external drives. These are stored in a case so in the event of an emergency I just grab that single case of drives and have all my data from all machines I care about. Also acts as a backup in case my server fails.
Backups to Dropbox / Gdrive / Facebook / Email
I use all of these methods also for creating backups. Google Drive is quite impressive. You get 15GB free, can drag and drop files easily and Gmail accounts are quick and easy to setup. I have 3 different ones and store different info across them all.
So my data lives in a minimum of 3 locations at one time and I consider this almost safe.
-
Stromkraft
- Posts: 7033
- Joined: Wed Jun 25, 2014 11:34 am
Re: Let’s talk back ups.
Important topic and great posts!
I have described my strategy before. I use Qrecall out of the main reason that it can deal with partial file updates, which is great for bigger binary files such as audio. Otherwise any big files with changes are re-added in full at any backup if it has been updated in any way.
I'd assume — haven't checked — the relevance of this depends in Live on how you edit as original files. I currently think anyway, the orignals are kept as is and changes are put recorded into .asd files. Nevertheless, exports under identical names alone, over time, add up to quite a few extra GB when re-added multiple times a day.
I have two archives:
In addition to this I make a Superduper backup of the whole disk at critical points. As I use it free this is a fresh copy only and not incremental steps. I dislike Time Machine and I don't know of any Trojans infecting the Superduper archives themselves, unlike TM. I used to use C.C.C. but it's not as well working as before and I've lost data with this some years ago.
At the moment I got a new 4GB drive and intend to maintain another copy of everything. Drives do fail.
I also plan to try to backup some stuff over the net, possibly to my own cloud elsewhere.
I have described my strategy before. I use Qrecall out of the main reason that it can deal with partial file updates, which is great for bigger binary files such as audio. Otherwise any big files with changes are re-added in full at any backup if it has been updated in any way.
I'd assume — haven't checked — the relevance of this depends in Live on how you edit as original files. I currently think anyway, the orignals are kept as is and changes are put recorded into .asd files. Nevertheless, exports under identical names alone, over time, add up to quite a few extra GB when re-added multiple times a day.
I have two archives:
- My work produce, i e what you find under ~/Documents. I have both my User Library and my Live project files here, not in the default Music folder. I also backup any presets, samples or anything similar that I make.
This is run every 8 hours typically to an external disk, or if traveling to USB stick. - My Music Library goes into it's own archive on another disk. This is run more seldom when I mount the disk
In addition to this I make a Superduper backup of the whole disk at critical points. As I use it free this is a fresh copy only and not incremental steps. I dislike Time Machine and I don't know of any Trojans infecting the Superduper archives themselves, unlike TM. I used to use C.C.C. but it's not as well working as before and I've lost data with this some years ago.
At the moment I got a new 4GB drive and intend to maintain another copy of everything. Drives do fail.
I also plan to try to backup some stuff over the net, possibly to my own cloud elsewhere.
Last edited by Stromkraft on Tue Sep 05, 2017 2:18 pm, edited 3 times in total.
Make some music!
Re: Let’s talk back ups.
I have two external hard drives to which I manually back up my data every now and then. I don't use any cloud solutions, because I don't find that viable with the amount of data we're talking here (all audio related stuff, including numerous huge Kontakt libraries, a huge photo library of raw files, graphics). My plan has been to keep an off-site drive too (at my parents'), but that system kind of broke when I had to upgrade from 2TB to 4TB backup disks (the old 2TB is now collecting dust at my parents').
Re: Let’s talk back ups.
Carbon Copy Cloner here; I clone on 2 different hard drives.
Ableton Forum Moderator
Re: Let’s talk back ups.
I use a mix of cloud back up and external drives myself. I have two 5TB USB3 external hard drives that are copies of each other, one of which also has my Time Machine back up on it. Everything I do gets backed up to both of these as soon as I'm done working on a project. Every couple of weeks I also back up to a 3rd external drive, and when this gets full I send it offsite to a family member or friend.
Important documents get zipped with encryption, and then are uploaded to my iCloud and Google Drive sites.
Important documents get zipped with encryption, and then are uploaded to my iCloud and Google Drive sites.
tarekith
https://tarekith.com
https://tarekith.com
-
re:dream
- Posts: 4598
- Joined: Fri Dec 28, 2007 9:42 am
- Location: Hoerikwaggo's sunset side...
- Contact:
Re: Let’s talk back ups.
My entire music library is inside a Dropbox folder so everything gets backed up to the cloud in real time.
This also means that all my music is always available to me no matter which computer I am working on - my laptop while traveling or my desktop at home.
It's well worth the monthly Dropbox fee.
Plus I also have a real time 'time machine' back up onto external hard drive.
OCD, I know, but hey.
This also means that all my music is always available to me no matter which computer I am working on - my laptop while traveling or my desktop at home.
It's well worth the monthly Dropbox fee.
Plus I also have a real time 'time machine' back up onto external hard drive.
OCD, I know, but hey.
-
fishmonkey
- Posts: 4478
- Joined: Wed Oct 24, 2007 4:50 am
Re: Let’s talk back ups.
i don't think it's OCD. backups are one of those annoying things like insurance. you pay for it year after year hoping you never need it. alternatively you choose to take the risk and suck it up when disaster strikes.
i have a friend who got hit by one of the Cryptolocker nasties a few months ago; no backups and lost everything. another friend lost half a book manuscript when a hard drive failed.
if you are serious about backups then you have at minimum two backup copies, with at least one offsite. i use a combo of Cloud backup (some encrypted) and external drives in multiple locations...
i have a friend who got hit by one of the Cryptolocker nasties a few months ago; no backups and lost everything. another friend lost half a book manuscript when a hard drive failed.
if you are serious about backups then you have at minimum two backup copies, with at least one offsite. i use a combo of Cloud backup (some encrypted) and external drives in multiple locations...
Re: Let’s talk back ups.
So how much are you guys shelling out for cloud backups? Are you putting all your data up there? How many gigabytes are we talking?
Re: Let’s talk back ups.
I have mine split between the 17GB I got for free from Google, and the 50GB I pay $12 a year for iCloud. For me that's enough for my important files and projects I might need to access anywhere. For client back ups and things like my song projects, I have those on enough HDs that I don't worry about uploading to the cloud for now.
tarekith
https://tarekith.com
https://tarekith.com
-
fishmonkey
- Posts: 4478
- Joined: Wed Oct 24, 2007 4:50 am
Re: Let’s talk back ups.
i don't have everything in the Cloud. stuff that i could download or install again (like sample libraries) i don't bother putting in the Cloud.
i have 1 TB with Dropbox. Dropbox is not fully secure, so i also use Arq to do some of my secure backups to Dropbox.
i also have a free account with Tresorit for some other critical stuff, because it is secure...
i have 1 TB with Dropbox. Dropbox is not fully secure, so i also use Arq to do some of my secure backups to Dropbox.
i also have a free account with Tresorit for some other critical stuff, because it is secure...
Re: Let’s talk back ups.
Dropbox for everything with a time machine back up on my local network...
Cheers
D
Cheers
D
Re: Let’s talk back ups.
With Time Machine, do you have a complete system backup, so that if your system drive fails, you could be up and running again quickly?
-
fishmonkey
- Posts: 4478
- Joined: Wed Oct 24, 2007 4:50 am
Re: Let’s talk back ups.
not as fast as recovering from a clone, but you can be up and running in an hour or two, depending on how much data you have and how fast your system and drives are.sporkles wrote:With Time Machine, do you have a complete system backup, so that if your system drive fails, you could be up and running again quickly?
i have clones, but also use Time Machine in two different locations. Time Machine is not bullet-proof, but when it is working it just does its thing with minimal intervention.
Re: Let’s talk back ups.
Yes, complete back up, and with two locations as well so they take it in turns to back up....
Cheers
D
Cheers
D