Database crash + re-scan -- I am alone in this ??

Discuss music production with Ableton Live.
mholloway
Posts: 1578
Joined: Sat Nov 29, 2008 7:24 pm
Location: Portland, Oregon, USA

Database crash + re-scan -- I am alone in this ??

Post by mholloway » Wed Sep 28, 2016 2:53 am

I just sent off an e-mail to support to try to get my old ticket on this re-opened (it was never solved to begin with).

NOTE: Basically I'm posting again here simply because I want to know if somehow, bizarrely, I'm the only one experiencing this issue with such incredible persistence. Support has already been contacted. I doubt anyone here can solve this, because support hasn't been able to, for 2+ years running. I just want to know, well, if I'm really alone.

The live database crashes -- a lot. Then it has to re-scan, which takes hours. During that time, I don't have access to any of my User Library -- which makes doing my (professional, mind you) work damn near impossible.

Currently it's happening about once a week, if I'm using Live regularly.

It usually happens while typing something into the search field at the top of the browser. A pop up suddenly appears, my stomach sinks and I go "aw fuck, here we go AGAIN," I'm told that the database has crashed and Live needs to be restarted and my database will be (slowly) re-scanned.

Before you blame my computer, please realize that this problem goes back so far that it was happening on my previous iMac, and now is happening exactly the same on my 2016 iMac. So, yeah, not just a "one bad system" thing.

Oh, and I'm still getting the "unknown file ID" error a lot still, too.

Anybody, ANYBODY else????

-Michael Very Frustrated Holloway

2016 iMac / El Capitan
my industrial music made with Ableton Live (as DEAD WHEN I FOUND HER): https://deadwhenifoundher.bandcamp.com/
my dark jazz / noir music made with Ableton Live: https://michaelarthurholloway.bandcamp. ... guilt-noir

fishmonkey
Posts: 4478
Joined: Wed Oct 24, 2007 4:50 am

Re: Database crash + re-scan -- I am alone in this ??

Post by fishmonkey » Wed Sep 28, 2016 4:24 am

sounds mighty frustrating.

there must be a way to make a backup of the Live database that can be quickly restored when you get a crash...

Tarekith
Posts: 19072
Joined: Fri Jan 07, 2005 11:46 pm
Location: Ableton Forum Administrator
Contact:

Re: Database crash + re-scan -- I am alone in this ??

Post by Tarekith » Wed Sep 28, 2016 5:49 am

Have you tried a completely fresh install of Live and all your packs? When you transferred to your newest computer, did you restore from a Time Machine back up or similar? Just wondering if you somehow had a corrupted library ages ago and keep propogating it to new set ups.
Tarekith
Ableton Forum Administrator
https://tarekith.com

jestermgee
Posts: 4500
Joined: Mon Apr 26, 2010 6:38 am

Re: Database crash + re-scan -- I am alone in this ??

Post by jestermgee » Wed Sep 28, 2016 9:49 am

Yeah somethings not quite right there.

I've had a db crash on the odd occasion but once in maybe a few months. I've just rebuilt my whole machine and reinstalled all my software from scratch which I like to do every 12 months so might be that time for you too.

Angstrom
Posts: 14921
Joined: Mon Oct 04, 2004 2:22 pm
Contact:

Re: Database crash + re-scan -- I am alone in this ??

Post by Angstrom » Wed Sep 28, 2016 10:52 am

I suspect something unique to your setup. Not your hardware, but your way of working.
Do you have some form of automated incremental backup system running? Or some form of filewrite auditing?

SQLite is Okayish, but its possible for apps to fumble it sometimes. That said I have written to the ableton"files.db" many hundreds of times with third party code while Ableton Live was using it and never corrupted it. It certainly ought to be robust in your more usual usage cases.

How to corrupt your database
https://www.sqlite.org/howtocorrupt.html

Quicker fixing : its most likely just an indexing issue. Index repair in a 3rd party SQLite app should be quick and youd get most of your DB back that way, and then let live reindex the incorrect or missing resources.
What I'd do is copy the db to a new location, open it in an SQL manager of your choice (or go commandline if you are hardcore)

Commandline or GUI based manager, rebuild the crashed DB to a copy and then put the repaired copy back in the Live DB folder. (As files.db)

I don't do this on a mac, so cant suggest a good freebie app. The commandline tools are all on
https://www.sqlite.org/download.html

mholloway
Posts: 1578
Joined: Sat Nov 29, 2008 7:24 pm
Location: Portland, Oregon, USA

Re: Database crash + re-scan -- I am alone in this ??

Post by mholloway » Thu Sep 29, 2016 12:15 am

Tarekith wrote:Have you tried a completely fresh install of Live and all your packs? When you transferred to your newest computer, did you restore from a Time Machine back up or similar? Just wondering if you somehow had a corrupted library ages ago and keep propogating it to new set ups.
Well, Live and all my packs were "freshly installed" when I bought my new imac. I didn't do any time machine stuff, I simply copied my LIve User LIbrary from the old system to the new one.

So, yeah, it does seem there's some form of corruption within the user library I guess, that got carried over. What to do about it, though, I have no idea. My user library is essential to my work.
my industrial music made with Ableton Live (as DEAD WHEN I FOUND HER): https://deadwhenifoundher.bandcamp.com/
my dark jazz / noir music made with Ableton Live: https://michaelarthurholloway.bandcamp. ... guilt-noir

mholloway
Posts: 1578
Joined: Sat Nov 29, 2008 7:24 pm
Location: Portland, Oregon, USA

Re: Database crash + re-scan -- I am alone in this ??

Post by mholloway » Thu Sep 29, 2016 12:17 am

Angstrom wrote:I suspect something unique to your setup. Not your hardware, but your way of working.
Do you have some form of automated incremental backup system running? Or some form of filewrite auditing?
I haven't been using anything like that, In fact I have no idea what those things are, ha.
Angstrom wrote:
SQLite is Okayish, but its possible for apps to fumble it sometimes. That said I have written to the ableton"files.db" many hundreds of times with third party code while Ableton Live was using it and never corrupted it. It certainly ought to be robust in your more usual usage cases.

How to corrupt your database
https://www.sqlite.org/howtocorrupt.html

Quicker fixing : its most likely just an indexing issue. Index repair in a 3rd party SQLite app should be quick and youd get most of your DB back that way, and then let live reindex the incorrect or missing resources.
What I'd do is copy the db to a new location, open it in an SQL manager of your choice (or go commandline if you are hardcore)

Commandline or GUI based manager, rebuild the crashed DB to a copy and then put the repaired copy back in the Live DB folder. (As files.db)

I don't do this on a mac, so cant suggest a good freebie app. The commandline tools are all on
https://www.sqlite.org/download.html
Thanks for the ideas, I dont' quite understand what SQlite is... I do agree that it seems to be an indexing issue. I'll look into some equivalent mac-based index repair, I suppose.
my industrial music made with Ableton Live (as DEAD WHEN I FOUND HER): https://deadwhenifoundher.bandcamp.com/
my dark jazz / noir music made with Ableton Live: https://michaelarthurholloway.bandcamp. ... guilt-noir

Stromkraft
Posts: 7033
Joined: Wed Jun 25, 2014 11:34 am

Re: Database crash + re-scan -- I am alone in this ??

Post by Stromkraft » Thu Sep 29, 2016 8:41 am

mholloway wrote:I just sent off an e-mail to support to try to get my old ticket on this re-opened (it was never solved to begin with).
I doubt anyone here can solve this, because support hasn't been able to, for 2+ years running.



Before you blame my computer, please realize that this problem goes back so far that it was happening on my previous iMac, and now is happening exactly the same on my 2016 iMac. So, yeah, not just a "one bad system" thing.

Oh, and I'm still getting the "unknown file ID" error a lot still, too.
This exact type of issue is something I haven't encountered and I've been moving the user library, as well as the complete native library one time, when migrating from OS X 10.6, to 10.9 and then to 10.11 during my time with Live 9 and I've changed disks at least 4 times.

I've overall had few or no problems with the index scanning. The "unknown file ID" happens to me once a year or so. I save quite a few presets, maybe 10-15 new a week, and usually each is saved many times.

I'd like to add that I've had at least one, maybe two, issues — unrelated to yours — running for several years and support has been unable to reproduce the issues as these have been intermittent and elusive also for me.

Some plug-in companies I purchase from have suggested to get inside my machine in order to solve specific unusual problems. Maybe that's the level Ableton needs to take problems that can't be reproduced on their machines? I don't know. I hope you get a solution for this as being cut off from your user library is not fun at all.
Last edited by Stromkraft on Sat Oct 01, 2016 10:09 am, edited 1 time in total.
Make some music!

Razz CE
Posts: 76
Joined: Mon May 11, 2015 12:53 am

Re: Database crash + re-scan -- I am alone in this ??

Post by Razz CE » Sat Oct 01, 2016 9:08 am

Dude I have a band-aid for this that's going to chance your life. Ready?

Back up your database file.

Just copy it onto another drive or another folder and then delete the corrupted file and paste the backup into its place. Voila - you get back 98% of your library instantly.

This happened to me once and I said "oh fuck if I'm ever going to deal with this again." For Windows the database file will be in C:\Users\[user name]\AppData\Roaming\Ableton\Live 9.x\Database\files.db
Win 10 Pro
Ryzen 9 3900XT
UAD Apollo/ART Voice Channel
Ableton Live 10/Cubase 10.5
Push 2/Moog Subsequent 37

Stromkraft
Posts: 7033
Joined: Wed Jun 25, 2014 11:34 am

Re: Database crash + re-scan -- I am alone in this ??

Post by Stromkraft » Sat Oct 01, 2016 10:11 am

@mholloway,

Have support looked at a copy of your database when it has crashed? What do they know about database corruption with the Live db? They told you anything about this?
Make some music!

tintala
Posts: 476
Joined: Mon Nov 29, 2010 5:37 pm

Re: Database crash + re-scan -- I am alone in this ??

Post by tintala » Sat Oct 01, 2016 8:38 pm

[quote="mholloway"]I just sent off an e-mail to support to try to get my old ticket on this re-opened (it was never solved to begin with).

NOTE: Basically I'm posting again here simply because I want to know if somehow, bizarrely, I'm the only one experiencing this issue with such incredible persistence. Support has already been contacted. I doubt anyone here can solve this, because support hasn't been able to, for 2+ years running. I just want to know, well, if I'm really alone.

The live database crashes -- a lot. Then it has to re-scan, which takes hours. During that time, I don't have access to any of my User Library -- which makes doing my (professional, mind you) work damn near impossible.



Anybody, ANYBODY else????

yes I posted this issue in the forum as well as some other users also, back on Sept 19th, I believe we are having the same issue? :?: :?
viewtopic.php?f=1&t=223223&hilit=+wholl ... ng+library

mholloway
Posts: 1578
Joined: Sat Nov 29, 2008 7:24 pm
Location: Portland, Oregon, USA

Re: Database crash + re-scan -- I am alone in this ??

Post by mholloway » Sat Oct 01, 2016 9:55 pm

Stromkraft wrote:@mholloway,

Have support looked at a copy of your database when it has crashed? What do they know about database corruption with the Live db? They told you anything about this?

Yes, they had me add something to options.txt that gives them additional info when I make a status report, then they had me send them a new status report immediately following the last crash -- just sent that off this week, waiting to hear back.

They haven't offered any explanation as to the Why of it so far, I think they might be stumped.

-M
my industrial music made with Ableton Live (as DEAD WHEN I FOUND HER): https://deadwhenifoundher.bandcamp.com/
my dark jazz / noir music made with Ableton Live: https://michaelarthurholloway.bandcamp. ... guilt-noir

steko
Posts: 3411
Joined: Thu Jan 22, 2009 8:29 pm

Re: Database crash + re-scan -- I am alone in this ??

Post by steko » Sat Oct 01, 2016 10:44 pm

Hey Michael, ever tried deleting the Live 9 database?:

For OS X the path to the database file is: /Users/[username]/Library/Application Support/Ableton/Live 9.x.x/Database/files.db

Quit Live before deleting.

Live 9 database disk image is malformed – Ableton AG

How to reset Live – Ableton AG


I think your database problem is probably caused by something else, but deleting the thing is worth a try…

And now you know which file to restore (quit Live first) from a backup...
Rahad Jackson wrote:My Awesome Mix Tape #6

mholloway
Posts: 1578
Joined: Sat Nov 29, 2008 7:24 pm
Location: Portland, Oregon, USA

Re: Database crash + re-scan -- I am alone in this ??

Post by mholloway » Sun Oct 02, 2016 6:23 am

steko wrote:Hey Michael, ever tried deleting the Live 9 database?:

For OS X the path to the database file is: /Users/[username]/Library/Application Support/Ableton/Live 9.x.x/Database/files.db

Quit Live before deleting.

Live 9 database disk image is malformed – Ableton AG

How to reset Live – Ableton AG


I think your database problem is probably caused by something else, but deleting the thing is worth a try…

And now you know which file to restore (quit Live first) from a backup...

Yeah, already went through that processes during my back and forth with support. Problem persisted nonetheless. Thanks though!

-M
my industrial music made with Ableton Live (as DEAD WHEN I FOUND HER): https://deadwhenifoundher.bandcamp.com/
my dark jazz / noir music made with Ableton Live: https://michaelarthurholloway.bandcamp. ... guilt-noir

steko
Posts: 3411
Joined: Thu Jan 22, 2009 8:29 pm

Re: Database crash + re-scan -- I am alone in this ??

Post by steko » Sun Oct 02, 2016 4:32 pm

Maybe one of your indexed directories/folders is corrupted, or contains (a) corrupted file(s. A tool like DiskWarrior would come in handy now…
Rahad Jackson wrote:My Awesome Mix Tape #6

Post Reply