Create a new param in GBAK to backup / restore tables/indexes in parallel (using all CPUs in CpuAffinity) [CORE1365]

[firebird-automations]

Submitted by: Eduardo Jedliczka (edujed)

Is duplicated by CORE3958
Relate to CORE2992

Votes: 1

I have a 8-core machine, and when I do a Gbak of a mid-size database (more than 15 GB) the restore time is near 2 hours using only one CPU, because the restoration process is linear.
If is possible, backuping / restoring data from 1 table per CPU concurrently (in other words, in parallel) this time will be strongly lower.
if is possible, rebuild 1 index per CPU can optimise this restore operation.

[firebird-automations]

Commented by: Sean Leyne (seanleyne)

Given that a restore is very disk intensive, I believe the bottleneck for the restore is not the CPU/processor but the disk subsystem.

Therefore, I don't see any benefit in trying to perform the restore from multiple threads.

Finally, you should investigate the new NBackup utility (new in v2.0) it performs backups/restores at the physical database level and runs at hardware speeds!

[firebird-automations]

Commented by: Eduardo Jedliczka (edujed)

in this case, the bottleneck is not disk... It is a RAID 0+1 with SAS disks 15k RPM with 2 GB of cache. But one CPU uses 100% in all time of restore process.

Of coarse, I know nbackup, but in case of corruption (hardware fail, energy loss) nbackup do not recovery the database.

[firebird-automations]

Commented by: @AlexPeshkoff

There are some problems with your suggestion.
First, in pre-vulcan/fb3 architecture you must use classic server (I suppose you already do on 8-CPU machine).
Next, gbak's backup format is strictly linear (it and contains data in DPB-like format - certainly, this is just analogy, but more or less correct), i.e. you can't say about the size of a table not reading all of it's data record by record.
And last seen at once problem - restoring database gbak creates it in single-user shutdown mode, i.e. no one (including sysdba) can attach before gbak completion. This solves a lot of problems happening due to attempts to attach to incomplete database. Such protection will have to be removed for multiway restore.

Anyway I like the suggestion - restore is really CPU intensive thing (for example, indices creation is also performed at the end of restore). Not sure all of your 8 CPU's will work efficiently, but certainly it may be done faster then now.

[firebird-automations]

Commented by: Eduardo Jedliczka (edujed)

Thanks for the answer.

Yes, you're wright, I user FireBird Classic Server.
Second, I understood the problem when a user connect in a database before the restore completion, I lost much time with it in FB 1.X, but it can be changed modifyind the way of restoration (only sub-processes - one for each index and/or CPU - of restoration can connect in a "in-restore" database)
Third, really, the greathest problem is the way of gbak works... DPB-like format.... Humm... initially I beleave gbak maybe work with pages (like FB) but it's nonsense.
Finally, I beleave, the most simple way of to that, is "sequential database restoration" with "parallel index creation".

[firebird-automations]

Modified by: @pcisar

Workflow: jira [ 12611 ] => Firebird [ 14213 ]

[firebird-automations]

Modified by: @dyemanov

Link: This issue is duplicated by CORE3958 [ CORE3958 ]

[firebird-automations]

Modified by: @dyemanov

Link: This issue relate to CORE2992 [ CORE2992 ]