Improved Open Source Backup:
Incorporating inline deduplication and sparse indexing solutions
G. P. E. Keeling
< Prev
Contents
Next >
Appendix G - Raw test data
Tests a1-a6 are backups of small files.
Tests b1-b6 are restores of small files.
Tests c1-c6 are backups of small files.
Tests d1-d6 are restores of small files.
amanda-3.3.1
This software uses either tar (in incremental mode) over ssh, or 'dump' to
retrieve files from the client. For the tests, I configured it to use tar.
I did not capture memory statistics for the use of tar and ssh, and due to
the complexity of the restore procedure, could not get server memory statistics
for the restore.
Each backup was performed as an incremental (amanda automatically promoted the
first of each sequence to a full).
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 32:56 - 23248 19183345724 17510752 30
a2 04:31 - 23344 107900600 17606856 36
a3 05:41 - 23328 656049880 18203648 42
a4 06:15 - 23360 655611900 18800440 48
a5 14:45 - 23252 6360937308 24597208 54
a6 02:22 - 23468 36993628 24630116 60
b1 30:19 15348 - 18903206284 - -
b2 31:51 15464 - 19009897796 - -
b3 33:08 15524 - 19544418832 - -
b4 33:12 15532 - 19545361840 - -
b5 41:58 15548 - 25166198412 - -
b6 45:47 15544 - 25179850864 - -
c1 32:36 - 23260 24220077632 22190460 30
c2 00:03 - 22184 39692 22190564 36
c3 32:31 - 23252 24227710432 44380932 42
c4 32:28 - 23272 24224534548 66571300 48
c5 32:30 - 23244 24229732244 88761668 54
c6 15:44 - 23260 11725042740 99503956 60
d1 33:30 4608 - 23807491248 - -
d2 33:37 4620 - 23804837640 - -
d3 32:56 4580 - 23806787244 - -
d4 33:00 4584 - 23818501728 - -
d5 1:05:58 4512 - 47611538272 - -
d6 49:04 4512 - 35333166188 - -
backshift-1.20
The server storage directory was mounted on the client using sshfs and
backshift was run on the client side, saving to the mounted directory.
Several hours into the second backup, where no files had been
touched in the backup set, testing backshift was abandoned.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 43:37:50 125132 - 6709214436 6314132 1290187
Note from the author of Backshift, August 2015
backuppc-3.2.1
This software uses tar over ssh to retrieve files from clients.
I did not capture memory statistics for the use of tar and ssh.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 1:23:49 - 284984 17901726148 9100204 2106563
a2 19:08 - 284972 28261484 9511692 2226526
a3 25:23 - 284968 542277068 10456412 2423805
a4 25:43 - 284968 542110772 10835368 2579155
a5 30:25 - 285008 5872127792 11253144 3188940
a6 19:52 - 110464 5854417192 11431256 3728321
b1 51:47 - 1172 18580045616 - -
b2 1:00:56 - 1168 18616293176 - -
b3 1:02:02 - 1168 18608841452 - -
b4 1:06:50 - 1172 18627536544 - -
b5 1:06:38 - 1168 18611113920 - -
b6 22:21 - 1160 6088763864 - -
c1 32:28 - 19644 24239202344 22190452 58
c2 00:04 - 19648 8680 22190484 71
c3 23:49 - 23704 53683896 44380820 96
c4 22:46 - 23704 53678308 44380864 114
c5 32:21 - 19620 24240577464 44380928 137
c6 15:47 - 19656 11734617416 55123184 161
d1 32:55 - 1144 23500251316 - -
d2 32:54 - 1140 23499752736 - -
d3 32:56 - 1144 23505114172 - -
d4 32:56 - 1140 23504421352 - -
d5 33:05 - 1148 23497633972 - -
d6 16:03 - 1152 11377405112 - -
bacula-5.2.13
This software has multiple components - the database, the director,
the storage daemon and the file daemon. MySQL was used as the database
software, and ran on the server along with the director and storage
daemon. The file daemon runs on the client side. Memory statistics were
recorded for all of these. MySQL disk space and nodes were also included in
the results.
Each backup was performed as an incremental (bacula automatically promoted the
first of each sequence to a full).
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 1:14:13 1188 183516 19415316808 16613252 28
a2 06:24 315816 212648 7339 16613252 28
a3 08:22 315712 212592 558876468 17090484 28
a4 08:21 315820 212852 558831313 17567712 28
a5 33:18 315932 212464 6558268748 23074912 28
a6 12:51 318896 212900 403522651 23214996 28
b1 37:49 4364 394420 18761519812 - -
b2 37:53 4364 394404 18761497347 - -
b3 37:41 4364 394348 18761753952 - -
b4 37:50 4360 394444 18760842419 - -
b5 38:41 4368 394380 18762252423 - -
b6 14:06 4372 279184 6143147781 - -
c1 36:36 4820 58416 24148768088 22207036 28
c2 00:01 13648 58980 7337 22207036 28
c3 35:27 14168 58416 24150990612 44413760 28
c4 36:00 14376 58792 24154976470 66620488 29
c5 36:06 14380 58496 24151605070 88827212 29
c6 17:27 48420 58224 11689696758 99577364 29
d1 32:28 4372 22583 24035061412 - -
d2 32:28 4360 58076 24033754963 - -
d3 32:27 4376 58328 24034781702 - -
d4 32:45 4360 58516 24094745919 - -
d5 32:29 4360 57904 24034420735 - -
d6 15:44 4356 57952 11634940513 - -
bup-0.25
For the tests, bup used ssh for the network transport. Memory usage
for ssh was not recorded. The bup 'split' command was used for backing up,
and 'join' for restore.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 1:05:25 50396 41564 13028027920 12024828 92
a2 47:36 36924 36976 22788 12054464 97
a3 47:41 43160 23764 552641840 12541864 98
a4 47:28 42076 15656 301338788 12818772 101
a5 51:40 50160 40564 3441960928 15993348 113
a6 16:46 43636 8828 123336 15993444 116
b1 27:49 8084 3687524 18721975020 - -
b2 28:12 8080 3689344 18727867332 - -
b3 27:31 8072 3688592 18720814964 - -
b4 27:15 8072 3689556 18725122480 - -
b5 27:49 8076 3446688 18731302336 - -
b6 09:57 8120 1689772 6136944368 - -
c1 37:06 54728 49860 14760055588 13673276 97
c2 19:30 43400 8214 32492 13647632 98
c3 19:38 45132 11000 44790792 13688684 101
c4 19:34 43888 8132 64096 13688720 104
c5 19:35 43892 34472 63836 13715140 109
c6 09:26 43872 8124 80428 13689000 110
d1 33:40 8116 3694364 23825024780 - -
d2 33:53 8116 3687716 23821504284 - -
d3 33:28 8084 3695360 23811360164 - -
d4 32:58 8084 3696864 23815830112 - -
d5 32:56 8084 3697192 23823484740 - -
d6 16:00 8120 3696736 11533677068 - -
obnam-1.1
The server storage directory was mounted on the client using sshfs and
obnam was run on the client side, saving to the mounted directory.
Testing on obnam was abandoned after it hadn't finished after
8 hours and had taken up more than 50GB of space on the server,
when there were only 22GB of files to bakup up on the client.
rdiff-backup
For the tests, rdiff-backup used ssh for the network transport. Memory usage
for ssh was not recorded. I was unable to record server memory statistics
for the server when restoring large files.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 2:08:09 155932 632632 19549959636 20492104 1624474
a2 25:17 154156 29548 732119660 20579220 1624480
a3 1:05:54 172512 73508 1330676312 21232008 1669655
a4 1:04:33 172576 73516 826853828 21492456 1714830
a5 10:56:29 156352 699136 13588674368 28216628 2763315
a6 12:28:14 65212 29532 1856630968 28095444 2763324
b1 2:20:46 287712 630348 18493237092 - -
b2 2:19:55 287832 630416 18493619200 - -
b3 2:19:32 287840 630028 18494316560 - -
b4 2:18:52 287960 630416 18494140436 - -
b5 2:18:07 289328 623580 18494732116 - -
b6 44:49 105024 255252 6056832020 - -
c1 1:40:30 10908 11256 24084094872 22190296 13
c2 00:05 10260 10396 68516 22190304 19
c3 42:21 17984 17292 104168884 22272648 27
c4 52:52 18364 18600 1999400 22272684 35
c5 2:09:00 11404 11092 24086698904 44462964 44
c6 35:39 17996 17996 2029536 41372716 52
d1 1:21:19 11416 - 23953765636 - -
d2 1:27:13 11412 - 23955454164 - -
d3 1:03:15 11416 - 23953752660 - -
d4 42:23 11416 - 23954690536 - -
d5 1:02:19 11412 - 23957680960 - -
d6 20:29 11412 - 11596528360 - -
rsync-link-dest
For the tests, rsync used ssh for the network transport. Memory usage for ssh
was not recorded.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 34:57 42224 49964 17921288120 20001104 1535719
a2 05:38 28316 28412 43588912 20360088 3071436
a3 08:11 28292 37028 560651704 21285484 4607153
a4 07:12 28580 37224 54572948 22210876 6142870
a5 14:36 34208 47856 5899248188 29008644 7678588
a6 01:51 16708 17624 14553352 29131180 8187702
b1 28:37 97540 138424 17396435028 - -
b2 28:29 97544 138532 17407095180 - -
b3 28:05 97540 136904 17394005376 - -
b4 27:54 97544 135868 17392775068 - -
b5 27:42 97544 138756 17391105544 - -
b6 08:46 69624 61852 5686955940 - -
c1 32:33 6588 1488 24254325112 22190276 4
c2 00:02 3188 1248 7316 22190280 6
c3 28:58 6312 9140 96024684 44380552 8
c4 29:41 3320 9144 2229896 66570824 10
c5 32:26 6852 1488 24242023980 88761096 12
c6 15:33 3216 9068 1887460 99503292 14
d1 32:31 7380 1256 23513700756 - -
d2 32:33 7376 1252 23515115572 - -
d3 32:40 7280 1244 23511657216 - -
d4 32:46 7316 1252 23529028584 - -
d5 32:54 7476 1248 23520135780 - -
d6 15:54 7256 1048 11383879512 - -
tar-over-ssh
Memory usage settings for ssh were not recorded. For the server, that meant
no memory usage results, since that was the only process running on the server.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 34:14 1416 - 18906459540 17457040 2
a2 32:46 1420 - 18901056092 34914076 3
a3 32:55 1416 - 18902683324 52371112 4
a4 32:53 1416 - 18901313052 69828148 5
a5 32:49 1436 - 18903018656 87286192 6
a6 11:27 1396 - 6192213332 93003248 7
b1 25:47 1120 - 18716011072 - -
b2 25:49 1116 - 18710756096 - -
b3 25:50 1124 - 18710245600 - -
b4 25:52 1124 - 18714191624 - -
b5 25:53 1128 - 18713813356 - -
b6 08:26 1118 - 6127883736 - -
c1 33:05 1204 - 23944470936 22190280 2
c2 32:40 1208 - 23944195580 44380556 3
c3 33:04 1208 - 23943847812 66570832 4
c4 32:47 1216 - 23943428308 88761108 5
c5 32:45 1216 - 23944522232 110951384 6
c6 15:54 1216 - 11591519516 121693580 7
d1 33:03 1152 - 23814698300 - -
d2 32:44 1148 - 23813943948 - -
d3 32:48 1152 - 23806047744 - -
d4 32:47 1148 - 23805303488 - -
d5 32:46 1156 - 23806356912 - -
d6 15:55 1160 - 11526917756 - -
urbackup-1.2.4
Note: Testing on urbackup was abandoned when it was discovered that
the only way to restore files was one at a time via its web interface.
This made restoring millions of files impractical.
burp-1.3.36
A one line patch was applied to burp-1.3.36 in order to turn off network
compression. Another patch was applied to prevent it compressing individual
files on restore.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 56:06 5136 3712 19073221246 20381968 1535686
a2 27:13 5124 3752 559650895 20762924 1535691
a3 35:12 5140 3844 1232999529 21720168 1580870
a4 38:03 5144 3828 764511451 22281984 1626049
a5 39:35 5128 3712 6630073883 29226320 2135147
a6 13:21 5124 3752 193122037 29355744 2135158
b1 39:58 3452 49216 18408345630 - -
b2 39:51 3448 49980 18408178034 - -
b3 43:34 3452 41896 18408416587 - -
b4 38:01 3452 35288 18403160479 - -
b5 37:05 3456 34568 18405181477 - -
b6 11:26 3448 13908 6025288820 - -
c1 32:56 3916 3560 24209453432 22190320 15
c2 0:04 3912 3652 10605 22190340 20
c3 49:24 10124 8536 125113914 22272284 32
c4 46:25 10124 8644 22968510 22272352 44
c5 32:56 3912 3560 24209415813 44462664 56
c6 39:41 10124 6788 22835167 41393692 68
d1 1:18:28 3456 3424 24168034544 - -
d2 1:12:39 3452 3424 24167416085 - -
d3 53:58 3456 3428 24167803329 - -
d4 33:13 3452 3424 24167751901 - -
d5 55:25 3448 3428 24167297229 - -
d6 16:08 3452 3428 11699804162 - -
burp-2.0.0
A one line patch was applied to burp-2.0.0 in order to turn off network
compression.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 50:12 936276 422436 6261782638 5321000 459
a2 6:01 560492 167004 463561084 5647588 466
a3 8:22 560604 191472 993873340 6447976 521
a4 7:38 560488 191036 479991676 6774508 528
a5 18:47 561020 267716 646705357 7102040 535
a6 2:17 188924 183604 154855021 7211576 542
b1 2:32:14 3392 1419164 18072607863 - -
b2 2:24:02 3396 1460392 18072342507 - -
b3 2:13:37 3392 1514896 18071492855 - -
b4 2:12:54 3396 1502624 18071360723 - -
b5 2:16:30 3396 1499876 18072840238 - -
b6 1:09:27 3396 925132 5919571611 - -
c1 44:27 166552 352840 14740348487 13442384 963
c2 00:22 3752 369148 10712 13529832 970
c3 32:45 96172 370104 333668235 13652820 981
c4 32:38 44292 370268 294625537 13740276 988
c5 32:42 44292 370012 294212673 13827728 995
c6 15:50 44284 370036 143341975 13870216 1004
d1 1:50:26 3392 929832 24079869152 - -
d2 1:49:51 3408 922188 24080071062 - -
d3 1:56:46 3408 901052 24081243535 - -
d4 2:00:49 3408 870040 24081042956 - -
d5 2:12:54 3412 842132 24081705105 - -
d6 41:56 3408 717240 11656952804 - -
burp-2.0.1
A one line patch was applied to burp-2.0.1 in order to turn off network
compression.
max mem (kb)
test time client server network (b) disk space (kb) nodes
a1 49:55 964824 222620 6333908776 5134556 1047
a2 5:26 546768 15616 459379807 5275764 1862
a3 7:30 546796 23460 982353947 5886076 2701
a4 7:01 546792 23440 470919618 6027296 3516
a5 17:36 547928 88208 631650761 6169096 4332
a6 1:57 184516 14256 151701404 6215876 4604
b1 2:03:56 3480 333452 18070868859 - -
b2 2:02:49 3464 333448 18070889282 - -
b3 2:00:17 3464 333452 18069833763 - -
b4 2:00:25 3460 333248 18070389579 - -
b5 1:59:24 3464 332836 18070507134 - -
b6 1:04:34 3460 333032 5919135548 - -
c1 46:07 166628 27076 15968721238 14442396 1317
c2 0:59 3836 23276 10660 14544832 2113
c3 33:22 166520 28208 380791042 14704548 2912
c4 33:30 137996 25148 351607673 14817148 3709
c5 33:30 137996 25264 349251193 14929748 4506
c6 16:18 122168 23076 169103345 14982380 4898
d1 1:08:10 3476 369672 24077829873 - -
d2 1:08:05 3464 369660 24076534634 - -
d3 1:10:13 3480 368820 24077401653 - -
d4 1:14:52 3476 368608 24078404161 - -
d5 1:13:54 3480 368612 24077496825 - -
d6 26:55 3460 362164 11655881312 - -
< Prev
Contents
Next >
|
