BURP - BackUp and Restore Program

ABOUT
WHY
FEATURES
CHANGELOG
NEWS
FAQ
DOCS
BURP-UI
BURP2
DOWNLOAD
LICENCE
CONTRIBUTORS
DONATIONS
SPONSORS
CONTACT

Improved Open Source Backup:
Incorporating inline deduplication and sparse indexing solutions

G. P. E. Keeling

< Prev Contents Next >

Appendix G - Raw test data

Tests a1-a6 are backups of small files.
Tests b1-b6 are restores of small files.
Tests c1-c6 are backups of small files.
Tests d1-d6 are restores of small files.

amanda-3.3.1
This software uses either tar (in incremental mode) over ssh, or 'dump' to
retrieve files from the client. For the tests, I configured it to use tar.
I did not capture memory statistics for the use of tar and ssh, and due to
the complexity of the restore procedure, could not get server memory statistics
for the restore.
Each backup was performed as an incremental (amanda automatically promoted the
first of each sequence to a full).
               max mem (kb)
test   time   client   server  network (b)   disk space (kb)   nodes
a1    32:56        -    23248  19183345724     17510752           30
a2    04:31        -    23344    107900600     17606856           36
a3    05:41        -    23328    656049880     18203648           42
a4    06:15        -    23360    655611900     18800440           48
a5    14:45        -    23252   6360937308     24597208           54
a6    02:22        -    23468     36993628     24630116           60
b1    30:19    15348        -  18903206284            -            -
b2    31:51    15464        -  19009897796            -            -
b3    33:08    15524        -  19544418832            -            -
b4    33:12    15532        -  19545361840            -            -
b5    41:58    15548        -  25166198412            -            -
b6    45:47    15544        -  25179850864            -            -
c1    32:36        -    23260  24220077632     22190460           30
c2    00:03        -    22184        39692     22190564           36
c3    32:31        -    23252  24227710432     44380932           42
c4    32:28        -    23272  24224534548     66571300           48
c5    32:30        -    23244  24229732244     88761668           54
c6    15:44        -    23260  11725042740     99503956           60
d1    33:30     4608        -  23807491248            -            -
d2    33:37     4620        -  23804837640            -            -
d3    32:56     4580        -  23806787244            -            -
d4    33:00     4584        -  23818501728            -            -
d5  1:05:58     4512        -  47611538272            -            -
d6    49:04     4512        -  35333166188            -            -

backshift-1.20 The server storage directory was mounted on the client using sshfs and backshift was run on the client side, saving to the mounted directory. Several hours into the second backup, where no files had been touched in the backup set, testing backshift was abandoned. max mem (kb) test time client server network (b) disk space (kb) nodes a1 43:37:50 125132 - 6709214436 6314132 1290187

Note from the author of Backshift, August 2015


backuppc-3.2.1 This software uses tar over ssh to retrieve files from clients. I did not capture memory statistics for the use of tar and ssh. max mem (kb) test time client server network (b) disk space (kb) nodes a1 1:23:49 - 284984 17901726148 9100204 2106563 a2 19:08 - 284972 28261484 9511692 2226526 a3 25:23 - 284968 542277068 10456412 2423805 a4 25:43 - 284968 542110772 10835368 2579155 a5 30:25 - 285008 5872127792 11253144 3188940 a6 19:52 - 110464 5854417192 11431256 3728321 b1 51:47 - 1172 18580045616 - - b2 1:00:56 - 1168 18616293176 - - b3 1:02:02 - 1168 18608841452 - - b4 1:06:50 - 1172 18627536544 - - b5 1:06:38 - 1168 18611113920 - - b6 22:21 - 1160 6088763864 - - c1 32:28 - 19644 24239202344 22190452 58 c2 00:04 - 19648 8680 22190484 71 c3 23:49 - 23704 53683896 44380820 96 c4 22:46 - 23704 53678308 44380864 114 c5 32:21 - 19620 24240577464 44380928 137 c6 15:47 - 19656 11734617416 55123184 161 d1 32:55 - 1144 23500251316 - - d2 32:54 - 1140 23499752736 - - d3 32:56 - 1144 23505114172 - - d4 32:56 - 1140 23504421352 - - d5 33:05 - 1148 23497633972 - - d6 16:03 - 1152 11377405112 - -
bacula-5.2.13 This software has multiple components - the database, the director, the storage daemon and the file daemon. MySQL was used as the database software, and ran on the server along with the director and storage daemon. The file daemon runs on the client side. Memory statistics were recorded for all of these. MySQL disk space and nodes were also included in the results. Each backup was performed as an incremental (bacula automatically promoted the first of each sequence to a full). max mem (kb) test time client server network (b) disk space (kb) nodes a1 1:14:13 1188 183516 19415316808 16613252 28 a2 06:24 315816 212648 7339 16613252 28 a3 08:22 315712 212592 558876468 17090484 28 a4 08:21 315820 212852 558831313 17567712 28 a5 33:18 315932 212464 6558268748 23074912 28 a6 12:51 318896 212900 403522651 23214996 28 b1 37:49 4364 394420 18761519812 - - b2 37:53 4364 394404 18761497347 - - b3 37:41 4364 394348 18761753952 - - b4 37:50 4360 394444 18760842419 - - b5 38:41 4368 394380 18762252423 - - b6 14:06 4372 279184 6143147781 - - c1 36:36 4820 58416 24148768088 22207036 28 c2 00:01 13648 58980 7337 22207036 28 c3 35:27 14168 58416 24150990612 44413760 28 c4 36:00 14376 58792 24154976470 66620488 29 c5 36:06 14380 58496 24151605070 88827212 29 c6 17:27 48420 58224 11689696758 99577364 29 d1 32:28 4372 22583 24035061412 - - d2 32:28 4360 58076 24033754963 - - d3 32:27 4376 58328 24034781702 - - d4 32:45 4360 58516 24094745919 - - d5 32:29 4360 57904 24034420735 - - d6 15:44 4356 57952 11634940513 - - bup-0.25 For the tests, bup used ssh for the network transport. Memory usage for ssh was not recorded. The bup 'split' command was used for backing up, and 'join' for restore. max mem (kb) test time client server network (b) disk space (kb) nodes a1 1:05:25 50396 41564 13028027920 12024828 92 a2 47:36 36924 36976 22788 12054464 97 a3 47:41 43160 23764 552641840 12541864 98 a4 47:28 42076 15656 301338788 12818772 101 a5 51:40 50160 40564 3441960928 15993348 113 a6 16:46 43636 8828 123336 15993444 116 b1 27:49 8084 3687524 18721975020 - - b2 28:12 8080 3689344 18727867332 - - b3 27:31 8072 3688592 18720814964 - - b4 27:15 8072 3689556 18725122480 - - b5 27:49 8076 3446688 18731302336 - - b6 09:57 8120 1689772 6136944368 - - c1 37:06 54728 49860 14760055588 13673276 97 c2 19:30 43400 8214 32492 13647632 98 c3 19:38 45132 11000 44790792 13688684 101 c4 19:34 43888 8132 64096 13688720 104 c5 19:35 43892 34472 63836 13715140 109 c6 09:26 43872 8124 80428 13689000 110 d1 33:40 8116 3694364 23825024780 - - d2 33:53 8116 3687716 23821504284 - - d3 33:28 8084 3695360 23811360164 - - d4 32:58 8084 3696864 23815830112 - - d5 32:56 8084 3697192 23823484740 - - d6 16:00 8120 3696736 11533677068 - - obnam-1.1 The server storage directory was mounted on the client using sshfs and obnam was run on the client side, saving to the mounted directory. Testing on obnam was abandoned after it hadn't finished after 8 hours and had taken up more than 50GB of space on the server, when there were only 22GB of files to bakup up on the client.
rdiff-backup For the tests, rdiff-backup used ssh for the network transport. Memory usage for ssh was not recorded. I was unable to record server memory statistics for the server when restoring large files. max mem (kb) test time client server network (b) disk space (kb) nodes a1 2:08:09 155932 632632 19549959636 20492104 1624474 a2 25:17 154156 29548 732119660 20579220 1624480 a3 1:05:54 172512 73508 1330676312 21232008 1669655 a4 1:04:33 172576 73516 826853828 21492456 1714830 a5 10:56:29 156352 699136 13588674368 28216628 2763315 a6 12:28:14 65212 29532 1856630968 28095444 2763324 b1 2:20:46 287712 630348 18493237092 - - b2 2:19:55 287832 630416 18493619200 - - b3 2:19:32 287840 630028 18494316560 - - b4 2:18:52 287960 630416 18494140436 - - b5 2:18:07 289328 623580 18494732116 - - b6 44:49 105024 255252 6056832020 - - c1 1:40:30 10908 11256 24084094872 22190296 13 c2 00:05 10260 10396 68516 22190304 19 c3 42:21 17984 17292 104168884 22272648 27 c4 52:52 18364 18600 1999400 22272684 35 c5 2:09:00 11404 11092 24086698904 44462964 44 c6 35:39 17996 17996 2029536 41372716 52 d1 1:21:19 11416 - 23953765636 - - d2 1:27:13 11412 - 23955454164 - - d3 1:03:15 11416 - 23953752660 - - d4 42:23 11416 - 23954690536 - - d5 1:02:19 11412 - 23957680960 - - d6 20:29 11412 - 11596528360 - -
rsync-link-dest For the tests, rsync used ssh for the network transport. Memory usage for ssh was not recorded. max mem (kb) test time client server network (b) disk space (kb) nodes a1 34:57 42224 49964 17921288120 20001104 1535719 a2 05:38 28316 28412 43588912 20360088 3071436 a3 08:11 28292 37028 560651704 21285484 4607153 a4 07:12 28580 37224 54572948 22210876 6142870 a5 14:36 34208 47856 5899248188 29008644 7678588 a6 01:51 16708 17624 14553352 29131180 8187702 b1 28:37 97540 138424 17396435028 - - b2 28:29 97544 138532 17407095180 - - b3 28:05 97540 136904 17394005376 - - b4 27:54 97544 135868 17392775068 - - b5 27:42 97544 138756 17391105544 - - b6 08:46 69624 61852 5686955940 - - c1 32:33 6588 1488 24254325112 22190276 4 c2 00:02 3188 1248 7316 22190280 6 c3 28:58 6312 9140 96024684 44380552 8 c4 29:41 3320 9144 2229896 66570824 10 c5 32:26 6852 1488 24242023980 88761096 12 c6 15:33 3216 9068 1887460 99503292 14 d1 32:31 7380 1256 23513700756 - - d2 32:33 7376 1252 23515115572 - - d3 32:40 7280 1244 23511657216 - - d4 32:46 7316 1252 23529028584 - - d5 32:54 7476 1248 23520135780 - - d6 15:54 7256 1048 11383879512 - -
tar-over-ssh Memory usage settings for ssh were not recorded. For the server, that meant no memory usage results, since that was the only process running on the server. max mem (kb) test time client server network (b) disk space (kb) nodes a1 34:14 1416 - 18906459540 17457040 2 a2 32:46 1420 - 18901056092 34914076 3 a3 32:55 1416 - 18902683324 52371112 4 a4 32:53 1416 - 18901313052 69828148 5 a5 32:49 1436 - 18903018656 87286192 6 a6 11:27 1396 - 6192213332 93003248 7 b1 25:47 1120 - 18716011072 - - b2 25:49 1116 - 18710756096 - - b3 25:50 1124 - 18710245600 - - b4 25:52 1124 - 18714191624 - - b5 25:53 1128 - 18713813356 - - b6 08:26 1118 - 6127883736 - - c1 33:05 1204 - 23944470936 22190280 2 c2 32:40 1208 - 23944195580 44380556 3 c3 33:04 1208 - 23943847812 66570832 4 c4 32:47 1216 - 23943428308 88761108 5 c5 32:45 1216 - 23944522232 110951384 6 c6 15:54 1216 - 11591519516 121693580 7 d1 33:03 1152 - 23814698300 - - d2 32:44 1148 - 23813943948 - - d3 32:48 1152 - 23806047744 - - d4 32:47 1148 - 23805303488 - - d5 32:46 1156 - 23806356912 - - d6 15:55 1160 - 11526917756 - - urbackup-1.2.4 Note: Testing on urbackup was abandoned when it was discovered that the only way to restore files was one at a time via its web interface. This made restoring millions of files impractical.
burp-1.3.36 A one line patch was applied to burp-1.3.36 in order to turn off network compression. Another patch was applied to prevent it compressing individual files on restore. max mem (kb) test time client server network (b) disk space (kb) nodes a1 56:06 5136 3712 19073221246 20381968 1535686 a2 27:13 5124 3752 559650895 20762924 1535691 a3 35:12 5140 3844 1232999529 21720168 1580870 a4 38:03 5144 3828 764511451 22281984 1626049 a5 39:35 5128 3712 6630073883 29226320 2135147 a6 13:21 5124 3752 193122037 29355744 2135158 b1 39:58 3452 49216 18408345630 - - b2 39:51 3448 49980 18408178034 - - b3 43:34 3452 41896 18408416587 - - b4 38:01 3452 35288 18403160479 - - b5 37:05 3456 34568 18405181477 - - b6 11:26 3448 13908 6025288820 - - c1 32:56 3916 3560 24209453432 22190320 15 c2 0:04 3912 3652 10605 22190340 20 c3 49:24 10124 8536 125113914 22272284 32 c4 46:25 10124 8644 22968510 22272352 44 c5 32:56 3912 3560 24209415813 44462664 56 c6 39:41 10124 6788 22835167 41393692 68 d1 1:18:28 3456 3424 24168034544 - - d2 1:12:39 3452 3424 24167416085 - - d3 53:58 3456 3428 24167803329 - - d4 33:13 3452 3424 24167751901 - - d5 55:25 3448 3428 24167297229 - - d6 16:08 3452 3428 11699804162 - -
burp-2.0.0 A one line patch was applied to burp-2.0.0 in order to turn off network compression. max mem (kb) test time client server network (b) disk space (kb) nodes a1 50:12 936276 422436 6261782638 5321000 459 a2 6:01 560492 167004 463561084 5647588 466 a3 8:22 560604 191472 993873340 6447976 521 a4 7:38 560488 191036 479991676 6774508 528 a5 18:47 561020 267716 646705357 7102040 535 a6 2:17 188924 183604 154855021 7211576 542 b1 2:32:14 3392 1419164 18072607863 - - b2 2:24:02 3396 1460392 18072342507 - - b3 2:13:37 3392 1514896 18071492855 - - b4 2:12:54 3396 1502624 18071360723 - - b5 2:16:30 3396 1499876 18072840238 - - b6 1:09:27 3396 925132 5919571611 - - c1 44:27 166552 352840 14740348487 13442384 963 c2 00:22 3752 369148 10712 13529832 970 c3 32:45 96172 370104 333668235 13652820 981 c4 32:38 44292 370268 294625537 13740276 988 c5 32:42 44292 370012 294212673 13827728 995 c6 15:50 44284 370036 143341975 13870216 1004 d1 1:50:26 3392 929832 24079869152 - - d2 1:49:51 3408 922188 24080071062 - - d3 1:56:46 3408 901052 24081243535 - - d4 2:00:49 3408 870040 24081042956 - - d5 2:12:54 3412 842132 24081705105 - - d6 41:56 3408 717240 11656952804 - -
burp-2.0.1 A one line patch was applied to burp-2.0.1 in order to turn off network compression. max mem (kb) test time client server network (b) disk space (kb) nodes a1 49:55 964824 222620 6333908776 5134556 1047 a2 5:26 546768 15616 459379807 5275764 1862 a3 7:30 546796 23460 982353947 5886076 2701 a4 7:01 546792 23440 470919618 6027296 3516 a5 17:36 547928 88208 631650761 6169096 4332 a6 1:57 184516 14256 151701404 6215876 4604 b1 2:03:56 3480 333452 18070868859 - - b2 2:02:49 3464 333448 18070889282 - - b3 2:00:17 3464 333452 18069833763 - - b4 2:00:25 3460 333248 18070389579 - - b5 1:59:24 3464 332836 18070507134 - - b6 1:04:34 3460 333032 5919135548 - - c1 46:07 166628 27076 15968721238 14442396 1317 c2 0:59 3836 23276 10660 14544832 2113 c3 33:22 166520 28208 380791042 14704548 2912 c4 33:30 137996 25148 351607673 14817148 3709 c5 33:30 137996 25264 349251193 14929748 4506 c6 16:18 122168 23076 169103345 14982380 4898 d1 1:08:10 3476 369672 24077829873 - - d2 1:08:05 3464 369660 24076534634 - - d3 1:10:13 3480 368820 24077401653 - - d4 1:14:52 3476 368608 24078404161 - - d5 1:13:54 3480 368612 24077496825 - - d6 26:55 3460 362164 11655881312 - -
< Prev Contents Next >
Donate with Bitcoin

Burp is open and free software. I work on it in my spare time. If you would like this work to continue, please consider making a small donation.


Burp, don't suck. Last updated: June 2016
By Graham Keeling
Hosted by 6sync