{"id":1776,"date":"2020-07-01T04:51:04","date_gmt":"2020-06-30T20:51:04","guid":{"rendered":"http:\/\/39.97.231.10\/?p=1776"},"modified":"2020-07-01T04:51:04","modified_gmt":"2020-06-30T20:51:04","slug":"hbase-hfile%e8%a7%a3%e6%9e%90","status":"publish","type":"post","link":"http:\/\/www.weijing.co\/?p=1776","title":{"rendered":"HBase HFile\u89e3\u6790"},"content":{"rendered":"<h1>\u57fa\u672c\u6982\u5ff5<\/h1>\n<article>\n<ul>\n<li>\u8868\uff08Table\uff09\uff0c\u6570\u636e\u7684\u7ec4\u7ec7\u5f62\u5f0f<\/li>\n<li>\u884c\uff08Row\uff09\uff0cTable\u4e2d\u7684\u6bcf\u4e00\u884c<\/li>\n<li>\u5217\u65cf\uff08Column Family\uff09\uff0c\u4e00\u884c\u4e2d\u6709\u591a\u4e2a\u5217\uff0c\u4ee5Column Family\u8fdb\u884c\u5206\u7ec4\uff0c\u540c\u4e00Column Family\u7684\u5217\u5b58\u50a8\u5728\u540c\u4e00\u4e2a\u5e95\u5c42\u6587\u4ef6\uff08HFile\uff09\u4e2d\uff0c\u6240\u4ee5Column Family\u4f1a\u5f71\u54cd\u6570\u636e\u7684\u7269\u7406\u5b58\u50a8\uff0c\u4e00\u822c\u5728\u8868\u521b\u5efa\u7684\u65f6\u5019\uff0c\u5c31\u9700\u8981\u6307\u5b9a\u597d\uff0c\u5e76\u4e14\u4e0d\u8981\u8f7b\u6613\u4fee\u6539\u3002<\/li>\n<li>\u5217\uff08Column Qualifier\uff09\uff0c\u8fd9\u4e2a\u4e0d\u9700\u8981\u521b\u5efa\u8868\u7684\u65f6\u5019\u5c31\u6307\u5b9a\u3002<\/li>\n<li>\u5355\u5143\u683c\uff08Cell\uff09\uff0c\u4e00\u7ec4Row\u3001Family\u3001Qualifier\u53ef\u4ee5\u5b9a\u4f4d\u4e00\u4e2aCell\u3002<\/li>\n<li>\u65f6\u95f4\u6233\uff08Timestamp\uff09\uff0c\u4e00\u4e2a\u5355\u5143\u683c\u4e2d\u7684\u6570\u636e\u662f\u7531\u7248\u672c\u7684\uff0c\u7248\u672c\u4ee5Timestamp\u6765\u533a\u5206\u3002\u9ed8\u8ba4\u5199\u5165\u7684\u662f\u5f53\u524d\u7684timestamp\uff0c\u8bfb\u53d6\u7684\u662f\u6700\u65b0\u7684timestamp\u7684\u6570\u636e\u3002HBase\u9ed8\u8ba4\u4e00\u4e2a\u5355\u5143\u683c\u4fdd\u5b58\u4e09\u4e2a\u7248\u672c\u3002<\/li>\n<\/ul>\n<p>\u6240\u4ee5\uff0c\u4e00\u4e2aHBase\u7684Table\u5b58\u53d6\u6a21\u5f0f\u4e3a\uff1a<\/p>\n<div>\n<pre>(Table, RowKey, Family, Qualifier, Timestamp) -&gt; Value\n<\/pre>\n<\/div>\n<p>\u4e00\u4e2aTable\u5b58\u53d6\u7b49\u4ef7\u4e8e\u9ad8\u7ea7\u8bed\u8a00\u7684\u4e00\u4e2amap\uff1a<\/p>\n<div>\n<pre>SortedMap&lt;\n    RowKey, List&lt;\n        SortedMap&lt;\n            Qualifier, List&lt;\n                Value, Timestamp\n            &gt;\n        &gt;\n    &gt;\n&gt;\n<\/pre>\n<\/div>\n<p>\u6216\u8005\uff0c\u518d\u76f4\u89c2\u4e00\u70b9\uff0c\u903b\u8f91\u4e0a\u7b49\u4ef7\u4e8e\u4e00\u4e2a\u56fa\u5b9a\u683c\u5f0f\u7684Json\uff1a<\/p>\n<div>\n<pre>{\n    \\\"RowKey1\\\" {\n        \\\"Family1\\\": {\n            \\\"Qualifier11\\\":{\n                \\\"Timestamp111\\\":\\\"Value111\\\"\n            }\n            \\\"Qualifier12\\\":{\n                \\\"Timestamp121\\\":\\\"Value121\\\",\n                \\\"Timestamp122\\\":\\\"Value122\\\"\n            }\n        }\n        \\\"Family2\\\" : {\n            \\\"Qualifier21\\\":{\n                \\\"Timestamp211\\\":\\\"Value211\\\"\n            }\n        }\n    }\n    \\\"RowKey2\\\" {\n        ...\n    }\n}\n<\/pre>\n<\/div>\n<h2>HFile\u5b58\u50a8\u5355\u5143<\/h2>\n<blockquote><p>\u4e5f\u53ebStoreFile\uff0c<a href=\\\"http:\/\/research.google.com\/archive\/bigtable.html\\\" target=\\\"_blank\\\" rel=\\\"nofollow noopener noreferrer\\\">google\u8bba\u6587\u4e2d\u662fSSTable<\/a>\uff0c\u662f\u6570\u636e\u5b58\u50a8\u7684\u5730\u65b9\uff0cHBase\u4e4b\u6240\u4ee5\u662f\u9762\u5411\u5217\u7684\u6570\u636e\u5e93\uff0c\u662f\u56e0\u4e3a\u6570\u636e\u4ee5key-value\u5f62\u5f0f\u5b58\u50a8\u7684\uff0c\u5217\u53ef\u4ee5\u52a8\u6001\u6269\u5c55<\/p><\/blockquote>\n<h4>KeyValue<\/h4>\n<p>\u5f53put\u5230hbase\u4e00\u4e2akey\u548cvalue\u7684\u65f6\u5019\uff0c\u4f1a\u589e\u52a0\u4e00\u6761\u8bb0\u5f55\uff1a<\/p>\n<div>\n<pre>(Table, RowKey, Family, Qualifier, Timestamp) -&gt; Value\n<\/pre>\n<\/div>\n<p>\u8be5\u8bb0\u5f55\u4ee5\u5b57\u8282\u6d41\u7684\u65b9\u5f0f\u5b58\u50a8\uff0c\u5bf9\u5e94\u5230\u78c1\u76d8\u4e2d\u7684\u5b58\u50a8\u683c\u5f0f\u4e3a\uff1a<\/p>\n<div>\n<div>\n<div><\/div>\n<div data-width=\\\"440\\\" data-height=\\\"224\\\"><img src=\\\"https:\/\/upload-images.jianshu.io\/upload_images\/13873072-e224ee0d18cfe5da.png?imageMogr2\/auto-orient\/strip|imageView2\/2\/w\/440\/format\/webp\\\" data-original-src=\\\"\/\/upload-images.jianshu.io\/upload_images\/13873072-e224ee0d18cfe5da.png\\\" data-original-width=\\\"440\\\" data-original-height=\\\"224\\\" data-original-format=\\\"image\/png\\\" data-original-filesize=\\\"35258\\\" data-image-index=\\\"0\\\" \/><\/div>\n<\/div>\n<div>image.png<\/div>\n<\/div>\n<ul>\n<li>Key Length\uff0832\u4f4d\u6574\u5f62\uff09<\/li>\n<li>Value Length\uff0832\u4f4d\u6574\u5f62\uff09<\/li>\n<li>Key\n<ul>\n<li>Row Length<\/li>\n<li>Row\uff08\u5373\uff1arowKey\uff09<\/li>\n<li>Column Family Length<\/li>\n<li>Column Family<\/li>\n<li>Column Qualifier\uff08Qulifier\u7684\u957f\u5ea6\u53ef\u4ee5\u901a\u8fc7Key Length\u3001Row Length\u3001Column Family Length\u3001Timestamp\u56fa\u5b9a\u957f\u5ea6\u3001KeyType\u56fa\u5b9a\u7c7b\u578b\u8ba1\u7b97\u51fa\u6765\uff09<\/li>\n<li>Timestamp<\/li>\n<li>KeyType \uff08Put, Delete, DeleteColumn, DeleteFamily\u7b49\u7c7b\u578b\uff09<\/li>\n<\/ul>\n<\/li>\n<li>Value KeyValue\u662f\u4e00\u4e2a\u57fa\u672c\u7684\u5355\u5143\u6a21\u5757\uff0c\u4e0d\u53ef\u518d\u5206\uff0c\u4f8b\u5982\uff1aBlockData\u9ed8\u8ba4\u4e3a64K\uff0c\u4f46\u662f\u5982\u679c\u6709\u4e00\u4e2aKeyValue\u662f8MB\uff0c\u4e00\u6837\u4f1a\u6574\u4f53\u5199\u8fdb\u53bb\uff0c\u8fd9\u4e2aBlockData\u7684\u5927\u5c0f\u662f\u540e\u68c0\u67e5\u7684\uff0c\u63d2\u5165\u6570\u636e\u540e\uff0c\u518d\u68c0\u67e5\u662f\u5426\u8d85\u8fc7\u9ed8\u8ba4\u503c\u3002<\/li>\n<\/ul>\n<h4>BlockData<\/h4>\n<p>\u591a\u4e2aKeyValue\u7ed3\u6784\uff0c\u6309\u7167Key\u9012\u589e\u7684\u987a\u5e8f\uff0c\u7ec4\u6210\u4e00\u4e2aBlockData\uff08\u9ed8\u8ba464K\uff09\uff0cBlockData\u5728HFile\u5b58\u5728\u78c1\u76d8\u4e2d\u7684\u5b58\u50a8\u65b9\u5f0f\u4e3a\uff1a<\/p>\n<div>\n<div>\n<div><\/div>\n<div data-width=\\\"326\\\" data-height=\\\"254\\\"><img src=\\\"https:\/\/upload-images.jianshu.io\/upload_images\/13873072-f8bc2ed167b55b2d.png?imageMogr2\/auto-orient\/strip|imageView2\/2\/w\/326\/format\/webp\\\" data-original-src=\\\"\/\/upload-images.jianshu.io\/upload_images\/13873072-f8bc2ed167b55b2d.png\\\" data-original-width=\\\"326\\\" data-original-height=\\\"254\\\" data-original-format=\\\"image\/png\\\" data-original-filesize=\\\"22617\\\" data-image-index=\\\"1\\\" \/><\/div>\n<\/div>\n<div><\/div>\n<\/div>\n<h4>DataBlock<\/h4>\n<p>DataBlock\u662f\u5bf9BlockData\u7684\u8fdb\u4e00\u6b65\u5c01\u88c5\uff0c\u5728BlockData\u57fa\u7840\u4e0a\uff0c\u589e\u52a0\u4e86\u4e00\u4e2a8\u5b57\u8282\u7684\u6807\u5fd7\u4f4d\uff0cmagic number\uff0c\u7528\u6765\u6807\u8bb0BlockData\u7684\u7c7b\u578b\uff0c\u5e38\u89c1\u7684\u6807\u8bb0\u7c7b\u578b\u6709\uff1a<\/p>\n<ul>\n<li>DATA\uff0c\u5c31\u662fBlockData\u7c7b\u578b<\/li>\n<li>LEAF_INDEX\uff0c\u53f6\u5b50\u7d22\u5f15\u5757<\/li>\n<li>BLOOM_CHUNK \uff0cBloom filter\u5757\uff08Bloom Filter\u662f\u91c7\u7528\u4f4dhash\u6807\u8bb0\uff0c\u5e38\u6570\u65f6\u95f4\u5185\u53ef\u4ee5\u5224\u65adkey\u662f\u5426\u5b58\u5728\u7684\u4e00\u79cd\u6570\u636e\u7ed3\u6784\uff09<\/li>\n<li>&#8230;\uff08\u7b49\u7b49\uff09\n<div>\n<div>\n<div><\/div>\n<div data-width=\\\"882\\\" data-height=\\\"314\\\"><img src=\\\"https:\/\/upload-images.jianshu.io\/upload_images\/13873072-0e926d5efc7e89d9.png?imageMogr2\/auto-orient\/strip|imageView2\/2\/w\/882\/format\/webp\\\" data-original-src=\\\"\/\/upload-images.jianshu.io\/upload_images\/13873072-0e926d5efc7e89d9.png\\\" data-original-width=\\\"882\\\" data-original-height=\\\"314\\\" data-original-format=\\\"image\/png\\\" data-original-filesize=\\\"179411\\\" data-image-index=\\\"2\\\" \/><\/div>\n<\/div>\n<div><\/div>\n<\/div>\n<\/li>\n<\/ul>\n<h4>HFile<\/h4>\n<p>\u7531\u4ee5\u4e0b\u56db\u90e8\u5206\u7ec4\u6210\uff1a<\/p>\n<ul>\n<li>Scanned Block section\uff0c\u626b\u63cf\u6570\u636e\u5b58\u50a8\u90e8\u5206\n<ul>\n<li>Data Block\uff0c\u6570\u636e\u5757<\/li>\n<li>leaf block index\uff0c\u7d22\u5f15\u5757<\/li>\n<li>Bloom chunk blocks\uff0cBloom chunk\u5757<\/li>\n<\/ul>\n<\/li>\n<li>Non-scanned block section\uff0c\u975e\u626b\u63cf\u6570\u636e\u5b58\u50a8\u90e8\u5206\n<ul>\n<li>Meta Block<\/li>\n<li>intermediate-level index blocks<\/li>\n<\/ul>\n<\/li>\n<li>load-on-open section\uff0c\u6253\u5f00HFile\u6587\u4ef6\u7684\u65f6\u5019\uff0c\u8fd9\u90e8\u5206\u9700\u8981\u52a0\u8f7d\u5230\u5185\u5b58\uff0c\u5305\u62ec\u6587\u4ef6\u4fe1\u606f\u548c\u7d22\u5f15\u4fe1\u606f\u3002<\/li>\n<li>Trailer\uff0c\u4e00\u4e2a\u56fa\u5b9a\u957f\u5ea6\uff0c\u8bb0\u5f55\u4e86HFile\u7684\u57fa\u672c\u4fe1\u606f\u3001\u5404\u4e2a\u90e8\u5206\u7684\u504f\u79fb\u503c\u548c\u5bfb\u5740\u4fe1\u606f\u3002<\/li>\n<\/ul>\n<div>\n<div>\n<div><\/div>\n<div data-width=\\\"566\\\" data-height=\\\"418\\\"><img src=\\\"https:\/\/upload-images.jianshu.io\/upload_images\/13873072-4ffd14f0301c5a78.png?imageMogr2\/auto-orient\/strip|imageView2\/2\/w\/566\/format\/webp\\\" data-original-src=\\\"\/\/upload-images.jianshu.io\/upload_images\/13873072-4ffd14f0301c5a78.png\\\" data-original-width=\\\"566\\\" data-original-height=\\\"418\\\" data-original-format=\\\"image\/png\\\" data-original-filesize=\\\"57858\\\" data-image-index=\\\"3\\\" \/><\/div>\n<\/div>\n<div><\/div>\n<\/div>\n<h4>HFile V1\u3001V2\u548cV3<\/h4>\n<ul>\n<li>V1\uff0cHBase0.92\u4e4b\u524d\uff0c\u7ed3\u6784\u7b80\u5355\uff0c\u5f53\u6570\u636e\u589e\u591a\u7684\u65f6\u5019\uff0c\u6bcf\u4e2aHFile\u7684BloomFilter\u4f1a\u6709100M\u591a\uff0c\u5f53\u8d85\u8fc720\u4e2aregion\u7684\u65f6\u5019\uff0c\u4f1a\u67092G\uff0cBlock indexes\u4f1a\u67096G\uff0cRegion\u9700\u8981\u5c06\u6240\u6709\u7684\u7d22\u5f15load\u5b8c\uff0c\u624d\u8ba4\u4e3a\u662f\u6b63\u786e\u52a0\u8f7d\u4e86\uff0c\u901f\u5ea6\u975e\u5e38\u6162\u3002<\/li>\n<li>V2\uff0c\u52a0\u901f\u4e86regions server\u7684\u542f\u52a8\uff0c\u540c\u65f6\uff0c\u91c7\u7528load-on-open\u8fd9\u4e2a\u90e8\u5206\u52a0\u8f7d\u5230\u5b8c\uff0cRegion\u5c31\u7b97\u662f\u6b63\u786e\u52a0\u8f7d\u4e86\u3002<\/li>\n<li>V3\uff0cHBase0.98\u4e4b\u540e\u5f15\u5165\uff0c\u5728File Trailer\u4e2d\u589e\u52a0\u4e86encryption_key\uff0c\u652f\u6301AES\u5bf9\u5f53\u524d\u7684HFile\u52a0\u5bc6\u3002<\/li>\n<\/ul>\n<h2>\u67e5\u627erowKey\u8fc7\u7a0b\uff1a<\/h2>\n<ul>\n<li>HFile\u5185\u90e8\u662f\u6709\u5e8f\u7684\uff0c\u4f46\u662fHFile\u4e4b\u95f4\u662f\u65e0\u5e8f\u7684<\/li>\n<li>\u5224\u65adkey\u662f\u5426\u5b58\u5728Hfile\u4e2d\uff0c\u53ef\u4ee5\u901a\u8fc7Leaf index\u6216\u8005Bloom Filter<\/li>\n<li>HBase\u5c06rowKey\u53ef\u80fd\u5b58\u5728\u7684\u6240\u6709HFile\u4e2d\u7684\u6700\u5c0f\u7684rowKey\uff0c\u8fdb\u884c\u6392\u5e8f\uff08rowKey\u3001column\u5347\u5e8f\uff0ctimestamp\u964d\u5e8f\uff09\uff0c\u7136\u540e\u653e\u5165\u961f\u5217\u4e2d<\/li>\n<\/ul>\n<\/article>\n","protected":false},"excerpt":{"rendered":"<p>\u57fa\u672c\u6982\u5ff5 \u8868\uff08Table\uff09\uff0c\u6570\u636e\u7684\u7ec4\u7ec7\u5f62\u5f0f \u884c\uff08Row\uff09\uff0cTable\u4e2d\u7684\u6bcf\u4e00\u884c \u5217\u65cf\uff08Column Famil &hellip;<\/p>\n<p class=\"read-more\"> <a class=\"\" href=\"http:\/\/www.weijing.co\/?p=1776\"> <span class=\"screen-reader-text\">HBase HFile\u89e3\u6790<\/span> \u67e5\u770b\u5168\u6587 &raquo;<\/a><\/p>\n","protected":false},"author":4,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"default","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","theme-transparent-header-meta":"default","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":""},"categories":[133,118],"tags":[],"uagb_featured_image_src":{"full":false,"thumbnail":false,"medium":false,"medium_large":false,"large":false,"1536x1536":false,"2048x2048":false,"mailpoet_newsletter_max":false,"woocommerce_thumbnail":false,"woocommerce_single":false,"woocommerce_gallery_thumbnail":false,"shop_catalog":false,"shop_single":false,"shop_thumbnail":false},"uagb_author_info":{"display_name":"weijing","author_link":"http:\/\/www.weijing.co\/?author=4"},"uagb_comment_info":0,"uagb_excerpt":"\u57fa\u672c\u6982\u5ff5 \u8868\uff08Table\uff09\uff0c\u6570\u636e\u7684\u7ec4\u7ec7\u5f62\u5f0f \u884c\uff08Row\uff09\uff0cTable\u4e2d\u7684\u6bcf\u4e00\u884c \u5217\u65cf\uff08Column Famil&hellip;","_links":{"self":[{"href":"http:\/\/www.weijing.co\/index.php?rest_route=\/wp\/v2\/posts\/1776"}],"collection":[{"href":"http:\/\/www.weijing.co\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.weijing.co\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.weijing.co\/index.php?rest_route=\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"http:\/\/www.weijing.co\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1776"}],"version-history":[{"count":0,"href":"http:\/\/www.weijing.co\/index.php?rest_route=\/wp\/v2\/posts\/1776\/revisions"}],"wp:attachment":[{"href":"http:\/\/www.weijing.co\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1776"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.weijing.co\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1776"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.weijing.co\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}